Watch the video carefully. DFlash2's tool call fails on python syntax.
Usually models in this class nail things like that 1 shot, which the other side did.
I don't know the cause. It may be nothing. But I'd like to see the model doing something where its path is a bit more constrained, to help out rule out such oddities.
zackangelo 9 hours ago [-]
DFlash is lossless so this would be a bug in the implementation if it is indeed a regression against the target model.
liuliu 9 hours ago [-]
Only if you do greedy sampling. With probabilisitic sampling (categorical sampling), you will end up with different trajectory just “mathematically equivalent”.
stymaar 8 hours ago [-]
Can you explain a little bit more please?
stkdump 5 hours ago [-]
Not OP, but you can influence how deterministic your LLM behaves using the temperature setting. The neural network doesn't directly output tokens, but logits which are then converted to probabilities and then a token is chosen at random, unless the temperature is 0 (i.e. greedy, we just always pick the most probable token without any randomness). All speculative decoding methods have to "commit" to a token though even when they don't know the actual logits of the full size NN yet. The question then is (and I don't know the answer): how do the common inference engines behave when the speculation landed on the most probable token, but the random choice still doesn't land on it? You can imagine that in the interest of performance as long as we stay reasonably inside the probability we just go ahead with the speculation. Not sure if thats implemented like that though.
Edit: I just looked up the math, and actually the idea of speculative decoding is done in a clever way that fully preserves the probability distribution while still maximizing the acceptance rate of draft tokens. So I would have to disagree with OP and say that no, non-greedy sampling doesn't influence the trajectories.
liuliu 3 hours ago [-]
Yes, it doesn’t impact the probability distribution due to verifier. However, remember how you use PRNG and effectively due to the drafter is sampled from a different distribution initially, a separate rejection sampling won’t be able to recover what the “old PRNG” would choose in a “without drafter” case. Hence in my original post, it is about different trajectories you will end up with, not the correctness of each stochastic sampling.
stkdump 2 hours ago [-]
If you assume that the RNG generates true randomness, then the two are identical. Only if you care about the determinism of the RNG (for example you want to use identical seeds and get the exact same generation between the two) it makes a real difference.
liuliu 2 hours ago [-]
Correct. I am trying to explain why even it is "exact", the generated text is different from the with / without DFlash2 runs, and potentially why the DFlash2 run will contain the invalid Python syntax.
stymaar 40 seconds ago [-]
If the underlying probability distributions are the same, then DFlash can lead to an invalid Python Syntax iif the autoregressive process could have generated one if the random sampling picked a different token.
If a model can output a “wrong” sequence with a certain probability p, then Dflash can also output the wrong sequence with the same probability. They wouldn't necessarily produce the same output from the same seed, but speculative decoding shouldn't be able to produce anything that the autoregressive model couldn't also produce when using a different seed.
Or am I misunderstanding something?
stkdump 37 seconds ago [-]
It will contain the same or different syntax with or without it. Also multiple runs without DFlash2 will contain the same or different syntax. And multiple runs with DFlash2 will have the same or different syntax with the same probability. DFlash2 literally has no influence (unless buggy). The difference is purely caused by the randomness.
hypfer 12 hours ago [-]
Amazing tech
> An agent writes in an afternoon what a chatbot writes in a month
But can you just.. not.
Your tech is so good, it speaks for itself. Don't ruin that.
adefa 12 hours ago [-]
I'm getting around 27 tokens per second decode using vLLM + Qwen 3.8 27b nvfp4 + DFlash 2 on the DGX Spark.
pdntspa 5 hours ago [-]
I am working this out with Fable right now, for getting this running on my DGX spark homelab; it mentioned that there might be issues with the 'optimized LM-head restrictions' that unsloth NVFP4 ships with. Have you had any issues here?
Are you trying this with vLLM? Or a different engine? I am getting about 15 tok/s on my spark on my current setup using the 0.26 nvidia vLLM image and MTP.
adefa 4 hours ago [-]
If you are using the Unsloth nvfp4 checkpoint, you need to patch vLLM+DFlash 2 to accept the quant's FP8 `lm_head`.
sarjann 12 hours ago [-]
Great news, has made low memory bandwidth model usage so much nicer.
Usually models in this class nail things like that 1 shot, which the other side did.
I don't know the cause. It may be nothing. But I'd like to see the model doing something where its path is a bit more constrained, to help out rule out such oddities.
Edit: I just looked up the math, and actually the idea of speculative decoding is done in a clever way that fully preserves the probability distribution while still maximizing the acceptance rate of draft tokens. So I would have to disagree with OP and say that no, non-greedy sampling doesn't influence the trajectories.
If a model can output a “wrong” sequence with a certain probability p, then Dflash can also output the wrong sequence with the same probability. They wouldn't necessarily produce the same output from the same seed, but speculative decoding shouldn't be able to produce anything that the autoregressive model couldn't also produce when using a different seed.
Or am I misunderstanding something?
> An agent writes in an afternoon what a chatbot writes in a month
But can you just.. not.
Your tech is so good, it speaks for itself. Don't ruin that.
Are you trying this with vLLM? Or a different engine? I am getting about 15 tok/s on my spark on my current setup using the 0.26 nvidia vLLM image and MTP.