Why is Speculative Decoding Fast?
- It's because you change the kind of work the GPU is doing.
- There is a common misconception that speculative decoding is fast because you do less work per token.
Unverified
- It's because you change the kind of work the GPU is doing.
- There is a common misconception that speculative decoding is fast because you do less work per token.
Sources: Jbarrow