Stop Thinking of LLMs as Next-Token Predictors
- gmcgoldr's blog View My GitHub Profile Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete.
- During pre-training, the model repeatedly takes some prior tokens, looks at the token that actually followed them, and makes that token more likely to be sampled next.
Unverified
- gmcgoldr's blog View My GitHub Profile Strictly speaking, the statement “LLMs are next-token predictors” isn’t wrong, but it’s incomplete.
- During pre-training, the model repeatedly takes some prior tokens, looks at the token that actually followed them, and makes that token more likely to be sampled next.
Sources: Github