Reducing Doom Loops with Final Token Preference Optimization
- A doom loop is a common failure mode during inference: the model emits a span (often something like “Wait, let me reconsider…”), then repeats the same span again and again, until the context window is exhausted.
- Small reasoning models are more prone to this behavior, especially on long thinking traces and hard problems [1].
- The commonly applied inference-time fix is to apply repetition_penalty to reweight the output distribution.
Unverified
- A doom loop is a common failure mode during inference: the model emits a span (often something like “Wait, let me reconsider…”), then repeats the same span again and again, until the context window is exhausted.
- Small reasoning models are more prone to this behavior, especially on long thinking traces and hard problems [1].
- The commonly applied inference-time fix is to apply repetition_penalty to reweight the output distribution.
Sources: Liquid