Artificial Analysis v5 and the benchmarker's dilemma
- Hill-climbing is easy, knowing what to hill-climb is hard.
- Making scores go up is expensive, but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today.
- Independent evaluation of language models is under-appreciated in how hard of a task it is.
Unverified
- Hill-climbing is easy, knowing what to hill-climb is hard.
- Making scores go up is expensive, but we have a much clearer playbook for it than we do for creating meaningful benchmarks at the true frontier of AI today.
- Independent evaluation of language models is under-appreciated in how hard of a task it is.
Sources: Interconnects