- DeepSeek V3.2-Exp (September 2025) — 80.00
- Kimi K2 Thinking (November 2025) — 72.59
- Gemini 2.5 Flash (May 2025) — 68.79
- GPT-5 (August 2025) — 23.44
- o3 (April 2025) — 22.57
- Gemini 2.5 Pro (March 2025) — 21.37
- Gemini 3 Pro (November 2025) — 16.93
- Claude 3.5 Sonnet (August 2025) — 11.67
- GPT-5 Pro (August 2025) — 1.96
The goal is to find out which models are the best practical performance per dollar spent on real world available API pricing. Right now it seems like DeepSeek and Kimi are winning, and Google has lost its way since Gemini 2.5 Flash.
This interests me because LLMs are global, especially if they're open sourced like Kimi K2 Thinking. Anyone can use the cheaper one just as easily, and no tariffs or policy can stop it.
This is problematic because MMLU is very easy, and I think SWE bench is too. Even 32B models can ‘saturate’ MMLU, hence Qwen 30B would obliterate other models in these charts if you added it.
They're also gamed/contaminated like crazy.
I'd lean more towards “private” benchmarks that aren’t contaminated, or gamed like LM Arena. For instance:
https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard
https://eqbench.com/
Long context benchmarks like RULER are good too, as that’s extremely difficult for many models now.
Okay here's the new bar chart with the more spread out weighting, and honestly it looks a lot more reasonable.
Thank you, I'll factor that into the index. If you have any other recommendations for how to make the index more robust let me know. The goal is to make this dependent on real world API costs. I don't care if the newest smartest model is released if it costs $100 every time to use.