technology

For business customers per token costs might not be a deal breaker, but for anything consumer facing it's a really tough sell in my opinion. I do expect that the cost of running models is going to come down significantly in the near future though. There is a whole bunch of recent research that identify some key optimizations that can be made. Some of the ones I've found particularly interesting here:

• Deepseek-mhc https://arxiv.org/abs/2512.24880
• a graph-guided code generation framework that could solve the problem of keeping a repo’s structure in context https://arxiv.org/abs/2509.16198
• Hierarchical Reasoning Model https://arxiv.org/abs/2506.21734
• MemOS https://arxiv.org/abs/2507.03724 (https://github.com/MemTensor/MemOS)
• https://github.com/BICLab/SpikingBrain-7B
• German researchers achieved 71.6% on ARC-AGI using a regular GPU for 2 cents per task https://arxiv.org/abs/2505.07859
• Affordable AI assistants with knowledge graph https://arxiv.org/abs/2504.02670 (https://github.com/spcl/knowledge-graph-of-thoughts)
• Nested Learning: A new ML paradigm for continual learning https://abehrouz.github.io/files/NL.pdf
• Continual Low-Rank Adaptation for Pre-trained Models https://arxiv.org/abs/2502.17920
• Recursive language models https://arxiv.org/pdf/2512.24601
• embeddings and context https://pub.sakana.ai/DroPE/
• DeepSeek Engram paper https://github.com/deepseek-ai/Engram/blob/main/Engram_paper.pdf
• a 32M parameter multi vector model outperforms 600M parameter models https://arxiv.org/abs/2601.08620

Once these ideas start getting integrated, I expect that we'll see much more capable models that can run on fairly cheap hardware. Even local models will likely be quite capable for a lot of tasks. And at that point running a model as a service and charging per token is going to be a dead end.