Basing costs on token consumption, whether it's for code suggestion, generation, or AI debugging, makes as much sense โ less, even โ than paying programmers per keystroke in and character out. That's even dafter than the lines of code per month metric for coder goodness, a concept so dumb it makes Juicero's backers look like Warren Buffett. There is no concept of usable work actually done, no sense that inefficiencies are rewarded, and no easy way to relate the price paid to the actual cost of production. But it's simple to understand and looks like any other prepayment limited use subscription model. Oddly, nobody seems minded to improve on this.
There are virtually no other metrics. You can measure tokens per second for a benchmark test case. You can measure the ratio of tokens out to tokens in, although it's not clearer why. At least with arguably comparable service models like cloud computing, you know what you're getting when you buy so much compute, memory, storage, and connectivity. You still have to watch automation or mismanagement, and Bill Shock still works at AWS, but you have a chance of linking results to costs. Good luck with LLM-based services, let alone AI agents.
Add this lack of value metrics to the ridiculous returns on investments the AI industry needs to show to make good on its promises, and we have a recipe for mounting TIBS inflation.
Vendors have an addiction to making everything a subscription, then frog-boiling subscribers, especially when they can incorporate an effective monopoly. Imagine the lock-in where an org has deskilled its code production humans and become reliant on a particular AI code gen chain.
...
If AI does result in deskilling the tech workforce and recapturing the engine of IT creation, it will be as if the mainframe era came at the end of semiconductor evolution, rather than the beginning. All that can be said about the evolutionary driver that will move things on is that it has yet to be invented, despite fifty years of looking.