this post was submitted on 03 Jul 2026
427 points (99.3% liked)
Technology
86074 readers
3114 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related news or articles.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 3 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
Something I’ve been noticing recently is that while the cost per token on specific models hasn’t gone up, the provided interfaces for using those models are starting to chew up significantly larger numbers of tokens for the same tasks that used fewer tokens with older versions of the interface software just a few months ago. Likely the interfaces are applying more expensive guardrail prompts and charging the end user for those tokens — but the end result is that it costs 4x as much to get the same work done.
A very large chunk of the improvements in the last year have come not from categorically better models, but from the circumstances of the models massively improving. For example, reasoning is just automatic prompt engineering, and eats a fuckton of tokens. Harnesses give LLMs tools, making it easier to turn nondeterminism into determinism (does this code compile is a decision the compiler can answer definitely). Then there's subagents, which is just automatic context engineering.
Basically, the price per token might not have changed, but in practice, the amount of tokens used to get "SOTA" performance has massively increased.
"Tokens" are just made up.
These "tokens" that are used to "measure" how much you use, they are not a real dimension that can be measured. Just an artificial counter that goes up when they decide that it should go up.
They can change the "size" of a "token" every day, and every second, and every microsecond....
Not entirely wrong, but tokens are not like an in-game currency, they're the fundamental "units" of data, both input and output, processed by the model. They're not completely untethered.
This is near the edge of my limited understanding, but AFAIK, yeah they can mess with token costs and billing schemes all they want. if they wanted to change the actual size/definition of what a token is though, that would require a whole new model (or at least a major update).
You aren't totally wrong. Such a unit exists and it is also called tokens, that can measure the capability of a model and the size of a running operation in a model.
But what they use for calculating your bill is something different today.
That doesn't make much sense. When Anthropic moved to Sonnet 5 they introduced a new tokenizer which increased token use up to 35%. If these would be unrelated kinds of tokens why would the usage go up when the process of tokenization changes?
I think you might have it mixed up with parameters, rather than tokens. Parameters are how big the model is, and are an indirect measure of how capable it is. Bigger models tend to be more capable.
The tokenizer varies a little, but I don't think it's changed measurably from tokens. You pay an amount for a million tokens worth of processing. The tokeniser difference just alters how text is converted to tokens, but the tokens themselves don't change all that much.
If anything, I'd honestly put the issue more with reasoning chains in models, where they basically babble to themselves inside of a tag, that most interfaces hide/collapse. It makes them work better, but vastly increases the amount of tokens per operation.
They have been getting longer and more sophisticated with newer models. So you might have a model now that basically repeats the output multiple times whilst refining and drafting the non-reasoning output.
If you're making it generate a lot, that'll balloon the usage, and thus price.
Yeah pretty much. Tokens are how models parse sentences.
It’s a wonky thing to charge by because each model tokenizes sentences differently. A sentence that would be 10 tokens in Claude can be 15 in OpenAI.
It’s why it’s crazy to try and charge by it and track employee usage by it.
Tokens are well-defined groups of bytes ranged by frequency of occurrence in texts to efficiently translate them into a sequence of 32 or 64-bit binary integers, an LLM-optimised form if compression. They are well-known, you can play with them here: https://gpt-tokenizer.dev/
People are arguing with you that tokens are "real" - they miss the point. You cannot predict how much "tokens" you will spend for a given prompt.
That's the problem you're highlighting, we are charged for a metric we cannot estimate before buying to make an informed decision how many we want to spend vs the quality of the outcome.
The charge and quality is arbitrary, and we can't trace if we actually spent as much as we were charged for.
Anyone who waffles about tokens being real should - I believe missed your point.
Maybe you're confusing tokens with the "credits" you pay for. Tokens have a technical meaning, but some companies are charging per AI credit, where they don't tell you the conversion rate of credits to tokens, so they can change this at any time, or vary it between models, etc.
Eh, they can be manipulated but I suggest you read on what a token is and how JTS used. What you are feeling here (with more being used for the same task) is multi modal llms working in unison, thus consuming more tokens for the same task to make your answers potentially better.
It's not like that. Tokens are an inherent computational property of how a model calculates the probabilities and such to generate text.
Having said that, what a token means in terms of computation varies wildly between models and is not directly comparable. So attributing a money value to tokens in general, independently of the model, is weird by nature.
And even within a model, the number of tokens needed to generate a response is very variable too, depending of the model itself and the parameters with which it has been configured (thinking mode, temperature, etc.).
So yeah, companies can pretty much set any price they want and there's not much anyone can do about it.
It does make sense for the provider as those for a specific model provide a good measure for computational effort, for that doecific model. That doesn't mean that token rate comparison between models give you a good picture.
“just made up” if you mean arbitrarily defined, sure. It’s not like a “bit” that has an irreducible objective definition.
However it does have a definition in whatever context you’re looking at and is very real, so I can’t really agree with your whole comment.
Yes the definition could be changed to jack up prices, but prices can also just be changed.
Dollars are “just made up” too and have varying value in different contexts to different parties. Are they “not real?” Would we hand wave away the entire financial section of the news and say bah, these “dollars” are fake anyway?
So do Chuck-E-Cheese's tokens.
Indeed. Your point?
For subscriptions they use a black box metric nobody knows. For usage credits, tokens are very measurable.
The subscriptions are much cheaper than usage credits but have been nerfed in the past and will be nerfed again in the future
This has 3 upvotes at time of writing in a technology community when it's so obviously ignorant of the actual technology that it should be an object of pity or mockery depending on the vibe.
Ignorance is a problem only if you are made aware of it and nothing changes.
https://blogs.nvidia.com/blog/ai-tokens-explained/
It doesn't matter how ignorant you are as long as you hate the right thing.
My CLAUDE.md file bloated significantly. It tried to load unnecessary skills and would retain throughout the whole session. Fixing that, maintaining good wikis and using clear often really helped fixed my personal token burn.
What do you use that file for? I see Claude.md thrown around and I'm a bit curious.
Consider each session with Claude a new employee. Your Claude.md file is what it knows and how it acts out the gate. You can save memory to it or save directory links so it knows where or how to look for something.
When you use the /init command in claude code, it'll scan your whole project and write a CLAUDE.md, which is basically an overview of the project contents and architecture that it uses as context when responding to queries.
LLMs generally work in one way. They get the prompt and give an answer. CLAUDE.md, system promp, rules, memory, tool defintions, mcps are different ways to prefix your prompt with extra information or context.
Skills, or plugins, are a way to inject less information until is needed (you can think about them as prefixing your prompt with "if you are asked about pizzas, add to context separate file pizza.md").
What you could add to CLAUDE.md depends on what you're doing. Generally it should be context LLM cannot infer relevant to all/most task performed in given project.
It's added in every chat you start with Claude for that project. It's useful for including context specific to your project that it couldn't otherwise know. High level stuff like what it's for, but also details about how the folders are organized. This saves time and tokens from rescanning the whole thing every time.
Oh, thanks! That's kind of neat, like not having to type "I'm on godot 4, c#" every time you ask about some quirk.
That's exactly what it's perfect for. If you go further and detail the intent of the project and give a high level overview of the architecture, it's even better at inferring what needs to be done without a bunch of expensive file reads and asking you repetitive questions
The models are evolving. Everything uses multi modal in the bavkend, eating up more and more tokens for the same task.
I switched to caveman on Claude Code. It cuts the token count; it's the same output, and it appears to me to be faster as well.