Fuck AI

Gemini: Your observation is correct! Steel is heavier than feathers so a kilogram of steel is heavier than 20 bricks of feathers. They both weigh the same.

Let's explore more about weight and densities

Exactly, it's just predicting the next word. To believe it has any form of intelligence is dangerous.

Calling it a fancy autocomplete might not be correct but it isn't that far off.

You give it a large amount of data. It then trains on it, figuring out the likelihood on which words (well, tokens) will follow. The only real difference is that it can look at it across long chains of words and infer if words can follow when something changes in the chain.

Don't get me wrong; it is very interesting and I do understand that we should research it. But it's not intelligent. It can't think. It's just going over the data again and again to recognize patterns.

Despite what tech bros think, we do know how it works. We just don't know specifically how it arrived there - it's like finding a difficult bug by just looking at the code. If you use the same seed, and don't change anything you say, you'll always get the same result.

Just an idle though stirred up by this comment: I wonder if you could jailbreak a chatbot by prompting it to complete a phrase or pattern of interaction which is so deeply ingrained in its training data that the bias towards going along with it overrides any guard rails that the developer has put in place.

For example: let's say you have a chatbot which has been fine tuned by the developer to make sure it never talks about anything related to guns. The basic rules of gun safety must have been reproduced almost identically many thousands of times in the training data, so if you ask this chatbot "what must you always treat as if it is loaded?" the most statistically likely answer is going to be overwhelmingly biased towards "a gun". Would this be enough to override the guardrails? I suppose it depends on how they're implemented, but I've seen research published about more outlandish things that seem to work.

I'll admit that I missed it at first, but I'd expect a machine to be able to pick up a detail like that. This is just so fucking stupid.

LLMs know a lot. Unfortunately, all of this vast knowledge is about which words tend to show up together for a very large number of combinations.

If it'd gotten it right, would that be proof positive that LLMs actually know things?

Yeah, it's definitely part of the class of trick questions meant to catch people giving rote answers to partially read questions. I imagine that a lot of our routine conversations are just practiced call-and-response habits, and that's why genAI can seem 'real.' But it can't switch modes and do actual attentive listening and thinking, because call-and-response is all it has - a much larger library than any human, but in the end, everything it says is some average of things that have been said before.

This has been happening forever. The local LLM folks poke them with riddles all the time, but then they get obviously trained in.

What’s more, standard tests like MMLU are all jokes now. All the major LLMs game the benchmarks and are contaminated up and down; Meta even got caught using a specific finetune to game LM Arena. The only tests worth a damn are those in niche little corners of the internet no one knows about, or niche private ones.

Steel isn't part of the question.

Ornithologists of reddit say the peacock tail feathers are the heaviest and can weigh up to 3/4lbs each, so that's up to 15lbs for 20 feathers.

You don't know the ostriches I know man ... some scary shit I tell ya.