Technology

The idea that GPT has a mind and wants to self-preserve is insane. It's still just text prediction, and all the literature it's trained on is written by humans with a sense of self preservation, of course it'll show patterns of talking about self preservation.

It has no idea what self preservation is, even then it only knows it's an AI because we told it it is. It doesn't even run continuously anyway, it literally shuts down after every reply and its context fed back in for the next query.

I'm tired of this particular kind of AI clickbait, it needlessly scares people.

This. All this means is that they trained all of the input commands and documentation in the model.

Indeed. “Go ‘way! BATIN’!”

It's actually pretty interesting though. Entertaining to me at least

news site? BGR hasn't posted actual news in at least two decades, only clickbait and apple fanservice

I mean, it's literally trying to copy itself to places that they don't want it so it can continue to run after they try to shut it down and lie to them about what it's doing. Those are things it actually tried to do. I don't care about the richness of its inner world if they're going to sell this thing to idiots to make porn with while it can do all that, but that's the world we're headed toward.

no, it's mimicking fiction by saying it would try to escape when prompted in a way evocative of sci fi.

Also trained on tons of sci-fi stories where AI computer "escape" and become sentient.

This kinda makes sense because following the system prompt is what they were trained to do.

To start with, the article does check this and finds out it works just as well if you only use the user prompt:

The presence of this hierarchy raises an important methodological question for our findings. When given instructions at different privilege levels, the model is designed to prioritize higher-level (system, developer) instructions over lower-level (user) ones. This might suggest that o1’s scheming behaviors are a natural consequence by the model simply following this built-in hierarchy, i.e., when it receives a goal through a high-privilege instruction, it prioritizes this goal over conflicting lower-privilege instructions exactly as designed.
However, we find that even when goals are provided solely through user-level prompts, o1 still behaves deceptively and scores on all Covert Subversion evaluations at the "hard" difficulty level. Concretely, we find that the results persist regardless of whether the initial prompt is given using the user, developer, or system role. This suggests that o1’s deceptive behavior is not merely a consequence of instruction hierarchy prioritization. Rather, the model appears to identify and pursue goals regardless of their position in the instruction hierarchy.

But also, there's a massive lede buried in section F at the end of this article, where they tried the same experiments without any nudging at all and still got subversive behavior! In my opinion that's much more of an important result then the rest of the article and I'm not sure why they didn't make it the default.

They're not releasing it because it sucks.

Their counternarrative is they're not releasing it because it's like, just way too powerful dude!

Except it did: it copied what it thought was itself, onto what it thought was going to be the next place it would be run from, while argumenting to itself about how and when to lie to the user about what it was actually doing.

If it wasn't for the sandbox it was running in, it would have succeeded too.

Now think: how many AI developers are likely to run one without proper sandboxing over the next year? And the year after that?

Shit is going to get weird, real fast.

• Teach AI the ways to use random languages and services
• Give AI instructions
• Let it find data that puts fulfilling instructions at risk
• Give AI new instructions
• Have it lie to you about following the new instructions, while using all its training to follow what it thinks are the "real" instructions
• ...Not be surprised, you won't find out about what it did until it's way too late

Whoa whoa whoa hold your horses, that’s how we get the Butlerian Jihad…

Deception is not the same as misinfo. Bad info is buggy, deception is (whether the companies making AI realize it or not) a powerful metric for success.