The ARC Prize organization designs benchmarks which are specifically crafted to demonstrate tasks that humans complete easily, but are difficult for AIs like LLMs, "Reasoning" models, and Agentic frameworks.
ARC-AGI-3 is the first fully interactive benchmark in the ARC-AGI series. ARC-AGI-3 represents hundreds of original turn-based environments, each handcrafted by a team of human game designers. There are no instructions, no rules, and no stated goals. To succeed, an AI agent must explore each environment on its own, figure out how it works, discover what winning looks like, and carry what it learns forward across increasingly difficult levels.
Previous ARC-AGI benchmarks predicted and tracked major AI breakthroughs, from reasoning models to coding agents. ARC-AGI-3 points to what's next: the gap between AI that can follow instructions and AI that can genuinely explore, learn, and adapt in unfamiliar situations.
You can try the tasks yourself here: https://arcprize.org/arc-agi/3
Here is the current leaderboard for ARC-AGI 3, using state of the art models
- OpenAI GPT-5.4 High - 0.3% success rate at $5.2K
- Google Gemini 3.1 Pro - 0.2% success rate at $2.2K
- Anthropic Opus 4.6 Max - 0.2% success rate at $8.9K
- xAI Grok 4.20 Reasoning - 0.0% success rate $3.8K.
(Logarithmic cost on the horizontal axis. Note that the vertical scale goes from 0% to 3% in this graph. If human scores were included, they would be at 100%, at the cost of approximately $250.)
https://arcprize.org/leaderboard
Technical report: https://arcprize.org/media/ARC_AGI_3_Technical_Report.pdf
In order for an environment to be included in ARC-AGI-3, it needs to pass the minimum “easy for humans” threshold. Each environment was attempted by 10 people. Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Many environments were solved by six or more people. As a reminder, an environment is considered solved only if the test taker was able to complete all levels, upon seeing the environment for the very first time. As such, all ARC-AGI-3 environments are verified to be 100% solvable by humans with no prior task-specific training
It's the fact of subjective experience - the warmth of a campfire, the bitterness of lemon, the greenness of green. We're essentially talking about consciousness here. The fact that there's something it is like to be.
While nobody knows what consciousness is or how it comes about, what I mean by it is best captured by the philosopher Thomas Nagel in his aforementioned essay "What Is It Like to Be a Bat?"
The premise still strikes me as odd. How can we know it’s like anything to be anything, if we can not know what it’s like to be anything else? Coming from a premise that, to truly understand anything, you must also understand what it is not.
Is it really fair to presume, from our biased perspective where “likeness” is an abstract quality of “being,” that everything ought have a manner of which it is like to be?
What about the totality of the universe, to include all its embedded agents. What would that be like? Would an ever small portion of that likeness include precisely what it’s like to be me?
Do you think it would be possible to qualitatively describe and differentiate between two distinct phenomenologies, one day? Not just behaviorally, but to actually differentiate between their internal processes — what it’s like to be them?
And what might it be like to be a whirlpool, lightning, or even an entire ecosystem? Would that strictly be as ludicrous as asking “what might it be like to be a rock,” or is there something else to be said given whirlpools, lightning, and ecosystems are more-or-less events rather than objects?
I don’t disagree with the argument you shared… I think there’s an obvious difference between what it’s like to be a bat versus a human, but I also feel like we’re missing something important that clearer terminology could work out.
Because it undeniably feels like something to be in this very moment from the perspective of my subjective experience. In fact, I'd even go as far as to claim that it's the only thing in the entire universe that cannot be an illusion. I could be a mind living in a simulated universe on an alien supercomputer, with every person I've ever interacted with just being a convincing AI, or I could be a Boltzmann brain - but what remains true despite all that is that something seems to be happening.
I think the closest we can get to true unconsciousness that you can still come back from is general anesthesia. It's nothing like sleep. It's like that period of time doesn't even exist. It's like the time before you were born.