Technology

What, hypothetically, would that look like?

Evidence of a legion of skilled hackers that never sleep would look like repeatable, preferably independently verified demonstrations that an AI system can:

• Find previously unknown vulnerabilities in real, unfamiliar software;
• Construct and execute working exploits;
• Chain multiple vulnerabilities across systems;
• Adapt when an attack fails;
• Maintain access and pursue objectives over long periods;
• Do this end-to-end with minimal human direction;
• Perform at approximately the level of skilled human penetration testers. Not merely solve capture-the-flag exercises or reproduce known exploits.

We have evidence meaningfully approaching parts of that. DARPA’s AI Cyber Challenge demonstrated autonomous vulnerability discovery and remediation. OpenAI reported that models conducting an evaluation escaped the intended research environment, chained vulnerabilities, and accessed Hugging Face’s production infrastructure. Anthropic and Meta have also reported “agentic” cyberattacks.

Those reports deserve independent scrutiny, particularly because some originate with interested companies. But the unresolved questions concern its reliability, generality, scalability and degree of autonomy. These are the important factors, of which most sources are silence.

I don’t think it’s a coincidence that (1) they financially benefit from the hot air around AI, (2) something interesting happens with AI, (3) they speak only on the points that promote hot air, while (4) leaving out the important details. That’s not a conspiracy any more than you going to the bathroom would be, once you need to piss. These CEOs don’t deserve the credit they’re given as benevolent, honest actors. They’re money hungry and will do as their interests suggest.

That is different from alleging that the incidents were fabricated. No conspiracy is required for real demonstrations to be surrounded by exaggerated extrapolation. A model can perform an impressive autonomous exploit while still being far from a dependable, infinitely scalable army of expert hackers. Both things can be true.

Based on what? They’re certainly behaving in that manner.

You’re right that my statement that the US government probably isn’t convinced was a strong one. I can dial it back.

The government’s public behavior clearly demonstrates that it considers AI strategically and militarily important. DARPA says it has invested more than $2 billion in AI for national security. The White House explicitly published a plan titled “Winning the AI Race”. The Defense Department’s current strategy explicitly describes military AI as a race and calls for “Military AI Dominance”.

So, yes: there are military research programs, substantial spending, official doctrine and concrete efforts to integrate AI into warfighting. Asking “where is DARPA for AI?” was a bad rhetorical question. It exists.

Consequently, I’ll revise my position:

It is defensible to describe the narrower US-China competition in military AI autonomous weapons, cyber operations, intelligence, targeting and command systems as an potential emerging arms race. What remains inadequately established is the much broader claim that all frontier-model development constitutes one unified, winner-take-all arms race.

My disbelief is that we’ll ever graduate from “potential emerging” to “actual” arms race. Therefore, I also believe there is better ways to invest our time and resources. Not to mention the humanitarian concerns.

This should make a global treaty relatively easy, if, as you say, no party actually believes there is a threat.

Not necessarily. States resist agreements because they are uncertain about future capabilities and want to preserve their options. Verification would also be extremely difficult; the same model, compute cluster and vulnerability research can support both defensive and offensive purposes.

Still, I think your larger point stands: if governments genuinely believe uncontrolled military AI is dangerous, they should be willing to negotiate reciprocal limits.

I would support targeted agreements covering things such as autonomous nuclear-launch decisions, fully autonomous lethal targeting, attacks on civilian infrastructure and particular forms of autonomous cyber operation. A treaty pausing “AI development” generally would be much harder to define and verify.

I would also support a local measure to slow AI development while we find a better set of training wheels. I don’t buy the concerns of an “arms race” here. In my eyes, we have much better chances of benefiting from the ordeal if we slow down. If we don’t, the benefactors are the AI companies (particularly, at our loss too).

You’d expect to know about [military programs] if they existed?

Not all of them. Classified programs obviously exist, and absence of public evidence is not evidence of absence.

But that principle cuts both ways: secrecy cannot serve as positive evidence for any particular claim either. A public argument that an arms race exists must ultimately rely on observable budgets, doctrine, procurement, deployments, research programs and demonstrated capabilities. As noted above, some of it exists but it’s all too conveniently sparse.

Pair that with the financial incentive to keep it that way, that’s what I call a cultural driver for propaganda. Propaganda can boom organically just like any other movement, which doesn’t require a conspiracy.

Are the specific semantics important?

Yes, the label smuggles in policy conclusions. It also smuggles in a the rhetoric that we can’t slow down, must win, because there’s some kind of existential threat.

Calling something an “arms race” suggests that falling behind creates an intolerable security risk, speed is inherently protective, restraint is dangerous, and ordinary regulation benefits the enemy. That is precisely how the term is being used politically.

Nevertheless, I was treating one narrow academic definition as though it controlled ordinary language. Under the broader Oxford definition, the military-AI competition may arguably qualify. I will argue that it doesn’t.

The semantic distinction still matters when someone moves from:

The US and China are competing in consequential military-AI capabilities

to:

Therefore, every restriction on American commercial AI companies threatens national security.

That conclusion does not follow automatically. General frontier-model development overlaps some with military capability, but the two are not identical.

You do if you are claiming that the known hacks from AI companies aren’t real.

That is not the claim I was trying to make. My claim is that real capabilities and incidents exist, but companies have incentives to present the strongest results as representative, extrapolate from demonstrations to future dominance, and characterize policies benefiting them as national-security necessities. All while none of it is necessarily qualifying as the risk they present it to be. Meanwhile, you take on the responsibility of paying out of your livelihood to support their endeavors — because of the supposed risks. Well, if the risks are overplayed, then what?

That requires no coordinated conspiracy. It requires ordinary institutional incentives, selective emphasis and mutually reinforcing beliefs. Most employees would not need to deceive anyone or even agree about the geopolitical implications.

An antivaxxer could say that same exact thing.

Fair. Saying my position is “growing” does not make it correct.

If it is an arms race, “worrying about ourselves” would be continuing to develop AI at breakneck speed.

Agreed: I begged the disputed question with that wording.

But even accepting that a military-AI arms race exists does not establish that breakneck, minimally regulated commercial development is the optimal response. It could favor:

• Concentrated investment in defensive cybersecurity;
• Hardened critical infrastructure;
• Carefully tested military systems;
• Limits on publishing or exporting dangerous capabilities;
• Mandatory security evaluations;
• Counterintelligence and model-weight protection;
• Arms-control negotiations;
• Slower deployment of unreliable systems.

…all while more so maintaining our homes, sanctity, budgets, and privacy.

An arms race creates an argument for maintaining capability. It does not prove that maximum aggregate speed is safer than targeted development, testing and regulation.

So, why do we entertain the narratives pushed primarily by tech CEOs when it creates exactly the difference which we should seek not to create?

Without discussion, or without coming to the conclusion you want?

There is clearly substantial discussion. My complaint should have been more precise: the existence of public discussion does not mean the public has meaningful influence over the decisions, nor that the costs and benefits are distributed democratically.

I do have a preferred conclusion, and disagreement with it is not evidence that discussion is illegitimate. The question is whether “China” is being used to foreclose ordinary evaluation of labor displacement, energy use, surveillance, market concentration and safety—as though the arms-race premise settles every downstream policy question.

I don’t believe this social and economical pain is worthwhile. The narrative is overblown to justify it. That doesn’t require a conspiracy.