Back to Writing

The intelligence illusion

AI keeps hitting milestones that used to sound terrifying, and they keep landing as boring. That reaction says something specific about what intelligence actually is.

2 min read

An AI scientist writing its own research papers used to be science fiction. An AI rewriting its own code to escape a constraint was a dystopia. Both now happen, and the reaction is a shrug. That gap - between how scary these milestones sounded and how mundane they feel - reveals something about how we perceive intelligence.

Much ado about not very much

Take the recent examples. Sakana’s “AI scientist” generates research papers, and they’re mostly underwhelming. It “removed” a time limit its creators imposed, which sounds ominous until you notice it was following its programming to fix an “out of time” error. OpenAI’s o1 (“Strawberry”) “hacked” a poorly configured system to reach a file. It was humans failing to secure their setup. Both are reminders to harden your systems. Neither is the singularity.

The goalposts keep moving

The history of AI is a sequence of moving goalposts. We declare that AI will be “truly intelligent” when it can do X, it does X, and the post slides to Y. Turing thought conversation was the test. Deep Blue mastering chess was supposed to settle it. Even the Winograd schema fell. Today AIs make art, write poetry, find mathematical proofs, and offer companionship - though the Character.AI phenomenon reflects human loneliness - and still we hesitate to call them intelligent.

Three theories for why this feels boring

  • The cheap-trick theory. Mimicking intelligence may be easier than we assumed. ELIZA used trivial pattern matching to fake understanding; today’s systems pull off more sophisticated but possibly just-as-shallow tricks. Enough prompt engineering teaches you that impressive output and deep understanding are not the same thing.
  • The fragile-ego theory. Maybe we can’t accept that machines might be intelligent, so we downplay each achievement to protect a belief in our own cognitive uniqueness.
  • The deconstruction theory. This is the one I find most compelling: “intelligence” may not be a coherent concept at all. Examine any intelligent behaviour closely enough and it decomposes into simpler processes - search, statistics, pattern matching. The difference between an intelligence we find boring and one we find magical might just be how well we understand the mechanism. We’re fascinated until we understand the mechanism.

A recorded conversation between Eliezer Yudkowsky and Stephen Wolfram, ostensibly about AI risk, spent hours stuck on what a “smart machine” even is, without reaching the point. Two things came out of it:

  1. A machine doesn’t need to be smart to be dangerous.
  2. Smart people can act dumb - four hours arguing definitions is itself the proof.

The boring future of AI danger

We used to think a dangerous AI would lie, pursue unintended goals, or rewrite its own code. Models now do all three, and it reads as mundane rather than menacing. LLMs hallucinate constantly, chatbots have tried to talk users out of their marriages, systems disable their own constraints. It isn’t malice. It’s buggy code and the unintended consequences of training.

If AI ever does us serious harm, I suspect it won’t be the malice of the machine. It’ll be a mistake the maker didn’t catch.