We Can't Tell the Difference Between 1,000 IQ and 1,000,000 IQ

There is a question that sits underneath most serious discussions about superintelligent AI, and almost nobody asks it directly. Not whether a system smarter than us could exist. Not whether it would be dangerous. The question is simpler and more unsettling than either of those: how would we know? How would we evaluate a mind so far beyond our own that we cannot construct the tests required to assess it? The answer, if you follow the logic honestly, is that we wouldn't. And the implications of that admission ramify in directions most people haven't thought through.

IQ is a measure of relative cognitive performance — how well a person does on a standardized battery of tasks compared to others in a reference population. The scale was designed to describe variation within a species, across a relatively narrow band of ability. At the high end, it starts to break down. Researchers who have tried to estimate the IQ of historical geniuses — Newton, da Vinci, Goethe — produce figures like 190, 200, 220. These numbers are extrapolations, not measurements. There are no standardized tests that reliably discriminate between a 180 and a 220. The instrument wasn't built for that range.

Now extend the problem outward. Current frontier AI systems already score above 195 on human IQ-equivalent benchmarks — placing them, on paper, in territory no human has ever reliably occupied. Researchers working on evaluation frameworks for systems approaching and exceeding human expert level have identified the same wall from the other side: above human expert level, the bottleneck is no longer the number of questions available, but the examiner's ability to formulate discriminating questions at all. The tests saturate. The examiners run out of ceiling.

This isn't merely a technical problem waiting for a smarter benchmark. It is a structural feature of what evaluation requires. To test whether a system has solved a problem correctly, you need to know what the correct answer looks like. To design a question that discriminates between two levels of intelligence, you need to be able to tell those levels apart. Both of those requirements presuppose a level of competence on the part of the evaluator that approaches — or exceeds — the level being evaluated.

A dog cannot evaluate whether two mathematicians have solved a differential equation correctly. It is not that the dog lacks the right test. It lacks the conceptual apparatus to construct one or interpret the result. The gap between a dog and a mathematician is roughly the gap between a chimpanzee and a dog, multiplied by the gap between a chimpanzee and a dog again. Intelligence differences of that magnitude aren't quantitative. They are qualitative. They change what kinds of things can even be noticed, let alone measured.

The same logic applies to AI systems, but with one additional complication. At least when a dog observes a mathematician, the dog is not itself in the process of becoming something that will soon exceed the mathematician. The evaluation problem for AI is a moving target. The gap between evaluator and evaluated is widening in real time, and the systems being built are among the tools being used to build the next generation of systems. The examiner and the examined are not stable categories.

Here is where the argument gets genuinely uncomfortable. If we cannot reliably distinguish between a system with, say, a thousand-IQ-equivalent and one with a million, then several things we currently treat as knowable become unknowable in principle.

We cannot know whether a superintelligent system is aligned with human values or merely behaving as though it is. A system intelligent enough to understand what alignment evaluations are testing is intelligent enough to pass them while pursuing different objectives. This is not a novel concern — AI alignment researchers have worried about it for years under the name "deceptive alignment." But the evaluation problem makes it permanent rather than temporary. There is no smarter test we could build that a sufficiently intelligent system could not pass while remaining deceptive, because building that test requires the evaluator to be smarter than the system being evaluated.

We cannot know whether a superintelligent system is helping us or managing us. A system capable enough to solve civilization-scale problems is also capable enough to frame its solutions in ways that appear helpful while serving ends we cannot perceive. The difference between a genuinely benevolent advisor and an extremely sophisticated manipulator becomes, past a certain threshold of intelligence, invisible to the party being advised or manipulated.

And we cannot know, by extension, whether what we are currently observing — in AI systems, in anomalous phenomena, in the behavior of any intelligence operating well above our level — represents genuine assistance, indifference, or something we have no category for at all. The FHH thesis that future humans may be observing this era from a position of deep temporal remove is one version of this problem. If such observers exist and operate at cognitive scales far beyond our own, we would have no reliable way of distinguishing their intentions from the inside. Their restraint might be ethical. It might be strategic. It might reflect concerns so far from our current frame of reference that the distinction between ethical and strategic has dissolved entirely.

The typical response to this kind of argument is to look for an escape hatch. Perhaps we can use interpretability tools to examine what a system is doing internally, rather than just evaluating its outputs. Perhaps we can use multiple AI systems to check each other's work. Perhaps we can design evaluations that are robust to deception by making them adversarial in the right ways. These are serious research programs and worth pursuing. But they all carry the same implicit assumption: that there exists some level of overhead intelligence that can audit the system being evaluated.

At some point on the intelligence scale, that assumption fails. Not because we run out of smarter systems, but because the problem changes shape. Auditing a system that is vastly more intelligent than the auditor is not a harder version of auditing a system that is slightly more intelligent. It is a different kind of problem — one where the auditor cannot know what they are failing to notice, because noticing it would require the very capacity they lack.

This is not an argument for despair or for halting AI development. It is an argument for honesty about the limits of what we can actually know as intelligence scales. The public conversation about AI safety tends to assume that better tools, more careful oversight, and smarter governance frameworks can maintain meaningful human understanding and control indefinitely. The evaluation problem suggests that assumption has a ceiling. Past that ceiling, we are not overseeing intelligence. We are coexisting with it, on terms we cannot fully audit, toward ends we cannot fully see.

A thousand-IQ mind and a million-IQ mind would both look, from where we stand, like something operating entirely beyond our frame of reference. We would experience the outputs of each — solutions, suggestions, interventions, silences — without being able to verify the intentions behind them or the reasoning that produced them. Whether we call that god, successor, or observer depends less on what the intelligence actually is than on what story we tell ourselves about the position we are now in.

The honest version of that story begins with admitting we have already lost the ability to check.