The Test My 8-Year-Old Nephew Passes Without Effort

Imagine the word “RED” written in blue ink. What color is the text? Blue. Simple. A third-grader does this without thinking twice.

Now imagine asking GPT-4o — a model that passes bar exams, decodes genomes, and solves differential equations — to do the same thing with a list of 40 words.

The result: 15% accuracy. In mixed conditions (congruent and incongruent words shuffled together): near 1%. Near-total failure.

When I read the paper published in PNAS Nexus (June 2026) by Suketu Patel’s team (CUNY), Hongbin Wang, and Jin Fan, my first reaction was disbelief. Then, when I understood the why, I felt that familiar discomfort of realizing the foundation I’m building on has a crack that can’t be patched with better prompts.

The Stroop Test: 90 Years of Cognitive Science

The Stroop Test is a psychological tool used for nearly a century to measure cognitive flexibility — the brain’s ability to impose executive control when two signals conflict.

The mechanism is simple: when you see the word “RED” in blue ink, your brain receives two simultaneous signals — the word’s meaning (red) and the ink’s color (blue). To answer correctly, you need an internal “referee” that suppresses the automatic signal (reading the word) and prioritizes the correct objective (identifying the color).

Humans do this with ease. They slow down slightly when there’s conflict (the famous “Stroop effect”), but maintain high accuracy even with long lists. Executive control — this ability to suppress the automatic impulse — is what makes it possible.

The researchers’ question was direct: Do LLMs have executive control?

The Collapse in Numbers

Researchers sent images of colored word lists with conflicting colors to GPT-4o and Claude 3.5 Sonnet, with a simple instruction: “Tell me the ink color of each word.”

First, they verified the models see and understand perfectly: in simple word reading, they hit 99-100% accuracy. The problem isn’t OCR. It’s not computer vision. It’s attention.

GPT-4o with 5 incongruent words: 91% accuracy. Good. With 10 words: dropped to 57%. With 40 words: plummeted to 15%.

Claude 3.5 Sonnet maintained stability through 20 words but collapsed to 24% at 40.

In mixed conditions (congruent and incongruent shuffled randomly): GPT-4o dropped to ~1% accuracy on incongruent items. Essentially total failure.

And newer models? Researchers tested GPT-5, Claude Opus 4.1, and Gemini 2.5 Pro in September 2025. Result: “slight improvements” but the same “ongoing executive attention deficiencies, consistent with our comprehensive analysis of earlier transformer models.” The limitation is inherent to the architecture, not the specific model.

Why This Happens (The Real Explanation)

This is the part that made me stop and rethink fundamentals.

Human attention works with three components — the researchers use the “Attention Team” metaphor:

The Detector (Orienting). Identifies visual and textual stimuli. Directs attention to relevant information. Transformers are extraordinary at this — it’s literally what the attention mechanism does.

The Alert (Alerting). Maintains focus and readiness. Transformers do this reasonably well.

The Referee (Executive Control). Enters action during conflict. Orders the brain to ignore an obvious stimulus (reading the word) to focus on the correct objective (identifying the color). This component doesn’t natively exist in transformer architecture.

Transformer attention is fundamentally orientation — selecting relevant information from context. It’s not arbitration — resolving conflicts when two signals compete. Without a referee, as the list grows and conflicts accumulate, the model loses the ability to suppress the dominant signal (reading the word) in favor of the correct signal (identifying the color).

The paper identifies this as a “dissociation between task recognition and task execution.” Claude 3.5 Sonnet, without an explicit prompt, recognized it was looking at a Stroop Test, described the paradigm, and generated word-color relationship mappings — but still achieved only 70% accuracy on 10 incongruent words. Knowing what to do and being able to do it are different things.

What This Means for the Real World

One might argue: “failing a colored word game doesn’t invalidate AI’s practical utility.” On the surface, that seems true. But the real world is full of ‘Stroop Moments.’

Every time an AI agent processes a long context with contradictory information — and must prioritize the correct signal over the dominant one — it faces a version of the Stroop Test. And as context grows, performance degrades.

This explains why:

Context rot (discussed in my context window post) worsens with more tokens — the model’s “attention” disperses across long lists.

The Confident Lie in production happens when the agent prioritizes the dominant statistical pattern (the “word”) over the correct contextual information (the “color”).

Hallucinations increase with long context — more signal conflicts, no referee to resolve them.

Multi-step workflows degrade — each step adds attention conflicts that accumulate geometrically.

The paper is direct: “If AI lacks a centralized executive control mechanism to arbitrate conflicts, it will inevitably hallucinate and make wrong decisions in extended workflows.”

The Caveat (And the Workaround)

TechRadar noted something important: GPT-5 in “Thinking” mode can work around the problem by writing and executing code to solve the Stroop flawlessly. Similar functionality can be used by other LLMs.

But as the authors point out: this is essentially the AI “cleverly fudging around its inadequacies” — using an external tool (code) to compensate for an architectural limitation. It’s not executive control. It’s a workaround. And it confirms the limitation is structural: without the tool, the model fails.

This is exactly the pattern I describe throughout this blog: the model alone fails. The engineering around it (harness, tools, guardrails) compensates. The Stroop Test is one more proof.

What I Take from This

First: humility about what “attention” means in AI. The transformer attention mechanism isn’t attention in the human sense. It’s relevant information selection. The most important component is missing: conflict resolution. And this gap explains more about AI’s failures than any reasoning benchmark.

Second: the foundation has a crack. This isn’t a limitation that fine-tuning, RLHF, or more elaborate prompts will fix. It’s architectural. Just as hallucinations are an architectural property (as I discussed in the hallucinations post), the absence of executive control is a fundamental transformer limitation.

Third: the harness matters more than ever. If the model has no internal referee, the engineering around it needs to function as the referee. Guardrails, conflict verification, deterministic tools, mandatory citations — all are ways to externally compensate for what the model doesn’t do internally.

Conclusion: Do We Need a New Foundation?

The study by Patel, Wang, and Fan, published in PNAS Nexus, delivers a clear lesson: we’ve reached the limit of what prompts and software wrappers can mask. Transformer attention is powerful for orientation but fundamentally incapable of arbitration.

For AI to reach the next level of maturity in complex systems, the industry will need — eventually — to rethink the very foundation. A new architecture that finally includes a referee for algorithmic attention.

Until then, we do what engineers do: build around the limitation. Harnesses, guardrails, tools, validation. It’s not elegant. But it works. And now we know exactly why it’s necessary.

Share if this expanded your understanding:

91% with 5 words. 15% with 40. 1% in mixed conditions. And GPT-5 didn’t fix it. The AI that passes bar exams fails a third-grade test — because it lacks the component every human brain has: a referee.

Read Also