The Question That Seemed to Have an Obvious Answer

If you were sick and needed an AI diagnosis, would you choose a generic chatbot or a billion-dollar platform designed exclusively for medicine, official partner of the New England Journal of Medicine?

Any sensible person would choose the specialist. I would too. And we’d be wrong.

On June 12, 2026, Nature Medicine — one of the world’s most prestigious scientific journals — published a study from NYU Langone that turned the medical AI world upside down: general-purpose models (GPT-5.2, Claude Opus 4.6, Gemini 3.1 Pro) outperformed specialized medical AIs across absolutely every test.

Not one test. All of them. The result was so devastating that OpenEvidence — one of the tools tested — formally requested the study’s retraction from Nature Medicine. The journal declined and pointed the company to its formal rebuttal process.

This story made me rethink everything I thought about “specialization” in AI.

The Trial by Fire: Impeccable Methodology

The NYU Langone team, led by Krithik Vishwanath and Eric Oermann, didn’t run a lab benchmark. They ran a real clinical test in three stages:

500 MedQA questions (USMLE-style — American medical licensing exam) testing pure medical knowledge. Results: Gemini scored 97.4%, GPT-5.2 scored 94.2%. OpenEvidence: 89.6%. UpToDate: 88.4%.

500 HealthBench items measuring alignment with clinician expectations. GPT led at 88.0. OpenEvidence: 62.6. UpToDate: 61.3.

100 real clinical queries (RCQ benchmark) — actual questions real physicians typed into an AI system while treating patients in NYU Langone’s HIPAA-compliant GPT instance. 12 senior clinicians evaluated each response in randomized, blinded fashion, producing 1,800 annotations. Frontier models formed the top performance tier across all evaluation axes.

The most humiliating detail for the specialized tools: they performed at the same level as Google Search AI Overview — Google’s free automatic search.

A $699/year per user platform… tied with Google Search.

Why Specialized AIs Lost

This is the question that interested me most — and the answer connects directly to everything I’ve written about RAG in recent months.

Specialized medical tools depend heavily on RAG (Retrieval-Augmented Generation). They’re programmed to search external medical databases and inject articles as context. The value proposition is: “we use medicine’s most trusted sources.”

The problem is exactly what I discussed in The Confident Lie and Chunking posts: if the retrieval system brings wrong, incomplete, or out-of-context information, the final answer will be bad. It doesn’t matter how prestigious the source is — if retrieval fails, the model hallucinates with academic citations.

Frontier models (GPT-5.2, Claude, Gemini) have massive medical knowledge directly integrated into native neural weights from base training. They don’t need to “search” the article because they already understand the correlations between medical concepts deeply. The knowledge isn’t consulted — it’s internalized.

As a Psychology Today analyst wrote: “My back-of-the-envelope calculation puts the incremental knowledge these specialized tools add at around one-tenth of one percent of what a standard model already knows. The specialized layer may contribute something at the margins. What this study suggests — counterintuitively — is that it’s no longer contributing enough to matter.”

The Controversy (Because Science Isn’t Simple)

It would be dishonest not to mention: OpenEvidence didn’t accept the result passively.

On June 15, OpenEvidence sent Nature Medicine a letter requesting retraction. They alleged contamination (frontier models may have seen MedQA questions during training), inadequate methodology (12 clinicians from one institution), and design bias.

Two weeks later, on June 27, OpenEvidence published a preprint on arXiv (not peer-reviewed) with 149 physicians evaluating in head-to-head comparisons. In their study, OpenEvidence won. But as Iatrox and Fix Health analyzed: the two studies measure different things, under different conditions, with different methodologies. “Both cannot be the whole truth, and neither is simply wrong.”

Judy//Arcade summarized the correct conclusion: “Vertical AI is not automatically better just because it is vertical. The wrong conclusion is that specialist medical AI is useless. The better conclusion is that specialization only matters when it improves the outcome.”

The Real Danger in Hospitals

The urgency raised by the Nature Medicine paper is a public health alert.

Tools like OpenEvidence and UpToDate Expert AI are already being purchased and used by hospitals. Real physicians are making clinical decisions based on these machines’ responses. And since UpToDate refused 19% of queries — more than any other model tested — one in five patients potentially receives no response.

These platforms were integrated into healthcare systems driven by marketing and corporate partnerships, without undergoing independent efficacy testing before implementation. We simply assumed “specialized means better” without demanding proof.

Senior author Eric Oermann and colleagues argue the results have direct implications for procurement, reimbursement, and regulatory oversight — and call for independent, real-world evaluation before clinical AI tools enter practice.

What I Take from This

Three lessons that go beyond medicine:

RAG doesn’t substitute reasoning. There’s no use giving access to the world’s largest library (via RAG) if the ability to reason about that data is limited. General models proved that cognitive flexibility and the ability to connect complex concepts are worth more than a restricted database.

“Specialized” has become a marketing argument, not a quality guarantee. The lesson for any industry — not just healthcare: if someone sells you a “specialized AI for X,” demand independent benchmarks. Specialization may be genuine — or it may be RAG on a generic LLM with premium branding.

Independent testing is non-negotiable. Especially in healthcare, but also in any high-risk domain. If the tool hasn’t undergone independent evaluation, it’s not validated — it’s just being sold.

Conclusion: Reasoning Beats Memorization

The corporate AI market needs to stop trusting marketing pitches and start demanding rigorous independent testing. When the subject is medicine, the difference between the best AI and the second-best can cost lives.

And the deepest lesson: in a world where frontier models already understand 99.9% of medical knowledge in their weights, what differentiates isn’t access to information — it’s the ability to reason about it.

Memorization lost. Reasoning won. And that should change how we buy, sell, and evaluate AI in every sector.

Share if this shifted your perspective:

A $699/year AI tied with free Google Search. A free AI beat both. Specialization that can’t prove itself in independent benchmarks isn’t specialization — it’s branding.

Read Also