Microsoft Says AI Beats Doctors; The Study Asked A Narrower Question

🔍 Read the full analysis: Microsoft Says AI Beats Doctors; The Study Asked A Narrower Question on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on office and shipping supplies

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Microsoft reported that its MAI-DxO diagnostic AI outscored generalist physicians on selected medical cases in a peer-reviewed Science paper published October 22. The study tested the system on historical, text-based case records—not patients receiving care—and compared it with individual generalists, so the results do not show how it would perform against specialist teams or in hospitals.

On 304 complex cases drawn from the New England Journal of Medicine, MAI-DxO recorded 83.2% diagnostic accuracy, according to the reported results. Generalist physicians scored 36.1% when allowed to consult textbooks and the internet, and 22.3% without those resources. On a separate set of 325 emergency department cases, the system scored 80.3%, compared with 46.8% for physicians. These figures are results reported by Microsoft and the study authors, not independent clinical outcome data.

The paper, “Sequential diagnosis with language models,” describes a benchmark in which a case begins with a short summary. A doctor or AI system must request additional information, such as a question’s answer or a test result, from a simulated gatekeeper holding the case file. The system is evaluated on its final diagnosis and the estimated cost of obtaining information. MAI-DxO uses a language model to coordinate virtual physician agents that propose diagnoses, select follow-up questions or tests, and review one another’s reasoning.

The comparison has important limits. The physicians were primary care doctors and hospital generalists working individually; the study did not include specialist teams. The benchmark used written records in a simulator, with no real patient interaction, physical examination or actual imaging. The cases were selected as difficult teaching cases, rather than a representative mix of patients presenting for care. The paper’s performance figures therefore describe results on that test, not a demonstrated level of accuracy in routine practice.

Clinical Performance Still Unproven

The results matter because diagnostic AI could affect how clinicians investigate complex cases, including which questions they ask and which tests they order. MAI-DxO’s benchmark is designed to measure that sequence, rather than simply asking a model to select a diagnosis when all case details are already provided. Its inclusion of estimated test costs also addresses a practical question: how much information the system requests before reaching an answer.

But the gap between a benchmark and patient care is substantial. Real clinicians work with incomplete or contradictory information, examine patients, use imaging and laboratory systems, and consult colleagues or specialists. They also have to account for common, harmless explanations and for the risks of unnecessary tests. The study’s difficult cases do not represent that full range. Its results may support further evaluation, but they do not establish that using MAI-DxO improves outcomes, reduces costs in practice or can safely replace clinical judgment.

The figures also do not isolate the value of Microsoft’s orchestration layer. Microsoft’s reported chart shows GPT-5 alone scoring 78.9% on the complex cases, compared with 83.2% for MAI-DxO. That suggests much of the measured performance came from the underlying model, while the system added a smaller accuracy gain and, according to Microsoft, reduced estimated testing costs. Those estimates should not be treated as observed savings in a health system.

How the Benchmark Was Built

The Science paper is the peer-reviewed version of work that appeared as a preprint in June 2025. The earlier version used a different model and reported a wider gap between the AI and doctors. Results from that preprint should not be substituted for the figures in the Science paper; details of the final study may also differ from the earlier account.

In the benchmark, a gatekeeper model supplies further details when requested. The source material reports that when a requested test result was absent from the case record, the gatekeeper generated a synthetic result. It also says a model judged answers against the published diagnosis, with physicians re-grading a sample and showing good, but not perfect, agreement. These features are part of the test design and can affect how closely its scores reflect real diagnostic work.

Estimated test prices were based on billing codes and a 2023 U.S. hospital price list, with a fixed $300 charge per physician visit, according to the source material. The authors described the costs as first-order approximations. That makes the cost comparisons useful as a feature of the experiment, but not as a direct estimate of what hospitals or patients would pay using the system.

““Building Medical Superintelligence””

— Microsoft, in its published description of the system

What the Scores Cannot Show

The study does not establish whether MAI-DxO can improve diagnoses or patient outcomes when used in live care. It also does not show how the system compares with specialists, multidisciplinary teams or doctors using the full tools available in hospitals. The benchmark’s selected cases do not include healthy patients or harmless explanations, according to the source material, so it could not measure how often the system would raise a false alarm in a broader population.

Some details reported about the benchmark come from the 2025 preprint because the full Science paper is behind a paywall; the final paper may differ on those points. The source material also says the system’s estimated test costs were model-based. Independent replication, performance across a wider range of cases and results from use with patients remain open questions.

Patient Study at Beth Israel

The next reported step is a prospective study at Beth Israel Deaconess Medical Center in Boston. Consenting patients are expected to interact with the system alongside their usual care, allowing researchers to evaluate it in a clinical setting rather than only against historical records. The source material does not provide a start date, enrollment target or expected results date.

That study will be a more direct test of how MAI-DxO works with patients, but it will not by itself answer every question about safety or wider deployment. Readers should look for details on the study design, what clinicians can see and do, how errors are handled, and whether results are independently reported. Until clinical evidence is available, the Science scores should be read as performance on a defined benchmark—not proof that the system is ready to diagnose patients on its own.

Key Questions

What did Microsoft’s MAI-DxO score on the complex cases?

Microsoft reported 83.2% accuracy on 304 complex cases from the New England Journal of Medicine. Generalist physicians scored 36.1% with textbooks and internet access and 22.3% without those resources in the reported comparison.

Did the study test the AI on real patients?

No. It used historical, text-based case records in a simulator. The evaluation did not involve real patient examinations, actual imaging or care decisions made in a hospital.

Was MAI-DxO compared with medical specialists?

The comparison group consisted of primary care doctors and hospital generalists working individually. The study did not compare the system with specialist teams or multidisciplinary groups.

Does the study show that MAI-DxO is ready for clinical use?

No. The paper reports performance on a benchmark, not evidence that the system improves patient outcomes or is safe as a clinical decision-maker. A prospective study at Beth Israel Deaconess Medical Center is the next reported step.

Source: ThorstenMeyerAI.com