iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A 2023 study found that ChatGPT answered about 72% of questions correctly across a set of fictional clinical cases. That result describes one early version of the chatbot working through text-based textbook vignettes—not its accuracy with real patients, and not proof that it can safely diagnose or treat anyone.
What the 72% result actually measured
Rao and colleagues evaluated ChatGPT using all 36 available clinical vignettes from the MSD Manual. The cases presented structured clinical scenarios and asked questions at successive points in a clinical workflow: differential diagnosis, diagnostic testing, final diagnosis, and management. The researchers removed questions that depended on images because the ChatGPT interaction they tested was text-based.
Three independent users tested prompts, and two independent scorers evaluated the answers before reaching consensus. The outputs came from the ChatGPT version available on January 9, 2023. Across the study’s clinical-reasoning tasks, the reported overall accuracy was 71.7% (95% CI 69.3%–74.1%). The study appeared in the Journal of Medical Internet Research; IEEE Spectrum’s account provides a plain-language description of the results.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteHere, accuracy means the share of answers in this selected set that evaluators judged correct. It is not a percentage of patients who would receive the right diagnosis, a measure of patient outcomes, or a head-to-head comparison with doctors in routine care.
#1 Best Overall
Performance varied by stage of the clinical workflow
ChatGPT did not perform equally well at every point in the cases. It was less accurate when asked to generate an initial differential diagnosis from limited information than when asked for a final diagnosis after more case details were available.
| Task in the vignette study | Reported accuracy | What it indicates |
|---|---|---|
| Initial differential diagnosis | 60.3% (95% CI 54.2%–66.6%) | Performance when proposing possible explanations early in the case. |
| Final diagnosis | 76.9% (95% CI 67.8%–86.1%) | Performance after the vignette supplied more information. |
| Testing recommendations and management/follow-up | About 69%, as summarized by IEEE Spectrum | Performance on questions about further tests and subsequent care in this study. |
| Miscellaneous clinical-detail questions | 76%, as summarized by IEEE Spectrum | Performance on other clinical questions in the workflow. |
The first two figures and their confidence intervals are reported in the primary study; the latter two are rounded figures from IEEE Spectrum’s report. The stages are not interchangeable: a model that answers a final-diagnosis question more accurately after receiving a fuller case has not thereby shown that it can reliably recognize what information is missing, choose the right tests, or guide care from the beginning.
Can ChatGPT make clinical decisions for actual patients?
The vignette study does not establish that. Its cases were fictional and standardized, its interaction was text-only, and its endpoint was judged answer accuracy. It did not test clinical outcomes or show that using ChatGPT improves care in ordinary practice. The authors also noted the possibility of hallucinations and uncertainty about the model’s training-data composition.
Recommended Free Tools
Evidence from a different kind of experiment offers a limited signal about potential assistance. A randomized pre-post study involving 50 US-licensed physicians found that providing ChatGPT-generated advice improved decision accuracy in a single chest-pain vignette scenario, without introducing or worsening the tested race or gender differences. The study was a 2023 medRxiv preprint. It measured physicians’ decisions in one controlled scenario; it does not show that patients can rely on ChatGPT or that the result generalizes to other clinical situations.
Rank #3
As Paul Root Wolpe, director of Emory University’s Center for Ethics, put it in IEEE Spectrum’s coverage: “I think that well-tested and designed chat programs can be an aid to physicians; they should never replace physicians.” That distinction fits the evidence: clinician assistance is a question for careful evaluation, not a reason to treat a chatbot as a clinician.
Does the evidence show ChatGPT is unbiased?
No. The 2023 vignette study’s analysis did not find age- or gender-associated differences in its particular setup, but that limited result cannot establish that the system is generally free of bias. A later study compared ChatGPT with 265 medical residents across five previously published experiments designed to induce bias. For case-intrinsic bias, diagnostic accuracy declined by an average of 12% for residents, 21% for ChatGPT 4.0, and 9% for ChatGPT 3.5. The authors reported that both ChatGPT and residents were susceptible when biasing information appeared in the patient history (Schmidt, Rotgans, and Mamede, Journal of General Internal Medicine, 2025).
These findings concern different models and experimental setups, so they should not be collapsed into one universal bias score. They do show why the absence of an observed age or gender difference in one vignette benchmark is not enough to support a broad claim of fairness.
What newer evidence does—and does not—add
A 2026 Communications Medicine article is described as evaluating 22 ChatGPT model versions on 45 real patient stories for care-seeking advice. That study addresses patient triage advice, not the 2023 study’s clinical-workflow benchmark. The accessible abstract does not establish detailed findings, so it cannot be used here to claim how accurately those models advised patients (Communications Medicine, 2026).
Best Value
More broadly, results from different studies are only meaningfully comparable when their model versions, dates, case sources, clinical tasks, data types, populations, and endpoints match. Answer accuracy on fictional cases, changes in a physician’s vignette decision, and outcomes for real patients are distinct forms of evidence.
Can ChatGPT replace a doctor?
The available evidence does not support treating ChatGPT as a replacement for a physician. The headline result comes from a specific early model completing structured fictional cases, while the clinician-assistance result comes from one controlled vignette scenario. Neither establishes safe, effective autonomous clinical decision-making in patient care. Marc Succi, a senior author of the 2023 study, told IEEE Spectrum: “Your physician isn’t going anywhere.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
