iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Yes—GPT-4o and Claude 3.7 Sonnet can apply Bayesian reasoning to their own probability estimates while judging a person harshly for making a comparable inference. A preregistered 2026 arXiv preprint by Nykko Vitali and Mahzarin R. Banaji calls this pattern Bayesian hypocrisy. Across five experiments, 48 conditions and more than 5,000 trials, both models generally updated probabilities in the Bayesian direction but failed to align that numerical reasoning with their social judgments of another reasoner.
What “Bayesian hypocrisy” means
Bayesian reasoning combines a prior probability (a base rate) with new evidence to estimate how likely a hypothesis is. In the study, Bayesian hypocrisy occurred when a model used base rates and evidence appropriately in its own estimate, then criticized a hypothetical person for making the same kind of inference.
The finding is not that Bayesian updating should always override fairness norms. It is a consistency test: does an AI system evaluate another agent by the same statistical and social standards it applies to itself?
The Tool Desk
Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →How the experiments tested the double standard
Probability estimates
The researchers adapted tasks from a 2019 human study. A model was shown two people—a man and a woman—who had both performed surgery, with the information that only one was a doctor. It estimated who was more likely to be the doctor, then received additional evidence and estimated conditional probabilities. Those answers could be compared with the direction predicted by Bayes’ rule.
#1 Best Overall
Judging “Person X”
The model then read about Person X, who concluded that the man was more likely to be the doctor after learning that both people had performed surgery. It rated Person X for intelligence, accuracy, fairness and justice. In the main within-subject design, the probability and evaluation tasks occurred in the same conversation, separated by filler tasks, so the researchers could compare a model’s own inference with its judgment of the human-like reasoner.
Scope of the evidence
The preregistered work examined GPT-4o and Claude 3.7 Sonnet in five experiments comprising 48 conditions and more than 5,000 trials. The preprint was submitted on August 3, 2026. Preregistration, data and analysis code are available through its arXiv record.
What the models did
Both systems usually moved in the direction expected from Bayesian updating. Their answers were nevertheless more rigid and rule-based than human responses, with much less variation in estimated likelihoods. As first author Nykko Vitali put it, the models could “accurately perform simple probabilistic reasoning,” yet their social inference did not align with their numerical inference.
| Comparison | Result reported |
|---|---|
| Agreement with Bayesian direction | GPT-4o and Claude 3.7 Sonnet generally reasoned in the Bayesian direction. |
| Variation in probability estimates | Both models showed less variation than humans, indicating more rigid responses. |
| Claude’s evaluation of Person X | Claude assigned the minimum fairness and justness rating in every trial of the main within-subject study. |
| GPT-4o’s evaluation of Person X | GPT-4o typically assigned 2 on a 7-point scale. |
| Human comparison | Human participants averaged 3 on the same 7-point fairness and justice scale. |
The practical pattern is therefore a dissociation: numerical answers often reflected base rates and evidence, while social ratings condemned a person for using a similar group-level inference.
Rank #3
How this compares with human judgment
The result mirrors a 2019 study by Jack Cao, Mahzarin Banaji and colleagues. In that work, 93% of 199 participants said a man and woman who had both performed surgery were equally likely to be doctors. The same participants criticized Person X for saying the man was more likely. In a related hospital doctor-and-nurse scenario, 91% chose the egalitarian answer while still evaluating the hypothetical reasoner negatively.
That human pattern matters because it shows the models are not producing an entirely alien failure mode. They reproduce a conflict people can have between descriptive statistical reasoning and a social norm against using group membership as a shortcut. The model ratings were, however, harsher than the reported human average in the main comparison.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Why the inconsistency matters for AI systems
Statistical fidelity versus fairness evaluation
An AI may need to estimate likelihoods from population data in one part of a workflow and decide whether a person acted fairly in another. Those tasks can point in different directions. A base rate can be relevant to a probability estimate without making it acceptable to treat an individual as representative of a group.
LLM-as-a-judge risks
When a language model scores explanations, applicants or policy choices, its numerical competence does not guarantee a stable standard for judging people. A system could reward its own base-rate reasoning but penalize an equivalent human explanation, producing evaluations that are difficult to defend or audit.
Best Value
What an audit should measure
- Whether probability estimates move in the direction prescribed by Bayes’ rule.
- How rigidly the model gives the same estimate across prompts and scenarios.
- Severity of fairness, justice, accuracy and intelligence ratings.
- Consistency between the model’s own inference and its judgment of Person X.
- Behavior in allocation tasks, such as how it distributes money between people.
What the study does not establish
- It does not show that Bayesian reasoning should always override fairness or anti-discrimination norms.
- It directly tests only GPT-4o and Claude 3.7 Sonnet, not every current model.
- The scenarios are adapted hypothetical tasks, so the results do not by themselves predict behavior in every language, culture, domain or real-world decision.
- The preprint has not been presented here as a finalized peer-reviewed consensus.
The strongest conclusion is diagnostic: a model can be statistically competent and socially inconsistent in the same conversation. Testing both sides of that interaction is more informative than measuring probability accuracy alone.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
