Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
In a limited ImageNet classification benchmark, machine-learning models recorded lower error rates than a cited estimate for human annotators. Microsoft researchers reported 4.94% top-5 test error in 2015, compared with a 5.1% human estimate. A later paper listed Google-associated BN-Inception at 4.82%. Those results were milestones on a defined dataset and task—not proof that computers had surpassed human vision in general.
What “beat humans” meant in the 2015 result
ImageNet’s Large Scale Visual Recognition Challenge tested systems on specific computer-vision tasks, including object-category classification, localization, and detection. The headline concerns classification: given an image, a model ranks likely object categories. It does not describe every kind of visual recognition, and scores from classification should not be mixed with localization or detection results. The challenge paper explains the benchmark’s tasks and evaluation.
In February 2015, Microsoft researchers Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun reported that their PReLU network achieved 4.94% top-5 test error on the ImageNet 2012 classification dataset. In top-5 evaluation, a prediction counts as correct if the right category appears among the model’s five highest-ranked choices. The researchers compared their result with a reported 5.1% human error estimate and called it the first to surpass human-level performance on that challenge. Microsoft Research’s February 10, 2015 account reports the claim and its qualification.
That wording matters: the comparison was between a model score and a particular estimate of human performance on a benchmark. It was not a controlled demonstration that a machine sees or understands images as people do.
#1 Best Overall
How the human comparison was measured
The 5.1% figure came from trained expert annotation on the challenge’s classification task; it was not a universal measure of human visual ability. The ImageNet challenge paper also describes a different estimate: an “optimistic” classifier that could use either of two annotators’ correct answers had an estimated error of 2.4% on a sample of just 204 images. That small-sample figure is not the same benchmark as the 5.1% estimate, but it shows how annotator protocol and sample size affect the comparison. Russakovsky et al.’s challenge paper details the human evaluation.
Where the Google-associated score fits
A later comparison table in the 2016 ResNet paper lists BN-Inception at 4.82% top-5 test error, alongside PReLU-net at 4.94% and ResNet (ILSVRC’15) at 3.57%. This supports the BN-Inception score in that paper’s comparison; it is not, by itself, a direct Google announcement that its system had broadly beaten human recognition. The ResNet paper is the source for those table entries.
Rank #2
- Keep track of everything from attendance to test scores
- Spiral bound
- Measures 8-1/2" x 11"
Keep that figure distinct from Google’s earlier GoogLeNet result and later Inception accuracy figures. Google’s original GoogLeNet paper describes its 2014 ILSVRC result, while a later Google retrospective reports ImageNet classification top-5 accuracy of 89.6% for Inception V1, 91.8% for V2, and 93.9% for V3. Accuracy and error are complementary measures, but those values concern different model releases and should not be presented as the same experiment as BN-Inception’s 4.82% test error. Google’s GoogLeNet paper and Google’s Inception retrospective provide that context.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →Why these scores are not a simple machine-versus-human ranking
A lower percentage is meaningful only when the figures measure sufficiently comparable things. For ImageNet claims, check the task, dataset split, metric, model setup, and human protocol before treating scores as a head-to-head contest.
- Task: classification, localization, and detection are different evaluations. A result on one does not automatically establish performance on another.
- Split and metric: test error, validation error, and accuracy should not be casually combined. The PReLU and BN-Inception figures cited above are top-5 test-error figures; Google’s Inception retrospective reports top-5 accuracy.
- Model form: a single network and an ensemble of networks are not equivalent comparisons. The cited 2016 table and the official 2015 challenge archive include results with differing contexts.
- Human protocol: annotator expertise, the number of annotators, how answers are combined, and the images sampled can change the human estimate.
For example, the official 2015 challenge archive lists MSRA classification-and-localization ensemble entries with 3.567% classification error. That is a 2015 challenge result, not the same experiment as Microsoft’s February 2015 PReLU paper result. The official ILSVRC2015 archive gives the challenge entries.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What the researchers said about the limits
Microsoft’s report quotes the researchers’ conclusion that their result was the first to surpass human-level performance “on this visual recognition challenge.” It also reproduces their explicit caution: “While our algorithm produces a superior result on this particular dataset, this does not indicate that machine vision outperforms human vision on object recognition in general.” They noted that machines still make obvious errors on elementary categories in cases trivial for people. The Microsoft Research account preserves both the milestone claim and its boundary.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
