iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Yes—but only when the claim is specific, the conditions are defined and the result is observable. AV-Comparatives argues that “AI security” is too broad to serve as one reliable, reproducible test category. A test can measure whether a security product detects malware or whether a defined control blocks a particular agent attack; neither result, by itself, proves that AI security as a whole has been tested.
Why “AI security” is too broad to test as one category
In an article published on 26 August 2026, AV-Comparatives explains why it does not treat “AI security” as a single, coherent test category. The label can cover very different things: AI-enabled security products, such as malware or phishing protection, and AI systems or agents that themselves need protection.
Even within a security product, machine learning may operate alongside behavioral analysis, anomaly detection and other mechanisms. When a product detects or blocks a threat, an outside evaluator may not be able to identify which component caused the result. AV-Comparatives therefore favors measuring the security outcome—such as detection or prevention—rather than attributing it to AI.
Free tools Windows power users keep installed
One-click scans. No signup required.
What a test can credibly measure
Security products that use AI
A product does not have to expose its internal workings for an evaluator to test an outcome. The question can be whether it detects malware, blocks an attack, avoids false positives or produces useful EDR telemetry. These results describe the product’s performance under the test conditions; they do not establish that AI, rather than another component, produced the result.
#1 Best Overall
This outcome-based approach fits AV-Comparatives’ existing testing methods. Its methodology page describes real-world protection, performance, anti-phishing and malware-removal tests. Its Advanced Threat Protection archive covers evaluations using hacking and penetration techniques against targeted threats, including exploits and fileless attacks. These tests assess security outcomes; they do not claim to isolate AI as the cause.
AI agents and their safeguards
An agent assessment can be meaningful when it tests a particular control against a specified threat. For example, a test could expose a product that claims to protect an agent from indirect prompt injection to controlled malicious content, then check whether the attack leads to an unauthorized action or data disclosure. A similarly bounded test could examine malicious tool use, unauthorized data access or attempted exfiltration.
The conclusion should be narrow: the control did or did not prevent the specified attack under the stated conditions. That is useful evidence, but it is not proof that “AI security” in general has been tested.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →Why agent-test results depend on the setup
Agent behavior can change with the model and version, system prompt, framework, available tools, permissions, memory, context, external information and configuration. Cloud-hosted systems may also change beyond a tester’s control. Those dependencies affect what a result means and whether another evaluator can reproduce it.
Attack scenarios matter just as much. AV-Comparatives identifies direct and indirect prompt injection, malicious retrieved content, tools and plugins (including MCP servers), memory manipulation and cross-agent communication as distinct areas to consider. A fixed prompt set may primarily show how one system handled those prompts in one environment; it may not predict how a different architecture or later version will behave.
Repeating runs and applying statistical analysis can reduce uncertainty, but they cannot by themselves resolve attribution. A system might resist an attack because the tested safeguard blocked it, because the model refused, or because of another feature of the environment. A precise-looking percentage does not remove that ambiguity.
Rank #4
How to judge an “AI security” test
When evaluating a test report, check whether its scope and evidence support the conclusion it draws:
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problems- Claim and outcome: Does the report identify a specific security claim and an observable result, such as prevention, detection or data disclosure?
- System and configuration: Does it identify the relevant model and version, agent framework, prompts, tools, permissions, memory and configuration?
- Attribution: Can the evaluator show that the security control caused the result, or could model refusal or another environmental factor explain it?
- Repeatability and scope: Were runs repeated, and is the conclusion limited to the systems and conditions actually tested?
- Representativeness and currency: Do the scenarios reflect the architectures and attacks of interest, and could changes to the model or threats make the result stale?
These checks reflect the issues AV-Comparatives raises; they are not a separate test or validation of any particular report.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.What AV-Comparatives says makes a test credible
AV-Comparatives says a credible assessment needs a clearly defined subject, measurable security outcomes and a methodology that is sufficiently objective, repeatable, representative and fair. It currently regards focused functional assessments, individual product reviews and dedicated research projects as more appropriate for emerging AI-security technologies than a broad “AI security test.”
The organization received 1,328 valid responses from 87 countries for its 2026 security survey, which also covered AI chatbot use and participants’ views on potential cyber threats. Those are survey participation facts, not evidence that settles how reliably AI security can be tested. Its central article publishes no quantitative result establishing the reliability of AI-security tests.
For vendors or enterprise decision-makers seeking an evaluation, AV-Comparatives describes its cybersecurity assessment services on its services page. Andreas Clementi, CEO and Co-Founder of AV-Comparatives, summarizes the organization’s position: “Independent testing should measure what can be demonstrated, not what is currently fashionable.”
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
