iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
An AI agent can surface contextual security issues in code changes, but one engineer’s four-month trial suggests the first pass is too noisy to trust on its own. In a workflow covering about 300 pull requests, a second agent that tried to disprove each finding narrowed the list for human review. The results are one practitioner’s account—not a benchmark or a general estimate of accuracy.
How the 300-PR review process worked
In a DEV Community post, author yureki_lab described using a dedicated security-review agent alongside a separate verifier. The reviewer ran through a Git hook and CI, examined the pull-request diff and context from changed files, and had read-only permissions. Documentation-only changes were skipped.
The setup was intentionally separate from the code-writing agent. Rather than asking for a general security review, the author gave the reviewer a list of threat classes to check. For every changed file, it had to address each class with either a finding or an explicit N/A:
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →- Authorization and ownership checks
- Injection
- Secrets
- Server-side request forgery (SSRF) and path handling
- Cryptography
- Deserialization
- Rate limiting and denial of service
A candidate finding needed to name the file and line, identify the attacker-controlled input, describe a concrete exploitation path, and explain its severity. That level of detail made the claims testable rather than just plausible-sounding warnings.
#1 Best Overall
A second agent tried to refute each finding
A separate verifier assessed every candidate and was instructed to default to refuting it. A finding survived only if the verifier could construct a concrete exploitation path using actual identifiers and call paths in the code. Survivors became blocking pull-request comments for a human to review; refuted candidates were logged instead of shown as alerts.
What the author reported
The author said the workflow reviewed about 300 pull requests over four months, mostly in TypeScript and Python services. The figures below are rounded counts manually tallied from the author’s log; they are not independently audited.
| Stage | Author-reported count |
|---|---|
| Pull requests reviewed | About 300 |
| Raw first-pass findings | 412 |
| Findings that survived verification | 47 |
| Findings the author confirmed as real | 11 |
| Issues the author considered serious | 3 |
The author described the three serious issues as a missing ownership check on a document endpoint, an error-path log that exposed a configuration object containing a token, and path traversal through a user-supplied export filename. These are examples from the author’s account, not independently reproduced vulnerabilities.
The counts do not establish a general precision or recall rate. The account does not provide an independently labeled set of missed vulnerabilities, so it cannot show how many real issues the workflow failed to find. The figures also reflect one person’s manual tally, not a controlled evaluation.
Rank #3
Why the extra verification step mattered
The first pass produced 412 candidates, while 47 survived the verifier and 11 were confirmed by the author. That gap illustrates why a plausible finding should not automatically become a blocking comment: a reviewer needs to connect an input to a real path through the code, and a human still needs to assess the findings that remain.
The author’s examples point to a kind of contextual review: checking not only whether a request is authenticated, for instance, but whether the user is authorized to access a particular document. The author said the agent surfaced multi-file or semantic issues, while established static-analysis tools in this workflow were better suited to known patterns. That is the author’s experience, not evidence that AI agents generally outperform static analysis.
Practical lessons from the workflow
- Separate review context from code generation. A dedicated reviewer avoids making the security check merely an extension of the code-writing agent’s task.
- Name the threat classes. A checklist helps make coverage explicit and makes it easier to see which areas the agent considered.
- Require an exploitable path. File, line, attacker-controlled input, call path, and severity rationale give a person something concrete to verify.
- Use a skeptical verifier. Ask a second agent to try to disprove candidates, with survival dependent on a concrete path through actual code identifiers.
- Keep the reviewer read-only. The security reviewer can inspect and report without changing the code it is assessing.
- Run static analysis alongside the agent. The author’s workflow treated the methods as complementary rather than presenting the agent as a replacement.
- Skip changes judged irrelevant to the threat model. This workflow excluded documentation-only diffs; that is a scope choice, not a universal rule for every repository.
- Keep a human responsible for triage. In this setup, verified candidates became comments for human review, not automatically accepted security defects.
What this account does—and does not—show
yureki_lab’s post is a single practitioner’s report, with rounded counts tallied by hand. It shows one way to use an AI agent as a security-review aid and gives examples of issues the author says it helped surface. It does not establish a general false-positive rate, recall, or superiority over static-analysis tools.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errorsThe author also described service-specific threat models and tracking findings across pull requests as future work being explored. The post did not report results for either idea, so they should be treated as open directions rather than demonstrated improvements.
Best Value
For the author’s description of the workflow and its results, see the DEV Community post by yureki_lab. The search result displays “Aug 22” but does not establish a publication year. The author’s version note identifies Claude Code (Aug 2026 release line), Node.js 22.x, Python 3.13, and Semgrep OSS.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
