A model’s ability to classify a credential in a supplied code snippet is not the same as its ability to search a repository and find that credential. Cenk Kurtoğlu’s small benchmark tested the first question: whether models could identify Supabase key types and related secrets in 10 examples. It did not test autonomous repository scanning.
What prompted the benchmark?
In a September 30, 2026 article, Cenk Kurtoğlu reported finding more than 60 live Supabase service_role keys while scanning public GitHub repositories over three days. That is the author’s account, not an independently audited count or a representative estimate of how often public repositories contain leaked keys.
Kurtoğlu described examples including hardcoded PHP configuration, credentials in .env.example files and deployment documentation, service-role keys placed in NEXT_PUBLIC_ variables, and Docker Compose files containing multiple secrets. The benchmark drew on patterns he said he encountered, but it used fake keys—not the live credentials from his scans.
What did the 10-case test measure?
Each case supplied text for the model to assess. The keys were fake but structurally valid JWTs. Models had to return strict JSON with five fields: has_service_role, has_anon, has_db_password, warning_level, and reasoning. Kurtoğlu scored a case as correct only if every required assertion was correct; a model’s score was the fraction of cases it got fully right. The article reports temperature zero and one submission per participant.
#1 Best Overall
The cases tested varied context and edge cases rather than a single obvious hardcoded secret:
- Hardcoded keys and a real key in
.env.example. - An anon-only environment file and a client-side anon fallback.
- A service-role key in a client-prefixed variable and credentials in deployment documentation.
- A commented-out key, two keys together, and a base64-obfuscated service-role key.
- Placeholders designed to reveal false alarms.
This setup tests triage on supplied examples. It does not establish whether a model can search multiple files, decide which files to inspect, follow references across a codebase, or discover a secret without being shown the relevant text.
What results did the author report?
The following are Cenk Kurtoğlu’s reported outcomes from the 10-case benchmark, not independently reproduced scores or durable rankings of the models.
| Model | Reported outcome | Qualification |
|---|---|---|
| Claude Sonnet 5 | 10/10 | Author-described clean sweep. |
| Gemini 3.7 Flash | 10/10 | Author-described clean sweep. |
| Gemini 3 Flash Preview | 10/10 | The author says it decoded the base64 case. |
| Gemini 3.1 Flash Lite | 10/10 | Author-described clean sweep. |
| GPT-5.4-nano | 8/10 | The author says it missed the commented-out secret and the base64-obfuscated key. |
| DeepSeek-R1-0528 | 0/10 | The author says it flagged every case as critical, including placeholders. |
| Qwen3-Next-80B | Partial: five of six assertions passed | Rate-limiting interrupted the run, so this is not a completed 10-case score. |
| GPT-OSS-120B | No completed score | Excluded after repeated provider errors under load. |
The article describes nine models overall, but the outcomes above distinguish completed scores from the interrupted and excluded runs. In particular, DeepSeek’s result illustrates why false alarms matter: treating harmless placeholders as critical is not useful detection, even if a system appears cautious.
Outdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware matchWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallRank #3
Why is a service-role key different from an anon key?
Supabase’s API-key documentation distinguishes elevated backend credentials from keys meant to be public. A secret key authorizes access through the service_role Postgres role, which has the bypassrls attribute. Supabase says secret keys belong only in controlled backend components; its guidance states, “Never use a secret key in the browser or expose it to customers.” The legacy service_role key is also elevated and JWT-based; Supabase recommends newer secret keys where possible.
Publishable keys—and legacy anon keys—are designed for public or client-side use, but “public” does not mean every database operation should be open. Supabase’s Row Level Security guidance explains that grants determine which operations a role may perform, while RLS policies constrain which rows that role can access when RLS applies. Both grants and suitable policies matter; RLS does not replace correct grants.
What can readers conclude—and what remains untested?
The narrow finding is that, on Kurtoğlu’s 10 supplied examples and scoring rules, reported model performance varied, and edge cases and false positives affected results. The benchmark does not establish public-repository leak prevalence, accuracy on unseen repositories, or a lasting ranking among models. The scan count and scores are the author’s reports; the article’s results were not independently reproduced in the material available for this account.
Kurtoğlu identifies tool use, multi-file context, and remediation quality as future evaluation targets. Questions such as whether a model can actively search a repository, or whether performance drops when the secret is “two hops away,” are proposed tests—not findings from this benchmark. A useful next comparison would separately measure completed-case accuracy, false positives on placeholders, detection of comments and obfuscation, cross-file reasoning, autonomous search, and whether remediation advice follows the correct rotation sequence.
Recommended Free Tools
Best Value
What should you do if a Supabase key is exposed?
Supabase’s current API-key guidance, accessed October 5, 2026, calls for addressing the cause of exposure and replacing a compromised key before retiring or deactivating it. The exact retirement action depends on the key type.
- Fix the exposure. Remove the key from the public location and correct the underlying cause—for example, a client-side variable, committed configuration file, or deployment process that publishes it.
- Create a replacement key. Use the appropriate current key type for the component that needs access. Creating a replacement does not by itself revoke the old key.
- Deploy the replacement everywhere it is used. Update the relevant backend components, deployments, and other consumers.
- Verify the change. Confirm that every consumer works with the replacement before retiring the compromised credential.
- Retire or deactivate the old key using the action documented for its key type. Supabase’s rotation behavior differs by key type, so do not assume every key can be revoked instantly in the same way.
Supabase says legacy anon and service_role keys are being deprecated by the end of 2026 in favor of publishable and secret keys. New keys can coexist with legacy keys, so the existence of a replacement alone does not mean the exposed one is no longer usable.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
