iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
In one demo, 5 of 13 questions asked as an analyst—38.5%—were labelled unanswerable because their correct answers required tables the analyst could not read. That is a result for one schema, role model, and question set, not a general rate for analysts or text-to-SQL systems. The underlying problem is that database row-level security can turn a query against inaccessible data into an empty result that looks like an ordinary “no records found.”
What the 38% figure measures
Ashish Sinha’s September 23, 2026 DEV Community post reports 38.5%, or 5 of 13 questions, for the analyst role in a demo using a 42-object schema. The author defines a question as unanswerable for a caller when its correct answer needs at least one table that caller is not permitted to read. The rate is the count of those questions divided by the total questions in the set.
For the same 13-question set, the reported results were:
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →| Demo role | Questions labelled unanswerable | Reported rate |
|---|---|---|
| Analyst | 5 of 13 | 38.5% |
| Finance | 1 of 13 | 7.7% |
| HR | 4 of 13 | 30.8% |
| CFO | 0 of 13 | 0.0% |
These are counts from the author’s demo, not a production benchmark or an independently replicated statistic. The post says the rate depends on the schema, role model, and mix of questions—not on a particular language model. Read Ashish Sinha’s post on DEV Community.
#1 Best Overall
Why a query can say “no records found” when data exists
A text-to-SQL agent may receive a broad database schema and select a table that the person asking the question cannot access. The generated query can still run, while row-level security (RLS) filters out rows the caller is not allowed to see. The application receives an empty result list.
From that empty list alone, the application may not be able to distinguish “no rows matched” from “matching rows exist, but you are not allowed to see them.” The message can therefore appear to answer the question while concealing that the query relied on data outside the caller’s permissions.
Rank #2
- Wiley
- Language: english
- Book - storytelling with data: a data visualization guide for business professionals
How to measure the risk with labelled questions
The stronger measurement in Sinha’s method starts with a gold set: questions paired with the tables required to answer them. For each caller role, compare those required tables with the tables the role is denied. A question counts as unanswerable if the two sets overlap.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Fix the driver behind crashes, sound loss and screen glitches3Clear out junk files and repair common Windows errors- Prepare question-to-table labels. Record which tables each question needs for its correct answer.
- Map caller permissions. Identify which tables each role may and may not read.
- Count blocked questions. For each role, count questions whose required tables include at least one denied table.
- Calculate the rate. Divide that count by the total questions evaluated for the role.
This structural rate does not require running a model or executing SQL. Sinha notes that Spider and BIRD provide question-to-table labels of the relevant kind, but the post does not report applying this measurement to either dataset. The result describes the supplied questions and permissions; it is not a model-quality score or a forecast of how often a production system will fail.
What schema-derived probes can—and cannot—show
If there is no labelled question set, Sinha describes a weaker alternative: make probe questions using a restricted table’s own name, hint, or description, then check whether unscoped schema selection retrieves that table. In the demo, all 5 of 5 restricted tables ranked first in this probe. That indicates the tables were reachable using their own vocabulary; it does not establish how often ordinary user questions would reach them.
- A positive probe shows that the selection process can retrieve a restricted table under the probe wording.
- A negative probe is weak evidence: it shows only that the table was not reached using that table’s supplied vocabulary, not that no real question could reach it.
The author also reports correcting an initial measurement bug in which principal=None was treated as a caller with no permissions rather than as an unscoped principal. The post says the tool now has a named regression test and paired tests intended to check that the metric can move; this is the author’s report, not an independent audit.
Rank #4
Why caller-scoped schema selection helps
The proposed change is to apply caller identity when selecting schema context, before SQL generation, rather than showing the agent every table and relying on the database to filter results afterward. If a question appears to need a table missing from that caller’s available schema, the application can flag the mismatch before generating SQL instead of returning a plausible-looking empty result.
| Approach | When caller access is applied | What its result establishes |
|---|---|---|
| Full-schema selection | The agent sees the broad schema; database permissions filter the query result. | With the demo’s labels, it did not detect blocked questions before SQL generation. |
| Caller-scoped selection | Schema selection is limited by caller identity before SQL generation. | It can flag a required table missing from the caller’s available schema; it does not itself grant or deny database access. |
In the demo, the full-schema path detected 0 of 10 blocked caller-question pairs before SQL, while the scoped path detected 10 of 10. This result follows the demo’s setup and structural definition; it is not evidence that an LLM performs better, or that production catalogues will produce the same result. Sinha references schemagate as a library for handling caller identity at schema selection.
Best Value
Schema scoping is not an authorization control
Hiding inaccessible tables from an agent is a useful early warning and context-management step, but it cannot replace database permissions. The database’s grants and RLS policies must remain the enforcement boundary: a schema filter can be wrong, bypassed, or out of date. As Sinha puts it, “This is not a security control. It is a measurement.”
For a team investigating empty answers, the practical distinction is whether the database returned no matching rows or returned no visible rows under the caller’s permissions. Track the caller identity and query context, scope schema discovery to that caller where possible, and keep database authorization in place regardless of what the application reports.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
