Quick wins for a faster PC:
Clear out junk files and repair common Windows errorsFree Scan →Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
MADQA is not a benchmark of embedding models in isolation. It is a multimodal, agentic document-question-answering benchmark: systems search a collection of heterogeneous PDFs, interpret text and visuals, gather evidence from multiple pages, and produce answers. Its results therefore describe the behavior of a complete document-QA setup—agent, model, retriever, file-search configuration and evaluation protocol—not the quality of a vector representation by itself.
That distinction matters when you are choosing a retrieval stack. MADQA can show how a tested system navigates difficult documents and how much search effort it uses. It cannot, on its own, predict latency, cost, access-control behavior or answer quality on your company’s corpus.
What does MADQA measure?
MADQA evaluates agents answering questions over a mixed collection of 800 heterogeneous PDF documents. The 2026 ICML paper by Łukasz Borchmann and co-authors reports 2,250 human-authored questions grounded in those documents. The project repository describes a workflow in which a system retrieves pages, reasons over textual and visual content, and aggregates evidence across pages.
This is broader than a conventional embedding test. A typical embedding evaluation may compare nearest-neighbor retrieval for labelled queries. MADQA instead observes a chain of decisions:
#1 Best Overall
- How the agent interprets the question and plans a search.
- Which retrieval or file-search tools it calls.
- Which pages and passages those tools return.
- Whether the system can read tables, figures, layouts and other visual elements.
- How it combines evidence from several pages.
- Whether the final answer is correct under the benchmark’s scoring rules.
The paper also uses an accuracy-effort trade-off protocol. In other words, a system is not judged only by its final answer; the amount and pattern of searching matter as well.
Is MADQA an embedding benchmark?
No. Embeddings may be one component of a system tested on MADQA, but the benchmark does not isolate them. The repository lists baselines that include BM25 search as well as managed Google and OpenAI file-search approaches. Those systems differ in indexing, query rewriting, ranking, context assembly, tool behavior and model orchestration. A score reflects the combined result.
The paper’s central question captures the broader target: “do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search?” That framing concerns navigation and decision-making over a document collection, not just the geometry of an embedding space.
Recommended Free Tools
There is no single universal “MADQA embedding score.” Results from different agents, retrievers, models or tool configurations are not interchangeable unless the rest of the evaluation setup is held constant. A higher score may come from better planning, stronger visual document handling, more effective reranking, a different answer model or simply more search attempts.
How does a MADQA evaluation work?
1. The system receives a document question
Each question is grounded in the benchmark’s PDF collection. Because the documents are heterogeneous, useful evidence may appear in ordinary paragraphs, tables, captions, charts or page layouts rather than in a single text chunk.
2. An agent searches the collection
The agent can issue retrieval or file-search actions and decide what to do with the returned pages. A strong answer may require reformulating a query, opening additional pages, or connecting evidence that is distributed across the collection.
Rank #2
- Use scikit-learn to track an example ML project end to end
- Explore several models, including support vector machines, decision trees, random forests, and ensemble methods
- Exploit unsupervised learning techniques such as dimensionality reduction, clustering, and anomaly detection
- Dive into neural net architectures, including convolutional nets, recurrent nets, generative adversarial networks, autoencoders, diffusion models, and transformers
- Use TensorFlow and Keras to build and train neural nets for computer vision, natural language processing, generative models, and deep reinforcement learning
3. The agent interprets text and visual content
The repository describes multimodal handling. The evaluated system therefore needs more than a text-only similarity match when a question depends on a figure, table structure or the relationship between elements on a page.
The Tool Desk
Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →Outbyte Driver Updater FREEFix the driver behind crashes, sound loss and screen glitchesFind Drivers →4. Evidence is aggregated into an answer
The final response is judged against the human-authored answer using the benchmark’s reported metrics. The process allows a system to be accurate for the wrong reason—for example, by repeatedly searching until it finds a useful page—so the benchmark also examines effort and behavior.
Which metrics does MADQA report?
The project’s evaluation code lists answer-level and page-level measures. Their presence is important because a final answer can look correct even when the system retrieved poor evidence, and a system can retrieve relevant pages without synthesizing them correctly.
| Metric | What it evaluates in context | How to read it |
|---|---|---|
| Accuracy | Whether the produced answer is judged correct. | A direct correctness result, but not a diagnosis of why the system succeeded or failed. |
| ANLS* | An answer-matching measure listed by the evaluation harness. | Interpret it with the benchmark’s implementation and split; do not assume it is interchangeable with another dataset’s similarly named score. |
| Semantic Accuracy | Whether the answer conveys the required meaning. | Useful when wording differs from a reference answer, but still an answer-level measure. |
| Page F1 | Overlap between pages selected by the system and pages marked as relevant. | Evidence-location quality, not proof that the final explanation is correct. |
| Kuiper | A distribution-based measure included in the repository’s evaluation code. | Use the repository’s definition and reporting conventions before comparing values. |
When publishing or comparing a result, state the split, evaluation script and metric definition. A bare number without those details is difficult to reproduce or interpret.
What did the 2026 paper find?
Best agents can match humans on raw accuracy
Borchmann and co-authors report that the best evaluated agents can match human searchers in raw accuracy. This is a benchmark-specific finding: it applies to the tested systems, questions and protocol, not to every production retrieval-augmented application.
Do these 3 things before closing this tab:
1Fix the driver behind crashes, sound loss and screen glitches2Repair Windows errors before they cause bigger problems3Scan for outdated or missing drivers - takes under a minuteHuman and agent successes differ
The paper also reports that agents and humans succeed on largely different questions. Equal aggregate accuracy can therefore hide different strengths. An agent may excel at exhaustive lookup while a person recognizes a document structure or makes a strategic jump that the agent misses.
Rank #3
Brute-force search compensates for weak planning
According to the paper, agents often rely on brute-force search to compensate for weak strategic planning. Repeated calls can eventually expose the right evidence, but that behavior may increase latency, tool charges and context-processing work in a deployed system.
A nearly 20% gap to oracle performance remains
The paper’s abstract reports that leading agents did not close a nearly 20% gap to oracle performance and persisted in unproductive loops. “Nearly” is part of the reported result; it should not be converted into an exact universal percentage. The oracle comparison is also specific to the paper’s experimental design.
What does a MADQA score tell me about retrieval?
A score can provide evidence that a complete system is capable of finding and using information in difficult PDF collections. Page F1 can help reveal whether the system tends to identify relevant evidence locations, while answer metrics show whether that evidence is turned into an acceptable response. Search-effort reporting can expose systems that achieve accuracy through excessive exploration.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
It does not tell you which layer deserves credit. Suppose two systems obtain the same accuracy. One may use a strong embedding model and a small number of targeted calls; another may combine ordinary retrieval with many retries. Without matched tool-call, agent and retriever details, the score alone cannot distinguish those explanations.
Likewise, a low score does not prove that an embedding model is weak. Failures may result from PDF parsing, missed visual information, poor query planning, inadequate reranking, context truncation or answer-generation errors. Diagnose the entire pipeline before replacing the embedding model.
How does MADQA measure search effort?
The benchmark’s accuracy-effort protocol treats searching as part of system quality. The repository and related evaluations report tool-call information, but you must check exactly what is counted. Some reports count internal agent rounds; others may expose only outer API calls. Those are not equivalent.
Rank #4
For a fair comparison, record at least:
- Total tool calls per question and the distribution, not only the average.
- Whether retries, parallel calls and hidden internal rounds are included.
- Elapsed latency and token or service usage under the same hardware and limits.
- Which calls returned useful evidence versus repeated or empty results.
- Stopping rules, maximum rounds and failure behavior.
A system that gains one point of accuracy by doubling search effort may be the wrong choice for an interactive product, while the same trade-off could be acceptable for an offline archival process.
PC Slower Than It Used to Be?
A free scan shows the junk files, broken settings and background clutter dragging Windows down - then fixes them in one click.Free scan · Windows 10 & 11Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteHow should I read vendor results?
Mixedbread publishes a vendor-maintained MADQA evaluation with model-plus-retriever configurations, accuracy figures and tool-call counts. Its page identifies the results as tested in May 2026 and explains how its agent tool calls are counted. Those details make the report useful as a dated example, but it is not an independent neutral ranking.
When you encounter a leaderboard or vendor article, check five things before drawing a conclusion:
- Configuration: identify the language model, embedding or retriever, reranker, file-search service and multimodal components.
- Metric: confirm whether the number is Accuracy, Semantic Accuracy, Page F1 or another measure.
- Counting rules: determine whether tool calls are internal rounds, external requests or a vendor-specific count.
- Evaluation date: software, models and service defaults can change; retain the stated test date.
- Independence: label vendor-run results as vendor evaluations rather than presenting them as an impartial ranking.
Use the public benchmark scripts and fixed split where possible. Do not compare a vendor’s optimized configuration with a paper baseline run under different limits and call the difference an embedding advantage.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Can I use benchmark rankings to choose an embedding model for my data?
Not by themselves. MADQA’s documents and questions may differ sharply from your corpus, language mix, terminology, permissions and user behavior. The benchmark does not establish your acceptable latency, per-query budget, retention policy, regional hosting requirement or access-control model.
Use MADQA as one piece of evidence about system design, then run a local evaluation before committing to an embedding or retrieval stack.
Best Value
Build a representative held-out set
Sample real documents and queries from the intended workload. Include routine lookups, multi-hop questions, tables, scans, diagrams, long documents and deliberately ambiguous wording. Keep a held-out set that is not used to tune prompts or retrieval settings.
Label more than the final answer
For each question, record an accepted answer, relevant pages or passages, required visual elements, access permissions and an expected abstention condition when evidence is insufficient. These labels let you separate retrieval failures from synthesis failures.
Compare full systems under matched conditions
Hold the document split, parsing pipeline, chunking policy, model limits, search budget and evaluation code constant. Change one component at a time where practical. Report answer correctness, evidence/page quality, search effort, latency and cost.
Free tools Windows power users keep installed
One-click scans. No signup required.
Test production constraints
Replay the same held-out queries with your authentication and filtering rules enabled. Measure p50 and tail latency, concurrent load, index-update time, multilingual behavior and the effect of restricted documents. A benchmark score cannot substitute for these tests.
What MADQA does not establish
- A universal embedding ranking: no single embedding score is defined for all MADQA systems.
- Performance on your private corpus: benchmark documents are not a proxy for every organization’s files.
- Production economics: the benchmark does not by itself establish your latency, token use, storage cost or tool charges.
- Security behavior: access controls, tenant isolation and redaction policies must be evaluated in your environment.
- Long-term reliability: a static benchmark does not reveal how quality changes as documents, models or service defaults are updated.
- Embedding-only causality: scores combine planning, retrieval, multimodal interpretation and answer generation.
A practical reporting template
If you publish or circulate a MADQA comparison, make the result auditable with a compact record:
| Field | Record |
|---|---|
| Dataset and split | MADQA split name, question and document counts, and any exclusions. |
| System | Agent model, embedding model, retriever, reranker, file-search service and multimodal handling. |
| Metrics | Accuracy, ANLS*, Semantic Accuracy, Page F1, Kuiper and the exact script version. |
| Effort | Tool-call definition, search-round limit, average and tail counts, latency and failure rate. |
| Provenance | Independent paper, repository run or vendor evaluation; include the test date. |
| Deployment evidence | Results on a representative local corpus, with measured cost, latency and access-control tests. |
This format prevents a benchmark number from being mistaken for a promise about a different workload.
Bottom line for embedding decisions
MADQA is valuable precisely because it tests more than vector similarity. It probes whether an agent can navigate heterogeneous documents, interpret multimodal evidence, stop searching intelligently and answer accurately. The 2026 study shows that strong agents can reach human-comparable raw accuracy while succeeding on different questions, using brute-force search and remaining nearly 20% short of oracle performance in the reported experiment.
Treat that result as system-level evidence. Use the benchmark scripts and metrics to understand retrieval and search behavior, inspect configuration and effort details, then validate the complete pipeline on representative documents and queries from your own deployment. Only that matched local test can tell you whether a particular embedding model is the right choice.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
