τ-bench
τ-bench: A focused, self-hosted benchmark for tool-using agent conversations. Ranked #33 of 39 in AI Model Evaluation Software by our editors (5.8/10); pricing: Free plan; best for tool-using agent conversation benchmarks.
At a glance
- Editor score5.8 / 10
- PricingFree plan
- Best forTool-using agent conversation benchmarks
- Free planYes
- Paid fromFree
- Safety & bias checksNo
- Founded2024
- Facts checked19 Nov 2026
Where it wins
- Simulates dynamic conversations between users and language agents
- Evaluates airline and retail API-tool interactions against policies
- Reports pass@k leaderboard metrics across configurable agent strategies
Where it doesn't
- Scope centers on airline and retail environments
- Self-hosted deployment requires setup and model-provider API keys
- Repository tasks are outdated; newer domains require τ³-bench
Our verdict on τ-bench
τ-bench is an open-source benchmark for evaluating language-model agents that converse with simulated users and call domain-specific API tools under policy guidelines. It is aimed at research and engineering teams studying conversational task success, tool adherence and agent behavior across small, mid-market and enterprise settings. The project is distributed as installable Python source and runs through self-hosted deployment.
Its strongest area is controlled interaction evaluation. Teams can configure model and user simulators, select task IDs, control concurrency and inspect historical trajectory files. Airline and retail environments provide concrete API-tool scenarios, while Act, ReAct and tool-calling strategies support different agent patterns. Policy-guideline scoring adds an adherence dimension, and pass@k leaderboard metrics give a consistent way to compare repeated task outcomes. The category specifications list model-provider support for OpenAI, Anthropic, Google, Mistral and AnyScale.
The published project is free and open source, but operating it means managing a self-hosted installation and supplying model-provider API keys. Community support is available rather than 24/7 coverage. τ-bench is a sensible choice when your evaluation program specifically needs simulated user-agent conversations with airline or retail tools, configurable runs and regression benchmarking. It is less suitable for safety-bias testing, human review workflows or evaluations outside its supplied interaction patterns. The maintainers state that the airline and retail tasks are outdated and direct users to τ³-bench for newer domains and fixed tasks, so teams needing current scenarios should consider that project instead.
Your product? Our editors can test τ-bench in depth and publish a review, labelled as paid · from $149/mo →
τ-bench pricing
τ-bench fact sheet
| Free plan | Yes |
|---|---|
| Paid from | Free |
| Evaluation types | tool-agent-user interaction; conversational task success; policy and API-tool adherence |
| Safety & bias checks | No |
| Regression benchmarks | Yes |
| Human review workflows | No |
| Included evaluation runs | Not verified |
| Deployment option | Self_hosted |
| Model providers | OpenAI; Anthropic; Google; Mistral; AnyScale |
| Deployment | Self-hosted |
| Support | Community |
| Built for | Small business, Mid-market, Enterprise (editorial estimate) |
| Pricing | Free plan |
| Website | github.com |
| Facts checked | 19 Nov 2026 |
Alternatives to τ-bench
- LangSmithA broad LangSmith toolkit for tracing, evaluating, and improving LLM applications.9.4
- RagasA flexible, self-hosted toolkit for metric-driven LLM evaluation.9.1
- BraintrustGoverned evaluation, tracing, and regression controls for teams operating AI applications.8.9
See all τ-bench alternatives →
Used τ-bench? Be the first to review it
The editor score above is our own research. What this page doesn't have yet is a reader's view — what you used τ-bench for, what worked and what didn't. No stars are seeded and no review is paid for; an editor reads every one before it appears.
Write a reviewTwo minutes · verified accounts only · read by an editor before it appears
Featured on iTechGuides
τ-bench is listed in our AI Model Evaluation Software directory. Add the badge to your site — it links back to this page.
<a href="https://www.itechguides.com/products/bench-ai-model-evaluation-tool/"><img src="https://www.itechguides.com/best/badge/bench-ai-model-evaluation-tool.svg" alt="Featured on iTechGuides" width="230" height="46"></a>
Reviewed by iTechGuides Editors · Editorial team · Updated Nov 2026
Advertiser disclosure: iTechGuides is reader-supported. Vendors can pay for top positions in our rankings and for a place on other products' pages, and we may earn a commission when you click some links. Bought places stand at the number bought; this page and our disclosure say so. Paying vendors may get a fuller, more detailed review from our editors, and that review can change their score. How we rank.
Last updated · How we research and update


