AgentBench review
A self-hosted benchmark for multi-turn agent evaluation across eight interactive environments.
Reviewed by iTechGuides Editors · Editorial team · Updated Oct 2026
AgentBench is an open-source benchmark and evaluation toolkit for assessing large language models as autonomous agents. It is designed for teams and researchers evaluating interactive behavior rather than for organizations seeking a hosted commercial SaaS product. Agents are evaluated through multi-turn interaction across eight environments, including operating systems, databases, knowledge graphs, digital card games, lateral-thinking puzzles, household tasks, web shopping, and web browsing.
Its strongest advantage is the breadth of its interactive evaluation setup. AgentBench provides development and test dataset splits, configurable agent and task definitions, evaluation scripts, and support for extending the benchmark with new tasks. Docker-based workers and services help package task environments, while API-based model clients allow different models to be evaluated. The repository also includes function-calling evaluation support, with containerized support for selected tasks. This makes the toolkit a fit for research workflows that need varied environments and multi-step interactions instead of single-turn prompt comparisons.
AgentBench is free and self-hosted on Linux, with the code available as an open-source project from THUDM. That deployment model gives technically capable teams control over their evaluation environment, but it also places responsibility for running the workers, services, datasets, and evaluation process on the user. The published positioning does not include verified judging, custom metrics, safety features, or CI features. Teams prioritizing those capabilities should choose an evaluation tool that provides them directly; teams focused on configurable, multi-turn agent benchmarks across diverse environments should consider AgentBench.
AgentBench pros and cons
- Where it wins
- Evaluates agents across eight interactive environments
- Supports multi-turn interactions, datasets, and configurable tasks
- Docker workers, API clients, and function-calling support included
- Where it doesn't
- Does not include verified judging
- Lacks custom metrics and safety features
- No built-in CI features for evaluation workflows
AgentBench fact sheet, pricing and score →
Advertiser disclosure: iTechGuides is reader-supported. Vendors can pay for top positions in our rankings and for a place on other products' pages, and we may earn a commission when you click some links. Bought places stand at the number bought; this page and our disclosure say so. Paying vendors may get a fuller, more detailed review from our editors, and that review can change their score. How we rank.
Last updated · How we research and update