Inspect AI
Inspect AI: A composable, self-hosted evaluation framework for researchers with broad testing needs. Ranked #9 of 29 in LLM Evaluation Tools by our editors (8.5/10); pricing: Open source; best for researchers needing many composable evaluations.
At a glance
- Editor score8.5 / 10
- PricingOpen source
- Best forResearchers needing many composable evaluations
- Paid fromFree
- Deployment optionsSelf-hosted
- Founded2024
- Facts checked20 Sep 2026
Where it wins
- More than 200 pre-built evaluations across varied AI tasks
- Composable agents, tools, solvers, datasets, scorers, and metrics
- Safety scanners, human review, sandboxing, logs, and CI workflows
Where it doesn't
- Self-hosted deployment requires teams to manage the environment
- Its breadth may be more than teams needing narrow evaluations
- Hosted SaaS workflows are not the product's operating model
Our verdict on Inspect AI
Inspect AI is an open-source framework for evaluating large language models, developed by the UK AI Security Institute and Meridian Labs. It is aimed at researchers and engineering teams evaluating coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. The framework can run evaluations against supported hosted or local models and is installed as a Python package. Its self-hosted design suits teams that want control over evaluation composition and execution rather than a hosted SaaS workflow.
The main strength is breadth with composability. Inspect AI includes more than 200 pre-built evaluations, while allowing teams to combine datasets, agents, tools, solvers, scorers, and metrics. Model-graded scorers support open-ended answers, and custom scorers and custom metrics support evaluation designs beyond the included library. For safety-focused work, transcript scanners address refusals, evaluation awareness, misconfiguration, and reward hacking. Human review of tool results adds a manual assessment path when automated scoring is not enough.
Inspect AI also connects evaluation work to development workflows. Sandboxing is available through Docker, Kubernetes, Modal, Proxmox, and Vagrant extensions, while web-based evaluation log viewing, VS Code integration, and local or remote log workflows support analysis across environments. CI-oriented evaluation subsets are useful for teams that want to include selected checks in ongoing development. Choose Inspect AI when you need a broad, programmable evaluation framework with safety, human review, and CI-oriented capabilities. Teams seeking a hosted product or a narrower, simpler evaluation tool should consider a different operating model.
Your product? Our editors can test Inspect AI in depth and publish a review, labelled as paid · from $149/mo →
Inspect AI pricing
Inspect AI fact sheet
| Free plan | Not verified |
|---|---|
| Paid from | Free |
| Deployment options | Self-hosted |
| Custom metrics | Yes |
| LLM-as-a-judge | Yes |
| Safety evaluations | Yes |
| Human review workflows | Yes |
| Prompt versioning | Not verified |
| CI/CD integration | Yes |
| Deployment | Self-hosted |
| Support | Docs |
| Built for | Solo, Small business, Mid-market, Enterprise (editorial estimate) |
| Pricing | Open source |
| Website | inspect.aisi.org.uk |
| Facts checked | 20 Sep 2026 |
Alternatives to Inspect AI
- DeepEvalA broad open-source toolkit for evaluating LLM apps, agents, RAG pipelines, and chatbots.9.4
- PromptfooA broad evaluation toolkit for teams that need testing, security scans, and CI/CD gates.9.3
- GalileoA mature evaluation platform spanning offline testing, production tracing, and runtime controls.9.1
See all Inspect AI alternatives →
Also Ranked In
Inspect AI is ranked in 3 of our directories.
Used Inspect AI? Be the first to review it
The editor score above is our own research. What this page doesn't have yet is a reader's view — what you used Inspect AI for, what worked and what didn't. No stars are seeded and no review is paid for; an editor reads every one before it appears.
Write a reviewTwo minutes · verified accounts only · read by an editor before it appears
Featured on iTechGuides
Inspect AI is listed in our LLM Evaluation Tools directory. Add the badge to your site — it links back to this page.
<a href="https://www.itechguides.com/products/inspect-ai/"><img src="https://www.itechguides.com/best/badge/inspect-ai.svg" alt="Featured on iTechGuides" width="230" height="46"></a>
Reviewed by iTechGuides Editors · Editorial team · Updated Oct 2026
Advertiser disclosure: iTechGuides is reader-supported. Vendors can pay for top positions in our rankings and for a place on other products' pages, and we may earn a commission when you click some links. Bought places stand at the number bought; this page and our disclosure say so. Paying vendors may get a fuller, more detailed review from our editors, and that review can change their score. How we rank.
Last updated · How we research and update