iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Measure passk when you need to know whether an AI agent can complete the same task successfully on every one of several attempts—not merely whether it can succeed once. It is a useful consistency measure for evaluating repeated work, but no passk score by itself certifies an agent as safe to run unattended overnight.
What passk measures
For a given task, passk is the probability that all k attempts succeed. That makes it different from pass@k, which asks whether at least one of the k attempts succeeds. Anthropic explains the distinction in its agent-evaluation guide.
| Metric | Question it answers | When it fits |
|---|---|---|
| pass@k | Did at least one of k attempts succeed? | When a successful result is useful even if other attempts fail. |
| passk | Did every one of k attempts succeed? | When the task needs to work consistently across repeated runs. |
The difference matters for unattended execution: a task that succeeds occasionally may look promising under pass@k, while passk exposes the chance of an unsuccessful attempt within the repeated run.
Why a strong single-run rate can still mean weak consistency
Anthropic illustrates the distinction with a 75% per-attempt success rate across three attempts. Assuming the attempts are independent and have that same success probability, the chance that all three succeed is 0.75 × 0.75 × 0.75, or about 42%. This is a mathematical illustration, not a measured result for a particular agent.
#1 Best Overall
- Ultimate To Do List with Multiple Sections: A to do list lover’s dream, our notepad offers multiple sections with ample space to write all your important tasks so you can organize and track your tasks better than with a regular list. Each page has a to do list as well as sections for top priorities, for tomorrow, and appointments/calls, making it easy to prioritize and stay organized. Say goodbye to feeling overwhelmed and hello to a more organized and productive you!
- Minimalist Design to Boost Productivity: Experience the perfect balance of minimalist and functional design with our daily to-do list notepad. Each notepad measures 6.5” x 9.8” and has 60 sheets, so there is enough space to write down everything you need to do. Featuring a minimalist black and white design and premium materials, our notepad is the perfect tool to keep you on track and motivated throughout the day!
- Spiral Bound with Protective Cover: Our twin spiral-bound notepad lets you start a new page while keeping old ones for reference. It makes it easy to flip through your to-do list. When you're done, do you want to remove your lists? No issue! They can be torn out as necessary. When you're on the go, the plastic cover on our notepad protects the pages from spills, scratches, and tears. Even better, the cover is see-through so you can quickly glance at your to-do list page as you go about your day.
- Premium, non-bleed pages: No more frustrations about pens or markers bleeding through flimsy paper! Our notepad is made with premium non-bleed 100 gsm paper to give you the best writing experience. Unlike with our competitors, these pages won’t bleed onto the next one, even if you write with a permanent marker.
- Sturdy Backing for Writing Anywhere: Our notepad is made with a thick backing that provides a sturdy surface for writing anytime, so you can take it on the go and never miss an important task again. Whether you're at home, in the office, or on the go, you'll always be able to capture your thoughts and stay on top of your daily routine.
That calculation also shows why you should not treat a high aggregate single-run score as an automatic passk result. A task suite may contain tasks with very different success rates, and a task-level consistency measure should preserve that variation rather than simply raising the suite-wide average to the kth power.
How passk is estimated
The τ-bench paper defines passk for k independent, identically distributed trials of the same task, then averages the task-level measure across tasks. For a task with c successful outcomes among n observed trials, its unbiased estimator is the number of ways to choose k successes from those c successes divided by the number of ways to choose k trials from all n observations:
Rank #2
- Organize Tasks Efficiently: This versatile To-Do List Notepad, a must-have office tool, features multiple sections with ample space for essential task tracking. Perfect for daily task management, helping you prioritize and manage office supplies effectively.
- Boost Your Productivity: Utilize our Daily Checklist Notepad to streamline planning and enhance efficiency. Begin your day on a positive note, gaining clarity on tasks and prioritizing effectively for steady progress towards your goals. Stay focused, motivated, and in control, maximizing productivity and achieving success.
- Stylish and Practical: This task planner showcases a clear PP cover, safeguarding inner pages from dirt and wear. The back cover, crafted from sturdy paperboard, offers stability for writing. Each notepad includes 52 pages of premium 100gsm paper, ensuring thickness and non-bleed properties for a seamless writing encounter.
- Multi-Purpose Essential: This daily to-do list notepad caters to various needs and audiences, spanning work, school, and home. Whether you're managing tasks, planning projects, or creating shopping lists, it's the ultimate organizational tool. With its versatile layout and ample space, it's a must-have for everyone seeking efficient organization.
- Ideal Present Choice: In search of a splendid present for your dear ones? Your quest ends here with our To-Do List Notepad! Its chic aesthetic and versatility make it a thoughtful addition to office and school supplies alike. Suited for students, professionals, and homemakers, it effortlessly aids in managing tasks and priorities.
pass^k estimate = C(c, k) / C(n, k)
Here, C(a, b) is the number of ways to choose b items from a. If fewer than k of the observed trials succeeded, the estimate for that task is zero. The estimator uses the observed repeated outcomes; it does not declare a task reliable because one run worked.
Do these 3 things before closing this tab:
1Scan for outdated or missing drivers - takes under a minute2Clear out junk files and repair common Windows errors3Fix the driver behind crashes, sound loss and screen glitchesThe paper reports using more than 40 GPT-4-turbo trials per τ-retail task during benchmark construction for tasks with zero or low success rates. That is a description of how those benchmark cases were constructed, not a universal minimum or a recommended sample size for evaluating your agent.
Rank #3
- TASK PLANNER NOTEPAD FOR WORK – Structured task planner and productivity planner designed as a task organizer and workflow planner with expanded layout
- REDUCE OVERWHELM & IMPROVE EXECUTION – Use this task planner, productivity planner, and task tracker to organize workflow and improve execution
- TASK PLANNING + ACTIVITY TRACKING IN ONE FORMAT – Use as a productivity planner, task organizer, and workflow planner to manage tasks and execution
- ANALOG TASK & ACTIVITY SYSTEM THAT WORKS – Writing tasks improves memory and reduces missed steps
- STRUCTURED CHECKLIST + LOG FORMAT – Combines task planning and activity tracking with expanded space for detailed workflow management
How to measure it for your agent
- Define the task and success rubric. Specify the inputs, expected outcome, and observable conditions that count as success. Apply the same definition to every repetition; otherwise, the trials are not measuring the same task.
- Choose k to reflect the repeated-work question. State how many attempts the metric covers. Report pass@k instead when any one successful attempt is enough for the use case; use passk when each attempt must succeed.
- Run repeated trials under comparable conditions. Record the outcome of every run and explain what counts as an independent run. The metric’s definition assumes i.i.d. trials, so changes in prompts, tools, task inputs, or run conditions can make results harder to interpret.
- Evaluate a relevant task suite. Use tasks that reflect the work the deployed agent is expected to do. The metric is task-based, but the cited sources do not prescribe a universal method for making a suite representative.
- Calculate task-level estimates, then report the aggregation. For each task, record c successes from n trials and estimate passk using the ratio above. When presenting an average across tasks, describe how tasks were weighted and include task-level outcomes so the aggregate does not conceal weak cases.
- Include the context needed to interpret the score. Report the task suite, k, observed trial count, success criteria, run conditions, and task-level results. If comparing agents, keep the task definitions, rubric, k, and conditions comparable.
What the score cannot tell you about an overnight run
passk describes repeated task success under the evaluated conditions. It does not, on its own, establish security, resilience to tool failures, or long-horizon operational safety. The broader topic of agent reliability is still being examined in research, including the 2026 reliability proceedings paper.
The cited materials do not set a universal passk threshold, confidence level, trial count, or operational checklist that certifies an agent for overnight use. Treat the metric as one piece of evidence, not a permission slip. The appropriate level of additional safeguards depends on what the agent can change, the cost of a mistake, and how errors can be detected and contained—questions a benchmark score alone cannot answer.
Quick Recap
Best Value
- Easily Stay On Track & Make The Most Of Your Time: ZICOTOs’ daily planner makes it easier than ever for you to stay organized, reduce stress & enjoy more free time! Arrange your schedule, priorities, to do’s and jot down plans & ideas on the daily notes section
- Smartly Plan Ahead & Boost Your Productivity: Absolutely clever & efficient! With the planner notebook you can break down your daily tasks into half-hourly focus blocks and map out priorities & follow-up duties to keep your day on track and enhance productivity
- Plenty Of Space For Efficient Planning: Stay focused & manage your time wisely! The 9.3x6.3” (inner pages) work planner & organizer notebook offers ample space for 80 days of life-changing planning with each day being spread across 2 pages - set yourself up for purposeful days
- Now Is The Best Time To Start: The daily planner is undated so you can start to add structure to your schedule and cultivate new planning habits right away! Beat procrastination, boost happiness & make each day count with the hourly planner
- Adds Beauty To Daily Planning: A gorgeous terracotta cover, chic gold foil letters, a golden ring wire and a clean, easy-to-use layout - enjoy the gorgeous and modern design of the undated daily planner!
Rank #4
- SIMPLE, ATTRACTIVE DESIGN - The notepad flaunts a design that's both stylish and fun, making it the perfect backdrop to track your daily tasks and other deliverables.
- 8.5" X 11": LETTER SIZE - With ample space for jotting down your activities/tasks, projects and deliverables, this notepad ensures you never miss a beat. The larger size offers room to log all your work.
- DOUBLE-WIRE SPIRAL BINDING - Tear off sheets as needed or keep them intact to be able to look back on your prior tasks. Whether you're at your desk, in a meeting, or on the go, your notepad is ready for you to use as you see fit.
- 50 SHEETS - Created so you can list all your tasks onto one sheet in order to keep track of time stamps as you note how you spend your time each day.
- VERSATILE FOR TRACKING ALL YOUR TASKS - Whether you're a professional who bills for your time, a student recording time spent on studies, or a volunteer tracking your hours, this notepad can help you record your time spent on any and all activities throughout your day.
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Free tools Windows power users keep installed
One-click scans. No signup required.
