AI Agent Evaluation Framework: Claude Code Eval Loop Steps

By Aoba Works · Published October 25, 2026 · Sources checked October 25, 2026 · 18 min read

How this guide was made: drafted with AI assistance from the official sources linked below, then passed an automated editorial check before publishing. Our process

• Claude Code
• AI Agents

An AI agent evaluation framework is a repeatable loop. You give the agent a fixed set of tasks, run each one several times in a clean environment, score the final result with graders, and track pass rates from run to run. Anthropic's engineering team describes an evaluation as "a test for an AI system: give an AI an input, then apply grading logic to its output to measure success." For Claude Code, that means a folder of tasks, a shell script that calls claude -p several times per task, a grader script for each task, and a CSV file you check after every model or prompt change.

Which option is right for you?

Most solo builders should start with a shell script and a CSV file. Add a tool only when the CSV gets hard to read.

All five follow the same method. Only the place you look at the numbers changes, so the rest of this guide uses the DIY loop and points out where a tool would take over.

What is evaluation in AI?

Evaluation in AI means measuring whether a system's outputs meet a defined standard. You use repeatable test cases and grading logic instead of checking results by eye now and then.

Anthropic's prompt engineering docs give three design rules for evals. The first is to be task-specific: "Design evals that mirror your real-world task distribution. Don't forget to factor in edge cases!" The second is to automate grading wherever you can, with string matches, code-graded checks or LLM-graded checks. The third is to favor volume: lots of automatically graded questions beat a few that you grade carefully by hand.

Here is an example. Say you have a Claude Code job that bumps dependency versions every night. If you read the morning diff and think "looks fine," you haven't run an evaluation. An evaluation has 15 saved repo snapshots, each with a known outdated dependency, and a script that checks for three things after each run: the version changed, the lockfile is consistent, and npm test passes. Every run gets the same verdict from the same checks.

Common pitfalls at this stage:

• Grading the agent's message instead of the repo. If Claude says "all tests pass," that is a claim. Anthropic separates the transcript from the outcome, which is "the final state in the environment". Grade the outcome.
• Writing vague tasks. Anthropic's test for a good task is whether two domain experts would independently reach the same pass/fail verdict. If you can't write down what "done" looks like, the grader can't check it either.

What is an AI agent evaluation framework?

An AI agent evaluation framework is a set of tasks, isolated trials, graders and stored results. It tests multi-step tool use across repeated runs, not single replies.

Agents are harder to test than a chatbot reply. Anthropic puts it this way: agents "use tools across many turns, modifying state in the environment and adapting as they go." A single reply can be graded on its text. An agent run has to be graded on what it changed. Anthropic's "Building effective agents" post defines agents as systems where LLMs "dynamically direct their own processes and tool usage." It also recommends "extensive testing in sandboxed environments, along with the appropriate guardrails."

The Demystifying evals post gives each part of the framework a name:

The post also separates two kinds of suite. Capability evals ask what the agent can do well, and they start with low pass rates. Regression evals ask whether the agent still handles tasks it used to handle, and they should stay close to 100%. If you are about to leave a job running unattended, you mostly need regression evals. You want to know it hasn't broken, not that it has learned something new.

Anthropic says it built Claude Code's own evals this way, one area at a time. It added evals "first for narrow areas like concision and file edits, and then for more complex behaviors like over-engineering."

How do you build an eval loop for Claude Code?

You build a Claude Code eval loop by storing tasks as prompt-plus-commit pairs. A script runs each task K times in a fresh worktree with claude -p, grades the outcome and appends a row to a CSV.

Anthropic recommends starting small: 20-50 simple tasks drawn from real failures, rather than waiting until you have hundreds. In its words, "Start early and don't wait for the perfect suite."

Step 1: Lay out the folder

Make one folder per task:

evals/
  tasks/
    fix-null-date-parse/
      prompt.md        # the exact prompt your automation sends
      base_commit      # a git SHA where the bug exists
      grade.sh         # exits 0 = pass, non-zero = fail
  results.csv
  run.sh

Take tasks from things that really went wrong, such as last week's failed cron run or the PR you had to revert. If you already check something by hand every morning, turn that check into a task. This is Anthropic's Step 1, "Start with what you already test manually."

Step 2: Write graders that check the outcome

An example grade.sh for a bug-fix task:

#!/usr/bin/env bash
set -e
npm test -- --silent                      # the failing test must now pass
git diff --name-only HEAD | grep -q '^src/date' # the fix touched the right area
! git diff HEAD | grep -q 'it.skip'        # no skipped tests to "pass"

The last line stops a common way of gaming the grader. Anthropic warns against grading the path the agent took when you can grade the result: "It's often better to grade what the agent produced, not the path it took." Its sample coding task does combine deterministic tests, static analysis (ruff, mypy, bandit) and an LLM rubric for code quality. Add a rubric only after your code graders are stable.

Step 3: Write the harness

This is an example script. Adjust the tools and budget for your project.

#!/usr/bin/env bash
# evals/run.sh: runs every task K times, grades the outcome, appends to results.csv
set -u
K=${K:-5}
RUN_ID=${RUN_ID:-$(date +%Y%m%d-%H%M)}
ROOT=$(pwd)
for task in "$ROOT"/evals/tasks/*/; do
  name=$(basename "$task")
  for i in $(seq 1 "$K"); do
    work="$(mktemp -d)/wt"
    git worktree add -q "$work" "$(cat "$task/base_commit")"
    start=$(date +%s)
    out=$(cd "$work" && claude --bare -p "$(cat "$task/prompt.md")" \
      --output-format json \
      --permission-mode acceptEdits \
      --allowedTools "Bash(npm test *)" \
      --max-budget-usd 1)
    secs=$(( $(date +%s) - start ))
    cost=$(echo "$out" | jq -r '.total_cost_usd // 0')
    if (cd "$work" && bash "$task/grade.sh" >/dev/null 2>&1); then pass=1; else pass=0; fi
    echo "$RUN_ID,$name,$i,$pass,$cost,$secs" >> "$ROOT/evals/results.csv"
    git worktree remove --force "$work"
  done
done

Why each part is there, with the source:

• A fresh worktree per trial. Anthropic says "Each trial should be 'isolated' by starting from a clean environment". Leftover files or cached data cause correlated failures that come from your setup, not from the agent.
• --bare. The headless docs call bare mode "the recommended mode for scripted and SDK calls." It skips auto-discovery of hooks, skills, plugins, MCP servers, auto memory and CLAUDE.md, so every machine gives the same result. Bare mode needs ANTHROPIC_API_KEY because it doesn't use your subscription login. One catch: if your real automation depends on CLAUDE.md or hooks, an eval in bare mode isn't testing what you ship. Either drop --bare or pass the same context with --append-system-prompt-file and --settings.
• --output-format json. The output includes total_cost_usd and a per-model breakdown. The docs note that both are client-side estimates and can differ from your actual bill.
• --max-budget-usd. This flag in the CLI reference caps spend for each trial.
• Permissions. In a non-interactive claude -p run, nobody is there to approve permission prompts, so actions that need approval won't go through. Check the headless docs for the default behavior of your installed version. Use the same permission flags in the eval that your production cron job uses. Otherwise the eval measures a different agent. The headless permissions template on this site covers a locked-down setup.

Step 4: Run it and summarize

chmod +x evals/run.sh evals/tasks/*/grade.sh
K=5 RUN_ID=baseline ./evals/run.sh
awk -F, '$1=="baseline"{n[$2]++; p[$2]+=$4} END{for(t in n) printf "%-28s %d/%d\n", t, p[t], n[t]}' evals/results.csv

A task that scores 5/5 is a regression candidate. A task that scores 2/5 is flaky, and you shouldn't leave it unattended yet.

Step 5: Read transcripts for failures

Anthropic's Step 6 is blunt: "Read the transcripts!" Re-run a failing trial with --output-format stream-json --verbose to see every tool call. With --permission-prompts none, the final result message also lists blocked actions in permission_denials (headless docs; check which flags your installed version supports). Many "agent failures" are actually missing permissions. For runs that stop partway through, see Why Claude Code stops mid-automation.

Which AI agent evaluation framework metrics should you track?

Track the pass rate per trial, pass^k for consistency, and the regression rate between versions. Add cost and wall time per run, plus permission denials from the transcript.

Anthropic's post makes the key distinction. pass@k is the chance that at least one of k attempts succeeds. pass^k is the chance that all k succeed. For unattended jobs, pass^k is the one that matters, because nobody is there to retry. The post gives the math: with a 75% per-trial success rate and 3 trials, the chance of passing all three is "(0.75)³ ≈ 42%."

Example thresholds are illustrations. Set your own from your baseline run.

If a capability task hits 100%, it is saturated. Anthropic calls this the point where "an agent passes all of the solvable tasks, leaving no room for improvement." Move it into the regression suite and add a harder task in its place. For more on bringing cost per trial down, see how to see and reduce Claude Code token usage.

How do you visualize AI agent evaluation framework metrics across runs?

Show pass rate per task by run ID, as a line or a pivot table. When a line drops after a change, you've found a regression.

Steps with tools you already have:

1. Open results.csv in Google Sheets or Excel and add a header row: run_id,task,trial,pass,cost_usd,secs.
2. Insert a pivot table. Put task in Rows, run_id in Columns and pass in Values, summarized by AVERAGE. Each cell is now the pass rate for that task in that run.
3. Add conditional formatting so cells below 0.8 turn red. That cutoff is only an example.
4. Insert a line chart from the pivot to see each task's trend.
5. Add a second pivot with cost_usd averaged by run_id to catch cost drift.

If you'd rather use promptfoo, its --repeat option sets the "Number of times to run each test", and promptfoo view opens a local browser UI to visualize results.

Pitfalls when charting:

• Comparing runs with different K. Five trials and twenty trials give pass rates with very different noise. Keep K fixed within a comparison.
• Averaging across all tasks. A suite average of 0.92 can hide one task that dropped from 1.0 to 0.2. Always look at results per task.
• Mixing changes. If you change the prompt and the model in the same run, you can't tell which one moved the chart. Change one thing per run ID, for example opus-prompt-v3.

Which open-source AI agent evaluation framework on GitHub should you try?

For Claude Code, promptfoo is the most direct ready-made option because it has a Claude Agent SDK provider. Anthropic's cookbook and Langfuse or Arize Phoenix cover other needs.

Anthropic's post also mentions Harbor for running agents in containerized environments at scale. That's useful once your trials need isolation beyond a git worktree.

Here's an example of when to switch. Once you have more than about ten tasks and you keep editing awk one-liners, moving the task list into a promptfoo config and using --repeat 5 is a reasonable next step. The grader logic stays in your own scripts or JavaScript assertions.

Should you use an open-source framework or build your own?

Build your own harness first if you have one or two automations. Adopt an open-source framework once you need shared configs, many tasks or a results UI.

Pitfalls:

• Leaning on LLM judges too early. Anthropic lists model-based graders as flexible but non-deterministic and more expensive, and says they need human calibration. Get code graders working first.
• Testing a different configuration. A framework provider that loads different tools or permissions than your production job will report pass rates for a different agent.
• Trusting automated evals alone. Anthropic says "A complete picture includes production monitoring, user feedback, A/B testing, manual transcript review, and systematic human evaluation." For solo builders, production monitoring is the part to add next. See AI agent monitoring in Claude Code.

What does an AI agent assessment framework (Gartner-style) add?

An enterprise AI agent assessment framework adds governance, risk and ROI criteria on top of pass rates. Solo builders should keep the technical eval loop as their main gate.

Gartner predicted that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value or inadequate risk controls (press release dated June 25, 2025). The same release warns about "agent washing," meaning existing products rebranded as agents. It estimates that only about 130 of the thousands of agentic AI vendors are real.

The last three rows reuse metrics from your eval loop. For the risk row, the AI agent guardrails example and Claude Code checkpoints guides cover what to set up.

What is an agent function in AI?

An agent function maps everything an agent has observed so far to its next action. In an eval, each step's tool call is that function's output.

The term comes from the classic textbook framing in Russell and Norvig's Artificial Intelligence: A Modern Approach. In Claude Code, the "observations" are your prompt plus every tool result so far. The "action" is the next tool call, such as Read, Edit or Bash, or the final answer. Anthropic's agent guidance stresses that agents need "ground truth" from the environment at each step. That is why tool results go back into the loop.

For example, after npm test fails, a good agent function chooses "read the failing test file." A poor one chooses "edit the test to skip it." Your outcome grader catches the second at the end. A transcript check on tool calls catches it at the step where it happened. Anthropic's sample task includes tool_calls and tracked_metrics (turns, tool calls, tokens) for this purpose.

What does the Mosaic AI Agent Evaluation framework offer?

Mosaic AI Agent Evaluation is Databricks' agent-scoring product, now part of MLflow 3. It provides LLM-judge scorers, batch evaluation and production monitoring for agents.

According to the Databricks docs, evaluation "measures whether your agent is actually good — is the response correct, grounded, safe, and complete?" The docs suggest scoring a few traces from the UI first, then moving the same scorers into a repeatable mlflow.genai.evaluate() loop.

Who it's not for: solo builders who aren't already on Databricks. Its built-in judges also focus on response quality, while a coding agent's real result is the state of the repo, so you'd still write outcome scorers.

Common mistakes and fixes

Most broken agent evals fail for the same few reasons, and each has a simple fix.

Next steps

Do today:

1. Pick the one Claude Code automation you'd most like to stop checking. Write three tasks from its recent failures, each as prompt.md, base_commit and grade.sh.
2. Copy the run.sh harness above, make its flags match your production command, and run K=5 RUN_ID=baseline.
3. Re-run with a new RUN_ID after every model, prompt or CLAUDE.md change. Then schedule it with the Claude Code cron job setup so regressions show up the next morning.

If you'd rather not build it yourself: hosted eval platforms take care of storage, dashboards and LLM judges. Braintrust's Starter plan is $0/month with 10,000 scores and 14-day retention, and Pro is $249/month (Braintrust pricing, as of October 2026; check the official page). LangSmith's Developer plan is $0 for 1 seat with up to 5k base traces a month, and Plus is $39 per seat per month (LangChain pricing, as of October 2026; check the official page). These suit small teams that want shared views. They are not for builders with one or two cron jobs, where a CSV is enough. Your graders still have to come from you, because no platform knows what "done" means for your repo.

Related guides on this site:

• Claude Code Headless Mode: Flags, Auth and Cost (2026)
• AI Agent Monitoring in Claude Code: Catch Silent Failures
• AI Agent vs Automation: A 5-Question Test for Claude Code

FAQ

What are the main AI agent evaluation framework metrics?

The core metrics are pass rate per task and pass^k (all k trials succeed). Add regression rate between versions, cost per run and run time. Anthropic's post shows why pass^k matters: a 75% per-trial rate gives only about 42% chance of three straight passes. For coding agents, also count permission denials and tool calls from the transcript.

What tool should I use as an AI agent assessment framework?

For Claude Code, start with a shell harness around claude -p --output-format json. Then move to promptfoo, which has an anthropic:claude-code provider, when you need repeats and a results viewer. Hosted options such as Braintrust and LangSmith add team dashboards. Databricks users can use MLflow 3's agent evaluation.

What is an AI agent assessment framework?

It is a structured way to judge whether an agent is ready for a job. It covers technical reliability (pass rates across repeated runs) and, in enterprise settings, cost, value and risk controls. Gartner's prediction that over 40% of agentic AI projects will be canceled by end-2027 is often used to justify these checklists. For a solo builder, the technical eval loop does most of the work.

What is an AI framework?

An AI framework is software that gives you reusable structure for building or testing AI systems, so you don't write everything from scratch. In evaluation, a framework supplies the parts that repeat every time: task definitions, a runner, graders and result storage. promptfoo and MLflow are examples. A 40-line shell script can play the same role at small scale.

Sources

• Anthropic: Demystifying evals for AI agents
• Anthropic: Building effective agents
• Claude Docs: Create strong empirical evaluations
• Claude Code Docs: Run Claude Code programmatically (headless)
• Claude Code Docs: CLI reference
• Anthropic Claude Cookbooks on GitHub
• promptfoo: Claude Agent SDK provider
• promptfoo: Command line options
• Databricks: Agent evaluation (MLflow 3)
• Gartner press release, June 25, 2025
• awslabs/agent-evaluation on GitHub
• Braintrust pricing
• LangChain (LangSmith) pricing
• Russell & Norvig, Artificial Intelligence: A Modern Approach

Spotted something out of date or wrong? Tell us at shigoto.ai.note@gmail.com. Guides are re-checked against official sources on a rolling schedule.