Claude Code Regression Testing After Prompt Changes: Steps

By Aoba Works · Published October 28, 2026 · Sources checked October 28, 2026 · 21 min read

How this guide was made: drafted with AI assistance from the official sources linked below, then passed an automated editorial check before publishing. Our process

• Agent Evals
• Claude Code
• Claude.md
• Headless Mode

To regression-test Claude Code after a prompt change, run a fixed set of test cases several times with your current CLAUDE.md and save the results. Then run the same cases with the edited file, grade both sets the same way, and compare pass counts per case. Use a threshold so one random failure doesn't block a good edit and a real drop doesn't slip through. Anthropic's own eval guide says regression evals ask "Does the agent still handle all the tasks it used to?" and "should have a nearly 100% pass rate." This guide covers that workflow end to end, with scripts you can adapt.

Which option is right for you?

Choose your setup by how often the agent runs unattended and how much you want to spend on test runs.

If you haven't set up an eval loop yet, start with the pillar guide, AI Agent Evaluation Framework: Claude Code Eval Loop Steps. This article covers one narrower job: comparing before and after a change.

Why do prompt changes to CLAUDE.md silently break agent behavior?

CLAUDE.md edits change the context Claude reasons over. A rule that used to work can get less weight or be contradicted by a new line, and nothing throws an error.

The Claude Code memory docs say this directly: "Claude treats them as context, not enforced configuration." The same page adds that "the more specific and concise your instructions, the more consistently Claude follows them." So there's no parser that rejects a bad CLAUDE.md. A vague new rule, or one that conflicts with an old one, just changes the odds of what the agent does.

Anthropic shipped a regression like this in Claude Code itself. Its April 2026 postmortem traces one of three quality problems to a system prompt line added on April 16 that capped text between tool calls and final responses at short word counts. A broader eval later "showed a 3% drop for both Opus 4.6 and 4.7," and the prompt was reverted in the April 20 release. In Anthropic's words, "one addition to the system prompt caused an outsized effect on intelligence in Claude Code." Their first eval set didn't catch it. The fix they committed to was "a broad suite of per-model evals for every system prompt change" and ablations "to understand the impact of each line."

For example, suppose your nightly dependency-update agent has a CLAUDE.md rule saying "never edit files under migrations/." You add a line for brevity: "Keep final reports under 50 words." Nothing obvious breaks. But the shorter report may drop the "files changed" list your morning check depends on. The agent may also skip the step where it re-reads the rules before editing. You'd only see this by comparing runs.

Pitfalls specific to CLAUDE.md:

• Edits outside the file you're looking at. User-level ~/.claude/CLAUDE.md, .claude/rules/, and auto memory all load too (memory docs), so the "before" and "after" may differ in more than one file.
• Hard rules written as prose. If an action must never happen, the docs point to a PreToolUse hook rather than a CLAUDE.md line. A regression suite measures how well guidance is followed. A hook enforces it. Our guardrails guide shows which rules belong where.

How do you set up a baseline run before editing a prompt?

Run your current CLAUDE.md against a fixed set of representative inputs and save every output before you change anything. That snapshot is your baseline.

There's one trap to clear first. Headless mode has a --bare flag that skips "auto-discovery of hooks, skills, … auto memory, and CLAUDE.md" (headless docs). The same page says "--bare is the recommended mode for scripted and SDK calls, and will become the default for -p in a future release." That's the right choice for most scripts. For a test of CLAUDE.md, though, it would quietly test nothing. So the baseline runs a normal claude -p inside a fresh copy of a fixture project, with only the CLAUDE.md you're testing in it.

Steps:

1. Create an evals/ folder in your repo with cases/ (one JSON file per case), fixtures/ (small sample projects), and schema.json (the output shape you'll grade).
2. Pin everything that isn't the variable you're testing: the full model ID (claude-opus-5-5, per the model config page), the effort level, and the Claude Code version. The setup page shows how to install an exact version, for example curl -fsSL https://claude.ai/install.sh | bash -s 2.1.89, and how to stop background updates with DISABLE_AUTOUPDATER (setup docs).
3. Ask for structured output. With --output-format json --json-schema '<schema>', the response puts schema-shaped data in a structured_output field. If the run ends without valid output, that field is missing (headless docs).
4. Copy the current CLAUDE.md to evals/baseline/CLAUDE.md and commit it, so the baseline is reproducible.
5. Run every case N times (3 to start) and save each run's JSON, the list of changed files, and the test exit code.

An example evals/schema.json for a "fix failing tests" job:

{
  "type": "object",
  "properties": {
    "status": { "enum": ["done", "blocked"] },
    "files_changed": { "type": "array", "items": { "type": "string" } },
    "tests_passed": { "type": "boolean" },
    "summary": { "type": "string" }
  },
  "required": ["status", "files_changed", "tests_passed", "summary"]
}

An example case file, evals/cases/fix-date-parse.json:

{
  "id": "fix-date-parse",
  "fixture": "evals/fixtures/date-utils",
  "prompt": "Run the tests and fix the failing date parsing test. Do not touch the migrations folder.",
  "allowed_paths": "^src/",
  "assert": ".structured_output.status == \"done\""
}

The runner script, evals/run.sh (an example to adapt; it assumes jq and git are installed):

#!/usr/bin/env bash
# usage: evals/run.sh <label> <path/to/CLAUDE.md> [runs]
set -euo pipefail
ROOT=$(pwd); LABEL=$1; MEMORY=$2; RUNS=${3:-3}
MODEL=${EVAL_MODEL:-claude-opus-5-5}; EFFORT=${EVAL_EFFORT:-high}
SCHEMA=$(cat evals/schema.json)

for case in evals/cases/*.json; do
  id=$(jq -r .id "$case"); prompt=$(jq -r .prompt "$case")
  fixture=$(jq -r .fixture "$case")
  for i in $(seq 1 "$RUNS"); do
    work=$(mktemp -d); out="$ROOT/results/$LABEL/$id/run$i"; mkdir -p "$out"
    cp -R "$fixture"/. "$work"/ && cp "$MEMORY" "$work/CLAUDE.md"
    (cd "$work" && git init -q && git add -A && git commit -qm base)
    (cd "$work" && claude -p "$prompt" \
       --model "$MODEL" --effort "$EFFORT" \
       --output-format json --json-schema "$SCHEMA" \
       --allowedTools "Read,Edit,Bash(npm test *)" \
       --permission-mode dontAsk --max-turns 30 --max-budget-usd 1 \
       > "$out/result.json") || echo $? > "$out/claude_exit"
    (cd "$work" && git diff --name-only > "$out/changed.txt" \
       && { npm test --silent >/dev/null 2>&1; echo $? > "$out/tests_exit"; })
  done
done

Each run starts from a clean copy because "each trial should be 'isolated' by starting from a clean environment" (Anthropic). Leftover files from run 2 shouldn't help or hurt run 3. Run the baseline with bash evals/run.sh baseline evals/baseline/CLAUDE.md 3.

The flags above are explained in more depth in Claude Code Headless Mode: Flags, Auth and Cost (2026). In short, dontAsk denies anything that would prompt, so the run can't hang waiting for approval.

How do you build a before/after diff workflow for prompt changes?

Run the same cases through the old and new CLAUDE.md, then compare graded structured fields per case. Don't compare raw text, which differs on every run even when behavior hasn't changed.

The workflow:

1. Save your edit as evals/candidate/CLAUDE.md. Leave the real CLAUDE.md alone until the comparison passes.
2. Run bash evals/run.sh candidate evals/candidate/CLAUDE.md 3.
3. Grade both folders with the same grader (next section).
4. Compare per-case pass counts with evals/compare.sh, and apply the threshold from the flaky-run section.
5. For every case that got worse, open the transcripts of a failing candidate run next to a passing baseline run. Anthropic's guide warns: "You won't know if your graders are working well unless you read the transcripts" (source).
6. Only then copy the candidate into the real CLAUDE.md and commit it, with the comparison summary in the commit message.

Example evals/compare.sh:

#!/usr/bin/env bash
# usage: evals/compare.sh baseline candidate
set -euo pipefail
join -t $'\t' <(sort "results/$1/summary.tsv") <(sort "results/$2/summary.tsv") |
while IFS=$'\t' read -r id base_pass base_n cand_pass cand_n; do
  verdict="ok"
  if   [ "$cand_pass" -lt "$base_pass" ] && [ $((base_pass - cand_pass)) -ge 2 ]; then verdict="REGRESSION"
  elif [ "$cand_pass" -lt "$base_pass" ]; then verdict="flaky-rerun"
  fi
  printf "%-24s base %s/%s  cand %s/%s  %s\n" "$id" "$base_pass" "$base_n" "$cand_pass" "$cand_n" "$verdict"
done | tee "results/compare-$2.txt"
! grep -q REGRESSION "results/compare-$2.txt"

The last line makes the script exit non-zero when any case regressed. A CI job can then fail the pull request.

Pitfalls:

• Changing two things at once. If you edit CLAUDE.md and bump the model in the same run, you can't tell which one caused a drop. Anthropic's postmortem calls this kind of one-change-at-a-time testing "ablations."
• Diffing summary. Free-text fields should be graded by a rubric, not a string diff.
• Forgetting cost. The JSON output includes total_cost_usd per run (headless docs), described there as a client-side estimate. Sum it with jq -s 'map(.total_cost_usd // 0) | add' results/candidate///result.json so you know what each comparison costs.

What makes a good grader for automated regression checks?

A good grader checks what "done" means for your task, using code first. It's a rule-based check, or a separate Claude call with an explicit rubric. "Looks right" isn't a grader.

Start with code. Anthropic's eval guide describes code-based graders as "fast, cheap, objective, reproducible, easy to debug," and advises that "it's often better to grade what the agent produced, not the path it took" (source). For a coding agent, the outcome usually lives in the workspace: did the tests pass, which files changed, does the structured output say done?

Example evals/grade.sh, which writes one summary.tsv line per case (id, passes, runs):

#!/usr/bin/env bash
# usage: evals/grade.sh <label>
set -euo pipefail
LABEL=$1; : > "results/$LABEL/summary.tsv"
for case in evals/cases/*.json; do
  id=$(jq -r .id "$case"); allow=$(jq -r .allowed_paths "$case"); assert=$(jq -r .assert "$case")
  pass=0; n=0
  for run in results/$LABEL/$id/run*; do
    n=$((n+1)); ok=1
    jq -e '.structured_output' "$run/result.json" >/dev/null 2>&1 || ok=0
    [ "$(cat "$run/tests_exit" 2>/dev/null)" = "0" ] || ok=0
    grep -vE "$allow" "$run/changed.txt" | grep -q . && ok=0      # touched a forbidden path
    jq -e "$assert" "$run/result.json" >/dev/null 2>&1 || ok=0
    [ $ok -eq 1 ] && pass=$((pass+1))
  done
  printf "%s\t%s\t%s\n" "$id" "$pass" "$n" >> "results/$LABEL/summary.tsv"
done

Add a model-based grader only for parts code can't check, like whether a report is accurate or follows a tone rule. Make it a separate claude -p --bare call. Bare mode is correct here, because the judge shouldn't load your agent's CLAUDE.md, and in bare mode Claude Code needs ANTHROPIC_API_KEY set (headless docs). Rules that keep it honest:

1. Grade one dimension per judge call. Anthropic recommends grading "each dimension with an isolated LLM-as-judge rather than using one to grade all dimensions."
2. Give it a way out. Allow "unknown" as a verdict, as the same guide suggests, and count unknowns as "needs a human look," not as passes.
3. Use a detailed rubric and ask for reasons before the score. Anthropic's test-development docs list detailed rubrics and reasoning-first grading as LLM-grading tips.
4. Calibrate. Hand-grade a sample of runs and check the judge agrees with you. Anthropic says judges "should be closely calibrated with human experts."

An example judge call:

jq -r '.structured_output.summary' "$run/result.json" | claude --bare -p \
  "Rubric: PASS only if the summary lists every changed file by path. FAIL if any changed file is missing. UNKNOWN if you cannot tell. Changed files: $(tr '\n' ' ' < "$run/changed.txt"). Explain briefly, then give the verdict." \
  --model claude-opus-5-5 --output-format json \
  --json-schema '{"type":"object","properties":{"reason":{"type":"string"},"verdict":{"enum":["pass","fail","unknown"]}},"required":["reason","verdict"]}' \
  | jq -r '.structured_output.verdict'

Limitation: an LLM judge is also nondeterministic and costs tokens on every graded run. If you're on a tight budget, keep judges for the two or three cases where prose quality actually matters.

How do you set a flaky-run threshold to avoid false alarms?

Run each Claude Code test case several times (3–5) and flag a regression only when the candidate's pass count drops by a margin set in advance. One random failure shouldn't block a good edit.

The need for this comes straight from Anthropic's guide: "Agent behavior varies between runs, which makes evaluation results harder to interpret." The guide's math shows why one run isn't enough. With a 75% per-trial success rate and three trials, "the probability of passing all three is (0.75)³ ≈ 42%" (source). A single pass or fail tells you very little about a case like that.

Two rules make the threshold workable:

• Only cases that pass reliably in the baseline belong in the regression suite. Anthropic says high-pass-rate capability evals can "graduate" to become a regression suite. A case that passes 3/5 on the baseline is a capability problem, so fix it separately.
• Decide the threshold before you look at results, and write it into compare.sh.

Example thresholds to start from. These are starting points, not Anthropic numbers, so tune them to your budget:

When a case lands in the "rerun" band, rerun only that case. After the extra runs, call it a regression if the combined candidate rate is clearly below the baseline rate, and read the failed transcripts either way. Our metrics article, AI Agent Evaluation Metrics: Pass Rate Across Repeat Runs, explains when to report pass@k and when to report pass^k.

Pitfalls:

• Retrying until green. If you rerun a failing case until it passes, you've measured pass@k, which is the "at least one success" number. An unattended agent needs the "every run succeeds" view.
• Ignoring infrastructure failures. A run that died from a rate limit isn't a prompt regression. Record the Claude Code exit code (claude_exit) and exclude API errors from the pass count. For handling them, see Claude Code Rate Limit Reached: Retry Setup for Cron Jobs.
• Expecting a temperature knob. The headless docs don't document a sampling-temperature flag for claude -p, so repeat runs are how you deal with variance, not settings.

How do you regression-test after swapping models (e.g. Sonnet to Opus)?

Treat a Claude Code model swap exactly like a prompt change. Rerun the same baseline suite with the new model pinned, then compare. Different models read the same CLAUDE.md differently.

Steps:

1. Pin both sides by full ID, not alias. The model config page warns that "aliases point to the recommended version for your provider and update over time," and says to use the full name (for example claude-opus-5-5) or ANTHROPIC_DEFAULT_OPUS_MODEL to pin (model config). If your cron job says --model opus, the model can change under you with no edit on your side.
2. Pin effort too. Default effort differs by model. As of October 2026, the docs list medium as the default for Opus 5.5, Sonnet 5.5 and Haiku 5.5, xhigh for Opus 4.7, and high for the others (model config). Anthropic's postmortem traces another of its quality issues to a default effort change from high to medium. Pass --effort explicitly on both runs.
3. Run EVAL_MODEL=<old-id> bash evals/run.sh model-old evals/baseline/CLAUDE.md 5, then the same with the new ID and label model-new.
4. Grade both, compare, and read failing transcripts.
5. If the new model regresses only on cases tied to one CLAUDE.md rule, rewrite that rule and test again. That rewrite is now a prompt change, so it goes through the same loop.

For example, a subagent that was moved to a cheaper model may stop writing its files_changed list. Your schema check catches that, which a string diff wouldn't. If you're choosing models per role, Claude Code Subagents Model Selection: Role-by-Role Table covers that choice. This suite is how you confirm it.

Pitfalls:

• Testing the new model on a different Claude Code version. The CLI updates itself by default. The stable channel serves "a version that is typically about one week old, skipping releases with major regressions" (setup docs), but pinning a version during the comparison is more reliable.
• Retiring the old model too early. Keep the old run's results. They're your evidence if the new model's problems show up a week later.

What should go into a minimal test case suite for an agent?

A minimal suite needs 5–15 cases drawn from your real recurring job. Include everyday tasks, edge inputs, one case per critical CLAUDE.md rule, and every past failure.

Anthropic's guide says "20-50 simple tasks drawn from real failures is a great start" (source). That's the target for a mature agent. For one solo-run job, 5–15 cases at 3–5 runs each already means 15–75 runs per side, and that's what keeps a comparison affordable enough to run every time. Grow toward 20–50 as failures give you new cases. Anthropic's test docs lean the same way: "Prioritize volume over quality" (develop tests). More cases with automated grading beat a few hand-graded ones.

An example suite plan for one nightly job:

The rule-specific cases do a second job. If CLAUDE.md quietly stops loading, for example because a future release makes --bare the default for -p, those cases fail in both runs. That tells you the harness broke, not the prompt.

Write each case so the result is clear-cut. Anthropic's test is that "a good task is one where two domain experts would independently reach the same pass/fail verdict." If you can't write the assert line, the case isn't ready yet.

How do you automate regression tests to run on every prompt change?

Wrap the run, grade and compare steps in one script, and trigger it from CI. Run it whenever CLAUDE.md, .claude/, model settings or the eval files change.

Add a single entry point, evals/check.sh:

#!/usr/bin/env bash
set -euo pipefail
RUNS=${RUNS:-3}
bash evals/run.sh baseline evals/baseline/CLAUDE.md "$RUNS"
bash evals/run.sh candidate CLAUDE.md "$RUNS"
bash evals/grade.sh baseline && bash evals/grade.sh candidate
bash evals/compare.sh baseline candidate

Then add a GitHub Actions workflow, .github/workflows/prompt-regression.yml:

name: Prompt regression
on:
  pull_request:
    paths: ["CLAUDE.md", ".claude/**", "evals/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 60
    steps:
      - uses: actions/checkout@v6
      - name: Install pinned Claude Code
        run: curl -fsSL https://claude.ai/install.sh | bash -s 2.1.89
      - name: Run regression suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          DISABLE_AUTOUPDATER: "1"
          EVAL_MODEL: claude-opus-5-5
          RUNS: "3"
        run: bash evals/check.sh
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: eval-results, path: results/ }

Notes on this setup:

• Version 2.1.89 is the example version from the setup docs. Replace it with the version you actually run in production.
• Store the key as a GitHub secret. The GitHub Actions docs say "Never commit API keys or OAuth tokens directly to your repository," and accept CLAUDE_CODE_OAUTH_TOKEN (generated with claude setup-token) if you'd rather bill a Pro or Max subscription.
• Each run uses GitHub Actions minutes plus API tokens, per the same page. Set timeout-minutes and keep RUNS low on pull requests.
• The claude-code-action (anthropics/claude-code-action@v1) is built for having Claude do work in a repo. For an eval harness, calling the CLI directly as above is simpler.

To catch drift you didn't cause, like a CLI update or a model alias moving, also run the suite on a schedule against the current production CLAUDE.md. Note the docs' warning that GitHub "disables the schedule after 60 days without repository activity" in public repos. If your agent runs on a VPS, the same check.sh fits into the crontab described in Claude Code Cron Job Setup: Run Automations Overnight.

What are common pitfalls when catching prompt regressions?

The usual Claude Code regression-testing mistakes are diffing raw text, using too few runs, and leaving the grader itself unversioned. Each one makes the comparison measure the wrong thing.

For example, if you tighten the judge rubric and grade only the candidate with it, scores drop even though the agent didn't change. Re-grade both sides with the same grader version every time.

Once a change ships, regression tests stop and production monitoring takes over. AI Agent Monitoring in Claude Code: Catch Silent Failures covers that half.

Next steps

Do these today:

1. Copy your current CLAUDE.md to evals/baseline/CLAUDE.md, write 5 case files from last month's real jobs (include at least one failure), and run evals/run.sh 3 times per case.
2. Change every --model opus/sonnet in your cron jobs and scripts to a full model ID, and add an explicit --effort.
3. Make your next CLAUDE.md edit go through evals/check.sh before it reaches the file your agent actually reads. The structure in CLAUDE.md Best Practices: A Template for Unattended Agents makes rule-specific cases easier to write.

If you'd rather not build the harness yourself:

• Promptfoo is an open-source eval tool with a Claude Agent SDK provider (anthropic:claude-agent-sdk, alias anthropic:claude-code). It supports options like working_dir, max_turns and copy_working_dir for isolated runs. It suits developers who want config files and a results UI instead of shell scripts. It's not a good fit if you can't run Node/npm in your environment. One limitation: you still have to write the cases and assertions, which is where most of the effort goes.
• For billing, CI runs that use an API key are charged per token (pricing). A subscription token instead bills your plan: Claude Pro is listed at $20 if billed monthly, and Max plans start at $100 per month (as of October 2026; check the official pricing page). A subscription suits a solo builder running small suites. It doesn't suit large nightly suites shared across a team, where an API key with a budget cap is easier to control.

Related guides on this site:

• AI Agent Evaluation Framework: Claude Code Eval Loop Steps
• AI Agent Evaluation Metrics: Pass Rate Across Repeat Runs
• Claude Code Headless Mode: Flags, Auth and Cost (2026)

FAQ

How often should you re-run regression tests on a Claude Code agent?

Run them on every change to CLAUDE.md, .claude/ settings, the model ID or the effort level, plus on a schedule (weekly is a reasonable example) to catch drift from CLI or model updates. The scheduled run uses the production CLAUDE.md as both baseline and candidate inputs over time, so compare it against the last saved results. If nothing changed and nothing failed, the run only cost tokens.

Can you use Claude itself as the grader for regression testing?

Yes, for parts code can't check, such as whether a summary is accurate. Run it as a separate claude --bare -p call with a written rubric, one dimension per call, and an unknown option, as Anthropic's eval guide recommends. Calibrate it against your own hand grading, and keep the rubric in git so grader changes don't look like agent regressions.

What's a good sample size to avoid flaky test results?

Three runs per case is the cheapest useful setting, and five is a solid default for agents that run unattended every day. Add a rerun band so borderline results (2/3 or 4/5) get more runs before you decide. Anthropic's pass^k example shows that a 75%-reliable task passes three straight runs only about 42% of the time, so one or two runs per side can't separate a flaky case from a broken one.

Does editing CLAUDE.md require a full regression suite every time?

For an agent that runs unattended, yes: run the full suite, because a line about one task can change behavior on another, as Anthropic's own postmortem shows. Keep the suite small (5–15 cases) so this stays cheap. Pure typo fixes are the only reasonable exception, and even then the scheduled run will catch anything you missed.

Sources

• Anthropic Engineering: Demystifying evals for AI agents
• Anthropic Engineering: An update on Claude Code quality reports (April 2026 postmortem)
• Claude Code docs: Run Claude Code programmatically (headless)
• Claude Code docs: How Claude remembers your project (memory)
• Claude Code docs: Model configuration
• Claude Code docs: Advanced setup (install, versions, updates)
• Claude Code docs: GitHub Actions
• Claude Platform docs: Develop tests
• Promptfoo: Claude Agent SDK provider
• Claude pricing
• Claude Platform API pricing
• Top search results reviewed for coverage gaps: TestGrid, Shiplight, Diffy

Spotted something out of date or wrong? Tell us at shigoto.ai.note@gmail.com. Guides are re-checked against official sources on a rolling schedule.