AI Agent Monitoring in Claude Code: Catch Silent Failures

By Aoba Works · Published October 23, 2026 · Sources checked October 23, 2026 · 17 min read

How this guide was made: drafted with AI assistance from the official sources linked below, then passed an automated editorial check before publishing. Our process

• Claude Code
• Hooks
• Cron
• Cost Tracking

AI agent monitoring means tracking what an autonomous agent did, what it produced and what it cost on every run, so you find out when it fails before your users or your data do. If you are a solo builder running Claude Code on a schedule, you can skip the enterprise OpenTelemetry dashboard. A Claude Code Stop hook that checks the agent's expected output, a cron wrapper that logs total_cost_usd, and a free heartbeat check will catch most silent failures, such as a nightly aggregation that "finished" but never wrote the file.

Which option is right for you?

Start with the cheapest setup that can tell you a job failed. Add tracing only when a single log line no longer explains why it failed.

Why these picks: the beginner row costs nothing and catches "never ran" and "ran but produced nothing". The daily-use row adds a history you can grep. The budget row stops runaway spend at the CLI level. Each row further down adds visibility but also adds a vendor account and more setup.

What Is AI Agent Monitoring?

AI agent monitoring means recording what an agent did, produced and finished on every run. That record lets you detect errors, drift and silent failures.

Classic uptime monitoring asks whether a process is alive. Agent monitoring asks whether the work got done. Datadog's AI observability explainer frames the difference as "answering not just whether a system is running, but whether its outputs are correct, grounded, safe, and useful."

For a solo builder, a run has three things to watch:

• Process outcome. Did claude -p exit 0? Claude Code exits with code 0 on success and a non-zero code when the run fails.
• Work outcome. Does the artifact exist, is it fresh, and does it contain plausible data? No vendor tool knows this for you. You have to define it.
• Cost and behavior. How much did the run spend, how many turns did it take, and did it hit an API error?

Example of a silent failure. Say a nightly job asks Claude Code to aggregate yesterday's orders into reports/daily-2026-10-22.csv. The source API returns an empty page, Claude writes a header-only CSV, explains that "no orders were found," and exits 0. Every process-level signal is green, and your weekly revenue chart quietly drops a day.

Pitfall: treating the agent's final message as the status. The model can sound confident while the file is empty. Check the file itself.

How Is AI Monitoring Different from AI Agent Monitoring?

AI monitoring is the umbrella term for watching any AI system, including models, data and outputs. AI agent monitoring narrows the focus to multi-step runs that call tools and act on their own.

Datadog's explainer describes the agent-specific part as tracing "multi-step decision chains, monitors tool selection and execution, detects agent loops and runaway behavior."

For example: if you only call the Claude API to classify support tickets, general AI monitoring (accuracy on a sample, latency, cost) is enough. Once Claude Code reads files, runs scripts and writes reports, you also need to know which step broke.

What Is a Monitoring Agent?

A monitoring agent is either collector software installed on a host, like the Datadog Agent, or an AI agent whose job is watching other agents.

The two meanings get mixed up in search results. The traditional one comes first: "The Datadog Agent is software that runs on your hosts." It collects metrics and logs and ships them to a backend, and no AI is involved.

The newer meaning is an LLM that judges another agent's work. Claude Code supports this directly. Besides shell command hooks, it has prompt and agent hook types, with default timeouts of 30 seconds and 60 seconds, against 600 seconds for command hooks (Claude Code hooks reference). A Stop hook of type prompt could ask a model whether the run really finished its task.

Pitfalls and fixes:

• The judge is nondeterministic and costs tokens. Use deterministic shell checks first: file exists, row count, valid JSON. Add an LLM judge only for questions a script can't answer, such as "is this summary faithful to the source?"
• The watcher fails silently too. If the monitoring agent crashes, nothing alerts you. Pair it with an external heartbeat (covered below) that expects a ping and alerts when none arrives.

Agent vs Agentless Monitoring: What's the Difference?

Agent-based monitoring installs a collector on the machine for deep, real-time data. Agentless monitoring sends or pulls data over APIs, with less setup and less detail.

The hook-based setup in this guide is agentless. Claude Code itself emits the data, either through hooks or its built-in OpenTelemetry exporter, and a curl sends a heartbeat. Claude Code's exporter can talk to a collector or straight to an OTLP endpoint. You set OTEL_EXPORTER_OTLP_ENDPOINT and pick otlp, prometheus, console or none for OTEL_METRICS_EXPORTER (Claude Code monitoring docs).

Pitfall: going agentless and then assuming you see host problems. A full disk or a cron daemon that died won't show up in Claude Code's telemetry because Claude Code never starts. Fix: the external heartbeat catches "nothing ran at all", which is exactly the gap agentless setups leave.

Single Agent vs Multi-Agent: Which Needs More Monitoring?

Multi-agent systems need more monitoring, because failures often happen at handoffs between agents. A single agent can be watched with one end-of-run check.

Inside Claude Code, SubagentStop receives the same fields as Stop, including last_assistant_message (hooks reference). In headless runs, --output-format stream-json tags each subagent message with parent_tool_use_id, so you can tell which subagent said what (headless docs).

Outside Claude Code, LangGraph apps send traces to LangSmith when the LangSmith environment variables are set. In the subagent tracing docs, you filter by the lc_agent_name metadata key. The vendor-neutral standard is OpenTelemetry's GenAI agent spans, such as invoke_agent and execute_tool. The spec is still marked "Development", so attribute names can change.

Example: a research subagent returns "no sources found" and the writer subagent drafts an article anyway. A SubagentStop hook that logs each subagent's last message would show the empty handoff in your run log. Model choice per role also matters for cost. See /claude-code-subagents-model-selection-role-by-role-table/.

How Do You Track AI Agent Costs?

To track AI agent costs, log total_cost_usd from each claude -p --output-format json run and add the numbers up per job.

With --output-format json, Claude Code's response "includes total_cost_usd and a per-model cost breakdown." The docs also warn that "Both figures are client-side estimates and can differ from your actual bill." For a baseline, Anthropic reports that across enterprise deployments "the average cost is around $13 per developer per active day," with 90% of users under $30 per active day (as of October 2026; check the official page). One unattended job should cost a small fraction of that. If it doesn't, investigate.

Steps:

1. Run your job with --output-format json and pipe the output through jq -r '.total_cost_usd'.
2. Append one CSV line per run: date, job name, exit code, cost.
3. Add a hard cap with --max-budget-usd (a documented CLI flag). For example, --max-budget-usd 2 for a job that normally costs $0.30. If the run hits the cap, Claude Code also stops any remaining background work (headless docs).
4. Once a week, sort the CSV by cost. A job whose cost doubles with no change in input is usually looping or retrying.
5. For authoritative numbers on API billing, use the Usage page in the Claude Console.

If you use a Pro or Max subscription instead of an API key, the dollar figure isn't your bill. The /usage docs say "Claude Max and Pro subscribers have usage included in their subscription." On those plans, check /usage for plan usage rather than relying on the dollar estimate (costs docs). For a full walkthrough, see /claude-code-how-to-see-token-usage-and-reduce-it/.

Pitfall: --continue and --resume report the whole conversation's total, earlier runs included (headless docs). Summing those values double-counts. Start each scheduled job as a fresh session, or log only the difference.

Which AI Agent Monitoring Tools Fit a Solo Builder?

For one person, a free heartbeat service plus Claude Code hooks covers most needs. Enterprise suites like Datadog, Splunk and AppDynamics make sense mainly if you already pay for them.

Datadog describes its product as one that "displays each step in a multi-agent workflow, including which tools agents used, handoffs between agents, retries, and errors." That level of detail helps when ten services call each other. It is more than one cron job needs.

Pitfall: buying a tracing product to fix a problem that is really "the job didn't run." Traces can't show a run that never started. A heartbeat can.

What Should an AI Agent Monitoring Dashboard Show?

A useful dashboard shows each run's status, whether the expected output passed its check, the cost, the duration and the last error. For one builder, that can be a log file.

The minimum fields per run:

Two ways to view it:

1. No-server option. Run jq -s 'group_by(.status) | map({status: .[0].status, n: length})' ~/.claude/monitor/runs.jsonl for a status count, and check the Healthchecks.io check list for up/down per job.
2. Charts option. Export Claude Code telemetry with CLAUDE_CODE_ENABLE_TELEMETRY=1, OTEL_METRICS_EXPORTER=otlp and OTEL_EXPORTER_OTLP_ENDPOINT set to the OTLP endpoint of your Grafana Cloud stack or collector. Then chart claude_code.cost.usage (USD) and claude_code.token.usage (tokens). Metrics export every 60 seconds by default and logs every 5 seconds (monitoring docs).

Pitfall: OTel metrics show spend and activity, not whether your CSV has rows. Keep the output check in your own log even if you add Grafana.

How Do You Build Lightweight Monitoring with Claude Code Hooks?

Use a Stop hook to check the output, and a cron wrapper to report results to a heartbeat. The hook confirms that the run's output exists and looks complete. The whole setup needs no monitoring stack.

The relevant behavior from the hooks reference: Stop fires "When Claude finishes responding," and "Stop hooks can block Claude from stopping and force the conversation to continue." Exiting with code 2 from a Stop hook prevents Claude from stopping. StopFailure fires "When the turn ends due to an API error". Its output is ignored, but the command still runs, so it can write a log line. For every event, see /claude-code-hooks-lifecycle-every-event-explained/.

Step 1: Write the Stop hook

Run mkdir -p ~/.claude/hooks ~/.claude/monitor and save the script below as ~/.claude/hooks/check-output.sh. It's an example; change the thresholds to fit your job.

#!/bin/bash
# Example Stop hook: verify the job's artifact; log every result.
input=$(cat)
session=$(echo "$input" | jq -r '.session_id')
reason=$(echo "$input" | jq -r '.stop_reason')
LOG="$HOME/.claude/monitor/runs.jsonl"
status="ok"; detail=""

# Only check when the cron wrapper set an expected file
if [ -n "$MONITOR_EXPECTED_FILE" ]; then
  f="$MONITOR_EXPECTED_FILE"
  min="${MONITOR_MIN_LINES:-2}"
  if [ ! -s "$f" ]; then
    status="fail"; detail="missing or empty: $f"
  elif [ "$(wc -l < "$f")" -lt "$min" ]; then
    status="fail"; detail="fewer than $min lines: $f"
  elif [ -n "$(find "$f" -mmin +"${MONITOR_MAX_AGE_MIN:-90}")" ]; then
    status="fail"; detail="stale file: $f"
  fi
fi

jq -nc --arg t "$(date -u +%FT%TZ)" --arg s "$session" --arg r "$reason" \
  --arg st "$status" --arg d "$detail" \
  '{time:$t, session:$s, stop_reason:$r, status:$st, detail:$d}' >> "$LOG"

# Give Claude one more attempt per session, then let it stop
marker="$HOME/.claude/monitor/retried-$session"
if [ "$status" = "fail" ] && [ ! -f "$marker" ]; then
  touch "$marker"
  echo "Output check failed ($detail). Re-check the source data and write the file, or explain exactly why it cannot be written." >&2
  exit 2
fi
exit 0

Make it executable with chmod +x ~/.claude/hooks/check-output.sh. The header-only CSV from the earlier example has one line, so MONITOR_MIN_LINES=2 flags it.

Step 2: Add a StopFailure logger

Save this as ~/.claude/hooks/log-failure.sh and run chmod +x on it as well:

#!/bin/bash
input=$(cat)
echo "$input" | jq -c '{time: (now|todate), event: .hook_event_name, session: .session_id, error: .error}' \
  >> "$HOME/.claude/monitor/runs.jsonl"

If the log shows rate_limit entries, set up retries as described in /claude-code-rate-limit-reached-retry-setup-for-cron-jobs/.

Step 3: Register both hooks

Add this to ~/.claude/settings.json (all projects) or .claude/settings.json (one project):

{
  "hooks": {
    "Stop": [
      { "hooks": [ { "type": "command", "command": "~/.claude/hooks/check-output.sh" } ] }
    ],
    "StopFailure": [
      { "hooks": [ { "type": "command", "command": "~/.claude/hooks/log-failure.sh" } ] }
    ]
  }
}

Open Claude Code and type /hooks. The docs describe it as "a read-only browser for your configured hooks." Confirm that both hooks appear under Stop and StopFailure, labeled with the right settings source.

Step 4: Wrap the job and add a heartbeat

Create a free check on Healthchecks.io and copy its ping URL. Healthchecks explains why you should report failures directly: "Actively signaling a failure minimizes the delay from your monitored service failing to you receiving an alert." The /start endpoint also lets it measure run time.

#!/bin/bash
# Example wrapper: ~/jobs/daily-orders.sh
export MONITOR_EXPECTED_FILE="$HOME/reports/daily-$(date +%F).csv"
export MONITOR_MIN_LINES=2
HC="https://hc-ping.com/YOUR-UUID"

curl -fsS -m 10 --retry 3 "$HC/start" > /dev/null
out=$(claude -p "Aggregate yesterday's orders into $MONITOR_EXPECTED_FILE" \
  --output-format json --max-budget-usd 2 \
  --allowedTools "Read,Write,Bash(python3 *)")
code=$?
cost=$(echo "$out" | jq -r '.total_cost_usd // 0')
echo "$(date -u +%FT%TZ),daily-orders,$code,$cost" >> "$HOME/.claude/monitor/costs.csv"

# Ground truth, independent of the agent and the hook
if [ "$code" -ne 0 ] || [ "$(wc -l < "$MONITOR_EXPECTED_FILE" 2>/dev/null || echo 0)" -lt 2 ]; then
  curl -fsS -m 10 --retry 3 "$HC/fail" > /dev/null
  exit 1
fi
curl -fsS -m 10 --retry 3 "$HC" > /dev/null

Schedule it with cron as shown in /claude-code-cron-job-setup-run-automations-overnight/. Now there are three layers. The hook gives Claude one chance to fix the output. The wrapper checks the file again on its own. The heartbeat alerts you if the wrapper never runs at all.

Common mistakes and fixes

• Running with --bare, so the hooks never fire. Bare mode skips "auto-discovery of hooks" (headless docs). Fix: pass the hooks with --settings <file>, or drop --bare for this job.
• The hook reads the transcript to find the final message. The transcript "may lag the in-memory conversation" (hooks reference). Fix: use last_assistant_message from the Stop input.
• Network calls in a SessionEnd hook get cut off. SessionEnd hooks share a 1.5-second budget by default (hooks reference). Fix: send pings from the wrapper, not from SessionEnd.
• The check fires during interactive coding. Stop fires after every response. Fix: the script only checks when MONITOR_EXPECTED_FILE is set, and only the cron wrapper sets it.
• Endless blocking. A Stop hook that always exits 2 keeps Claude working. Fix: the per-session marker file allows one retry. For jobs that stall in other ways, see /claude-code-stuck-on-compacting-conversation-cron-watchdog/.
• Untrusted project hooks in -p mode. A -p session runs the hooks in a project's .claude/settings.json "even in a folder you've never trusted." Fix: keep monitoring hooks in your own ~/.claude/settings.json and review any repo's hooks before you schedule it.

Next steps

Things you can do today:

1. Choose the one recurring Claude Code job whose failure would hurt most. Write down the file it must produce and the minimum number of rows that output should have.
2. Install the Stop hook and the wrapper above, run the job once by hand, then check ~/.claude/monitor/runs.jsonl and costs.csv.
3. Test the alert: point MONITOR_EXPECTED_FILE at a path that doesn't exist and confirm that Healthchecks.io marks the check as failed.

If you'd rather not build it yourself: Healthchecks.io Business ($20/month as of October 2026; check the official page) adds capacity beyond the free 20 jobs. It suits people with many cron jobs. If you need to see inside a run, it won't help. LangSmith Plus ($39/seat/month) suits LangGraph apps that need step traces, and it adds little for plain claude -p jobs. Datadog or Splunk agent observability is worth it if your company already pays for the platform. For one person it is usually overpriced and more complex than the problem.

Related guides on this site:

• /claude-code-hooks-examples-7-copy-paste-scripts/
• /why-claude-code-stops-mid-automation-and-how-to-fix-it/
• /claude-code-permissions-best-practices-headless-template/
• /ai-agent-guardrails-example-copy-paste-claude-code-rules/

FAQ

What is AI agent studio monitoring and evaluation?

The phrase refers to the monitoring and evaluation features built into agent-building platforms. Monitoring watches live runs: errors, latency, cost. Evaluation scores outputs against test cases or rubrics. Splunk, for example, now points users to Splunk Agent Observability "to monitor, evaluate, and configure guardrails" for agentic apps. For a Claude Code job, your output check is a small evaluation that runs on every execution.

Do I need a monitoring platform for a single Claude Code agent?

Usually not. One Stop hook, a wrapper that checks the exit code and the output file, and a free heartbeat cover "didn't run", "failed" and "ran but produced nothing". Add a platform once you have several agents passing work to each other, or once teammates need a shared view.

How much does AI agent cost monitoring cost to run?

The DIY version costs nothing beyond your normal Claude usage: Healthchecks.io Hobbyist and Grafana Cloud Free are both $0 (as of October 2026; check the official pages). Hosted tracing starts at LangSmith Plus, $39 per seat per month. Datadog doesn't publish a public price for its agent product, so contact sales.

Can Splunk or AppDynamics monitor custom AI agents out of the box?

Only partly. Splunk's AI Agent Monitoring relies on OpenTelemetry GenAI instrumentation and is scheduled for deprecation. AppDynamics needs Python Agent 25.10.0 or higher configured to publish LLM events (AppDynamics docs). A Claude Code CLI job isn't a Python app, so its OTel export is the closer fit.

Sources

• Claude Code: Hooks reference
• Claude Code: Run Claude Code programmatically (headless)
• Claude Code: Manage costs effectively
• Claude Code: Monitoring (OpenTelemetry)
• Claude Console usage page
• Datadog: What Is AI Observability?
• Datadog: Monitor, troubleshoot, and improve AI agents
• Datadog Agent documentation
• Splunk AI Agent Monitoring docs
• Splunk AppDynamics AI Agent Monitoring docs
• OpenTelemetry GenAI agent spans spec
• OpenTelemetry GenAI semantic conventions
• LangChain: Trace Deep Agents applications
• LangChain pricing (LangSmith)
• Healthchecks.io pricing
• Healthchecks.io pinging API
• Grafana Cloud pricing
• Fin.ai: AI Agent Monitoring and Observability
• Dust: AI Agent Monitoring guide
• Mercor: AI Agent Monitoring & Observability Guide

Spotted something out of date or wrong? Tell us at shigoto.ai.note@gmail.com. Guides are re-checked against official sources on a rolling schedule.