Questions? Call 1.800.716.8955

Chapter 5

Eval, Testing, and Debugging (2.6%)

What the exam tests: identifying error types, picking a recovery strategy, reading traces to find failure modes, and deciding whether a problem comes from your integration code or from the model's output. Only one or two items, but evaluation thinking runs through the whole exam.

First question: integration or model?

Most debugging items hinge on isolating where a problem originates.

Symptom Likely origin First check

HTTP 4xx or 5xx error Integration layer Status code, error type, request_id Truncated answer Integration (your max_tokens) stop_reason == "max_tokens"

  • JSON parse error on model output Model output or missing con‐
  • straint
  • Use structured outputs or strict tools

Tool never called Tool definition or prompt Tool description quality, tool_choice Tool called with wrong arguments Schema or description Schema clarity, enums, strict: true Correct tool result, wrong final an‐

swer

Model reasoning or context Trace: was the result in context? Too much noise?

Works in dev, fails in prod Configuration Model version, prompt version, env vars Quality dropped after an upgrade Model change Pinned version, eval comparison

Error types and recovery strategies

Figure 5. Match the recovery to the class of error

Error class Example Recovery

Transient 429, 500, 529, timeouts Retry with exponential backoff and jitter; honor retry-after Client 400, 401, 403, 404, 413 Do not retry unchanged; fix request, key, model ID, or size Model out‐

  • put
  • Invalid JSON, missing field,
  • refusal

Validate, then retry with feedback, constrain output, or fall back Tool execu‐

tion

Downstream API failed Return tool_result with is_error: true and a useful message so Claude can adapt

Logic Agent loops forever Cap iterations, add a task budget, add a stop condition Use the SDK's typed exceptions (for example RateLimitError, APIStatusError, APIConnectionError in Python), catching the most specific first. Log the request_id from every error; Anthropic support needs it.

Trace analysis

A trace is the full record of an agent run: every request, response, tool call, tool result, token count, and latency. Read traces to find failure modes such as:

The model never saw the needed fact (retrieval or context problem).

The fact was in context but buried in noise (context bloat).

A tool returned an error the model misread.

The agent repeated the same tool call (missing feedback or state).

EVAL, TESTING, AND DEBUGGING (2.6%)

Instructions conflicted between system prompt and CLAUDE.md or tool descriptions. Instrument from day one. The SDKs support OpenTelemetry, and Claude Managed Agents provides a session viewer in the Console.

Evaluations in brief

An eval is a fixed set of inputs plus a grading method that you run every time you change a prompt, model, or tool. Grading methods, cheapest first:

Code-based: exact match, regex, JSON schema validation, unit tests.

LLM-as-judge: a separate Claude call scores output against a rubric. Validate the judge against human labels.

Human review: the gold standard, expensive, used to calibrate the other two. Good evals use real or realistic inputs, include edge cases and adversarial cases, and keep a held-out test set so you do not overfit the prompt. In production, sample live traffic for quality monitoring.

Worked example 5.1: error handling that separates retryable from fatal import logging

  • import anthropic
  • client = anthropic.Anthropic(max_retries=3, timeout=120.0)
  • log = logging.getLogger("claude")
  • def ask(messages):
  • try:

resp = client.messages.create(model="claude-sonnet-5-5", max_tokens=2048, messages=messages)

except anthropic.BadRequestError as e: # 400: our bug, do not retry log.error("bad request %s: %s", e.request_id, e.message)

  • raise
  • except anthropic.AuthenticationError: # 401: config problem
  • log.critical("API key invalid or missing")
  • raise

except anthropic.NotFoundError as e: # 404: usually a wrong model ID log.error("not found %s", e.request_id)

raise

except anthropic.RateLimitError as e: # 429: SDK already retried log.warning("rate limited after retries %s", e.request_id)

raise RetryLater() from e

except anthropic.APIStatusError as e: # other HTTP errors, e.g. 500, 529 log.warning("server error %s %s", e.status_code, e.request_id)

raise RetryLater() from e

except anthropic.APIConnectionError: # network failure or timeout raise RetryLater()

  • if resp.stop_reason == "max_tokens":
  • log.info("truncated response %s", resp.id)
  • elif resp.stop_reason == "refusal":
  • log.info("refusal %s", resp.stop_details)
  • return resp
  • What the exam wants you to see:

Most specific first. RateLimitError is a subclass of APIStatusError; catching the parent first would hide the distinction.

The SDK already retries 408, 409, 429, and 5xx responses (twice by default). Your code handles what is left after those retries.

Log the request_id. It is the key Anthropic support uses to find a failed request. A 200 response can still be a failure. Truncation and refusal arrive as normal responses; only stop_reason reveals them.

Worked example 5.2: reading a trace

An internal HR agent answered "You have 0 vacation days left" to an employee who had 12. Here is the trimmed trace:

  • Step Event Detail
  • 1 user "How many vacation days do I have left?"
  • 2 assistant tool_use get_balance(employee_id="E-1042")
  • 3 tool_result {"error": "timeout", "balance": 0}
  • 4 assistant "You have 0 vacation days left."

Diagnosis: the model read the tool result faithfully. The fault is in the integration layer: the tool wrapper returned a default balance: 0 alongside the error instead of an error result. Claude had no way to know 0 was fake.

Fix: on failure, return is_error: true with a message like "Balance service timed out; ask the user to try again later." Never return fabricated defaults. Also add a test that simulates the timeout.

The exam lesson is problem origin isolation: before changing the prompt or the model, confirm what the model actually saw.

Worked example 5.3: a second trace, a model-side failure

A contract-extraction workflow returns the wrong governing law for some contracts.

  • Step Event Detail
  • 1 user 45-page contract text plus "Extract governing law"
  • 2 assistant {"governing_law": "Delaware"}

EVAL, TESTING, AND DEBUGGING (2.6%)

The contract says Delaware in a template header but Ontario in clause 31.2, which governs. The input was complete and correct, so this is a model-output problem. Fixes, in order of effort: Clarify the prompt: "The governing law is set in the operative clause, usually titled Governing Law or Jurisdiction. Ignore template headers." Ask for the clause number and quote alongside the answer, so the output can be checked. Add three contracts with misleading headers to the eval set.

Raise effort for this step if errors persist.

Worked example 5.4: a minimal eval harness

  • import json
  • import anthropic
  • client = anthropic.Anthropic()
  • SYSTEM = open("prompts/classifier.v7.md").read()
  • def classify(ticket: str) -> str:
  • r = client.messages.create(
  • model="claude-haiku-5-5", max_tokens=20, system=SYSTEM,
  • messages=[{"role": "user", "content": ticket}],
  • )
  • return r.content[0].text.strip().lower()

cases = [json.loads(line) for line in open("evals/tickets.jsonl")] # {"text":..., "label":...}

results = [(c, classify(c["text"])) for c in cases]

accuracy = sum(pred == c["label"] for c, pred in results) / len(cases)

  • print(f"accuracy: {accuracy:.1%} on {len(cases)} cases")
  • for c, pred in results:
  • if pred != c["label"]:

print(f"MISS expected={c['label']} got={pred}: {c['text'][:80]}")

This is code-based grading: exact match on a label. It is cheap, deterministic, and should run on every prompt change. For larger runs, send the eval through the Batch API to halve the cost.

Worked example 5.5: an LLM-as-judge rubric

For open-ended outputs such as support replies, exact match does not work. Use a separate grading call with a precise rubric:

You are grading a customer-support reply. Score each criterion 1 (yes) or 0 (no).

<criteria>
  • 1. Answers the customer's actual question.
  • 2. States no facts that contradict the provided policy.
  • 3. Gives a clear next step.
  • 4. Uses a polite, non-defensive tone.
  • 5. Is under 150 words.
</criteria>
<policy>{policy}</policy>
<customer_message>{message}</customer_message>
<reply>{reply}</reply>

Return JSON: {"scores": [..5 integers..], "reason": "one sentence"} Good practice: binary criteria rather than 1 to 10 scales, structured output so scores parse reliably, and a calibration step where humans grade 50 replies and you check the judge agrees. Criterion 5 should be checked in code, not by the judge.

Scenario drills

A new prompt version scores 94% on the eval versus 91% for the old one, but customers complain more. What might be wrong with the eval?

An agent returns a 400 error only for some users. What is your first step?

Responses are fine in testing but often end mid-sentence in production. What do you check? A tool wrapper catches all exceptions and returns an empty list. Why is that dangerous? Your LLM judge gives nearly every output 9 out of 10. What do you change?

Answers:

The eval set may not represent real traffic (synthetic, too easy, or missing new categories); refresh it from anonymized production samples.

Log and inspect the failing request body and request_id; something user-specific (a huge attachment, an unsupported file type, malformed history) is breaking the request. stop_reason; production inputs are probably longer and hitting max_tokens.

Claude cannot distinguish "no results" from "the search failed," so it will confidently report nothing found.

Switch to specific binary criteria, add examples of bad outputs, and calibrate against human labels.

Common traps

Retrying a 400 in a loop.

Blaming the model when the trace shows the tool result never reached it.

Upgrading a model without re-running evals.

Using an LLM judge with no check against human judgment.

Being skeptical of nothing: confident output is not correct output; validate it.

EVAL, TESTING, AND DEBUGGING (2.6%)

An agent sometimes ends with "I couldn't find the order," yet the logs show the order API returned it. Where do you look first?

A nightly job fails with 529 errors at peak times. What is the right handling? Which grading method should you try first for a field-extraction task?

Answers: The trace: whether the tool result was appended correctly to the conversation and whether it was buried in noise. Retry with exponential backoff (or move the job to the Batch API). Code-based grading: compare extracted fields to expected values.

Use ← → to move between sections