Evaluating and validating output
Exam domain 2: Output Evaluation and Validation (21%, the largest domain)
You own every output you use, so verify anything that matters before it leaves your hands. Expect about 13 questions on judging accuracy, spotting hallucinations and bias, fact-checking, deciding when a human must review, adapting outputs to an audience, and choosing output formats.
Why Claude gets things wrong
Hallucination. Claude can produce fluent, confident, specific-sounding content that is false: invented citations, statistics, quotes, legal subsections, product features or URLs. Stale knowledge. Without web search, Claude knows only what was in its training data up to a cutoff date.
Missing context. Claude fills gaps with plausible assumptions when you did not give it the facts.
Misreading sources. Long or messy documents, scanned PDFs and complex tables can be misread or partly ignored.
Calculation slips. Mental arithmetic in text can be wrong. Use code execution for anything numeric that matters.
Bias. Outputs can reflect stereotypes or one-sided framing found in training data. The trap the exam sets. Confidence is not evidence. Asking Claude "Are you sure?" or "Rate your confidence" does not verify anything. Anthropic's own sample question marks those options wrong and rewards checking the citation against the official source.
Red flags that call for verification
- Specific numbers, dates, names, percentages or quotes
- Citations, case law, regulation sections, page numbers or URLs
- Claims about recent events
Anything legal, medical, financial, safety-related or compliance-related Claims that are surprising, or that conveniently confirm what you hoped
Internal inconsistencies, such as a total that does not match its parts
Validation techniques
- Technique How it works
- Check against the primary
- source
Open the regulation, contract, dataset or original document and confirm the claim Ask for sources, then open
them
Use Research or web search for citations, then click through; a citation that does not exist is a hallucination
Ask Claude to quote "Quote the exact sentence from the attached contract that supports each point." Quotes are easy to check
Recompute numbers Have Claude use code execution, or check in a spreadsheet Cross-check in a fresh chat Ask the question again with no prior context, or have Claude critique the first answer Ask what could be wrong "List the assumptions you made and the claims most likely to be wrong." Treat this as a pointer, not as proof
Spot-check samples For bulk outputs, such as 500 classified tickets, review a random sample before trusting the rest
Subject-matter review Route high-stakes content to a qualified human
When a human must review
Human review is required, not optional, when the output:
goes to customers, regulators, the public or senior leadership;
makes or supports legal, financial, medical, HR or safety decisions; affects individuals' rights or opportunities, such as hiring, performance or credit; touches regulated data or contractual commitments; or
will be acted on automatically or at scale.
The higher the stakes and the harder an error is to reverse, the more review you need.
Spotting bias
Check whether the output assumes a gender, culture, age or ability without reason. Check whether it presents one side of a contested issue as settled, or uses different language for comparable groups. Fixes include asking for balanced perspectives, removing irrelevant demographic details from inputs, and having diverse reviewers check sensitive content.
Adapting output for its audience
The same analysis may need to become an executive summary, a detailed appendix and a customer FAQ. Ask Claude to adapt it, then review each version against its reader: length, jargon, tone and what decision they need to make. When comparing several drafts, give Claude the criteria and ask for a side-by-side comparison, then make the final call yourself.
Choosing an output format
Format Best for
Inline text Short answers, explanations, quick drafts
Artifact Longer documents, reusable templates, diagrams, code, interactive tools; anything you will iterate on or share
Structured data (table, CSV,
JSON)
Comparisons, extracted fields, data you will paste into another tool Created file (docx, xlsx, pptx,
pdf)
A finished deliverable in the tool your audience uses
A five-question evaluation checklist
Before using any output that matters, run through these questions. They map directly to the Domain 2 task statements. Accurate? Are the facts, figures, names and citations correct? Which ones have I actually checked?
Complete? Did Claude answer every part of the request? Did it skip a document, a section or an edge case?
Consistent? Does the output contradict itself, the sources or earlier decisions? Fair? Is the language balanced and free of unjustified assumptions about people? Fit for the reader? Is the length, tone, format and level of detail right for who will use it? If any answer is "I don't know," either verify or flag the output as unverified when you share it.
Matching verification effort to risk
Not every output deserves the same scrutiny. The exam rewards proportionate review.
Evaluating and validating output
Figure 3. Review effort rises with the stakes of the output
- Risk
- level Example outputs Minimum review
- Low Brainstormed ideas, personal notes, first drafts for
- your own use
- Quick read for sense
- Medi‐
- um
- Internal reports, team emails, meeting summaries
- others rely on
- Check key facts and figures; confirm nothing important
- is missing
- High Customer communications, published content, ex‐
- ecutive or board material
- Verify every specific claim at the source; second re‐
- viewer for sensitive content
- Critical Legal, financial, medical, safety, HR decisions; regu‐
- latory filings
- Qualified expert review and sign-off; Claude output
- treated as a draft only
Worked examples
Worked example 4.1: The invented citation
Situation. Jordan, a compliance coordinator, asks Claude to summarize data-retention requirements for financial records in her province and to cite the relevant section of the statute. The answer is fluent and cites "Section 14(3)(b)."
What Jordan does.
Opens the official statute on the government's legislation site.
Finds that Section 14 exists but has no subsection (3)(b). The retention rule is actually in Section 9. Asks Claude: "The statute is attached. Quote the exact text that sets the retention period, with its section number." Claude now quotes Section 9 correctly.
Notes the correction in her summary and has the legal team confirm before it goes into policy.
More review
Lesson. Specific-looking citations are a classic hallucination. The fix is to check the primary source and, where possible, give Claude the source so it quotes rather than recalls.
Worked example 4.2: Numbers that do not add up
Situation. A marketing analyst asks Claude to summarize campaign results. The summary says, "Total leads: 4,820. Email: 2,100; Social: 1,650; Paid search: 1,270."
Check. 2,100 + 1,650 + 1,270 = 5,020, not 4,820. Something is wrong.
Fix. The analyst uploads the raw data and asks Claude to recompute using code execution and to show the calculation. The correct total is 5,020; Claude had misread one row in the earlier summary.
Lesson. Internal inconsistency is a red flag. For figures that matter, have Claude calculate with code, and reconcile totals yourself.
Worked example 4.3: Sampling a bulk task
Situation. Claude has categorized 900 expense claims into policy categories.
Approach.
Pull a random sample of 50 claims.
Check each against the policy. Result: 46 correct, 4 wrong (92% accuracy).
All four errors involve "client entertainment" versus "team meals."
Add a clear definition and two examples of each to the instructions, then re-run that category. Re-sample 30 claims from the affected category: 30 out of 30 correct.
Lesson. Sampling gives you a measured error rate and shows where to improve the prompt. It is the proportionate answer between "check everything" and "check nothing."
Worked example 4.4: Catching bias in a draft
Situation. An HR coordinator asks Claude for interview questions for a warehouse supervisor role. One question asks whether the candidate "has family commitments that might affect shift work."
Issue. The question invites information about family status, which is irrelevant to ability and may breach employment law in many places.
Fix. Replace it with a neutral, job-related question: "This role includes rotating night shifts. Are you able to meet that schedule?" Then have HR review the full set.
Lesson. Bias can be subtle and legally risky. Review people-related outputs with fairness and law in mind.
Worked example 4.5: One analysis, three audiences
Evaluating and validating output
Situation. An operations lead has a detailed analysis of warehouse delays.
Audience Prompt adjustment Output
Executive team "Five bullets: the problem, cost impact, root cause, recommendation, decision needed."
- Half a page
- Warehouse su‐
- pervisors
"Practical changes to shift handover, in plain language, with a checklist." One page and a checklist
Customers "A short, honest service update that apologizes, explains the fix and gives the new delivery window. No internal detail."
120-word notice
The operations lead checks each version for accuracy and makes sure the customer note contains no internal data.
Worked example 4.6: Comparing drafts against criteria
Situation. A communications team has three draft press-release openings.
Prompt.
Compare these three openings against our criteria: (1) news value in the first sentence, (2) under 40 words, (3) no superlatives, (4) includes the launch date. Make a table scoring each criterion as Met or Not met, with a one-line reason. Do not choose a winner. Lesson. Claude can apply criteria consistently and quickly, and the table makes its judgment checkable. The final choice stays with the team.
Scenario walkthroughs
Scenario A. Claude's summary of a customer interview includes a quote the customer never said.
Analysis. This is a hallucinated quote. Never publish quotes without checking the transcript. Ask Claude to use only verbatim text from the transcript and to mark paraphrases clearly. Scenario B. A research brief from Research mode cites ten sources. You have time to check only three.
Analysis. Prioritize the sources behind the claims your decision depends on, the most surprising claims, and any specific statistics. Flag the remaining claims as unverified in your notes. Scenario C. A colleague says, "Claude said it was 95% confident, so it is fine." Analysis. Self-reported confidence is not a reliable measure of accuracy. Verification against sources is.
Scenario D. Claude's market sizing assumes the target market is all businesses in the country, when your product serves only firms with more than 50 employees.
Analysis. This is an unstated assumption, not a calculation error. Asking Claude to "list all assumptions" surfaces this kind of issue. Correct the assumption and recompute. Scenario E. A board report draft is accurate, but it is 14 pages long and the board asked for two. Analysis. Accuracy is not enough; the output must fit its audience. Ask for a two-page executive version with an appendix, then check that nothing material was lost.
Key points
Confidence is not accuracy. Verify against primary sources.
Citations, numbers and regulated topics are top-priority checks.
Scale review to stakes and reversibility.
Pick the output format your reader will actually use.
Common traps
Treating Claude's stated confidence, or a repeated answer, as verification.
Reformatting or rounding a doubtful figure instead of checking it.
Reviewing every item of a bulk task by hand when a measured sample would do, or reviewing none at all.
Check yourself
Name three types of content that should always trigger verification.
Why is asking Claude "Are you sure?" not a valid check?
You have 2,000 Claude-classified records. What is the proportionate way to validate them?
Which output format suits a comparison of five vendors on six criteria?
An accurate analysis is too technical for its audience. Is it fit for use?
Answers. (1) Any three of: specific numbers, citations, quotes, legal or medical or financial claims, recent events, surprising claims. (2) Claude's confidence comes from the same process that produced the answer, so it does not independently verify it. (3) Review a random sample, measure the error rate, fix the prompt for error patterns, then re-check. (4) A table or structured data. (5) No; it must be adapted to the reader.
Evaluating and validating output
Exercise
Ask Claude a factual question in your field that has a precise answer, such as a regulation or standard. Verify every specific claim against the primary source and note any errors.