Sample data
Illustrative input and output
Not real customer data — shown to illustrate the shape of a run.
Sample input CSV — support_qa_testset.csv
test_id,category,input_prompt,expected_answer
TC-001,billing,"How do I update my payment method?","Guide the user to Account > Billing > Payment Methods"
TC-002,billing,"Why was I charged twice this month?","Explain proration and offer to check the invoice"
TC-003,technical,"The app crashes when I export a report","Ask for steps to reproduce, offer workaround, escalate if unresolved"
TC-004,account,"How do I delete my account permanently?","Explain the deletion process and 30-day recovery window"
TC-005,technical,"API returns a 429 error intermittently","Explain rate limits and suggest exponential backoff"
Sample run output
| Test ID | Category | Judge: Accuracy | Judge: Tone | Verdict | Review status |
|---|
| TC-001 | billing | 9 / 10 | 9 / 10 | Pass | Reviewed — accepted |
| TC-002 | billing | 6 / 10 | 8 / 10 | Borderline | Reviewed — overridden to Fail |
| TC-003 | technical | 8 / 10 | 7 / 10 | Pass | Reviewed — accepted |
| TC-004 | account | 4 / 10 | 6 / 10 | Fail | Reviewed — accepted |
| TC-005 | technical | 9 / 10 | 9 / 10 | Pass | Pending |
Rows with a copper left border are disagreement rows — flagged automatically wherever attached judge configs diverge, matching the Atlas app's report view.
Sample judge configuration (JSON)
{
"name": "strict-accuracy-v2",
"judge_model": "claude-sonnet",
"scoring_scale": "1-10",
"dimensions": ["accuracy", "completeness", "tone"],
"pass_threshold": 7,
"rubric_prompt": "Score the candidate response strictly on factual correctness against the expected answer. Penalize hedging."
}
Sample agent configuration
Fields shown as they appear in the app's agent editor.
| Field | Example value |
|---|
| generation_mode | llm_prompt |
| generation_model | claude-sonnet |
| temperature | 0.20 |
| system_prompt | "You are a billing support assistant. Be concise, cite the relevant account setting, and never promise a refund." |
| skills attached | prompt_cache, token_budget |
| primary judge | strict-accuracy-v2 (pass_threshold 7, type: rubric) |
Sample run report
Aggregate view for one finished run — report.html's per-judge summary and flaky assertions.
Disagreement count
2 / 200
| Judge | avg_score | pass_rate | pass_threshold |
|---|
| strict-accuracy-v2 | 8.1 / 10 | 89% | 7 |
| tone-check | 8.6 / 10 | 94% | 6 |
| Flaky assertion | fail_count | total | fail_rate |
|---|
| contains_all(required_terms) | 6 | 200 | 3.0% |
| regex_match(order_id_format) | 3 | 200 | 1.5% |
Sample comparison report
Baseline vs. candidate run, matched by match_key — from compare.html.
Verdict flips
3 pass→fail · 9 fail→pass
| Key | Baseline score | Candidate score | Delta | Verdict flip |
|---|
| TC-014 | 6 / 10 | 9 / 10 | +3 | fail → pass |
| TC-057 | 8 / 10 | 5 / 10 | −3 | pass → fail |
| TC-091 | 7 / 10 | 7 / 10 | 0 | — |
Sample drift dashboard
Judge calibration over time and by category — from drift.html.
Highest flip rate
tone-check · 11.3%
| Judge | N | Override % | Pass flip % | Harsh % | Lenient % |
|---|
| strict-accuracy-v2 (#14) | 640 | 5.9% | 3.1% | 2.2% | 3.7% |
| tone-check (#9) | 640 | 9.4% | 11.3% | 8.0% | 3.3% |
All values on this page are illustrative examples matching the app's real field and column names — not figures from a live deployment.
