Batch-test prompts against CSV test cases, score responses automatically with configurable judge rubrics, and put a mandatory human in the loop before anything counts as "final." Juri replaces ad-hoc console testing with a repeatable, auditable evaluation workflow.
Manually eyeballing whether an LLM's response is "good enough" doesn't scale past a handful of examples. Juri turns that judgment call into a structured pipeline: define a prompt once as a versioned Script, attach one or more Judge Configurations (rubric + scoring model), run it against a CSV of test cases, and get consistent, comparable, human-reviewed results every time.
Every prompt, every generation parameter, and every judge rubric is versioned. A run always points back to the exact script version that produced it — no more "which prompt was this?"
Uses the Claude Agent SDK to both generate candidate responses and score them against configurable rubrics — accuracy, relevance, completeness, clarity, or fully custom dimensions.
No run is ever "final" purely from automated judging. Every row is reviewed, overridable, and requires a reviewer comment — a durable, resumable, auditable trail.
The core loop, from prompt to finalized, exportable report.
Create a Script — system prompt, generation model, temperature/max tokens — and attach one or more Judge Configurations with rubric + pass threshold.
Upload test cases: input prompt, optional expected answer, test case ID, category/tag.
Juri generates a candidate response per row and scores it against every attached judge config. Partial row failures don't abort the batch.
Every row enters mandatory pending review. Reviewers accept, override scores/verdicts, and must comment.
Finalize the run, view the multi-judge comparison report, and export as CSV, JSON, or HTML.
What's shipped and usable today, grouped by area.
Named, versioned prompt bundles with generation params (model, system prompt, temperature, max tokens) and full version history per run.
1..N judge configs per script, each with its own judge model, rubric text, scoring dimensions, scoring scale (1–5, 1–10, pass/fail), and optional custom output schema. Inherits from an org-level default unless overridden.
Row-by-row generate → judge loop with live progress ("87/200 processed") and per-row failure isolation.
Side-by-side prompt, candidate response, and all judge scores/justifications. Accept, override any dimension or the overall verdict, comment required.
Per-row side-by-side scores across judge configs, disagreement rows surfaced prominently, and aggregate averages to spot systematically harsh or lenient configs.
CSV and JSON for raw data, HTML for shareable comparison/full reports. Print-to-PDF from the browser covers PDF needs — no lock-in.
Run-to-run comparison and judge drift tracking to see how scoring shifts as prompts or rubrics evolve over time.
Pending runs start in a shared team queue. Reviewers can claim; admins can assign or unassign. Assigned runs are visible only to the assignee and admins.
Per-script review policy: full, random sample, borderline-margin, or failures-only — snapshotted per run so mid-run edits don't change that batch.
One deployment, multiple teams within a single organization. Tool admins manage the org-level default judge; teams manage their own scripts and runs.
A workspace for iterating on prompts and judge rubrics before committing them to a versioned script.
A narrow, scoped HTTP API (/api/v1/integrations/{app_id}/…) for evaluate-only
runs where a caller supplies its own candidate response — for teams wiring Juri's judging
into an existing pipeline without adopting it as a full CI product.
Concrete scenarios Juri is built to support for QA and Dev teams working with LLM-backed features.
A team updates a customer-support system prompt. Before merging, they re-run the existing CSV of 200 support questions against both the old and new script version, and use the run-to-run comparison to confirm the new prompt doesn't quietly regress accuracy or tone.
Two judge configurations — one scoring strictly on factual accuracy, one on a broader helpfulness rubric — are attached to the same script. The comparison report highlights rows where they disagree, so the team can tune thresholds or retire the rubric that's miscalibrated.
Before moving a production prompt from one model to another, a script is duplicated with the new generation model and run against the same test CSV, letting the team compare pass rates and judge scores side by side prior to cutover.
Every reviewed row carries a mandatory comment and a persisted human decision. For teams in regulated environments, the finalized run plus HTML export becomes the evidence that a human confirmed AI-generated output before it was trusted.
A team already generates candidate responses in their own CI job and just needs consistent
scoring. They register an Integration App, post candidate_response values to
Juri's evaluate-only API, and get judged results back with an optional signed callback.
A 500-row run lands in the shared pending-review queue. Reviewers claim batches of rows, an admin reassigns a stuck batch to a teammate going on leave, and the run only finalizes once every required row has a human decision.
Illustrative CSV test-case input and the kind of run output Juri produces. Not real customer data.
support_qa_testset.csv| Test ID | Category | Judge: Accuracy | Judge: Tone | Verdict | Review status |
|---|---|---|---|---|---|
| TC-001 | billing | 9 / 10 | 9 / 10 | Pass | Reviewed — accepted |
| TC-002 | billing | 6 / 10 | 8 / 10 | Borderline | Reviewed — overridden to Fail |
| TC-003 | technical | 8 / 10 | 7 / 10 | Pass | Reviewed — accepted |
| TC-004 | account | 4 / 10 | 6 / 10 | Fail | Reviewed — accepted |
| TC-005 | technical | 9 / 10 | 9 / 10 | Pass | Pending review |
Set per script, snapshotted per run, so a policy change mid-run doesn't affect a batch already in flight.
| Mode | Behavior |
|---|---|
| full | Every row requires human review before the run can finalize. |
| sample | A random percentage of rows require review; failed and rate-limited rows are always included. |
| borderline | Only rows scoring within a configurable margin of the judge's pass threshold require review. |
| failures_only | Only rows a judge marked as fail (plus failed/rate-limited rows) require review. |
A pragmatic, single-organization internal tool — not built for multi-tenant SaaS scale, and deliberately so.
Juri's build has progressed through three broad phases.
Default judge, scripts, CSV runs, mandatory review, multi-judge comparison report, CSV/JSON export.
Dashboards, run-to-run compare, judge drift, sampling review modes, user admin, review assignment (claim/assign), judge config history with version diffs and pass/override rates.
Teams, advanced judge types, optional Celery workers for concurrent processing (docker compose --profile scale up).
Narrow, scoped HTTP API for evaluate-only runs where a caller supplies its own candidate response — deliberately not a full CI/CD product.
Local smoke test via Docker Compose.
For a full blank-machine runbook (Docker install, git clone, MySQL, host Claude OAuth mounts),
see docs/INSTALL.md in the repository. Without an ANTHROPIC_API_KEY,
generation and judging fall back to the Claude Agent SDK's ambient claude login
OAuth session.