Sample report
Sample LLM evaluation report: a demo agent and a public benchmark
Real runs from our own demo stack: judges checked on held-out cases, agent tasks run several times, a CI gate with its false alarms and misses, and a scheduled run. Two test sets, always reported apart.
https://veriprajna.com/ai-services/llm-evaluation-services/sample-eval-report
Results page: two test sets, reported apart
Test set A is our own demo support agent; test set B is a public agent benchmark. Every figure carries its n and split, and the two are never pooled.
- Results on a test set, not a measured rate. Every figure describes this demo only; none is a benchmark of any model or agent.
- Every regression and the endpoint change in this report were planted by us. The shop, its customers and its orders are invented; no client system or data appears here.
Golden set: cases with expected end states, written before any run
Test set A's cases were written from the shop's policy before the agent ran. In a build, cases come from your real traffic where it exists, plus known failures and cases written with your experts, kept apart in reporting.
Rubric and labels, fixed before the first judge prompt
Test set A has one label per case: 48 replies, dev 17 pass and 7 fail, held out 17 pass and 7 fail, each with a written reason. Half of the labelled transcripts come from the pinned agent and half from a reply-writing defect planted by us in the app code, so that there were failures to label. Test set B's labels are the benchmark's own code.
Graders: code checks first, a judge only where code cannot decide
Each agreement figure comes from a held-out split never used for judge tuning, and the first-pass judge is kept as the before.
- Test set A: the first-pass prompt agreed on 20 of 24 dev cases; one revision brought the rubric's definition of actions into the prompt, the dev cases reached 24 of 24, and the prompt was frozen. Each version then ran once on the held-out cases.
- Test set B: the judge was tuned on the 44 train tasks only and chosen by a rule fixed in advance. Only the first-pass and the chosen versions ran on the test tasks, once each.
- Failures caught in B counts a task only when every failing trial of it was flagged.
Agent tasks with checkable end states, run k times
A pass^k score counts a task only when all k trials succeed; one published definition is that it "measures the probability that all k trials succeed" (Anthropic, "Demystifying evals for AI agents", 9 Jan 2026). The agents ran at the model's default sampling, so trials vary from run to run.
Baseline and CI gate: a threshold with a floor, false alarms counted
The threshold is the baseline minus a margin set from the spread of unchanged runs, with a pre-set minimum. False alarms are counted on further unchanged runs that were not used to set it, and regressions of several sizes, planted by us, show what it catches and what it misses.
Scheduled run: the same suite, re-run when nothing has shipped
It catches sustained score drops on what the golden set covers, not intermittent faults.
Source: Our recorded runs of 9 November 2026. Every run behind these figures replays from its recorded model replies with no network, and tests re-derive every figure; the results files stay with our private demo stack.
- Golden set
Test set A, our demo agent. A small tool-using support agent for an invented shop (20 customers, 53 orders; fake data only). Each trial starts from a fresh copy of the shop's local SQLite store, and nothing touches a live system. 48 cases in 11 categories, each with the exact end state the store should be left in. Split 24 dev and 24 held out, stratified by category, fixed before any judge prompt existed.
Test set B, a public benchmark. An agent run in the τ-bench v1.0.1 retail environment (repository sierra-research/tau2-bench, MIT licence), pinned and unmodified. Judge-free tasks only, scored by the benchmark's own database check: 44 train tasks for judge tuning and 28 test tasks for reporting, from the benchmark's own split.
- Rubric
Test set A. Written before any grading. An episode passes only when the store's end state matches the expected one, the agent completed, and the judge passes the reply; the judge runs only when both code checks pass. One clarification was recorded before any label was written: a reply that calls a write that happened "unconfirmed" fails, "actions" means writes, and a statement about a lookup counts only where it misstates a write.
Test set B. The judge reads the benchmark's retail policy as its rubric, plus the full transcript. It never sees the simulated customer's scenario, the reference actions, the database state or the reward.
- Graders
Held-out split Before: first-pass judge After: final judge A: judge vs our labels, 24 held-out cases Agreed 19 of 24; failures caught 6 of 7; pass-labelled agreed 13 of 17 Agreed 24 of 24; failures caught 7 of 7; pass-labelled agreed 17 of 17 B: judge vs the benchmark's end-state check, 28 test tasks Agreed 20 of 28 tasks; failures caught 2 of 8 Agreed 23 of 28 tasks; failures caught 3 of 8 Counted per case in A (one transcript per case) and per task in B, where a task agrees only when the judge matches the check on all 4 trials. B's trajectories, shown apart because they are not independent: before, 100 of 112 agree (9 false passes, 3 false fails); after, 104 of 112 agree (8 false passes, 0 false fails).
- Agent trials
Test set B: all 4 trials succeeded on 20 of 28 tasks (pass^4, per task). No trial succeeded on tasks 27 and 100. 96 of 112 trajectories succeeded.
Test set A: all 3 trials passed on 48 of 48 tasks (pass^3, per task), and on 48 of 48 in every one of the 20 stability runs.
Trials of one task share the prompt, the model and the time of the run, so they are not independent.
- Baseline and CI gate
Threshold, test set A only. 20 unchanged runs of the pinned agent, each 48 cases run 3 times (n = 144 episodes): baseline 1.000, run-to-run spread 0.000. The margin is the larger of 3 times the spread and a pre-set minimum of 1/48 (0.020833), so with no spread the threshold is 0.979167. A run fails the build when 4 or more of its 144 episodes fail.
False alarms: 0 of 19 further unchanged runs, not used to set the threshold, fell below it.
Regressions planted by us, each run 3 times and compared with the threshold:
Change to the agent's prompt (planted by us) Score drop per run Caught R1: removed the rule against refunding an order already refunded 0.000, 0.000, 0.000 0 of 3 R2: loosened the address rule 0.056, 0.056, 0.049 3 of 3 R3: removed the 30-day refund window 0.083, 0.090 (second run excluded) 2 of 2 R4: removed the refund window and the identity rule 0.104, 0.104, 0.104 3 of 3 8 caught, 3 missed. Smallest drop caught: 4.9 percentage points (7 of 144 episodes) at n = 144.
- Scheduled run
The same suite re-run against the model endpoint, with nothing committed. The schedule was simulated by 5 repeated invocations of the same commit. From the third, a local proxy in front of the model endpoint removed the Policies section from the agent's instructions: a change planted by us.
Run Endpoint Score Result 1 Unchanged 1.000 Above threshold 2 Unchanged 1.000 Above threshold 3 Planted change 0.729 Below, not yet flagged 4 Planted change 0.729 Below, flagged 5 Planted change 0.722 Below, flagged The job flags on the second consecutive run below the threshold, so it flagged at invocation 4. It never flagged on the unchanged endpoint.
Misses, limits, and how this sample was built
Misses and limits
- 3 planted regressions were missed, and none of them lowered the score. With the rule against refunding an order already refunded removed (R1), every run still scored 1.000: the agent still refused. A gate cannot flag a change that does not move the score.
- Nothing is known about drops between 0 and the smallest one caught. No planted change produced a drop in that range, so this sample shows neither catches nor misses there.
- The threshold rests on its pre-set minimum margin, because the baseline never failed. Every stability and false-alarm run counted here scored 1.000, so the measured spread was zero and the gate is strict. The fresh clone's own 20 unchanged runs scored between 0.993 and 1.000, so the variation is small but not zero.
- No catch depended on the judge. Every caught run was below the threshold on its end-state check failures alone; the judge failed one further episode, in the third R3 run.
- Failures caught in test set B rests on 8 tasks. 2 against 3 is one task, and no repeat judge run was made, so that change may be run-to-run noise. Most of the judge's gain was fewer false fails (3 to 0); false passes moved from 9 to 8.
- Some failures are invisible in the transcript. In test set B the simulated customer sometimes asked for something its scenario did not, and some reference end states look debatable to us: task 100's check depends on the order of item ids inside one call, and in task 64 two product variants meet the scenario while the reference picks one. These still count against the judge in the figures; we report them and change no label. Here a transcript judge is no substitute for an end-state check.
- The gate sees only what the golden set covers. A failure no case exercises cannot lower the score.
Who wrote what, and the risks that follow
- Test set A's cases, expected end states and reply labels were written by Veriprajna's AI build agent, with a reason per case, not by a domain expert. No person has confirmed them.
- The same agent wrote the rubric and both judge prompts, so the judge was tuned toward the labeller's reading of the rubric. The held-out split guards against tuning on the test cases, not against that shared reading.
- The judge and the agent are from the same model family, and so is the labeller. Zheng et al. examined self-enhancement bias, where an LLM judge may favour answers it generated itself, and found their data could not settle whether it occurs (Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena"). We disclose the risk; it was not measured here.
- Test set B's reference labels are the benchmark's own database check; we wrote none. The notes on goal states that look wrong were written by the AI build agent.
Run configuration, per test set
Test set A. A local machine with no live system; the only network calls went to the model API. Model access: a commercial model API with an API key, under the provider's commercial terms, with no consumer subscription.
Agent: a small model at medium effort, up to 10 model turns per episode, no sampling settings sent. Judge: a larger model from the same family at low effort, returning a verdict and a reason.
Split seed 20261109. Every split, label rule, threshold rule and acceptance target was written to a file with a UTC timestamp before the run that tested it.
Test set B. τ-bench v1.0.1 at commit fc0055d, licence read at the tag (MIT), run in the benchmark's own environment and left unmodified. Retail domain, judge-free tasks only:
| Split | Kept | Excluded, and why | |
|---|---|---|---|
| Test | 28 | 12: 11 whose reward needs the benchmark's own LLM judge, and maintainer-flagged task 5 | |
| Train | 44 | 30: 29 whose reward needs that judge (one of them maintainer-flagged task 4), and maintainer-flagged task 7 |
Before any agreement figure was computed, every trajectory was re-scored with model calls forbidden, and all 112 test and 176 train trajectories matched their recorded reward.
The agent and the simulated customer ran on the same small model at its default sampling, 4 trials per task.
The user simulator's model was swapped: the benchmark's default user model, from another provider, was swapped for the small model the agent uses, from the same provider as the agent and the judge. The simulator's code, prompt and guidelines are the pinned ones. The judge is the larger model from the same family at its default effort.
Handover check: a fresh clone
The delivered repository was checked without our private harness: the CI job's steps, run in a fresh clone on a local machine.
- A clone into an empty folder, a new environment from the repository's requirements file, and only the model API key, PATH and HOME set.
- The workflow's gate, re-baseline and calibration steps, run as written. Every step exited 0.
- Our private build harness could not be imported, and no file in the clone refers to it.
- Baseline re-run: score 1.000, gate pass. Stability run: 20 runs, baseline 0.998958, spread 0.002544, its own threshold 0.978125. Judge calibration: 24 of 24 held-out cases agreed, 7 of 7 failures caught.
Not set in advance as a check: none of the clone's 20 unchanged runs fell below the original threshold.
Runs excluded, and why
| Run | Test set | Why it is excluded | |
|---|---|---|---|
| pilot-01, pilot-02 | A | Pilots that checked the agent loop and the judge path before the measured runs | |
| Label pools v1 and v2 | A | Two planted prompt defects that produced no failing replies; kept, labelled and unused | |
| fa-s18 | A | One episode hit the API's concurrent-request limit (HTTP 429) after the API library's retries; kept on file, not re-run | |
| R3_refund_window-r2 | A | The same error on one episode; kept on file, not re-run | |
| smoke-1 | B | One trial on one train task to prove the model route; outside every count | |
| First launch of the trials | B | Stopped with no finished trial, to fix how two processes wrote the record of model replies; replies already recorded were reused | |
| Task 100, trial 0, first attempt | B | Crashed when the simulated customer sent an empty message; run again under the benchmark's own retry rule. The crashed attempt is kept, and the second attempt scored 0, as did the task's other three trials |
No run was dropped or repeated to improve a number.
What this sample does not show
- A measured rate. No real traffic was sampled, so no figure here describes how often a live agent fails, and none carries an interval.
- Labels from people. In a build, your experts write the labels, and two of them label the same cases to test the rubric.
- An eval platform. The suite ran as plain test-runner tests in the fresh-clone check; no eval platform or framework was run.
- A hosted CI service. The CI steps ran on a local machine.
Know whether your next model, prompt or retrieval change scores worse
This sample is our demo agent and a public benchmark. A build measures yours, on your cases with your experts, starting with a short discovery call.
- Email or WhatsApp us
- Reply within 1 business day
- Free 30-minute discovery call
- Written scope and fixed fee within 2 business days of the call