LLM Evaluation & Regression Harness

LLM Evaluation & Regression Harness

A statistically honest A/B gate for LLM systems, and a report that measures how often the usual approach gets the answer wrong.

The library is small: a paired bootstrap, a permutation test, a power calculation, judge-agreement diagnostics, a content-addressed dataset, and a compare() that turns all of it into one of five verdicts. The interesting part is docs/results.md, which is generated by running the library and which contains 17 written-in-advance predictions, 3 of which the measurements contradicted.

python run_eval.py --out docs/results.md     # the full report (~5 minutes)
python run_eval.py --quick --section 7       # one section, fast, to stdout
python -m pytest tests -q                    # 299 tests
./test.ps1                                   # six-stage verification

The problem

Teams shipping LLM features run an eval set before and after a prompt change, compare two averages, and ship if the number went up. Every part of that is defensible in isolation and the combination is unsound:

• The eval set is too small to see the effect being claimed. Detecting a +0.01 quality improvement at 80% power needs 2,309 items on the paired variance measured here. Teams use 50 and report three-decimal deltas.
• A "no significant difference" result is reported as "no regression." These are different claims and the second one does not follow.
• The judge is validated with a statistic that cannot detect the failure it actually has. Cohen's kappa collapses under skew; Gwet's AC1 was built not to, which means it also does not collapse under degeneracy. A judge that answers "pass" to everything scores AC1 above 0.94 on 95%-pass data.
• The winner is chosen from many candidates and its measured gain is reported as its true gain. The selection is the bias.

docs/results.md puts a number on each of these.

Selected findings

The gate

result = compare(baseline_run, candidate_run, dataset)
if result.blocked:
    raise SystemExit(result.summary())

Five verdicts, and the fourth is the one that does not usually exist:

Calibration on 200 trials per scenario at n=120: a genuinely harmful change (-0.06) is caught 97% of the time and never called an improvement; a genuine +0.06 is found 92% of the time; a no-op change is called an improvement 1.5% of the time and UNDERPOWERED 92% of the time.

Why the systems are simulated, and why that is the stronger choice

No LLM is called anywhere in this repository. That is not a limitation worked around; it is what makes the report mean anything.

Every claim here is of the form "the standard method reports X when the truth is Y." That sentence requires knowing Y. With a real model behind an API there is no ground truth to compare against -- you would get a table of measurements and no way to say which were wrong, which is precisely the epistemic position this repository is arguing is untenable. §9 states that a judge reported +0.0711 for a true effect of +0.0021. That claim is only available because the +0.0021 was set, not estimated.

The simulation is also the only way to answer how often. "A 50-item eval set can mislead you" is a truism. "It reports a real +0.04 improvement as a decline in 12% of runs" required 10,000 evaluations under a known effect. At real API prices and latencies that experiment does not get run, which is a large part of why these numbers are not already common knowledge.

The design decision that carries the weight is the shared/private variance decomposition in SimulatedModel. An item's quality splits into an effect belonging to the item -- this question is hard, and it is hard for every model -- and an effect belonging to the model on that item. Only the second differs between two systems being compared. That ratio is the intraclass correlation, and it is the entire reason paired analysis works. A simulation with independent models would make paired statistics look worthless; one with perfectly correlated models would make them look magical. Both would be useless. Here it is a parameter, and §1 varies it deliberately to show that eval set adequacy is a property of how similar your two systems are, not a property of the number 50.

The statistics are not simulated. The bootstrap, the BCa correction, the permutation test, the Student-t quantiles and the exact sign test are the real implementations, written from primitives (no scipy), validated against published tables, and exercised by 299 tests.

What this does not establish is anything about a specific real model's quality. See docs/known-limitations.md, which is deliberately blunt.

What is in here

evalharness/
  stats.py       bootstrap, BCa, permutation, BH, power, MDE, required-n,
                 Student-t via a continued-fraction incomplete beta,
                 exact sign test
  agreement.py   kappa, AC1, prevalence/bias indices, the baseline-margin
                 check that neither statistic performs
  dataset.py     content-addressed eval sets; order-independent fingerprints
  systems.py     simulated models and judges with known ground truth
  compare.py     the gate
  corpus.py      dataset generation
  report.py      the expect/found DSL that makes predictions falsifiable
run_eval.py      14 experiments -> docs/results.md

report.py enforces the discipline the report depends on: a prediction must be written before the measurement is taken, exactly one finding may answer it, and render() refuses to produce a document with an open prediction. It is not possible to quietly drop a prediction that turned out wrong.

Documentation

• docs/results.md -- the report. Start at §1 and §7.
• docs/known-limitations.md -- what this does not establish.
• docs/adr/ -- five decisions and what they cost.
• docs/portfolio/ -- four essays, including 04-bugs-the-experiment-found.md, which catalogues seven defects the measurements found in code that already had passing tests -- including one in the mutation suite that verifies the tests.

Requirements

Python 3.12+, numpy. No scipy: every statistical function is implemented from primitives, which was originally a constraint and became the most useful part of the exercise.