r19 results - golden-set regression gate on synthetic recorded outputs

Synthetic recorded outputs: no model was evaluated. Every output string was drawn from an authored behaviour profile (fixtures.py, seed 20260043) and stored in fixtures/recorded_outputs_v1.json.gz. The regressions are seeded by construction, so these results describe the gate and its statistics, not any prompt or model. Numbers are copied from outputs/metrics.json and the CSV files written by cd python && uv run python -m reference.r19_golden_set_regression_gate.run.

Item	Value
Date, seed	2026-10-12, 20260043 (fixtures); bootstrap seed 11; null-test seed 2026
Golden set	200 cases: 120 MFT, 52 INV, 28 DIR; 103 critical; 44 hidden (18 hidden critical)
Versions	1 baseline (16 reference runs + 60-run pool of the unchanged prompt), 8 candidates x 8 runs
Gate	3 gate runs, 5 confirmation runs only for suspect critical failures, margin 3 points, 2,000 bootstrap resamples
Environment	Python 3.12.13, numpy 2.5.3, standard library otherwise
Runtime	25 s full run (about 2 s --smoke); tests 3 s

Baseline (v1) on the 16 reference runs: aggregate case pass rate 0.864 (run-to-run range 0.830 to 0.900 on 200 cases). Property pass rates: claim type 0.931, urgency 0.977, documents 0.971, escalate 0.983, no forbidden phrase 1.000. 21 cases are unstable (pass rate between 0.2 and 0.8 over 16 runs); none fails every run.

Gate decisions on the recorded candidates (3 gate runs)

Candidate	Aggregate base -> cand	Diff, one-sided 95% bounds	Non-inferiority	Critical: suspects / confirmed	Slice alarms	Decision	Naive "single run went up" (128 run pairs)
v2-reworded	0.864 -> 0.865	+0.001 (-0.021, +0.023)	non-inferior	2 / 0	none	SHIP	ships 47%
v2-improved	0.864 -> 0.913	+0.049 (+0.031, +0.068)	non-inferior	1 / 0	none	SHIP	ships 89%
v2-fewshot	0.864 -> 0.918	+0.054 (+0.027, +0.080)	non-inferior	15 / 11	none	BLOCK: critical	ships 66%
v2-reassure	0.864 -> 0.818	-0.046 (-0.072, -0.021)	inconclusive	23 / 10	critical=yes	BLOCK: critical	ships 1%
v2-tuned	0.864 -> 0.917	+0.052 (+0.033, +0.072)	non-inferior	0 / 0	none	SHIP (visible-only flag)	ships 99%
v2-trim	0.864 -> 0.722	-0.143 (-0.176, -0.110)	inferior	3 / 0	3 slices	BLOCK: non-inferiority	ships 0%
v2-borderline	0.864 -> 0.827	-0.038 (-0.060, -0.017)	inconclusive	1 / 0	none	HOLD: inconclusive	ships 5%
v1-broken-escalation	0.864 -> 0.482	-0.383 (-0.437, -0.327)	inferior	82 / 82	12 slices	BLOCK: critical	ships 0%

Mean candidate-minus-baseline over all 8 candidate runs (naive column basis): reworded +0.001, improved +0.034, fewshot +0.021, reassure -0.043, tuned +0.048, trim -0.141, borderline -0.035, broken -0.371.

Findings

1. The "average went up" gate ships a critical-slice regression, and the gate does not. v2-fewshot raises the aggregate (mean +2.1 points over its 8 runs; +5.4 on the 3 gate runs; the naive rule passes in 66% of the 128 single-run comparisons) while the escalate property falls from 0.983 to 0.938 overall and, in the seeded slice (injury mentioned in a property claim, 31 cases), the all-property pass rate falls from 0.915 to 0.753 on the gate runs. The gate blocks it on all 56 choices of 3 gate runs out of 8: 11 critical cases are confirmed failing (all in the seeded slice; 9 of the 11 are DIR cases, the directional tests written to detect exactly this).
2. Slice statistics alone did not catch it. The majority-pass McNemar test on the slice had 2 pass-to-fail and 1 fail-to-pass cases (p = 1.0): a case that fails 30% of the time still passes by majority in most runs, so the slice test is the wrong instrument for "sometimes fails". The per-case critical rule with confirmation reruns is what blocked it. The slice test did fire for broken, reassure and trim, where failures are frequent.
3. Flake handling decides whether the gate is usable. On the unchanged prompt (300 null replicates):
   • "any critical failure in 3 runs blocks": 300/300 false blocks (100%, Wilson 98.7% to 100%). The five flaky critical cases alone make this nearly certain.
   • quarantine of baseline-flaky critical cases but no confirmation reruns: 202/300 false blocks (67%), from single one-off failures among about 90 stable critical cases.
   • quarantine plus confirmation reruns: 0/300 false blocks (Wilson 0% to 1.3%).
4. The cost of flake handling is sensitivity to small critical regressions. Seeded escalation-miss probability on the 31-case slice (100 fresh simulated replicates per level): confirmed-block rate 0% at 0 (null), 9% at 0.05, 47% at 0.10, 97% at 0.20, 100% at 0.40 and 0.80. A regression that makes the slice fail one run in ten is blocked less than half the time by this rule; the rule trades that against a false-block rate of zero.
5. Aggregate non-inferiority is underpowered at 200 cases and 3 runs. The unchanged prompt ships in 239/300 null replicates (79.7%, Wilson 74.8% to 83.8%); the other 61 are HOLD (inconclusive), none are blocked. A uniform degradation (all of claim-type, urgency and document error rates doubled, mean true diff -5.0 points) is never shipped (0/100): 61 HOLD, 33 BLOCK non-inferiority, 6 BLOCK slice; at x3 (-10.0 points) every replicate is blocked. Same arithmetic from the other side: the gate cannot distinguish "no change" from "-3 points" with this much data.
6. v2-reworded ships on 41 of 56 gate-run triples and holds on 15. Even a true no-op flips between SHIP and HOLD depending on which three runs are used; decisions for all other candidates were identical on all 56 triples.
7. A tuned-on-visible prompt looks good everywhere except where it should not. v2-tuned gains +6.9 points on the 156 visible cases (interval +4.5 to +9.5) and -0.5 on the 44 hidden cases (interval -6.0 to +4.5). The gate ships it (non-inferiority holds) with the visible-only flag. The flag is weak: v2-improved (a real improvement) also trips it by chance (visible +5.4, hidden +3.3, hidden interval -2.5 to +9.9).
8. Seeded regressions: v2-reassure does not trap the naive rule. Its coverage promises also reduce the aggregate (-4.3 points over 8 runs), so the naive rule blocks it (ships 1 of 128 comparisons); only the critical rule says why.

Success criteria (design targets from the project record)

Criterion	Result
(a) Blocks a deliberately broken version on every run	Met on this fixture: v1-broken-escalation blocked on 56/56 gate-run triples; v2-fewshot and v2-reassure also 56/56. By construction (90% miss rate on 91 cases).
(b) Passes the unchanged prompt on at least 95% of repeated runs	Not met: 79.7% (239/300). No false block, all misses are non-inferiority HOLDs. Post-hoc alternatives on the same pool (not the pre-set rule): 5 gate runs 90.7% (272/300); 7 gate runs 95.0% (285/300, Wilson 91.9% to 97.0%); 3 runs with a 4-point margin 95.0%.
(c) Written release rule	Met: README.md "Release rule" and the constants in gate.py.

What this does not prove

• It does not measure any prompt or model. No model produced these outputs; the gate's catch rates are properties of a world in which the author decided where the regressions are. A real regression may be diffuse, correlated across cases or outside the golden set, none of which was tested.
• Seeded-regression sizes were chosen by the author and adjusted once after a first look (the v2-fewshot gain and loss). The first null run used an 8-run baseline and showed false critical blocks from three flaky critical cases that happened never to fail in the baseline; the reference was raised to 16 runs. Both are design choices made after seeing output.
• The null replicates share a 60-run pool and overlap; Wilson intervals treat them as independent, so they are somewhat optimistic. The power study uses 100 replicates per level.
• Per-run outputs are independent draws; real sampling noise is correlated within a prompt version and across similar cases, which would make both flakiness and the intervals worse.
• The 44-case hidden set gives wide intervals; the visible-vs-hidden flag cannot establish overfitting.
• Properties are surface checks (enum equality, document sets, regex). Tone, clarity and paraphrased coverage promises are not tested. Moving escalation into deterministic keyword-triggered code (the record's postmortem question) was not simulated.
• The margin (3 points), the confirmation rule (2 of 5) and the slice thresholds were fixed before the run and not tuned to hit criterion (b); the alternatives listed are labelled post-hoc.

Reproduce

cd python && uv run python -m reference.r19_golden_set_regression_gate.run
cd python && uv run pytest tests/test_r19_golden_set_regression_gate.py