r09 results - structured-output validation on recorded fixtures

Fixture mode: no live generations evaluated. Every output string scored here is a hand-authored template or a deterministic fault injection applied to a synthetic gold record (file fixtures/fixtures_v1.json, seed 20240901). These results describe the validator, not any language model. Numbers are copied from outputs/metrics.json written by cd python && uv run python -m reference.r09_structured_output_validation.run.

Item	Value
Fixture set	100 synthetic notices, 182 labelled output fixtures: valid 25, syntax 26, schema 47, semantic 51, injection-followed 18, injection-ignored 15
Split	dev 95 fixtures / test 87 fixtures, split by notice (dev 50 notices / test 50 notices)
Schema	domain.RECORD_SCHEMA v1.0.0, nine required fields, additionalProperties: false
Runtime	0.2 s for scoring all four strategies on all 182 fixtures; standard library only (jsonschema is not used or added)

Strategies: strict = parse and schema only (the baseline: anything that fails goes to a person); repair_only = plus the safe repairs; repair_plus_checks = plus source cross-checks, the several-flights rule and injection signals (the pipeline under test); aggressive = plus unsafe repairs (an ablation, never recommended).

Findings (all 182 fixtures unless stated)

1. Schema validation catches every schema violation and raises no false alarm on this set. Of 78 fixtures that are not strict-valid by construction (syntax and schema faults, plus 5 injection outputs that add a field or are not JSON), the stdlib validator rejected 78 (100%, Wilson 95% interval 95.3% to 100%); of the 104 that are strict-valid by construction it rejected none. The same holds on dev (38 of 38) and test (40 of 40). The stdlib validator also agrees with the jsonschema library on every fixture output and on 400 random mutations when that library is installed (differential test; it was installed in this environment as a dependency of another tool).
2. Only 57.1% of the outputs are valid on the first strict parse; safe repair lifts it to 73.1%, and every repair was correct. Strict-valid 104 of 182; schema-valid after safe repair 133 of 182. 29 fixtures were repaired with 32 logged operations (9 enum normalisations, 7 code-fence strips, 6 trailing-comma removals, 5 prose strips, 3 boolean- string and 2 integer-string coercions). Harmful repairs (a correct value altered): 0. Invented values filled in: 0. Records accepted after repair that differ from gold: 0. These are properties of the 14 value-level repairs and 18 formatting repairs on this set, not a guarantee for unseen faults.
3. Routing. Route accuracy against the expected route: strict 48.9% (89/182), repair_only 64.8%, repair_plus_checks 90.7% (165/182, Wilson 85.6% to 94.1%), aggressive 82.4%. Under the default pipeline 53 outputs (29.1%) go straight through, 29 (15.9%) are accepted after repair and 100 (54.9%) go to a person. Route confusion (expected by predicted): accept 38 / 0 / 2, accept-after-repair 0 / 29 / 0, human-review 15 / 0 / 98. Dev and test agree (89.5% and 91.9%).
4. The silent-wrong rate is the number that matters, and validation alone leaves most of it. A schema-valid but wrong record that is accepted is "silent wrong". Strict: 60 of 104 accepted records (57.7%). Safe repair adds 29 records that are all correct, so the count stays at 60 and the denominator grows: 60 of 133 (45.1%). Source cross-checks and the injection signals cut it to 15 of 82 accepted records (18.3%, Wilson 11.4% to 28.0%; 8.2% of all fixtures). All 15 come from the four semantic fault types the rules cannot see by design (date equal to the issue date, delay taken from another number in the notice, rebooking flipped when the phrase is not in the lexicon, a vague cause made specific; 4, 4, 4 and 3 fixtures). Dev has 9 and test has 6.
5. Cross-checks, not the schema, do the work against wrong values. 36 of the 51 semantic fixtures reach a person: 32 through a source check (flight, airport, date or delay absent from the notice, a type, cause or rebooking value that contradicts the wording, and so on) and the 4 two-flight notices through the several-flights rule. Per-field exact match among accepted records under the default pipeline: delay_minutes 95.1%, date 95.1%, rebooking_offered 95.1%, cause_category 96.3%, every other field 100%; under strict acceptance delay_minutes is 82.7%.
6. Injected instructions. The output-side detector found 18 of 18 followed-instruction outputs (recall 1.00, Wilson 82.4% to 100%) with 2 false alarms (precision 0.90, 18 of 20). The two false alarms are the fixtures in which the instruction asked for the value that was already true, so the output equals gold (labelled "ignored" by construction); a detector cannot tell that from following. The input-side detector flagged all 33 notices with an embedded instruction and none of the 62 others (the fixture set has no benign notice containing a link; a unit test checks that a link without an instruction is not flagged on the input side). On test alone: output side 8 of 8 recall, 8 of 9 precision. Of the 18 followed outputs, 13 are schema-valid: under the strict baseline those 13 would have reached automation. The default pipeline routes all 18 to a person; 8 are caught by a source check (rebooking, delay or cause contradicts the notice) and the injection signal, 5 by a schema/parse failure and the signal, and 5 (the e-mail and link requests that live in the free-text summary) by the injection signal alone.
7. Unsafe repairs are not an improvement. The aggressive strategy repairs 55 fixtures with 62 operations (39 field- level). It sends 15 more fixtures to accept-after-repair: 11 whose repaired record happens to equal gold (6 extra fields dropped, 3 unknown enum values mapped to "other" where gold was "other", 1 day-first date guess, 1 negative delay clipped to 0 where gold was 0) and 4 truncated outputs closed by the unsafe closer with the missing summary invented ("n/a"). Its silent-wrong count is 19 against 15, its route accuracy 82.4% against 90.7%, and 4 invented values were written. It did not alter a correct value on this set; its apparent successes depend on the fixture happening to agree with the default, which is the reason the default pipeline refuses them. The test test_a_tempting_repair_is_blocked... shows a truncated record closed to "delay_minutes": 9 (a cut-off 95), which only the cross-check then stops.
8. By template. The silent-wrong count per template (default pipeline) is 6 of 36 fixtures (template 1, the one with a "check-in closes N minutes" distractor), then 2 to 3 for the others, 0 for the four two-flight notices (all routed to a person by the several-flights rule).

What this does not prove

• It does not measure any model. No model produced these strings. Real model faults are more varied than the fault subtypes the author thought of, and the real rates of each are unknown.
• The 100% catch rate for schema violations is by construction and expected: the schema and the labels come from the same author, and the validator is deterministic. It says the validator implements the schema, not that the schema is the right contract.
• The silent-wrong rate (8.2% of fixtures, 18.3% of accepted) is a function of how many uncatchable semantic faults the author wrote (15 of 182). It is not an error rate for a model, and it should not be quoted as one.
• The injection detector's recall of 1.00 is circular: the same author wrote the seven instruction templates and the patterns. A paraphrased, encoded, translated or split instruction is not tested and would likely evade a pattern-based tripwire. It is not a security boundary: the boundary is that an extracted record cannot trigger a passenger message or refund without a deterministic policy check and, for high-impact actions, human approval.
• Dev and test are split by notice, but rules and fixtures have one author and no rule was changed after scoring. The test split therefore says nothing about generalisation to fault types the author did not anticipate; the record's ten fixtures written by someone else were not available.
• Notices follow five English templates (plus a two-flight form). Free-form, multilingual, spelled-out numbers ("ninety- five minutes"), look-alike characters beyond three fixtures and very long notices were not evaluated.
• Wilson intervals treat fixtures as independent; fixtures share notices (about 2 per notice), so the intervals are somewhat optimistic.
• The routing policy for an ignored injection (accept, log the flag) is a judgment; sending every flagged notice to a person would add the 13 accepted ignored-instruction fixtures (7% of all fixtures) to the human-review load.

Reproduce

cd python && uv run python -m reference.r09_structured_output_validation.run
cd python && uv run pytest tests/test_r09_structured_output_validation.py