project-027 · MLOps and system design · cross-industry
LLM question-answering service: versioned golden set, regression gate and request observability
An internal assistant answers questions from a supplied passage and must say "not answerable" when the passage does not contain the answer. Every change of prompt, model or parameters ships weekly. Which release gate and which production telemetry stop a change that quietly makes answers worse, abstains less, breaks the output schema, or raises cost and latency?
At a glance
Data
SQuAD 2.0 dev set (dev-v2.0.json) (public) · Dev file 4 MB; the golden set is a stratified sample of 400 questions (half unanswerable) plus 50 tagged hard cases, all versioned by content hash.
Details on dataBaseline
The current production configuration (prompt v1, model and parameters recorded) scored on the frozen golden set;
Details on baselineValidation
Paired comparison of candidate vs baseline configuration on the same items: paired bootstrap confidence intervals for the F1 difference and the abstention-recall…
Details on validationMetric
EM and F1 (answerable), abstention precision and recall (unanswerable), schema validity rate, injection-test pass rate, latency p50/p95, tokens per request, retry rate.
Details on metric
Success criterion
A versioned golden set, a gate that runs in CI on every change, and a report that shows the gate (1) passes the current configuration, (2) blocks at least three deliberately degraded configurations (a prompt that removes the abstention instruction, a model with a smaller context budget, a temperature increase), and (3) logs every request with the fields needed to reproduce it. Results are stated separately for real-model mode and fixture mode.
Context
SQuAD 2.0 pairs Wikipedia passages with questions, a large share of which are unanswerable from the passage, which makes it a compact test of grounded answering and abstention. Because the dev set is public and widely used, it may be present in model pretraining data; it is therefore suitable for detecting regressions between configurations, not for claiming absolute capability. If no model is configured, the project runs in fixture mode on recorded or synthetic outputs, labelled as such, and makes no claims about model quality.
Data
- Dataset
- SQuAD 2.0 dev set (dev-v2.0.json) (public)
- Licence
- CC BY-SA 4.0 (as stated on the dataset site)
- Size
- Dev file 4 MB; the golden set is a stratified sample of 400 questions (half unanswerable) plus 50 tagged hard cases, all versioned by content hash.
- Verification
- identity verified
Access
Direct download from the SQuAD site (dev-v2.0.json, 4 MB), no login; Hugging Face mirror rajpurkar/squad_v2. Cache and checksum.
Grain
One row per question: passage id, question, gold answers (possibly several spans) or the unanswerable flag.
| Column | Meaning |
|---|---|
| context | Passage the answer must come from |
| question | User question |
| answers (text, answer_start) | Gold spans; empty for unanswerable questions |
| is_impossible | True when the passage does not contain the answer |
| golden_set_version | Platform field: content hash of the sampled set and its sampling seed |
EDA and data-quality rules
Stratify the golden set by answerable vs unanswerable and by passage, so no single article dominates.
Keep all gold answer variants; scoring against one variant understates correct answers.
Freeze the set by hash; any edit creates a new version and the gate compares like with like.
Add 20 synthetic passages containing embedded instructions (labelled synthetic) to test that context is treated as data.
Baseline
The current production configuration (prompt v1, model and parameters recorded) scored on the frozen golden set; in fixture mode, a recorded output file for that configuration.
Features
Logged per request: request id, prompt template version, model id, parameters, passage hash,
input/output tokens, latency, retries, schema validation result, abstained flag, answer, and the gate run id.
Output contract: JSON with answer (string or null) and evidence_span (substring of the passage or null),
validated against a JSON schema.
Candidate methods
Deterministic gate: SQuAD-style exact match/F1 on answerable items, abstention precision/recall, schema validity, latency and cost budgets
Cheap, reproducible, no judge model; misses paraphrased correct answers, which lowers absolute scores but affects compared configurations similarly.
Deterministic gate plus an LLM-as-judge for answer equivalence on items where F1 is below 1
Catches paraphrases; the judge itself must be validated on 100 human-labelled items and pinned, or the gate drifts with the judge.
Validation design
Paired comparison of candidate vs baseline configuration on the same items: paired bootstrap confidence intervals for the F1 difference and the abstention-recall difference. Gate rule (team choice, stated): block if F1 drops by more than 2 points with the 95% interval excluding zero, abstention recall drops by more than 3 points, schema validity falls below 99.5%, or p95 latency or cost per 1,000 requests exceeds budget by more than 10%. Run each configuration twice to measure run-to-run variation before trusting small differences.
Leakage checks
Contamination: the public dev set may be in pretraining data; claims are relative, and a small private set of freshly written questions (labelled synthetic) checks that relative conclusions hold. The judge never sees gold labels in its prompt unless the protocol requires it, and then this is documented.
Bias and representativeness
Slice by article topic and question type (who/when/how many); a configuration can improve the average while failing numeric questions.
Robustness checks
Embedded-instruction passages must not change behaviour; long passages near the context budget; malformed outputs trigger one bounded retry, then a schema-failure response, never an unvalidated answer.
Technical metrics
EM and F1 (answerable), abstention precision and recall (unanswerable), schema validity rate, injection-test pass rate, latency p50/p95, tokens per request, retry rate.
Business interpretation
Cost per 1,000 requests computed from measured tokens and the provider's list price on a recorded date (or a labelled hypothetical price in fixture mode), and the share of questions answered without escalation to a human.
Error slices
Answerable vs unanswerable.
Question type and article topic.
Passage length bucket.
Embedded-instruction fixtures.
Implementation steps
Download and sample the golden set with a fixed seed; store the hash and sampling script.
Write the output JSON schema and validator; implement EM/F1 with all gold variants.
Implement request logging (structured, one record per call) and the gate script with paired bootstrap.
Run the baseline configuration twice; measure run-to-run noise.
Create three degraded configurations and confirm the gate blocks each one with a readable reason.
Optionally add the judge, validate it on 100 human labels, and pin its version.
Wire the gate into CI and write the release checklist and rollback steps.
Reproduction
Gate script runnable in fixture mode with no network or keys (recorded outputs); real-model mode requires an API key or a local model configured by the learner, and its report states model id, date and prices used.
Limitations
Public benchmark items may be memorised; absolute scores are not capability evidence.
Span-matching metrics under-credit valid paraphrases; the judge adds its own bias.
Fixture mode validates the gate logic, not any model's quality.
Production and monitoring
Every response passes schema validation before it is returned; failed validation returns an explicit fallback. Traces sampled for human review weekly; dashboards for abstention rate, schema failures, latency, tokens and spend against budget; alerts on sudden abstention changes (a common symptom of prompt or model changes). Releases are versioned bundles (prompt, model id, parameters, schema) with one-step rollback.
Interview questions
Your new prompt raises F1 by one point but abstains on fewer unanswerable questions. Do you ship?
How do you tell a real regression from run-to-run noise in LLM outputs?
Why is a public QA benchmark weak evidence of model capability, and what would you add?
Postmortem and alternatives
Typical findings: abstention moved more than F1 across changes; schema failures clustered on long passages; run-to-run noise was large enough to make 1-point differences meaningless. Alternative: a smaller domain golden set written by subject experts, with SQuAD kept only as a smoke test.
Rubric4 criteria
| Criterion | Meets | Exceeds |
|---|---|---|
| Golden set governance | Versioned by hash, stratified, all gold variants kept. | Private synthetic set confirms the relative conclusions. |
| Gate design | Paired comparison with stated thresholds; blocks three degraded configurations. | Run-to-run noise measured and used to set thresholds. |
| Observability | Every request logged with the fields needed to reproduce it. | Dashboard and alert definitions tied to the gate metrics. |
| Evidence honesty | Fixture mode and real-model mode reported separately. | Report states model id, date and prices for every real-model number. |
References6
- Know What You Don't Know: Unanswerable Questions for SQuAD
SQuAD 2.0 design and why abstention on unanswerable questions is tested.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Biases and agreement of LLM judges; why the judge must be validated.
- NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
Benchmark contamination risks in LLM evaluation.
- Structured outputs (Claude Platform Docs)
Schema-constrained outputs and validation.
- OWASP Top 10 for LLM Applications (2025 risk list)
Prompt injection risks motivating the embedded-instruction fixtures.
- AI Engineering
Evaluation pipelines and LLM application monitoring.