S20 · experiment simulation

Golden set regression for a refund-email extractor

People, data and outputs in this scenario are simulated. Each stage is recorded when you commit; the reveal comes after.

4stages
40 minabout
0stages committed

The brief

The synthetic airline's customer-care team receives refund emails. An LLM step outputs JSON: booking_ref (6 characters), reason (one of: schedule_change, medical, bereavement, voluntary, duplicate_charge, other), requested_action (refund, voucher, rebook, unclear) and a 40-word summary. Agents act on these fields.

Planned changes: prompt v3 to v4 (new instructions and examples), and a cheaper model (model S) instead of the current model L. Model names, emails and all results are simulated.

Stage 1 of 4

You have 40,000 historical emails with agent-corrected fields and a budget to label 300 for the golden set.

Which design choices belong in the golden set? Select all that apply.

Stratify by reason and language so rare, costly reasons (medical, bereavement) have enough cases.Include adversarial cases, such as emails containing embedded instructions to approve a refund.Use the emails that were used to write the prompt's few-shot examples, because their labels are verified.Version the set, its labelling guide and the acceptance rule per field, and freeze it for a release cycle.Sample only the last two weeks of email, so the set is current and reflects live traffic exactly.