Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Do not treat a model's stated reasoning as an audit trail
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
hardempirical finding - Calibrate LLM confidence on labelled data before using it as a threshold
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
hardempirical finding - Compare chunk sizes at the same context token budget, not the same k
You are running a chunk-size experiment, for example 200 versus 800 tokens, and comparing recall@5 for each.
hardpractitioner heuristic - Compare RAG variants with paired per-question differences on small golden sets
Variant B scores 0.78 and variant A 0.75 on an 80-question golden set, and someone proposes shipping B.
hardpractitioner heuristic - Run a new write-capable agent in shadow mode and compare its proposals with human actions first
A team wants to give a drafting agent permission to apply refunds or rebookings after good offline test results.
hardpractitioner heuristic - Assert required steps and forbidden actions in a trajectory instead of matching an exact path
An agent that issues refunds reaches the right final state, but sometimes skips the identity check the policy requires.
hardpractitioner heuristic - Debias a judge-scored pass rate with a small human-labelled sample instead of reporting it raw
A judge scores thousands of production conversations cheaply, but you also hold human labels for a small subset.
harddocumented fact - Rank many variants with a Bradley-Terry fit on sampled pairs rather than all-pairs win counts
You compare eight or more prompt or model variants by pairwise judging and the all-pairs table has dozens of cells with few items each.
harddocumented fact - Cluster the standard errors when eval questions share a document, ticket or customer
An eval has 300 questions written from 30 documents, ten per document, and you compute a score interval as if the 300 were independent.
hardempirical finding - Gate on a non-inferiority margin for the paired difference, not on a raw score threshold
A CI gate fails the build when the candidate scores below the baseline by any amount on a 400-item set.
hardpractitioner heuristic - Measure guardrail false positives on benign traffic, because rare attacks make most blocks wrong
A classifier blocks prompts it believes are attacks or policy violations, and its quality is reported as recall on a balanced test set.
hardpractitioner heuristic