Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Do not treat a model's stated reasoning as an audit trail
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
hardempirical finding - Consider continued pretraining only when you hold a large unlabelled domain corpus
An insurer's claims text is full of domain jargon, and a general model handles it poorly even after prompt work.
hardempirical finding - Expect LoRA to learn less and forget less than full fine-tuning
A team uses LoRA to teach a model a large new skill domain, such as a proprietary programming language, and gains are small.
hardempirical finding - Calibrate LLM confidence on labelled data before using it as a threshold
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
hardempirical finding - Locate RLHF failures in feedback, reward model or policy before changing the recipe
After preference tuning, an internal assistant becomes evasive on some topics and overconfident on others, and the team considers switching alignment methods.
hardpractitioner heuristic - Track a gold metric while optimising against a reward model, and stop before it turns down
During RL fine-tuning, the reward-model score keeps rising run after run, and the team keeps training.
hardempirical finding - Choose DPO for simplicity, but check for distribution shift in the preference data
A team with a fixed preference dataset picks DPO because it avoids training a reward model and running reinforcement learning.
hardempirical finding - Compare chunk sizes at the same context token budget, not the same k
You are running a chunk-size experiment, for example 200 versus 800 tokens, and comparing recall@5 for each.
hardpractitioner heuristic - Compare RAG variants with paired per-question differences on small golden sets
Variant B scores 0.78 and variant A 0.75 on an 80-question golden set, and someone proposes shipping B.
hardpractitioner heuristic - Decompose multi-hop questions instead of hoping one retrieval finds everything
Questions combine facts from different documents, such as "which of our A321 routes are affected by the new crew-rest rule?"
hardpractitioner heuristic - Treat writes to long-term agent memory as untrusted input with provenance
An assistant saves "facts" or preferences from conversations, documents or web pages into persistent memory reused in later sessions.
hardpractitioner heuristic - Close exfiltration channels such as auto-loaded links and images in model output
The chat interface renders model output as Markdown, including images and links to arbitrary domains, and the model can see private data.
hardpractitioner heuristic - Keep the model that reads untrusted content away from tools and private data
One agent reads incoming customer emails, can query customer records, and can send messages externally.
harddocumented fact - Fix the plan before reading untrusted data when the task allows it
An agent must process external content, for example summarise today's supplier emails into a report, with a predictable set of actions.
harddocumented fact - Have the model quote the relevant evidence before it answers from a long input
Retrieval is perfect in a 40,000-token prompt, yet multi-step answers over the supplied evidence are worse than on a trimmed version of the same prompt.
hardempirical finding - Re-check preconditions at write time with a version token, not only at read time
An agent reads seat availability or a fare early in a long run and makes the booking change many steps later.
harddocumented fact - Record the undo step for every write and place irreversible steps last
An agent must perform a sequence of writes, such as hold a seat, charge a fee and send a confirmation, and a later step may fail.
harddocumented fact - Run a new write-capable agent in shadow mode and compare its proposals with human actions first
A team wants to give a drafting agent permission to apply refunds or rebookings after good offline test results.
hardpractitioner heuristic - Assert required steps and forbidden actions in a trajectory instead of matching an exact path
An agent that issues refunds reaches the right final state, but sometimes skips the identity check the policy requires.
hardpractitioner heuristic - Debias a judge-scored pass rate with a small human-labelled sample instead of reporting it raw
A judge scores thousands of production conversations cheaply, but you also hold human labels for a small subset.
harddocumented fact - Rank many variants with a Bradley-Terry fit on sampled pairs rather than all-pairs win counts
You compare eight or more prompt or model variants by pairwise judging and the all-pairs table has dozens of cells with few items each.
harddocumented fact - Cluster the standard errors when eval questions share a document, ticket or customer
An eval has 300 questions written from 30 documents, ten per document, and you compute a score interval as if the 300 were independent.
hardempirical finding - Report code-generation pass@k from n samples with the unbiased estimator, at the k users actually get
A code or SQL assistant is scored by "any of k samples passes the tests", using k samples per task exactly.
harddocumented fact - Control who can write to the corpus, because a few planted passages can steer answers
A RAG assistant indexes a wiki, a ticket system and a shared drive that many employees and contractors can edit.
hardpractitioner heuristic - Restrict tool access for the rest of a session once untrusted text has entered the context
One long agent session reads external content such as email, web pages or tickets and also holds state-changing tools.
hardpractitioner heuristic - Measure guardrail false positives on benign traffic, because rare attacks make most blocks wrong
A classifier blocks prompts it believes are attacks or policy violations, and its quality is reported as recall on a balanced test set.
hardpractitioner heuristic - Do not extrapolate small-model 8-bit results to large models, because outlier features break naive quantisation
A model that quantised cleanly at a few billion parameters is scheduled for the same naive 8-bit recipe at a much larger size.
hardempirical finding - Reserve a small random slice of routed traffic for the other model so you can measure what routing hides
A router sends easy requests to a small model and the team reports quality from the traffic the router chose.
hardpractitioner heuristic