Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Slice LLM accuracy by entity popularity because long-tail facts fail first
A travel assistant answers well about major hubs in demos, and the team plans to launch it for all airports in the network.
mediumempirical finding - For fixed-label text classification, benchmark a small encoder before a generative LLM
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
mediumpractitioner heuristic - Triage hallucinations as contradictions or unsupported additions because the fixes differ
Reviewers label 40 generated summaries of delay-compensation cases simply as hallucinated or fine.
mediumpractitioner heuristic - Score long answers claim by claim instead of with one overall verdict
A disruption-report generator writes 300-word summaries and reviewers mark each summary acceptable or not.
mediumempirical finding - Give the model a scored way to abstain, or your eval rewards confident guessing
An eval for a policy assistant scores answers only as right or wrong, so a reply of insufficient information counts as wrong.
mediumpractitioner heuristic - Seed eval sets with questions that invite common misconceptions
A revenue-management help bot is evaluated only on questions written by the people who built it.
mediumempirical finding - Keep your hypothesis out of the prompt when asking an LLM to judge evidence
An analyst asks a model to confirm that a revenue drop on one route was caused by a competitor's new capacity.
mediumempirical finding - Balance and shuffle few-shot examples, then test several orders
A few-shot prompt for classifying refund requests lists five approve examples followed by one reject example.
mediumempirical finding - Evaluate prompts across several formatting variants before comparing models
Two candidate models are compared with one prompt template that was tuned on model A.
mediumempirical finding - Never draw few-shot demonstrations from the evaluation set
To improve a classification prompt quickly, an engineer copies the hardest misclassified eval cases into the prompt as examples.
mediumpractitioner heuristic - Require a fine-tune to beat the best prompt on a frozen eval set before you own it
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
mediumpractitioner heuristic - Deduplicate fine-tuning data against the eval set before training
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
mediumpractitioner heuristic - Name the missing capability before choosing prompting, retrieval or fine-tuning
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
mediumpractitioner heuristic - Anchor synthetic training data to real data and track its provenance
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
mediumempirical finding - Treat public benchmark scores as possibly contaminated and keep a private held-out set
A vendor's model posts a high score on a well-known public benchmark, and the team expects similar quality on its own tasks.
mediumempirical finding - Put a confidence interval on every eval score and size the set for the difference you care about
A prompt revision raises accuracy on a 200-item eval set from 80 to 83 percent, and the team announces an improvement.
mediumdocumented fact - Compare two models on the same items with a paired analysis
Model A and model B are run on the same 300 eval items, and their accuracies are compared as two independent proportions.
mediumdocumented fact - Run every pairwise LLM judgment in both orders and count only consistent verdicts
An LLM judge compares old and new assistant answers, always shown in the same order, and prefers the new one 62 percent of the time.
mediumempirical finding - Control for answer length when an LLM judge or reward model scores outputs
A new prompt makes answers 40 percent longer, and the LLM judge's win rate jumps.
mediumempirical finding - Avoid letting a model family judge its own outputs
A team uses the same model to generate answers and to judge them against a competitor's answers.
mediumempirical finding - Validate an LLM judge against human labels before trusting its scores
A team plans to replace human review of complaint replies with an LLM judge to speed up evaluation.
mediumpractitioner heuristic - Use pairwise judging to rank close variants and rubric grading to set pass bars
A team asks an LLM judge to score answers from 1 to 10 to decide between two similar prompt versions.
mediumpractitioner heuristic - Split the eval set into a development set for iteration and a locked test set for decisions
A team iterates on prompts for weeks against one 300-item eval set and reports the final score as expected production quality.
mediumpractitioner heuristic - Check whether a preference-tuning gain is just longer answers
A preference-tuned model wins more human and model comparisons, and its answers are noticeably longer than before.
mediumempirical finding - Report quality and coverage separately when evaluating generated samples
Two image generators are compared by a single FID score, and the lower score wins.
mediumdocumented fact - Do not assume generated data is private; test it for memorised training records
A team proposes sharing synthetic images or records produced by a generative model trained on customer data, on the grounds that the data is synthetic.
mediumempirical finding - Choose the embedding model on your own labelled queries, not a public leaderboard
The team picks an embedding model because it tops a public retrieval benchmark.
mediumempirical finding - Score citation recall and citation precision separately
You report a single "citation accuracy" number for a cited-answer system.
mediumdocumented fact - Use nDCG when some retrieved passages are partly relevant
Labellers say some passages fully answer the question, some are related background, and some are outdated versions of the right rule.
mediumdocumented fact - Locate RAG failures with a retrieval-by-answer two-way table
A RAG assistant gives wrong answers on some questions and the team debates whether to change the retriever or the prompt.
mediumpractitioner heuristic - Measure faithfulness and correctness separately, because faithful answers can be wrong
Your RAG evaluation uses a faithfulness score (is every claim supported by the retrieved context?) as the main quality metric.
mediumdocumented fact - Include unanswerable questions and measure how often the assistant abstains
The golden set contains only questions that the corpus can answer.
mediumpractitioner heuristic - Run RAG ablations one component at a time with per-stage metrics
A new RAG version changes the chunker, the embedding model and the prompt at once, and end-to-end accuracy improves.
mediumpractitioner heuristic - Slice retrieval metrics by query type before declaring a retriever better
A hybrid retriever raises average recall@5 by two points over the dense baseline.
mediumpractitioner heuristic - Route counting and aggregation questions to SQL, not to chunk retrieval
Users ask the document assistant "how many flights to FRA were cancelled last month?" or "average fare on IST-ADB in July".
mediumpractitioner heuristic - Score agents on the end state they produce, not on how the transcript reads
Agent evaluation consists of reading transcripts or asking a judge model whether the conversation "looks successful".
mediumdocumented fact - Run each agent test several times and report consistency, not one lucky pass
An agent passes a test task once in a demo and the task is marked solved.
mediumempirical finding - Budget GenAI features by cost per successful task, not price per token
A business case multiplies expected requests by the list price per million tokens and calls that the cost.
mediumpractitioner heuristic - Expect multi-turn chat input cost to grow faster than the number of turns
A chat assistant resends the full conversation history on every turn, and the cost model assumes a fixed price per message.
mediumpractitioner heuristic - Route easy requests to a small model and escalate only when a check fails
All traffic goes to the largest available model, although most requests are simple lookups or classifications.
mediumempirical finding - Pick model configurations from the quality-cost frontier, not by model size
Model selection asks "which is the best model?" and defaults to the largest one available.
mediumpractitioner heuristic - Pick few-shot examples by similarity to each request and keep the fixed set as the baseline
A classifier for passenger messages uses the same six hand-picked examples for every request, and accuracy is uneven across message types.
mediumempirical finding - Track refusals on a benign in-domain prompt set for every model or prompt change
A safety-tuned model update ships to an airline help assistant, and the tests cover only prompts that should be refused.
mediumempirical finding - Evaluate instruction language and user language as separate factors before translating the prompt
A Turkish-language assistant's system prompt is translated into Turkish because the model should be addressed in its users' language.
mediumempirical finding - Set the abstain threshold from a coverage-versus-error curve on labelled answerable and unanswerable questions
A RAG assistant refuses when the top retrieval score is below a threshold someone picked by eye, and product wants fewer refusals.
mediumempirical finding - Treat stored embeddings as sensitive data, not as an anonymised derivative of the text
A team keeps embeddings of customer messages in a shared vector service, reasoning that vectors of numbers are not readable text.
mediumempirical finding - Count unjudged retrieved passages as unknown, and judge a new retriever's top-k before comparing it
The golden set's relevant passages were labelled from the old retriever's top 10, and a new retriever is scored against it.
mediumdocumented fact - Validate an LLM relevance labeller against chance-corrected agreement with human labels before it expands the golden set
Labelling passages for retrieval evaluation is slow, and the team wants an LLM to grade query-passage pairs at scale.
mediumempirical finding - Do not build the retrieval eval set only from questions generated out of the chunks
To get 500 labelled queries quickly, an LLM writes a question for each chunk, and that chunk becomes the gold answer.
mediumpractitioner heuristic - Add hard-negative passages to RAG tests, such as the superseded rule or the neighbouring route
RAG tests give the generator only the gold passage plus unrelated filler, and the system passes, then fails in production on look-alike text.
mediumempirical finding - Run an oracle-context test to separate the retrieval gap from the generation ceiling
RAG answer accuracy is 62% and the team splits between improving retrieval and improving the prompt without agreement.
mediumpractitioner heuristic - Label failed agent runs with a shared failure taxonomy at the first wrong step
A team reviews a pile of failed agent traces and writes a different free-text diagnosis for each.
mediumpractitioner heuristic - Build the golden set from stratified production traffic, not from questions developers imagine
Engineers wrote the eval questions themselves, and the assistant scores well offline but users complain about requests nobody anticipated.
mediumpractitioner heuristic - Compute headroom before trusting a saturated eval, and harden or retire it
Several prompt or model variants all score above 95% on your eval and the differences between them look tiny.
mediumpractitioner heuristic - Audit the items every variant fails before concluding the models are the problem
A handful of eval items are failed by every prompt version and model you have tried.
mediumempirical finding - Check judge recall on failing answers and chance-corrected agreement, because skewed pass rates inflate raw agreement
A judge reports 90% agreement with human reviewers and 90% of answers are acceptable.
mediumdocumented fact - Reweight offline slice scores to the production traffic mix before forecasting online quality
The offline score is 85% but production satisfaction looks different, and the golden set over-represents hard or rare cases.
mediumpractitioner heuristic - Read thumbs-up and thumbs-down rates as a selection-biased signal and sample the silent majority
A dashboard tracks the thumbs-down rate of an assistant and the team treats it as the quality metric.
mediumpractitioner heuristic - Pilot an annotation guide on 30 items and treat low agreement as a spec defect
Two annotators label LLM outputs for "helpfulness" and agree only slightly better than chance.
mediumpractitioner heuristic - Redact personal data before the prompt leaves your boundary and keep the mapping local
Support tickets and call transcripts with names, emails and booking references go to a hosted model for summarising or classifying.
mediumpractitioner heuristic - Keep an evidence pack indexed to a risk framework, and label it evidence rather than compliance
Security, privacy or audit teams ask for proof that an LLM feature was assessed, tested and approved.
mediumpractitioner heuristic - Deduplicate the fine-tuning set against itself, not only against the evaluation set
A support-ticket corpus of resolved cases becomes a fine-tuning set, and thousands of tickets are templated near-copies of the same outage.
mediumpractitioner heuristic - Distil a small model on teacher outputs from your own traffic and judge it on the tail the teacher handles
A large model gives good answers on a narrow, high-volume task and the team wants a small model trained on its outputs to cut cost.
mediumpractitioner heuristic - Forecast token spend from the per-request length distribution, not from the median request
A cost forecast for a new assistant multiplies expected requests by the length of a typical prompt seen in the pilot.
mediumpractitioner heuristic - Measure language-ID routing on your own short and mixed messages, and give it a default path
An assistant routes each message to a language-specific prompt, analyzer or model after a detector labels the language, and the detector was chosen from its published score.
mediumpractitioner heuristic - Pin and publish the eval harness version, because the same model scores differently across implementations
A team compares its own benchmark number for an open model with a published figure, or moves an eval from one framework to another.
mediumempirical finding - Seed facts into long documents to measure how many survive each level of a summary tree
A 400-page report is summarised in chunks, the chunk summaries are summarised again, and the top-level summary is accepted because it reads well.
mediumpractitioner heuristic - Score generated SQL on several database variants, not on the one database you have
A text-to-SQL feature is scored by running the generated and reference queries on one database and comparing results.
mediumpractitioner heuristic