Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Slice LLM accuracy by entity popularity because long-tail facts fail first
A travel assistant answers well about major hubs in demos, and the team plans to launch it for all airports in the network.
mediumempirical finding - For fixed-label text classification, benchmark a small encoder before a generative LLM
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
mediumpractitioner heuristic - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Triage hallucinations as contradictions or unsupported additions because the fixes differ
Reviewers label 40 generated summaries of delay-compensation cases simply as hallucinated or fine.
mediumpractitioner heuristic - Score long answers claim by claim instead of with one overall verdict
A disruption-report generator writes 300-word summaries and reviewers mark each summary acceptable or not.
mediumempirical finding - Give the model a scored way to abstain, or your eval rewards confident guessing
An eval for a policy assistant scores answers only as right or wrong, so a reply of insufficient information counts as wrong.
mediumpractitioner heuristic - Seed eval sets with questions that invite common misconceptions
A revenue-management help bot is evaluated only on questions written by the people who built it.
mediumempirical finding - Keep your hypothesis out of the prompt when asking an LLM to judge evidence
An analyst asks a model to confirm that a revenue drop on one route was caused by a competitor's new capacity.
mediumempirical finding - Do not treat a model's stated reasoning as an audit trail
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
hardempirical finding - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic - Balance and shuffle few-shot examples, then test several orders
A few-shot prompt for classifying refund requests lists five approve examples followed by one reject example.
mediumempirical finding - Evaluate prompts across several formatting variants before comparing models
Two candidate models are compared with one prompt template that was tuned on model A.
mediumempirical finding - Never draw few-shot demonstrations from the evaluation set
To improve a classification prompt quickly, an engineer copies the hardest misclassified eval cases into the prompt as examples.
mediumpractitioner heuristic - Supply context instead of personas, because role prompts do not add knowledge
A prompt starts by telling the model it is a world-class revenue-management expert, and the team expects better pricing analysis as a result.
easyempirical finding - Require a fine-tune to beat the best prompt on a frozen eval set before you own it
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
mediumpractitioner heuristic - Deduplicate fine-tuning data against the eval set before training
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
mediumpractitioner heuristic - Name the missing capability before choosing prompting, retrieval or fine-tuning
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
mediumpractitioner heuristic - Anchor synthetic training data to real data and track its provenance
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
mediumempirical finding - Choose models on a task-specific eval set, not on public leaderboard rank
A team picks the model at the top of a public leaderboard for a booking-change assistant.
easypractitioner heuristic - Treat public benchmark scores as possibly contaminated and keep a private held-out set
A vendor's model posts a high score on a well-known public benchmark, and the team expects similar quality on its own tasks.
mediumempirical finding - Put a confidence interval on every eval score and size the set for the difference you care about
A prompt revision raises accuracy on a 200-item eval set from 80 to 83 percent, and the team announces an improvement.
mediumdocumented fact - Compare two models on the same items with a paired analysis
Model A and model B are run on the same 300 eval items, and their accuracies are compared as two independent proportions.
mediumdocumented fact - Run every pairwise LLM judgment in both orders and count only consistent verdicts
An LLM judge compares old and new assistant answers, always shown in the same order, and prefers the new one 62 percent of the time.
mediumempirical finding - Control for answer length when an LLM judge or reward model scores outputs
A new prompt makes answers 40 percent longer, and the LLM judge's win rate jumps.
mediumempirical finding - Avoid letting a model family judge its own outputs
A team uses the same model to generate answers and to judge them against a competitor's answers.
mediumempirical finding - Validate an LLM judge against human labels before trusting its scores
A team plans to replace human review of complaint replies with an LLM judge to speed up evaluation.
mediumpractitioner heuristic - Use pairwise judging to rank close variants and rubric grading to set pass bars
A team asks an LLM judge to score answers from 1 to 10 to decide between two similar prompt versions.
mediumpractitioner heuristic - Split the eval set into a development set for iteration and a locked test set for decisions
A team iterates on prompts for weeks against one 300-item eval set and reports the final score as expected production quality.
mediumpractitioner heuristic - Calibrate LLM confidence on labelled data before using it as a threshold
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
hardempirical finding - Check whether a preference-tuning gain is just longer answers
A preference-tuned model wins more human and model comparisons, and its answers are noticeably longer than before.
mediumempirical finding - Report quality and coverage separately when evaluating generated samples
Two image generators are compared by a single FID score, and the lower score wins.
mediumdocumented fact - Do not assume generated data is private; test it for memorised training records
A team proposes sharing synthetic images or records produced by a generative model trained on customer data, on the grounds that the data is synthetic.
mediumempirical finding - Compare chunk sizes at the same context token budget, not the same k
You are running a chunk-size experiment, for example 200 versus 800 tokens, and comparing recall@5 for each.
hardpractitioner heuristic - Keep a BM25 baseline when queries contain codes, identifiers or rare terms
Users search with fare basis codes, error codes, part numbers or policy clause ids, and the team plans a dense-only retriever.
easyempirical finding - Choose the embedding model on your own labelled queries, not a public leaderboard
The team picks an embedding model because it tops a public retrieval benchmark.
mediumempirical finding - Score citation recall and citation precision separately
You report a single "citation accuracy" number for a cited-answer system.
mediumdocumented fact - Report recall@k at the k you actually pass to the model
A retrieval dashboard shows recall@100, while the generation prompt receives the top 5 chunks.
easydocumented fact - Use MRR when one right passage exists and how high it ranks matters
Each question has a single authoritative passage, such as one clause of a contract, and you want one number for ranking quality.
easydocumented fact - Use nDCG when some retrieved passages are partly relevant
Labellers say some passages fully answer the question, some are related background, and some are outdated versions of the right rule.
mediumdocumented fact - Locate RAG failures with a retrieval-by-answer two-way table
A RAG assistant gives wrong answers on some questions and the team debates whether to change the retriever or the prompt.
mediumpractitioner heuristic - Measure faithfulness and correctness separately, because faithful answers can be wrong
Your RAG evaluation uses a faithfulness score (is every claim supported by the retrieved context?) as the main quality metric.
mediumdocumented fact - Label the evidence passages in your golden set, not only the expected answers
You are building the first evaluation set for a document assistant from real user questions.
easypractitioner heuristic - Include unanswerable questions and measure how often the assistant abstains
The golden set contains only questions that the corpus can answer.
mediumpractitioner heuristic - Run RAG ablations one component at a time with per-stage metrics
A new RAG version changes the chunker, the embedding model and the prompt at once, and end-to-end accuracy improves.
mediumpractitioner heuristic - Compare RAG variants with paired per-question differences on small golden sets
Variant B scores 0.78 and variant A 0.75 on an 80-question golden set, and someone proposes shipping B.
hardpractitioner heuristic - Slice retrieval metrics by query type before declaring a retriever better
A hybrid retriever raises average recall@5 by two points over the dense baseline.
mediumpractitioner heuristic - Route counting and aggregation questions to SQL, not to chunk retrieval
Users ask the document assistant "how many flights to FRA were cancelled last month?" or "average fare on IST-ADB in July".
mediumpractitioner heuristic - Write the workflow in code when the steps are known, and use the model only at fuzzy steps
A team proposes an autonomous agent for a process whose steps are already documented, such as refund eligibility checks or disruption rebooking.
easypractitioner heuristic - Score agents on the end state they produce, not on how the transcript reads
Agent evaluation consists of reading transcripts or asking a judge model whether the conversation "looks successful".
mediumdocumented fact - Run each agent test several times and report consistency, not one lucky pass
An agent passes a test task once in a demo and the task is marked solved.
mediumempirical finding - Report steps, tool calls and tokens per task alongside agent success rate
Two agent designs reach similar task success, and the comparison stops there.
easypractitioner heuristic - Run GenAI design reviews against a shared threat checklist, not ad-hoc brainstorming
A GenAI feature reaches design review and the security section says "prompt injection mitigated by system prompt".
easydocumented fact - Version prompt, retrieval settings and model together as one release
Prompts live in code, retrieval parameters in a config service and the model name in an environment variable, each changed independently.
easypractitioner heuristic - Budget GenAI features by cost per successful task, not price per token
A business case multiplies expected requests by the list price per million tokens and calls that the cost.
mediumpractitioner heuristic - Expect multi-turn chat input cost to grow faster than the number of turns
A chat assistant resends the full conversation history on every turn, and the cost model assumes a fixed price per message.
mediumpractitioner heuristic - Send offline workloads such as eval runs and nightly enrichment through a batch API
Golden-set evaluations, document classification backfills or nightly summaries run through the real-time API with custom concurrency code.
easydocumented fact - Route easy requests to a small model and escalate only when a check fails
All traffic goes to the largest available model, although most requests are simple lookups or classifications.
mediumempirical finding - Pick model configurations from the quality-cost frontier, not by model size
Model selection asks "which is the best model?" and defaults to the largest one available.
mediumpractitioner heuristic - Pick few-shot examples by similarity to each request and keep the fixed set as the baseline
A classifier for passenger messages uses the same six hand-picked examples for every request, and accuracy is uneven across message types.
mediumempirical finding - Track refusals on a benign in-domain prompt set for every model or prompt change
A safety-tuned model update ships to an airline help assistant, and the tests cover only prompts that should be refused.
mediumempirical finding - Check which sampling parameters each model accepts before running a temperature study
An evaluation harness sweeps temperature from 0 to 1 across hosted models, including one released this quarter.
easydocumented fact - Evaluate instruction language and user language as separate factors before translating the prompt
A Turkish-language assistant's system prompt is translated into Turkish because the model should be addressed in its users' language.
mediumempirical finding - Set the abstain threshold from a coverage-versus-error curve on labelled answerable and unanswerable questions
A RAG assistant refuses when the top retrieval score is below a threshold someone picked by eye, and product wants fewer refusals.
mediumempirical finding - Treat stored embeddings as sensitive data, not as an anonymised derivative of the text
A team keeps embeddings of customer messages in a shared vector service, reasoning that vectors of numbers are not readable text.
mediumempirical finding - Count unjudged retrieved passages as unknown, and judge a new retriever's top-k before comparing it
The golden set's relevant passages were labelled from the old retriever's top 10, and a new retriever is scored against it.
mediumdocumented fact - Validate an LLM relevance labeller against chance-corrected agreement with human labels before it expands the golden set
Labelling passages for retrieval evaluation is slow, and the team wants an LLM to grade query-passage pairs at scale.
mediumempirical finding - Do not build the retrieval eval set only from questions generated out of the chunks
To get 500 labelled queries quickly, an LLM writes a question for each chunk, and that chunk becomes the gold answer.
mediumpractitioner heuristic - Add hard-negative passages to RAG tests, such as the superseded rule or the neighbouring route
RAG tests give the generator only the gold passage plus unrelated filler, and the system passes, then fails in production on look-alike text.
mediumempirical finding - Run an oracle-context test to separate the retrieval gap from the generation ceiling
RAG answer accuracy is 62% and the team splits between improving retrieval and improving the prompt without agreement.
mediumpractitioner heuristic - Multiply per-step reliability before promising an end-to-end agent success rate
A team demos an agent whose individual steps look reliable and plans to promise a task success rate to the business.
easypractitioner heuristic - Run a new write-capable agent in shadow mode and compare its proposals with human actions first
A team wants to give a drafting agent permission to apply refunds or rebookings after good offline test results.
hardpractitioner heuristic - Label failed agent runs with a shared failure taxonomy at the first wrong step
A team reviews a pile of failed agent traces and writes a different free-text diagnosis for each.
mediumpractitioner heuristic - Assert required steps and forbidden actions in a trajectory instead of matching an exact path
An agent that issues refunds reaches the right final state, but sometimes skips the identity check the policy requires.
hardpractitioner heuristic - Build the golden set from stratified production traffic, not from questions developers imagine
Engineers wrote the eval questions themselves, and the assistant scores well offline but users complain about requests nobody anticipated.
mediumpractitioner heuristic - Compute headroom before trusting a saturated eval, and harden or retire it
Several prompt or model variants all score above 95% on your eval and the differences between them look tiny.
mediumpractitioner heuristic - Audit the items every variant fails before concluding the models are the problem
A handful of eval items are failed by every prompt version and model you have tried.
mediumempirical finding - Check judge recall on failing answers and chance-corrected agreement, because skewed pass rates inflate raw agreement
A judge reports 90% agreement with human reviewers and 90% of answers are acceptable.
mediumdocumented fact - Debias a judge-scored pass rate with a small human-labelled sample instead of reporting it raw
A judge scores thousands of production conversations cheaply, but you also hold human labels for a small subset.
harddocumented fact - Rank many variants with a Bradley-Terry fit on sampled pairs rather than all-pairs win counts
You compare eight or more prompt or model variants by pairwise judging and the all-pairs table has dozens of cells with few items each.
harddocumented fact - Cluster the standard errors when eval questions share a document, ticket or customer
An eval has 300 questions written from 30 documents, ten per document, and you compute a score interval as if the 300 were independent.
hardempirical finding - Score extraction field by field with typed comparators instead of whole-record exact match
An extraction pipeline is graded by exact string equality of the whole JSON output against gold records with eight fields.
easypractitioner heuristic - Gate on a non-inferiority margin for the paired difference, not on a raw score threshold
A CI gate fails the build when the candidate scores below the baseline by any amount on a 400-item set.
hardpractitioner heuristic - Reweight offline slice scores to the production traffic mix before forecasting online quality
The offline score is 85% but production satisfaction looks different, and the golden set over-represents hard or rare cases.
mediumpractitioner heuristic - Read thumbs-up and thumbs-down rates as a selection-biased signal and sample the silent majority
A dashboard tracks the thumbs-down rate of an assistant and the team treats it as the quality metric.
mediumpractitioner heuristic - Pilot an annotation guide on 30 items and treat low agreement as a spec defect
Two annotators label LLM outputs for "helpfulness" and agree only slightly better than chance.
mediumpractitioner heuristic - Blind and shuffle annotation tasks, and seed known-answer items to catch drift
Reviewers compare outputs from two systems, with system names visible and a fixed order, over several days of labelling.
easypractitioner heuristic - Publish an eval report with set version, judge version, runs, intervals and the number of variants tried
An internal eval result is shared as "new prompt: 84% vs 81%" with no further context.
easypractitioner heuristic - Redact personal data before the prompt leaves your boundary and keep the mapping local
Support tickets and call transcripts with names, emails and booking references go to a hosted model for summarising or classifying.
mediumpractitioner heuristic - Record vendor data terms per endpoint, because retention differs inside one provider
Procurement approved a model provider, and teams now use its chat, batch, file, fine-tuning and embedding endpoints.
easypractitioner heuristic - Measure guardrail false positives on benign traffic, because rare attacks make most blocks wrong
A classifier blocks prompts it believes are attacks or policy violations, and its quality is reported as recall on a balanced test set.
hardpractitioner heuristic - Keep an evidence pack indexed to a risk framework, and label it evidence rather than compliance
Security, privacy or audit teams ask for proof that an LLM feature was assessed, tested and approved.
mediumpractitioner heuristic - Check which tokens carry loss before supervised fine-tuning, because the default depends on data format
An instruction-tuning set is loaded into a trainer and the same recipe is reused with a different dataset layout.
easypractitioner heuristic - Deduplicate the fine-tuning set against itself, not only against the evaluation set
A support-ticket corpus of resolved cases becomes a fine-tuning set, and thousands of tickets are templated near-copies of the same outage.
mediumpractitioner heuristic - Distil a small model on teacher outputs from your own traffic and judge it on the tail the teacher handles
A large model gives good answers on a narrow, high-volume task and the team wants a small model trained on its outputs to cut cost.
mediumpractitioner heuristic - Read the provider's terms before training a model on its outputs
A plan to distil a hosted model's answers into an in-house small model is approved on technical grounds.
easypractitioner heuristic - Forecast token spend from the per-request length distribution, not from the median request
A cost forecast for a new assistant multiplies expected requests by the length of a typical prompt seen in the pilot.
mediumpractitioner heuristic - Capture the human's edit of a model draft as feedback, because the diff shows what was wrong
Agents edit AI-drafted customer replies before sending, and the only feedback collected is a thumbs-up on the draft.
easypractitioner heuristic - Measure language-ID routing on your own short and mixed messages, and give it a default path
An assistant routes each message to a language-specific prompt, analyzer or model after a detector labels the language, and the detector was chosen from its published score.
mediumpractitioner heuristic - Permute the options of every multiple-choice eval and report the spread, not one order
A model is scored on a four-option multiple-choice set, or asked to pick a label from a lettered list, and the options appear in one fixed order.
easyempirical finding - Pin and publish the eval harness version, because the same model scores differently across implementations
A team compares its own benchmark number for an open model with a published figure, or moves an eval from one framework to another.
mediumempirical finding - Seed facts into long documents to measure how many survive each level of a summary tree
A 400-page report is summarised in chunks, the chunk summaries are summarised again, and the top-level summary is accepted because it reads well.
mediumpractitioner heuristic - Score generated SQL on several database variants, not on the one database you have
A text-to-SQL feature is scored by running the generated and reference queries on one database and comparing results.
mediumpractitioner heuristic