Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Count tokens per language before sizing prompts, budgets and context limits
An airline assistant must answer customer messages in English, Turkish, Arabic and German, and the team sized its context and cost budget from English test prompts.
easyempirical finding - Test effective context at your own input lengths instead of trusting the advertised window
A team plans to paste a whole ground-handling contract (a hypothetical 300 pages) into one prompt because it fits inside the documented context window.
mediumempirical finding - Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Slice LLM accuracy by entity popularity because long-tail facts fail first
A travel assistant answers well about major hubs in demos, and the team plans to launch it for all airports in the network.
mediumempirical finding - Separate total from active parameters when sizing mixture-of-experts models
A team compares a mixture-of-experts (MoE) model with a dense model for self-hosting using only the headline parameter count.
mediumdocumented fact - For fixed-label text classification, benchmark a small encoder before a generative LLM
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
mediumpractitioner heuristic - Match checkpoint type and chat template to the job before judging an open model
An engineer downloads open model weights, sends plain instructions and gets rambling, off-task text.
easydocumented fact - Set decoding parameters per task instead of one global temperature
One shared configuration sets a high temperature for every LLM call, from marketing copy to fare-rule extraction.
easyempirical finding - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Use majority voting over sampled answers for verifiable reasoning tasks, and price it
An LLM computes refund amounts from multi-step fare rules and is wrong on a noticeable share of eval cases with a single chain-of-thought answer.
mediumempirical finding - Check the stop reason before parsing, and treat truncated output as a failure
A batch job extracts structured itinerary changes and occasionally receives JSON that ends mid-object.
easydocumented fact - Triage hallucinations as contradictions or unsupported additions because the fixes differ
Reviewers label 40 generated summaries of delay-compensation cases simply as hallucinated or fine.
mediumpractitioner heuristic - Score long answers claim by claim instead of with one overall verdict
A disruption-report generator writes 300-word summaries and reviewers mark each summary acceptable or not.
mediumempirical finding - Use disagreement across samples as a hallucination flag when no reference exists
An assistant drafts airport transit guidance for which no structured source of truth exists yet.
mediumempirical finding - Replace "check your answer" prompts with an external verifier
To cut errors, a team adds a second prompt asking the model to review and fix its own fare calculation.
mediumempirical finding - Give the model a scored way to abstain, or your eval rewards confident guessing
An eval for a policy assistant scores answers only as right or wrong, so a reply of insufficient information counts as wrong.
mediumpractitioner heuristic - Verify every identifier the model emits against the system of record
An assistant drafts rebooking notes that include flight numbers, fare basis codes and policy section numbers.
easypractitioner heuristic - Seed eval sets with questions that invite common misconceptions
A revenue-management help bot is evaluated only on questions written by the people who built it.
mediumempirical finding - Keep your hypothesis out of the prompt when asking an LLM to judge evidence
An analyst asks a model to confirm that a revenue drop on one route was caused by a competitor's new capacity.
mediumempirical finding - Do not treat a model's stated reasoning as an audit trail
A reviewer accepts an LLM's credit-limit recommendation because its step-by-step explanation looks sound.
hardempirical finding - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic - Balance and shuffle few-shot examples, then test several orders
A few-shot prompt for classifying refund requests lists five approve examples followed by one reject example.
mediumempirical finding - Use few-shot examples to show format and label space, not to teach new rules
A team tries to teach the model a new internal overbooking-compensation rule by adding three worked examples to the prompt.
mediumempirical finding - Evaluate prompts across several formatting variants before comparing models
Two candidate models are compared with one prompt template that was tuned on model A.
mediumempirical finding - Use provider-enforced schemas instead of asking for JSON in the prompt
A pipeline extracts passenger name, flight and requested change from emails by asking the model to reply in JSON, then parses with a try/except.
easydocumented fact - Validate business rules after the schema passes, because schemas only check shape
A structured-output extractor for refund requests returns perfectly valid objects, and the team stops checking them.
easypractitioner heuristic - Give every extracted field an explicit unknown value so the model is not forced to invent
An extraction schema requires a booking reference, a fare class and a travel date for every complaint email.
easypractitioner heuristic - Repair failed outputs with the validator's error message and a retry cap
A structured extraction fails business-rule validation on a few percent of documents, and an engineer adds code that silently coerces bad values into defaults.
easypractitioner heuristic - Prune long system prompts by ablating rules against the eval set
A customer-service system prompt has grown to dozens of rules added after individual complaints, some of them contradicting each other.
mediumpractitioner heuristic - Extract from long documents section by section with provenance, then aggregate in code
An analyst asks a model to list every penalty clause from a long supplier contract in one call.
mediumpractitioner heuristic - Separate instructions from pasted data with clear delimiters, but do not call it security
A prompt mixes instructions with a pasted customer email, and the model sometimes follows sentences inside the email as if they were instructions.
easypractitioner heuristic - Let the model reason before you force strict formats on reasoning-heavy tasks
A multi-step fare-difference calculation is moved into a strict JSON output mode, and accuracy on the eval set drops.
mediumempirical finding - Never draw few-shot demonstrations from the evaluation set
To improve a classification prompt quickly, an engineer copies the hardest misclassified eval cases into the prompt as examples.
mediumpractitioner heuristic - Supply context instead of personas, because role prompts do not add knowledge
A prompt starts by telling the model it is a world-class revenue-management expert, and the team expects better pricing analysis as a result.
easyempirical finding - Measure chain-of-thought prompting per task instead of switching it on everywhere
After reading that step-by-step prompting improves reasoning, a team adds it to every prompt, including simple classification.
easyempirical finding - Require a fine-tune to beat the best prompt on a frozen eval set before you own it
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
mediumpractitioner heuristic - Deduplicate fine-tuning data against the eval set before training
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
mediumpractitioner heuristic - Name the missing capability before choosing prompting, retrieval or fine-tuning
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
mediumpractitioner heuristic - Do not fine-tune to inject facts that change; retrieve them instead
A team fine-tunes a model on this quarter's fare rules so agents can ask about them without retrieval.
mediumempirical finding - Fine-tune for consistent behaviour that long prompts cannot hold
A prompt for drafting delay notices needs a long style guide and many examples, yet tone and structure still drift across thousands of daily messages.
mediumpractitioner heuristic - Curate a small, consistent fine-tuning set before collecting a large noisy one
A team plans to fine-tune on 200,000 historical agent replies exported without review.
mediumempirical finding - Rerun safety and refusal tests after every fine-tune, even on benign data
A team fine-tunes an aligned model on harmless customer-service transcripts and evaluates only task quality.
mediumempirical finding - Consider continued pretraining only when you hold a large unlabelled domain corpus
An insurer's claims text is full of domain jargon, and a general model handles it poorly even after prompt work.
hardempirical finding - Anchor synthetic training data to real data and track its provenance
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
mediumempirical finding - Do the LoRA parameter arithmetic before planning hardware
A team asks whether LoRA lets them fine-tune a model on the GPU they already have.
mediumdocumented fact - Expect LoRA to learn less and forget less than full fine-tuning
A team uses LoRA to teach a model a large new skill domain, such as a proprietary programming language, and gains are small.
hardempirical finding - Evaluate the exact artefact you will serve, not the training checkpoint
A LoRA adapter scores well during training, then is merged into the base weights and quantized for serving without re-evaluation.
easypractitioner heuristic - Judge a compressed model on your tasks, not on perplexity alone
A self-hosted model is quantized or pruned to cut serving memory, and the team signs off because perplexity barely moved.
mediumempirical finding - Choose models on a task-specific eval set, not on public leaderboard rank
A team picks the model at the top of a public leaderboard for a booking-change assistant.
easypractitioner heuristic - Treat public benchmark scores as possibly contaminated and keep a private held-out set
A vendor's model posts a high score on a well-known public benchmark, and the team expects similar quality on its own tasks.
mediumempirical finding - Put a confidence interval on every eval score and size the set for the difference you care about
A prompt revision raises accuracy on a 200-item eval set from 80 to 83 percent, and the team announces an improvement.
mediumdocumented fact - Compare two models on the same items with a paired analysis
Model A and model B are run on the same 300 eval items, and their accuracies are compared as two independent proportions.
mediumdocumented fact - Run every pairwise LLM judgment in both orders and count only consistent verdicts
An LLM judge compares old and new assistant answers, always shown in the same order, and prefers the new one 62 percent of the time.
mediumempirical finding - Control for answer length when an LLM judge or reward model scores outputs
A new prompt makes answers 40 percent longer, and the LLM judge's win rate jumps.
mediumempirical finding - Avoid letting a model family judge its own outputs
A team uses the same model to generate answers and to judge them against a competitor's answers.
mediumempirical finding - Validate an LLM judge against human labels before trusting its scores
A team plans to replace human review of complaint replies with an LLM judge to speed up evaluation.
mediumpractitioner heuristic - Use pairwise judging to rank close variants and rubric grading to set pass bars
A team asks an LLM judge to score answers from 1 to 10 to decide between two similar prompt versions.
mediumpractitioner heuristic - Check everything verifiable with code before reaching for an LLM judge
An eval pipeline sends every output to an LLM judge, including checks such as valid JSON, word limits and whether the booking reference matches the input.
easypractitioner heuristic - Turn every production failure into a regression test that gates changes
A model upgrade reintroduces a bug that was fixed months ago, quoting outdated pet-travel rules.
easypractitioner heuristic - Split the eval set into a development set for iteration and a locked test set for decisions
A team iterates on prompts for weeks against one 300-item eval set and reports the final score as expected production quality.
mediumpractitioner heuristic - Calibrate LLM confidence on labelled data before using it as a threshold
A claims-triage assistant auto-approves cases when the model's token probability or stated confidence exceeds 0.9.
hardempirical finding - Locate RLHF failures in feedback, reward model or policy before changing the recipe
After preference tuning, an internal assistant becomes evasive on some topics and overconfident on others, and the team considers switching alignment methods.
hardpractitioner heuristic - Track a gold metric while optimising against a reward model, and stop before it turns down
During RL fine-tuning, the reward-model score keeps rising run after run, and the team keeps training.
hardempirical finding - Choose DPO for simplicity, but check for distribution shift in the preference data
A team with a fixed preference dataset picks DPO because it avoids training a reward model and running reinforcement learning.
hardempirical finding - Measure output diversity after preference tuning when variety is part of the product
A preference-tuned model generates marketing copy for 300 routes, and the drafts start to sound alike.
mediumempirical finding - Check whether a preference-tuning gain is just longer answers
A preference-tuned model wins more human and model comparisons, and its answers are noticeably longer than before.
mediumempirical finding - Re-measure core capabilities after alignment tuning to catch an alignment tax
A model tuned for helpfulness and tone on support conversations is promoted after winning preference comparisons.
mediumempirical finding - Pick a generative model family by the constraint that binds, not by popularity
A retailer wants synthetic product images for layout testing and assumes the newest diffusion model is the only option.
mediumdocumented fact - Sweep guidance strength as a fidelity-versus-diversity knob, like temperature
A team generating synthetic training images sets a high guidance scale because individual samples look sharpest.
mediumempirical finding - Do not assume generated data is private; test it for memorised training records
A team proposes sharing synthetic images or records produced by a generative model trained on customer data, on the grounds that the data is synthetic.
mediumempirical finding - Chunk along the document's own structure before tuning chunk size
You are ingesting manuals, policies or contracts that have headings, numbered clauses or tables into a RAG index.
easypractitioner heuristic - Give each chunk the context it lost when it was cut out of the document
Many chunks say things like "this fee is waived" or "revenue grew 3%" without naming the product, route or period they refer to.
mediumempirical finding - Compare chunk sizes at the same context token budget, not the same k
You are running a chunk-size experiment, for example 200 versus 800 tokens, and comparing recall@5 for each.
hardpractitioner heuristic - Enforce validity dates and scope with metadata filters, not with the prompt
The corpus holds several versions of the same policy, for example baggage rules effective before and after 2024-03-01, or tariffs per country.
mediumpractitioner heuristic - Apply document permissions inside the retriever, before the model sees anything
An internal assistant indexes documents with different audiences, such as HR cases, board papers and public FAQs, in one vector store.
mediumpractitioner heuristic - Keep a BM25 baseline when queries contain codes, identifiers or rare terms
Users search with fare basis codes, error codes, part numbers or policy clause ids, and the team plans a dense-only retriever.
easyempirical finding - Fuse lexical and vector results by rank, not by adding raw scores
You combine BM25 and embedding search into a hybrid retriever and need one ranked list.
mediumempirical finding - Choose the embedding model on your own labelled queries, not a public leaderboard
The team picks an embedding model because it tops a public retrieval benchmark.
mediumempirical finding - Retrieve wide with a cheap retriever, then rerank narrow with a cross-encoder
The right passage is often in the top 50 but rarely in the top 5 you pass to the model.
mediumempirical finding - Stop adding retrieved passages once answer quality stops rising
Answers miss facts, and the proposed fix is to raise k from 20 to 50 because the context window allows it.
mediumempirical finding - Put the strongest evidence at the start of the context, not buried in the middle
Context assembly concatenates passages in arbitrary order, for example by document id or ingestion time.
easyempirical finding - Drop weakly relevant chunks instead of always filling k slots
Retrieval always returns exactly k chunks, even for questions where only one or none is relevant.
mediumempirical finding - Deduplicate near-identical chunks before they crowd out other evidence
The corpus contains copies of the same policy in several places: the PDF, an intranet page and an email announcement.
easypractitioner heuristic - Validate citation ids in code before showing a cited answer
Your RAG prompt asks the model to cite sources as [doc-id] markers.
easyempirical finding - Score citation recall and citation precision separately
You report a single "citation accuracy" number for a cited-answer system.
mediumdocumented fact - Report recall@k at the k you actually pass to the model
A retrieval dashboard shows recall@100, while the generation prompt receives the top 5 chunks.
easydocumented fact - Use MRR when one right passage exists and how high it ranks matters
Each question has a single authoritative passage, such as one clause of a contract, and you want one number for ranking quality.
easydocumented fact - Use nDCG when some retrieved passages are partly relevant
Labellers say some passages fully answer the question, some are related background, and some are outdated versions of the right rule.
mediumdocumented fact - Locate RAG failures with a retrieval-by-answer two-way table
A RAG assistant gives wrong answers on some questions and the team debates whether to change the retriever or the prompt.
mediumpractitioner heuristic - Measure faithfulness and correctness separately, because faithful answers can be wrong
Your RAG evaluation uses a faithfulness score (is every claim supported by the retrieved context?) as the main quality metric.
mediumdocumented fact - Label the evidence passages in your golden set, not only the expected answers
You are building the first evaluation set for a document assistant from real user questions.
easypractitioner heuristic - Include unanswerable questions and measure how often the assistant abstains
The golden set contains only questions that the corpus can answer.
mediumpractitioner heuristic - Run RAG ablations one component at a time with per-stage metrics
A new RAG version changes the chunker, the embedding model and the prompt at once, and end-to-end accuracy improves.
mediumpractitioner heuristic - Compare RAG variants with paired per-question differences on small golden sets
Variant B scores 0.78 and variant A 0.75 on an 80-question golden set, and someone proposes shipping B.
hardpractitioner heuristic - Slice retrieval metrics by query type before declaring a retriever better
A hybrid retriever raises average recall@5 by two points over the dense baseline.
mediumpractitioner heuristic - Decompose multi-hop questions instead of hoping one retrieval finds everything
Questions combine facts from different documents, such as "which of our A321 routes are affected by the new crew-rest rule?"
hardpractitioner heuristic - Monitor the ingestion pipeline, because parse failures drop evidence silently
A RAG index is rebuilt nightly from PDFs, HTML and spreadsheets owned by other teams.
mediumpractitioner heuristic - Route counting and aggregation questions to SQL, not to chunk retrieval
Users ask the document assistant "how many flights to FRA were cancelled last month?" or "average fare on IST-ADB in July".
mediumpractitioner heuristic - Write the workflow in code when the steps are known, and use the model only at fuzzy steps
A team proposes an autonomous agent for a process whose steps are already documented, such as refund eligibility checks or disruption rebooking.
easypractitioner heuristic - Give every agent run hard budgets and explicit stopping conditions
An agent loop runs "until the task is done" with no limit on iterations, tokens, spend or wall-clock time.
easypractitioner heuristic - Expose only the tools a task needs, as an explicit allow-list per route
One agent configuration registers every available tool (email, database, payments, file system) for all requests.
easydocumented fact - Prefer narrow, purpose-built tools over generic ones like raw SQL or HTTP
A customer-service agent gets a runsql tool or a generic httprequest tool "so it can handle anything".
mediumpractitioner heuristic - Make tools ask for missing inputs instead of acting on values the model filled in
A user says "rebook me on the next flight" without a date or booking reference, and the rebooking tool requires both.
mediumpractitioner heuristic - Require human approval for irreversible actions, showing the exact action rather than the model's summary
An agent can issue refunds, send emails to customers, cancel bookings or change prices.
mediumpractitioner heuristic - Make side-effecting tools idempotent so retries cannot double-act
A booking or payment tool call times out, and the agent or the client library retries it.
mediumpractitioner heuristic - Return structured, classified tool errors so the agent can recover or stop
Tool failures reach the model as raw stack traces or a generic "error occurred".
easypractitioner heuristic - Persist agent state as explicit records, not only as a growing transcript
A long-running agent (hours, many tool calls) keeps all of its progress inside the conversation history.
mediumpractitioner heuristic - Treat writes to long-term agent memory as untrusted input with provenance
An assistant saves "facts" or preferences from conversations, documents or web pages into persistent memory reused in later sessions.
hardpractitioner heuristic - Score agents on the end state they produce, not on how the transcript reads
Agent evaluation consists of reading transcripts or asking a judge model whether the conversation "looks successful".
mediumdocumented fact - Run each agent test several times and report consistency, not one lucky pass
An agent passes a test task once in a demo and the task is marked solved.
mediumempirical finding - Report steps, tool calls and tokens per task alongside agent success rate
Two agent designs reach similar task success, and the comparison stops there.
easypractitioner heuristic - Test agents against simulated tools in a sandbox before they touch real systems
An agent is ready for a pilot, and its only testing so far has been against live APIs with real customer records.
mediumpractitioner heuristic - Prove one agent is insufficient before splitting work across several agents
A design proposes separate planner, researcher, critic and executor agents for a task one agent has not yet attempted.
mediumopinion - Write tool descriptions like an interface spec and test them with varied inputs
Tools are registered with one-line descriptions such as "search bookings", and the agent misuses them.
easypractitioner heuristic - Cap and shape tool results before they enter the context window
A tool returns a full API payload, a 2,000-row query result or an entire web page to the agent.
easypractitioner heuristic - Treat retrieved and tool-returned text as data, and enforce limits outside the model
A RAG assistant or agent reads emails, web pages, tickets or uploaded files written by people you do not control.
mediumdocumented fact - Assume the system prompt will leak, so never put secrets or access rules only there
A system prompt contains API keys, internal URLs, discount codes, or the only statement of which users may see what.
easydocumented fact - Close exfiltration channels such as auto-loaded links and images in model output
The chat interface renders model output as Markdown, including images and links to arbitrary domains, and the model can see private data.
hardpractitioner heuristic - Keep the model that reads untrusted content away from tools and private data
One agent reads incoming customer emails, can query customer records, and can send messages externally.
harddocumented fact - Fix the plan before reading untrusted data when the task allows it
An agent must process external content, for example summarise today's supplier emails into a report, with a predictable set of actions.
harddocumented fact