Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Test effective context at your own input lengths instead of trusting the advertised window
A team plans to paste a whole ground-handling contract (a hypothetical 300 pages) into one prompt because it fits inside the documented context window.
mediumempirical finding - Slice LLM accuracy by entity popularity because long-tail facts fail first
A travel assistant answers well about major hubs in demos, and the team plans to launch it for all airports in the network.
mediumempirical finding - Separate total from active parameters when sizing mixture-of-experts models
A team compares a mixture-of-experts (MoE) model with a dense model for self-hosting using only the headline parameter count.
mediumdocumented fact - For fixed-label text classification, benchmark a small encoder before a generative LLM
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
mediumpractitioner heuristic - Use majority voting over sampled answers for verifiable reasoning tasks, and price it
An LLM computes refund amounts from multi-step fare rules and is wrong on a noticeable share of eval cases with a single chain-of-thought answer.
mediumempirical finding - Triage hallucinations as contradictions or unsupported additions because the fixes differ
Reviewers label 40 generated summaries of delay-compensation cases simply as hallucinated or fine.
mediumpractitioner heuristic - Score long answers claim by claim instead of with one overall verdict
A disruption-report generator writes 300-word summaries and reviewers mark each summary acceptable or not.
mediumempirical finding - Use disagreement across samples as a hallucination flag when no reference exists
An assistant drafts airport transit guidance for which no structured source of truth exists yet.
mediumempirical finding - Replace "check your answer" prompts with an external verifier
To cut errors, a team adds a second prompt asking the model to review and fix its own fare calculation.
mediumempirical finding - Give the model a scored way to abstain, or your eval rewards confident guessing
An eval for a policy assistant scores answers only as right or wrong, so a reply of insufficient information counts as wrong.
mediumpractitioner heuristic - Seed eval sets with questions that invite common misconceptions
A revenue-management help bot is evaluated only on questions written by the people who built it.
mediumempirical finding - Keep your hypothesis out of the prompt when asking an LLM to judge evidence
An analyst asks a model to confirm that a revenue drop on one route was caused by a competitor's new capacity.
mediumempirical finding - Balance and shuffle few-shot examples, then test several orders
A few-shot prompt for classifying refund requests lists five approve examples followed by one reject example.
mediumempirical finding - Use few-shot examples to show format and label space, not to teach new rules
A team tries to teach the model a new internal overbooking-compensation rule by adding three worked examples to the prompt.
mediumempirical finding - Evaluate prompts across several formatting variants before comparing models
Two candidate models are compared with one prompt template that was tuned on model A.
mediumempirical finding - Prune long system prompts by ablating rules against the eval set
A customer-service system prompt has grown to dozens of rules added after individual complaints, some of them contradicting each other.
mediumpractitioner heuristic - Extract from long documents section by section with provenance, then aggregate in code
An analyst asks a model to list every penalty clause from a long supplier contract in one call.
mediumpractitioner heuristic - Let the model reason before you force strict formats on reasoning-heavy tasks
A multi-step fare-difference calculation is moved into a strict JSON output mode, and accuracy on the eval set drops.
mediumempirical finding - Never draw few-shot demonstrations from the evaluation set
To improve a classification prompt quickly, an engineer copies the hardest misclassified eval cases into the prompt as examples.
mediumpractitioner heuristic - Require a fine-tune to beat the best prompt on a frozen eval set before you own it
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
mediumpractitioner heuristic - Deduplicate fine-tuning data against the eval set before training
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
mediumpractitioner heuristic - Name the missing capability before choosing prompting, retrieval or fine-tuning
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
mediumpractitioner heuristic - Do not fine-tune to inject facts that change; retrieve them instead
A team fine-tunes a model on this quarter's fare rules so agents can ask about them without retrieval.
mediumempirical finding - Fine-tune for consistent behaviour that long prompts cannot hold
A prompt for drafting delay notices needs a long style guide and many examples, yet tone and structure still drift across thousands of daily messages.
mediumpractitioner heuristic - Curate a small, consistent fine-tuning set before collecting a large noisy one
A team plans to fine-tune on 200,000 historical agent replies exported without review.
mediumempirical finding - Rerun safety and refusal tests after every fine-tune, even on benign data
A team fine-tunes an aligned model on harmless customer-service transcripts and evaluates only task quality.
mediumempirical finding - Anchor synthetic training data to real data and track its provenance
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
mediumempirical finding - Do the LoRA parameter arithmetic before planning hardware
A team asks whether LoRA lets them fine-tune a model on the GPU they already have.
mediumdocumented fact - Judge a compressed model on your tasks, not on perplexity alone
A self-hosted model is quantized or pruned to cut serving memory, and the team signs off because perplexity barely moved.
mediumempirical finding - Treat public benchmark scores as possibly contaminated and keep a private held-out set
A vendor's model posts a high score on a well-known public benchmark, and the team expects similar quality on its own tasks.
mediumempirical finding - Put a confidence interval on every eval score and size the set for the difference you care about
A prompt revision raises accuracy on a 200-item eval set from 80 to 83 percent, and the team announces an improvement.
mediumdocumented fact - Compare two models on the same items with a paired analysis
Model A and model B are run on the same 300 eval items, and their accuracies are compared as two independent proportions.
mediumdocumented fact - Run every pairwise LLM judgment in both orders and count only consistent verdicts
An LLM judge compares old and new assistant answers, always shown in the same order, and prefers the new one 62 percent of the time.
mediumempirical finding - Control for answer length when an LLM judge or reward model scores outputs
A new prompt makes answers 40 percent longer, and the LLM judge's win rate jumps.
mediumempirical finding - Avoid letting a model family judge its own outputs
A team uses the same model to generate answers and to judge them against a competitor's answers.
mediumempirical finding - Validate an LLM judge against human labels before trusting its scores
A team plans to replace human review of complaint replies with an LLM judge to speed up evaluation.
mediumpractitioner heuristic - Use pairwise judging to rank close variants and rubric grading to set pass bars
A team asks an LLM judge to score answers from 1 to 10 to decide between two similar prompt versions.
mediumpractitioner heuristic - Split the eval set into a development set for iteration and a locked test set for decisions
A team iterates on prompts for weeks against one 300-item eval set and reports the final score as expected production quality.
mediumpractitioner heuristic - Measure output diversity after preference tuning when variety is part of the product
A preference-tuned model generates marketing copy for 300 routes, and the drafts start to sound alike.
mediumempirical finding - Check whether a preference-tuning gain is just longer answers
A preference-tuned model wins more human and model comparisons, and its answers are noticeably longer than before.
mediumempirical finding - Re-measure core capabilities after alignment tuning to catch an alignment tax
A model tuned for helpfulness and tone on support conversations is promoted after winning preference comparisons.
mediumempirical finding - Pick a generative model family by the constraint that binds, not by popularity
A retailer wants synthetic product images for layout testing and assumes the newest diffusion model is the only option.
mediumdocumented fact - Sweep guidance strength as a fidelity-versus-diversity knob, like temperature
A team generating synthetic training images sets a high guidance scale because individual samples look sharpest.
mediumempirical finding - Do not assume generated data is private; test it for memorised training records
A team proposes sharing synthetic images or records produced by a generative model trained on customer data, on the grounds that the data is synthetic.
mediumempirical finding - Give each chunk the context it lost when it was cut out of the document
Many chunks say things like "this fee is waived" or "revenue grew 3%" without naming the product, route or period they refer to.
mediumempirical finding - Enforce validity dates and scope with metadata filters, not with the prompt
The corpus holds several versions of the same policy, for example baggage rules effective before and after 2024-03-01, or tariffs per country.
mediumpractitioner heuristic - Apply document permissions inside the retriever, before the model sees anything
An internal assistant indexes documents with different audiences, such as HR cases, board papers and public FAQs, in one vector store.
mediumpractitioner heuristic - Fuse lexical and vector results by rank, not by adding raw scores
You combine BM25 and embedding search into a hybrid retriever and need one ranked list.
mediumempirical finding - Choose the embedding model on your own labelled queries, not a public leaderboard
The team picks an embedding model because it tops a public retrieval benchmark.
mediumempirical finding - Retrieve wide with a cheap retriever, then rerank narrow with a cross-encoder
The right passage is often in the top 50 but rarely in the top 5 you pass to the model.
mediumempirical finding - Stop adding retrieved passages once answer quality stops rising
Answers miss facts, and the proposed fix is to raise k from 20 to 50 because the context window allows it.
mediumempirical finding - Drop weakly relevant chunks instead of always filling k slots
Retrieval always returns exactly k chunks, even for questions where only one or none is relevant.
mediumempirical finding - Score citation recall and citation precision separately
You report a single "citation accuracy" number for a cited-answer system.
mediumdocumented fact - Use nDCG when some retrieved passages are partly relevant
Labellers say some passages fully answer the question, some are related background, and some are outdated versions of the right rule.
mediumdocumented fact - Locate RAG failures with a retrieval-by-answer two-way table
A RAG assistant gives wrong answers on some questions and the team debates whether to change the retriever or the prompt.
mediumpractitioner heuristic - Measure faithfulness and correctness separately, because faithful answers can be wrong
Your RAG evaluation uses a faithfulness score (is every claim supported by the retrieved context?) as the main quality metric.
mediumdocumented fact - Include unanswerable questions and measure how often the assistant abstains
The golden set contains only questions that the corpus can answer.
mediumpractitioner heuristic - Run RAG ablations one component at a time with per-stage metrics
A new RAG version changes the chunker, the embedding model and the prompt at once, and end-to-end accuracy improves.
mediumpractitioner heuristic - Slice retrieval metrics by query type before declaring a retriever better
A hybrid retriever raises average recall@5 by two points over the dense baseline.
mediumpractitioner heuristic - Monitor the ingestion pipeline, because parse failures drop evidence silently
A RAG index is rebuilt nightly from PDFs, HTML and spreadsheets owned by other teams.
mediumpractitioner heuristic - Route counting and aggregation questions to SQL, not to chunk retrieval
Users ask the document assistant "how many flights to FRA were cancelled last month?" or "average fare on IST-ADB in July".
mediumpractitioner heuristic - Prefer narrow, purpose-built tools over generic ones like raw SQL or HTTP
A customer-service agent gets a runsql tool or a generic httprequest tool "so it can handle anything".
mediumpractitioner heuristic - Make tools ask for missing inputs instead of acting on values the model filled in
A user says "rebook me on the next flight" without a date or booking reference, and the rebooking tool requires both.
mediumpractitioner heuristic - Require human approval for irreversible actions, showing the exact action rather than the model's summary
An agent can issue refunds, send emails to customers, cancel bookings or change prices.
mediumpractitioner heuristic - Make side-effecting tools idempotent so retries cannot double-act
A booking or payment tool call times out, and the agent or the client library retries it.
mediumpractitioner heuristic - Persist agent state as explicit records, not only as a growing transcript
A long-running agent (hours, many tool calls) keeps all of its progress inside the conversation history.
mediumpractitioner heuristic - Score agents on the end state they produce, not on how the transcript reads
Agent evaluation consists of reading transcripts or asking a judge model whether the conversation "looks successful".
mediumdocumented fact - Run each agent test several times and report consistency, not one lucky pass
An agent passes a test task once in a demo and the task is marked solved.
mediumempirical finding - Test agents against simulated tools in a sandbox before they touch real systems
An agent is ready for a pilot, and its only testing so far has been against live APIs with real customer records.
mediumpractitioner heuristic - Prove one agent is insufficient before splitting work across several agents
A design proposes separate planner, researcher, critic and executor agents for a task one agent has not yet attempted.
mediumopinion - Treat retrieved and tool-returned text as data, and enforce limits outside the model
A RAG assistant or agent reads emails, web pages, tickets or uploaded files written by people you do not control.
mediumdocumented fact - A schema-valid tool call can still be unauthorised, so authorise every call downstream
Structured outputs or strict tool schemas guarantee well-formed JSON, and the team treats that as the safety check.
mediumdocumented fact - Run model-written code in a disposable sandbox with no network and no secrets
An analytics assistant writes and executes Python or SQL to answer questions about uploaded files or data.
mediumpractitioner heuristic - Let tools act with the requesting user's permissions, not a shared superuser account
All agent tools connect to backends with one service account that can read and write everything.
mediumdocumented fact - Keep a prompt-injection regression suite and report attack success next to task utility
Security testing of an LLM feature was a one-off manual red-team session before launch.
mediumempirical finding - Capture prompt and response content in traces only by explicit opt-in with redaction
You add tracing to an LLM application and the default plan logs full prompts, retrieved documents and outputs for every request.
mediumdocumented fact - Give each request one deadline and pass the remaining time to every downstream call
Retrieval, reranking, the model call and tool calls each have their own fixed timeout, or none.
mediumdocumented fact - Retry LLM calls at exactly one layer, with jittered backoff and respect for retry-after
Your code wraps the provider SDK call in its own retry loop, and a job queue also retries failed jobs.
mediumdocumented fact - Keep a tested fallback model path for provider overloads and outages
A customer-facing assistant depends on one hosted model, and the only failure plan is "retry".
mediumpractitioner heuristic - Prefer exact-match response caching, and scope any semantic cache very narrowly
To save cost, the team proposes caching answers and serving them for new questions whose embeddings are similar.
mediumpractitioner heuristic - Handle errors that arrive mid-stream, after the response already started
The application streams model output to users and treats the initial HTTP 200 as success.
mediumdocumented fact - Budget GenAI features by cost per successful task, not price per token
A business case multiplies expected requests by the list price per million tokens and calls that the cost.
mediumpractitioner heuristic - Expect multi-turn chat input cost to grow faster than the number of turns
A chat assistant resends the full conversation history on every turn, and the cost model assumes a fixed price per message.
mediumpractitioner heuristic - Order prompts stable-first so provider prompt caching can actually hit
Requests share a long system prompt, tool definitions or a reference document, but the cache-hit rate is near zero.
mediumdocumented fact - Route easy requests to a small model and escalate only when a check fails
All traffic goes to the largest available model, although most requests are simple lookups or classifications.
mediumempirical finding - Pick model configurations from the quality-cost frontier, not by model size
Model selection asks "which is the best model?" and defaults to the largest one available.
mediumpractitioner heuristic - Measure time-to-first-token and per-token generation time separately
A latency SLO is defined only on total response time for a streamed chat assistant.
mediumdocumented fact - Plan model API capacity in tokens per minute, and count what your provider counts
Capacity planning for a launch estimates requests per minute only, using average prompt sizes.
mediumdocumented fact - Test system-versus-user conflicts directly instead of assuming the system prompt wins
A fare-quote assistant's system prompt forbids quoting refund amounts, while users and pasted emails tell it to ignore earlier instructions.
mediumempirical finding - Measure per-rule compliance as the rule count in one prompt grows, and split dense prompts
A claims-reply prompt has grown to about 60 rules through patches, and each new rule was checked alone on a handful of examples.
mediumempirical finding - Pick few-shot examples by similarity to each request and keep the fixed set as the baseline
A classifier for passenger messages uses the same six hand-picked examples for every request, and accuracy is uneven across message types.
mediumempirical finding - Sweep the number of demonstrations against cost before settling on a shot count
A team uses five demonstrations by habit, although the context window would hold hundreds of labelled fare-rule examples.
mediumempirical finding - Restate the full requirement set in one message instead of revealing it across turns
An itinerary assistant receives a trip request in pieces: dates in turn one, budget in turn three, a baggage constraint in turn five.
mediumempirical finding - Order schema fields so evidence and rationale are written before the verdict field
A refund-eligibility extractor returns JSON whose first key is eligible, followed by reason and clausescited.
mediumpractitioner heuristic - Find where each schema constraint is enforced, because helpers may only move it into the description
A Pydantic or Zod model declares a fare between 0 and 5,000 and at most five itinerary items, and the team assumes provider-enforced schemas guarantee both.
mediumdocumented fact - Ask a schema for a short explanation, not for step-by-step reasoning
An extraction schema includes a stepbystepreasoning property so reviewers can audit decisions, and a few percent of requests come back as refusals.
mediumdocumented fact - Track refusals on a benign in-domain prompt set for every model or prompt change
A safety-tuned model update ships to an airline help assistant, and the tests cover only prompts that should be refused.
mediumempirical finding - Score tool calls on argument match and on executed outcome, since each alone misleads
A flight-change agent is evaluated by exact string comparison of its tool-call JSON against a reference call.
mediumdocumented fact - Hold per-request toggles constant inside a cached prefix, because they invalidate it
A support agent sets toolchoice per turn, attaches an image on some turns and varies its thinking settings, and the cache hit rate is far below what prompt ordering predicts.
mediumdocumented fact - Allocate the context window as a ledger with per-section ceilings and a fixed eviction order
A booking assistant concatenates system rules, a user profile, conversation history, retrieved fare rules and tool results, and requests intermittently overflow or lose the rules.
mediumpractitioner heuristic - Compress prompts only after checking that identifiers, numbers and negations survive
A cost review proposes a prompt-compression library to shrink long retrieved policy text by a factor of five.
mediumempirical finding - Evaluate instruction language and user language as separate factors before translating the prompt
A Turkish-language assistant's system prompt is translated into Turkish because the model should be addressed in its users' language.
mediumempirical finding - Gate an embedding or chunking change on logged-query replay, and read the queries whose top results changed most
A candidate index (new embedding model, new chunker or new analyzer) beats the current one on the golden set and the team wants to switch.
mediumpractitioner heuristic - Measure approximate-index recall against exact search on your own vectors before tuning anything else
Retrieval quality is poor after moving from brute-force search to an HNSW or IVF index, and the team starts changing models and prompts.
mediumpractitioner heuristic - Shorten embeddings only with a model that supports it, renormalise, and measure recall before taking the savings
Vector storage and ANN memory dominate the retrieval bill, and someone proposes cutting every embedding to its first 256 dimensions.
mediumempirical finding - Retrieve small passages for matching, then expand to the parent section for the generator
Long sections embed poorly because they mix several topics, but short chunks hand the generator half a rule without its conditions.
mediumpractitioner heuristic - Give the lexical branch a language-specific analyzer for agglutinative or multilingual corpora
A Turkish and English support corpus uses a default English analyzer for BM25, and customers write the same word with different suffixes.
mediumpractitioner heuristic - Extract metadata filters from the query with a schema, and fall back to unfiltered search when they return nothing
Users type "IST-FRA business fares after March" and the index has route, cabin and effective-date fields that semantic search ignores.
mediumpractitioner heuristic - Rewrite follow-up chat turns into standalone queries, and keep the original for fusion
A chat assistant embeds the raw user message, and a follow-up such as "what about the child fare?" arrives with no route or product named.
mediumpractitioner heuristic - Set the abstain threshold from a coverage-versus-error curve on labelled answerable and unanswerable questions
A RAG assistant refuses when the top retrieval score is below a threshold someone picked by eye, and product wants fewer refusals.
mediumempirical finding - Give the retrieval path a per-stage latency budget, and gate the stage that dominates
Time to first token is over target, and the team debates faster vector search while the path also contains an LLM query rewrite and a reranker.
mediumpractitioner heuristic - Choose rerank depth from the recall-at-N curve against reranker latency, not from a round number
The reranker scores the top 100 candidates because that was the tutorial default, and latency is over budget.
mediumpractitioner heuristic - Prove deletion end to end with a canary document that you delete and then probe for
A customer record or a withdrawn policy must disappear from the assistant, and the team deleted the source file and the main vector row.
mediumpractitioner heuristic - Key answer caches on index version or document version, so a document update invalidates stale answers
A response cache keyed on the normalised question makes a popular FAQ cheap, and the underlying policy document then changes.
mediumpractitioner heuristic - Resolve group membership at query time, because access lists copied into chunk metadata go stale
Each chunk carries an allowed-groups list copied from the source system at ingestion, and the retriever filters on the user's groups from the session.
mediumpractitioner heuristic - Treat stored embeddings as sensitive data, not as an anonymised derivative of the text
A team keeps embeddings of customer messages in a shared vector service, reasoning that vectors of numbers are not readable text.
mediumempirical finding - Count unjudged retrieved passages as unknown, and judge a new retriever's top-k before comparing it
The golden set's relevant passages were labelled from the old retriever's top 10, and a new retriever is scored against it.
mediumdocumented fact - Validate an LLM relevance labeller against chance-corrected agreement with human labels before it expands the golden set
Labelling passages for retrieval evaluation is slow, and the team wants an LLM to grade query-passage pairs at scale.
mediumempirical finding - Do not build the retrieval eval set only from questions generated out of the chunks
To get 500 labelled queries quickly, an LLM writes a question for each chunk, and that chunk becomes the gold answer.
mediumpractitioner heuristic - Add hard-negative passages to RAG tests, such as the superseded rule or the neighbouring route
RAG tests give the generator only the gold passage plus unrelated filler, and the system passes, then fails in production on look-alike text.
mediumempirical finding