Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Count tokens per language before sizing prompts, budgets and context limits
An airline assistant must answer customer messages in English, Turkish, Arabic and German, and the team sized its context and cost budget from English test prompts.
easyempirical finding - Test effective context at your own input lengths instead of trusting the advertised window
A team plans to paste a whole ground-handling contract (a hypothetical 300 pages) into one prompt because it fits inside the documented context window.
mediumempirical finding - Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Separate total from active parameters when sizing mixture-of-experts models
A team compares a mixture-of-experts (MoE) model with a dense model for self-hosting using only the headline parameter count.
mediumdocumented fact - Estimate key-value cache memory before promising long contexts on self-hosted models
An ML engineer must serve a self-hosted model with long contexts to many concurrent users on a fixed GPU budget.
harddocumented fact - For fixed-label text classification, benchmark a small encoder before a generative LLM
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
mediumpractitioner heuristic - Match checkpoint type and chat template to the job before judging an open model
An engineer downloads open model weights, sends plain instructions and gets rambling, off-task text.
easydocumented fact - Set decoding parameters per task instead of one global temperature
One shared configuration sets a high temperature for every LLM call, from marketing copy to fare-rule extraction.
easyempirical finding - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Use majority voting over sampled answers for verifiable reasoning tasks, and price it
An LLM computes refund amounts from multi-step fare rules and is wrong on a noticeable share of eval cases with a single chain-of-thought answer.
mediumempirical finding - Check the stop reason before parsing, and treat truncated output as a failure
A batch job extracts structured itinerary changes and occasionally receives JSON that ends mid-object.
easydocumented fact - Use disagreement across samples as a hallucination flag when no reference exists
An assistant drafts airport transit guidance for which no structured source of truth exists yet.
mediumempirical finding - Replace "check your answer" prompts with an external verifier
To cut errors, a team adds a second prompt asking the model to review and fix its own fare calculation.
mediumempirical finding - Verify every identifier the model emits against the system of record
An assistant drafts rebooking notes that include flight numbers, fare basis codes and policy section numbers.
easypractitioner heuristic - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic - Use provider-enforced schemas instead of asking for JSON in the prompt
A pipeline extracts passenger name, flight and requested change from emails by asking the model to reply in JSON, then parses with a try/except.
easydocumented fact - Validate business rules after the schema passes, because schemas only check shape
A structured-output extractor for refund requests returns perfectly valid objects, and the team stops checking them.
easypractitioner heuristic - Give every extracted field an explicit unknown value so the model is not forced to invent
An extraction schema requires a booking reference, a fare class and a travel date for every complaint email.
easypractitioner heuristic - Repair failed outputs with the validator's error message and a retry cap
A structured extraction fails business-rule validation on a few percent of documents, and an engineer adds code that silently coerces bad values into defaults.
easypractitioner heuristic - Separate instructions from pasted data with clear delimiters, but do not call it security
A prompt mixes instructions with a pasted customer email, and the model sometimes follows sentences inside the email as if they were instructions.
easypractitioner heuristic - Let the model reason before you force strict formats on reasoning-heavy tasks
A multi-step fare-difference calculation is moved into a strict JSON output mode, and accuracy on the eval set drops.
mediumempirical finding - Require a fine-tune to beat the best prompt on a frozen eval set before you own it
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
mediumpractitioner heuristic - Deduplicate fine-tuning data against the eval set before training
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
mediumpractitioner heuristic - Name the missing capability before choosing prompting, retrieval or fine-tuning
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
mediumpractitioner heuristic - Do not fine-tune to inject facts that change; retrieve them instead
A team fine-tunes a model on this quarter's fare rules so agents can ask about them without retrieval.
mediumempirical finding - Fine-tune for consistent behaviour that long prompts cannot hold
A prompt for drafting delay notices needs a long style guide and many examples, yet tone and structure still drift across thousands of daily messages.
mediumpractitioner heuristic - Curate a small, consistent fine-tuning set before collecting a large noisy one
A team plans to fine-tune on 200,000 historical agent replies exported without review.
mediumempirical finding - Rerun safety and refusal tests after every fine-tune, even on benign data
A team fine-tunes an aligned model on harmless customer-service transcripts and evaluates only task quality.
mediumempirical finding - Consider continued pretraining only when you hold a large unlabelled domain corpus
An insurer's claims text is full of domain jargon, and a general model handles it poorly even after prompt work.
hardempirical finding - Anchor synthetic training data to real data and track its provenance
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
mediumempirical finding - Do the LoRA parameter arithmetic before planning hardware
A team asks whether LoRA lets them fine-tune a model on the GPU they already have.
mediumdocumented fact - Expect LoRA to learn less and forget less than full fine-tuning
A team uses LoRA to teach a model a large new skill domain, such as a proprietary programming language, and gains are small.
hardempirical finding - Tune learning rate and target modules before raising LoRA rank
A LoRA fine-tune underperforms and the first reaction is to raise the rank from 8 to 128.
mediumpractitioner heuristic - Use QLoRA to fit a larger base model, then evaluate the quantized result honestly
The model size that best fits quality targets cannot be fine-tuned in 16-bit on the available GPU.
mediumempirical finding - Evaluate the exact artefact you will serve, not the training checkpoint
A LoRA adapter scores well during training, then is merged into the base weights and quantized for serving without re-evaluation.
easypractitioner heuristic - Judge a compressed model on your tasks, not on perplexity alone
A self-hosted model is quantized or pruned to cut serving memory, and the team signs off because perplexity barely moved.
mediumempirical finding - Budget about 16 bytes per parameter before activations for full fine-tuning with Adam
A team asks whether full fine-tuning of a 7-billion-parameter model fits on one 80 GB GPU.
mediumdocumented fact - Choose models on a task-specific eval set, not on public leaderboard rank
A team picks the model at the top of a public leaderboard for a booking-change assistant.
easypractitioner heuristic - Check everything verifiable with code before reaching for an LLM judge
An eval pipeline sends every output to an LLM judge, including checks such as valid JSON, word limits and whether the booking reference matches the input.
easypractitioner heuristic - Turn every production failure into a regression test that gates changes
A model upgrade reintroduces a bug that was fixed months ago, quoting outdated pet-travel rules.
easypractitioner heuristic - Locate RLHF failures in feedback, reward model or policy before changing the recipe
After preference tuning, an internal assistant becomes evasive on some topics and overconfident on others, and the team considers switching alignment methods.
hardpractitioner heuristic - Track a gold metric while optimising against a reward model, and stop before it turns down
During RL fine-tuning, the reward-model score keeps rising run after run, and the team keeps training.
hardempirical finding - Choose DPO for simplicity, but check for distribution shift in the preference data
A team with a fixed preference dataset picks DPO because it avoids training a reward model and running reinforcement learning.
hardempirical finding - Measure output diversity after preference tuning when variety is part of the product
A preference-tuned model generates marketing copy for 300 routes, and the drafts start to sound alike.
mediumempirical finding - Check whether a preference-tuning gain is just longer answers
A preference-tuned model wins more human and model comparisons, and its answers are noticeably longer than before.
mediumempirical finding - Re-measure core capabilities after alignment tuning to catch an alignment tax
A model tuned for helpfulness and tone on support conversations is promoted after winning preference comparisons.
mediumempirical finding - Pick a generative model family by the constraint that binds, not by popularity
A retailer wants synthetic product images for layout testing and assumes the newest diffusion model is the only option.
mediumdocumented fact - Sweep guidance strength as a fidelity-versus-diversity knob, like temperature
A team generating synthetic training images sets a high guidance scale because individual samples look sharpest.
mediumempirical finding - Report quality and coverage separately when evaluating generated samples
Two image generators are compared by a single FID score, and the lower score wins.
mediumdocumented fact - Do not assume generated data is private; test it for memorised training records
A team proposes sharing synthetic images or records produced by a generative model trained on customer data, on the grounds that the data is synthetic.
mediumempirical finding - Chunk along the document's own structure before tuning chunk size
You are ingesting manuals, policies or contracts that have headings, numbered clauses or tables into a RAG index.
easypractitioner heuristic - Give each chunk the context it lost when it was cut out of the document
Many chunks say things like "this fee is waived" or "revenue grew 3%" without naming the product, route or period they refer to.
mediumempirical finding - Enforce validity dates and scope with metadata filters, not with the prompt
The corpus holds several versions of the same policy, for example baggage rules effective before and after 2024-03-01, or tariffs per country.
mediumpractitioner heuristic - Apply document permissions inside the retriever, before the model sees anything
An internal assistant indexes documents with different audiences, such as HR cases, board papers and public FAQs, in one vector store.
mediumpractitioner heuristic - Keep a BM25 baseline when queries contain codes, identifiers or rare terms
Users search with fare basis codes, error codes, part numbers or policy clause ids, and the team plans a dense-only retriever.
easyempirical finding - Fuse lexical and vector results by rank, not by adding raw scores
You combine BM25 and embedding search into a hybrid retriever and need one ranked list.
mediumempirical finding - Choose the embedding model on your own labelled queries, not a public leaderboard
The team picks an embedding model because it tops a public retrieval benchmark.
mediumempirical finding - Retrieve wide with a cheap retriever, then rerank narrow with a cross-encoder
The right passage is often in the top 50 but rarely in the top 5 you pass to the model.
mediumempirical finding - Stop adding retrieved passages once answer quality stops rising
Answers miss facts, and the proposed fix is to raise k from 20 to 50 because the context window allows it.
mediumempirical finding - Put the strongest evidence at the start of the context, not buried in the middle
Context assembly concatenates passages in arbitrary order, for example by document id or ingestion time.
easyempirical finding - Drop weakly relevant chunks instead of always filling k slots
Retrieval always returns exactly k chunks, even for questions where only one or none is relevant.
mediumempirical finding - Deduplicate near-identical chunks before they crowd out other evidence
The corpus contains copies of the same policy in several places: the PDF, an intranet page and an email announcement.
easypractitioner heuristic - Validate citation ids in code before showing a cited answer
Your RAG prompt asks the model to cite sources as [doc-id] markers.
easyempirical finding - Report recall@k at the k you actually pass to the model
A retrieval dashboard shows recall@100, while the generation prompt receives the top 5 chunks.
easydocumented fact - Locate RAG failures with a retrieval-by-answer two-way table
A RAG assistant gives wrong answers on some questions and the team debates whether to change the retriever or the prompt.
mediumpractitioner heuristic - Run RAG ablations one component at a time with per-stage metrics
A new RAG version changes the chunker, the embedding model and the prompt at once, and end-to-end accuracy improves.
mediumpractitioner heuristic - Decompose multi-hop questions instead of hoping one retrieval finds everything
Questions combine facts from different documents, such as "which of our A321 routes are affected by the new crew-rest rule?"
hardpractitioner heuristic - Monitor the ingestion pipeline, because parse failures drop evidence silently
A RAG index is rebuilt nightly from PDFs, HTML and spreadsheets owned by other teams.
mediumpractitioner heuristic - Write the workflow in code when the steps are known, and use the model only at fuzzy steps
A team proposes an autonomous agent for a process whose steps are already documented, such as refund eligibility checks or disruption rebooking.
easypractitioner heuristic - Give every agent run hard budgets and explicit stopping conditions
An agent loop runs "until the task is done" with no limit on iterations, tokens, spend or wall-clock time.
easypractitioner heuristic - Expose only the tools a task needs, as an explicit allow-list per route
One agent configuration registers every available tool (email, database, payments, file system) for all requests.
easydocumented fact - Prefer narrow, purpose-built tools over generic ones like raw SQL or HTTP
A customer-service agent gets a runsql tool or a generic httprequest tool "so it can handle anything".
mediumpractitioner heuristic - Make tools ask for missing inputs instead of acting on values the model filled in
A user says "rebook me on the next flight" without a date or booking reference, and the rebooking tool requires both.
mediumpractitioner heuristic - Require human approval for irreversible actions, showing the exact action rather than the model's summary
An agent can issue refunds, send emails to customers, cancel bookings or change prices.
mediumpractitioner heuristic - Make side-effecting tools idempotent so retries cannot double-act
A booking or payment tool call times out, and the agent or the client library retries it.
mediumpractitioner heuristic - Return structured, classified tool errors so the agent can recover or stop
Tool failures reach the model as raw stack traces or a generic "error occurred".
easypractitioner heuristic - Persist agent state as explicit records, not only as a growing transcript
A long-running agent (hours, many tool calls) keeps all of its progress inside the conversation history.
mediumpractitioner heuristic - Treat writes to long-term agent memory as untrusted input with provenance
An assistant saves "facts" or preferences from conversations, documents or web pages into persistent memory reused in later sessions.
hardpractitioner heuristic - Score agents on the end state they produce, not on how the transcript reads
Agent evaluation consists of reading transcripts or asking a judge model whether the conversation "looks successful".
mediumdocumented fact - Report steps, tool calls and tokens per task alongside agent success rate
Two agent designs reach similar task success, and the comparison stops there.
easypractitioner heuristic - Test agents against simulated tools in a sandbox before they touch real systems
An agent is ready for a pilot, and its only testing so far has been against live APIs with real customer records.
mediumpractitioner heuristic - Prove one agent is insufficient before splitting work across several agents
A design proposes separate planner, researcher, critic and executor agents for a task one agent has not yet attempted.
mediumopinion - Write tool descriptions like an interface spec and test them with varied inputs
Tools are registered with one-line descriptions such as "search bookings", and the agent misuses them.
easypractitioner heuristic - Cap and shape tool results before they enter the context window
A tool returns a full API payload, a 2,000-row query result or an entire web page to the agent.
easypractitioner heuristic - Treat retrieved and tool-returned text as data, and enforce limits outside the model
A RAG assistant or agent reads emails, web pages, tickets or uploaded files written by people you do not control.
mediumdocumented fact - Assume the system prompt will leak, so never put secrets or access rules only there
A system prompt contains API keys, internal URLs, discount codes, or the only statement of which users may see what.
easydocumented fact - Close exfiltration channels such as auto-loaded links and images in model output
The chat interface renders model output as Markdown, including images and links to arbitrary domains, and the model can see private data.
hardpractitioner heuristic - Keep the model that reads untrusted content away from tools and private data
One agent reads incoming customer emails, can query customer records, and can send messages externally.
harddocumented fact - Fix the plan before reading untrusted data when the task allows it
An agent must process external content, for example summarise today's supplier emails into a report, with a predictable set of actions.
harddocumented fact - Encode or parameterise model output before it reaches SQL, HTML or a shell
Generated text is inserted into a web page, a database query, a template or a command line.
easydocumented fact - A schema-valid tool call can still be unauthorised, so authorise every call downstream
Structured outputs or strict tool schemas guarantee well-formed JSON, and the team treats that as the safety check.
mediumdocumented fact - Run model-written code in a disposable sandbox with no network and no secrets
An analytics assistant writes and executes Python or SQL to answer questions about uploaded files or data.
mediumpractitioner heuristic - Let tools act with the requesting user's permissions, not a shared superuser account
All agent tools connect to backends with one service account that can read and write everything.
mediumdocumented fact - Keep a prompt-injection regression suite and report attack success next to task utility
Security testing of an LLM feature was a one-off manual red-team session before launch.
mediumempirical finding - Run GenAI design reviews against a shared threat checklist, not ad-hoc brainstorming
A GenAI feature reaches design review and the security section says "prompt injection mitigated by system prompt".
easydocumented fact - Set per-user quotas and spend limits as a security control, not only as finance hygiene
A public or partner-facing LLM endpoint has no per-user request, token or cost limits.
easydocumented fact - Capture prompt and response content in traces only by explicit opt-in with redaction
You add tracing to an LLM application and the default plan logs full prompts, retrieved documents and outputs for every request.
mediumdocumented fact - Give each request one deadline and pass the remaining time to every downstream call
Retrieval, reranking, the model call and tool calls each have their own fixed timeout, or none.
mediumdocumented fact - Retry LLM calls at exactly one layer, with jittered backoff and respect for retry-after
Your code wraps the provider SDK call in its own retry loop, and a job queue also retries failed jobs.
mediumdocumented fact - Branch error handling on the error type, because two 429s can need opposite responses
All non-200 responses from the model API go through one generic "wait and retry" handler.
easydocumented fact - Trace each LLM request as spans per stage with model, tokens and finish reason
A user reports a slow or wrong answer, and logs show only the final response and total latency.
easydocumented fact - Version prompt, retrieval settings and model together as one release
Prompts live in code, retrieval parameters in a config service and the model name in an environment variable, each changed independently.
easypractitioner heuristic - Pin dated model versions and schedule migrations before retirement dates
Production calls a moving model alias, or a pinned model that the provider has deprecated.
easydocumented fact - Keep a tested fallback model path for provider overloads and outages
A customer-facing assistant depends on one hosted model, and the only failure plan is "retry".
mediumpractitioner heuristic - Prefer exact-match response caching, and scope any semantic cache very narrowly
To save cost, the team proposes caching answers and serving them for new questions whose embeddings are similar.
mediumpractitioner heuristic - Handle errors that arrive mid-stream, after the response already started
The application streams model output to users and treats the initial HTTP 200 as success.
mediumdocumented fact - Expect multi-turn chat input cost to grow faster than the number of turns
A chat assistant resends the full conversation history on every turn, and the cost model assumes a fixed price per message.
mediumpractitioner heuristic - Order prompts stable-first so provider prompt caching can actually hit
Requests share a long system prompt, tool definitions or a reference document, but the cache-hit rate is near zero.
mediumdocumented fact - Send offline workloads such as eval runs and nightly enrichment through a batch API
Golden-set evaluations, document classification backfills or nightly summaries run through the real-time API with custom concurrency code.
easydocumented fact - Route easy requests to a small model and escalate only when a check fails
All traffic goes to the largest available model, although most requests are simple lookups or classifications.
mediumempirical finding - Pick model configurations from the quality-cost frontier, not by model size
Model selection asks "which is the best model?" and defaults to the largest one available.
mediumpractitioner heuristic - Measure time-to-first-token and per-token generation time separately
A latency SLO is defined only on total response time for a streamed chat assistant.
mediumdocumented fact - Plan model API capacity in tokens per minute, and count what your provider counts
Capacity planning for a launch estimates requests per minute only, using average prompt sizes.
mediumdocumented fact - Test system-versus-user conflicts directly instead of assuming the system prompt wins
A fare-quote assistant's system prompt forbids quoting refund amounts, while users and pasted emails tell it to ignore earlier instructions.
mediumempirical finding - Measure per-rule compliance as the rule count in one prompt grows, and split dense prompts
A claims-reply prompt has grown to about 60 rules through patches, and each new rule was checked alone on a handful of examples.
mediumempirical finding - Sweep the number of demonstrations against cost before settling on a shot count
A team uses five demonstrations by habit, although the context window would hold hundreds of labelled fare-rule examples.
mediumempirical finding - Put long documents first and the question last, then test the reverse on your own task
A contract-review prompt places the instructions and question at the top and pastes 150 pages of terms underneath.
easyempirical finding - Have the model quote the relevant evidence before it answers from a long input
Retrieval is perfect in a 40,000-token prompt, yet multi-step answers over the supplied evidence are worse than on a trimmed version of the same prompt.
hardempirical finding - Restate the full requirement set in one message instead of revealing it across turns
An itinerary assistant receives a trip request in pieces: dates in turn one, budget in turn three, a baggage constraint in turn five.
mediumempirical finding - Order schema fields so evidence and rationale are written before the verdict field
A refund-eligibility extractor returns JSON whose first key is eligible, followed by reason and clausescited.
mediumpractitioner heuristic