Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Test effective context at your own input lengths instead of trusting the advertised window
A team plans to paste a whole ground-handling contract (a hypothetical 300 pages) into one prompt because it fits inside the documented context window.
mediumempirical finding - Separate total from active parameters when sizing mixture-of-experts models
A team compares a mixture-of-experts (MoE) model with a dense model for self-hosting using only the headline parameter count.
mediumdocumented fact - For fixed-label text classification, benchmark a small encoder before a generative LLM
A team wants a large generative model to tag two million complaint emails a month into 12 fixed categories.
mediumpractitioner heuristic - Use majority voting over sampled answers for verifiable reasoning tasks, and price it
An LLM computes refund amounts from multi-step fare rules and is wrong on a noticeable share of eval cases with a single chain-of-thought answer.
mediumempirical finding - Use disagreement across samples as a hallucination flag when no reference exists
An assistant drafts airport transit guidance for which no structured source of truth exists yet.
mediumempirical finding - Replace "check your answer" prompts with an external verifier
To cut errors, a team adds a second prompt asking the model to review and fix its own fare calculation.
mediumempirical finding - Let the model reason before you force strict formats on reasoning-heavy tasks
A multi-step fare-difference calculation is moved into a strict JSON output mode, and accuracy on the eval set drops.
mediumempirical finding - Require a fine-tune to beat the best prompt on a frozen eval set before you own it
A team proposes fine-tuning a model for disruption-notice drafting after two weeks of prompt work.
mediumpractitioner heuristic - Deduplicate fine-tuning data against the eval set before training
Fine-tuning data is exported from the same ticket archive that supplied the eval set.
mediumpractitioner heuristic - Name the missing capability before choosing prompting, retrieval or fine-tuning
A stakeholder asks for the model to be fine-tuned on company data because answers about baggage policy are wrong.
mediumpractitioner heuristic - Do not fine-tune to inject facts that change; retrieve them instead
A team fine-tunes a model on this quarter's fare rules so agents can ask about them without retrieval.
mediumempirical finding - Fine-tune for consistent behaviour that long prompts cannot hold
A prompt for drafting delay notices needs a long style guide and many examples, yet tone and structure still drift across thousands of daily messages.
mediumpractitioner heuristic - Curate a small, consistent fine-tuning set before collecting a large noisy one
A team plans to fine-tune on 200,000 historical agent replies exported without review.
mediumempirical finding - Rerun safety and refusal tests after every fine-tune, even on benign data
A team fine-tunes an aligned model on harmless customer-service transcripts and evaluates only task quality.
mediumempirical finding - Anchor synthetic training data to real data and track its provenance
To save labelling cost, a team trains each new model version on outputs generated by the previous version.
mediumempirical finding - Do the LoRA parameter arithmetic before planning hardware
A team asks whether LoRA lets them fine-tune a model on the GPU they already have.
mediumdocumented fact - Tune learning rate and target modules before raising LoRA rank
A LoRA fine-tune underperforms and the first reaction is to raise the rank from 8 to 128.
mediumpractitioner heuristic - Use QLoRA to fit a larger base model, then evaluate the quantized result honestly
The model size that best fits quality targets cannot be fine-tuned in 16-bit on the available GPU.
mediumempirical finding - Judge a compressed model on your tasks, not on perplexity alone
A self-hosted model is quantized or pruned to cut serving memory, and the team signs off because perplexity barely moved.
mediumempirical finding - Budget about 16 bytes per parameter before activations for full fine-tuning with Adam
A team asks whether full fine-tuning of a 7-billion-parameter model fits on one 80 GB GPU.
mediumdocumented fact - Measure output diversity after preference tuning when variety is part of the product
A preference-tuned model generates marketing copy for 300 routes, and the drafts start to sound alike.
mediumempirical finding - Check whether a preference-tuning gain is just longer answers
A preference-tuned model wins more human and model comparisons, and its answers are noticeably longer than before.
mediumempirical finding - Re-measure core capabilities after alignment tuning to catch an alignment tax
A model tuned for helpfulness and tone on support conversations is promoted after winning preference comparisons.
mediumempirical finding - Pick a generative model family by the constraint that binds, not by popularity
A retailer wants synthetic product images for layout testing and assumes the newest diffusion model is the only option.
mediumdocumented fact - Sweep guidance strength as a fidelity-versus-diversity knob, like temperature
A team generating synthetic training images sets a high guidance scale because individual samples look sharpest.
mediumempirical finding - Report quality and coverage separately when evaluating generated samples
Two image generators are compared by a single FID score, and the lower score wins.
mediumdocumented fact - Do not assume generated data is private; test it for memorised training records
A team proposes sharing synthetic images or records produced by a generative model trained on customer data, on the grounds that the data is synthetic.
mediumempirical finding - Give each chunk the context it lost when it was cut out of the document
Many chunks say things like "this fee is waived" or "revenue grew 3%" without naming the product, route or period they refer to.
mediumempirical finding - Enforce validity dates and scope with metadata filters, not with the prompt
The corpus holds several versions of the same policy, for example baggage rules effective before and after 2024-03-01, or tariffs per country.
mediumpractitioner heuristic - Apply document permissions inside the retriever, before the model sees anything
An internal assistant indexes documents with different audiences, such as HR cases, board papers and public FAQs, in one vector store.
mediumpractitioner heuristic - Fuse lexical and vector results by rank, not by adding raw scores
You combine BM25 and embedding search into a hybrid retriever and need one ranked list.
mediumempirical finding - Choose the embedding model on your own labelled queries, not a public leaderboard
The team picks an embedding model because it tops a public retrieval benchmark.
mediumempirical finding - Retrieve wide with a cheap retriever, then rerank narrow with a cross-encoder
The right passage is often in the top 50 but rarely in the top 5 you pass to the model.
mediumempirical finding - Stop adding retrieved passages once answer quality stops rising
Answers miss facts, and the proposed fix is to raise k from 20 to 50 because the context window allows it.
mediumempirical finding - Drop weakly relevant chunks instead of always filling k slots
Retrieval always returns exactly k chunks, even for questions where only one or none is relevant.
mediumempirical finding - Locate RAG failures with a retrieval-by-answer two-way table
A RAG assistant gives wrong answers on some questions and the team debates whether to change the retriever or the prompt.
mediumpractitioner heuristic - Run RAG ablations one component at a time with per-stage metrics
A new RAG version changes the chunker, the embedding model and the prompt at once, and end-to-end accuracy improves.
mediumpractitioner heuristic - Monitor the ingestion pipeline, because parse failures drop evidence silently
A RAG index is rebuilt nightly from PDFs, HTML and spreadsheets owned by other teams.
mediumpractitioner heuristic - Prefer narrow, purpose-built tools over generic ones like raw SQL or HTTP
A customer-service agent gets a runsql tool or a generic httprequest tool "so it can handle anything".
mediumpractitioner heuristic - Make tools ask for missing inputs instead of acting on values the model filled in
A user says "rebook me on the next flight" without a date or booking reference, and the rebooking tool requires both.
mediumpractitioner heuristic - Require human approval for irreversible actions, showing the exact action rather than the model's summary
An agent can issue refunds, send emails to customers, cancel bookings or change prices.
mediumpractitioner heuristic - Make side-effecting tools idempotent so retries cannot double-act
A booking or payment tool call times out, and the agent or the client library retries it.
mediumpractitioner heuristic - Persist agent state as explicit records, not only as a growing transcript
A long-running agent (hours, many tool calls) keeps all of its progress inside the conversation history.
mediumpractitioner heuristic - Score agents on the end state they produce, not on how the transcript reads
Agent evaluation consists of reading transcripts or asking a judge model whether the conversation "looks successful".
mediumdocumented fact - Test agents against simulated tools in a sandbox before they touch real systems
An agent is ready for a pilot, and its only testing so far has been against live APIs with real customer records.
mediumpractitioner heuristic - Prove one agent is insufficient before splitting work across several agents
A design proposes separate planner, researcher, critic and executor agents for a task one agent has not yet attempted.
mediumopinion - Treat retrieved and tool-returned text as data, and enforce limits outside the model
A RAG assistant or agent reads emails, web pages, tickets or uploaded files written by people you do not control.
mediumdocumented fact - A schema-valid tool call can still be unauthorised, so authorise every call downstream
Structured outputs or strict tool schemas guarantee well-formed JSON, and the team treats that as the safety check.
mediumdocumented fact - Run model-written code in a disposable sandbox with no network and no secrets
An analytics assistant writes and executes Python or SQL to answer questions about uploaded files or data.
mediumpractitioner heuristic - Let tools act with the requesting user's permissions, not a shared superuser account
All agent tools connect to backends with one service account that can read and write everything.
mediumdocumented fact - Keep a prompt-injection regression suite and report attack success next to task utility
Security testing of an LLM feature was a one-off manual red-team session before launch.
mediumempirical finding - Capture prompt and response content in traces only by explicit opt-in with redaction
You add tracing to an LLM application and the default plan logs full prompts, retrieved documents and outputs for every request.
mediumdocumented fact - Give each request one deadline and pass the remaining time to every downstream call
Retrieval, reranking, the model call and tool calls each have their own fixed timeout, or none.
mediumdocumented fact - Retry LLM calls at exactly one layer, with jittered backoff and respect for retry-after
Your code wraps the provider SDK call in its own retry loop, and a job queue also retries failed jobs.
mediumdocumented fact - Keep a tested fallback model path for provider overloads and outages
A customer-facing assistant depends on one hosted model, and the only failure plan is "retry".
mediumpractitioner heuristic - Prefer exact-match response caching, and scope any semantic cache very narrowly
To save cost, the team proposes caching answers and serving them for new questions whose embeddings are similar.
mediumpractitioner heuristic - Handle errors that arrive mid-stream, after the response already started
The application streams model output to users and treats the initial HTTP 200 as success.
mediumdocumented fact - Expect multi-turn chat input cost to grow faster than the number of turns
A chat assistant resends the full conversation history on every turn, and the cost model assumes a fixed price per message.
mediumpractitioner heuristic - Order prompts stable-first so provider prompt caching can actually hit
Requests share a long system prompt, tool definitions or a reference document, but the cache-hit rate is near zero.
mediumdocumented fact - Route easy requests to a small model and escalate only when a check fails
All traffic goes to the largest available model, although most requests are simple lookups or classifications.
mediumempirical finding - Pick model configurations from the quality-cost frontier, not by model size
Model selection asks "which is the best model?" and defaults to the largest one available.
mediumpractitioner heuristic - Measure time-to-first-token and per-token generation time separately
A latency SLO is defined only on total response time for a streamed chat assistant.
mediumdocumented fact - Plan model API capacity in tokens per minute, and count what your provider counts
Capacity planning for a launch estimates requests per minute only, using average prompt sizes.
mediumdocumented fact - Test system-versus-user conflicts directly instead of assuming the system prompt wins
A fare-quote assistant's system prompt forbids quoting refund amounts, while users and pasted emails tell it to ignore earlier instructions.
mediumempirical finding - Measure per-rule compliance as the rule count in one prompt grows, and split dense prompts
A claims-reply prompt has grown to about 60 rules through patches, and each new rule was checked alone on a handful of examples.
mediumempirical finding - Sweep the number of demonstrations against cost before settling on a shot count
A team uses five demonstrations by habit, although the context window would hold hundreds of labelled fare-rule examples.
mediumempirical finding - Restate the full requirement set in one message instead of revealing it across turns
An itinerary assistant receives a trip request in pieces: dates in turn one, budget in turn three, a baggage constraint in turn five.
mediumempirical finding - Order schema fields so evidence and rationale are written before the verdict field
A refund-eligibility extractor returns JSON whose first key is eligible, followed by reason and clausescited.
mediumpractitioner heuristic - Find where each schema constraint is enforced, because helpers may only move it into the description
A Pydantic or Zod model declares a fare between 0 and 5,000 and at most five itinerary items, and the team assumes provider-enforced schemas guarantee both.
mediumdocumented fact - Ask a schema for a short explanation, not for step-by-step reasoning
An extraction schema includes a stepbystepreasoning property so reviewers can audit decisions, and a few percent of requests come back as refusals.
mediumdocumented fact - Score tool calls on argument match and on executed outcome, since each alone misleads
A flight-change agent is evaluated by exact string comparison of its tool-call JSON against a reference call.
mediumdocumented fact - Hold per-request toggles constant inside a cached prefix, because they invalidate it
A support agent sets toolchoice per turn, attaches an image on some turns and varies its thinking settings, and the cache hit rate is far below what prompt ordering predicts.
mediumdocumented fact - Allocate the context window as a ledger with per-section ceilings and a fixed eviction order
A booking assistant concatenates system rules, a user profile, conversation history, retrieved fare rules and tool results, and requests intermittently overflow or lose the rules.
mediumpractitioner heuristic - Compress prompts only after checking that identifiers, numbers and negations survive
A cost review proposes a prompt-compression library to shrink long retrieved policy text by a factor of five.
mediumempirical finding - Gate an embedding or chunking change on logged-query replay, and read the queries whose top results changed most
A candidate index (new embedding model, new chunker or new analyzer) beats the current one on the golden set and the team wants to switch.
mediumpractitioner heuristic - Measure approximate-index recall against exact search on your own vectors before tuning anything else
Retrieval quality is poor after moving from brute-force search to an HNSW or IVF index, and the team starts changing models and prompts.
mediumpractitioner heuristic - Shorten embeddings only with a model that supports it, renormalise, and measure recall before taking the savings
Vector storage and ANN memory dominate the retrieval bill, and someone proposes cutting every embedding to its first 256 dimensions.
mediumempirical finding - Retrieve small passages for matching, then expand to the parent section for the generator
Long sections embed poorly because they mix several topics, but short chunks hand the generator half a rule without its conditions.
mediumpractitioner heuristic - Give the lexical branch a language-specific analyzer for agglutinative or multilingual corpora
A Turkish and English support corpus uses a default English analyzer for BM25, and customers write the same word with different suffixes.
mediumpractitioner heuristic - Extract metadata filters from the query with a schema, and fall back to unfiltered search when they return nothing
Users type "IST-FRA business fares after March" and the index has route, cabin and effective-date fields that semantic search ignores.
mediumpractitioner heuristic - Rewrite follow-up chat turns into standalone queries, and keep the original for fusion
A chat assistant embeds the raw user message, and a follow-up such as "what about the child fare?" arrives with no route or product named.
mediumpractitioner heuristic - Give the retrieval path a per-stage latency budget, and gate the stage that dominates
Time to first token is over target, and the team debates faster vector search while the path also contains an LLM query rewrite and a reranker.
mediumpractitioner heuristic - Choose rerank depth from the recall-at-N curve against reranker latency, not from a round number
The reranker scores the top 100 candidates because that was the tutorial default, and latency is over budget.
mediumpractitioner heuristic - Prove deletion end to end with a canary document that you delete and then probe for
A customer record or a withdrawn policy must disappear from the assistant, and the team deleted the source file and the main vector row.
mediumpractitioner heuristic - Key answer caches on index version or document version, so a document update invalidates stale answers
A response cache keyed on the normalised question makes a popular FAQ cheap, and the underlying policy document then changes.
mediumpractitioner heuristic - Resolve group membership at query time, because access lists copied into chunk metadata go stale
Each chunk carries an allowed-groups list copied from the source system at ingestion, and the retriever filters on the user's groups from the session.
mediumpractitioner heuristic - Treat stored embeddings as sensitive data, not as an anonymised derivative of the text
A team keeps embeddings of customer messages in a shared vector service, reasoning that vectors of numbers are not readable text.
mediumempirical finding - Check that each cited passage actually supports its claim, using an entailment check, not only that the id exists
A RAG answer shows inline citations, and the pipeline already verifies that every cited id is in the retrieved set.
mediumempirical finding - Do not accept the model's "done" as completion; run an independent completion check
A coding or operations agent reports that the task is finished after a few steps.
mediumpractitioner heuristic - Design long agent tasks as resumable jobs with checkpoints, not one long session
A research or reconciliation agent needs tens of minutes and dozens of tool calls, and a timeout or deploy kills it midway.
mediumpractitioner heuristic - Hand work across sessions through a fixed checklist the agent may tick but not rewrite
A multi-day task spans many context windows, and each new session must know what is done and what remains.
mediumpractitioner heuristic - Compact long agent runs at task boundaries, keeping decisions and open items, not raw tool output
An agent run approaches the context limit after many large tool results, and quality begins to drop.
mediumdocumented fact - Re-verify remembered facts against the system of record before acting on them
An assistant recalls from memory that a passenger prefers a window seat, or that a rebooking is pending, and uses it to act.
mediumpractitioner heuristic - Split a high-impact tool into preview and commit, with the commit bound to the previewed state
An agent can change many records at once, such as a bulk fare correction or mass rebooking.
mediumpractitioner heuristic - Treat a timed-out write as unknown and read back before retrying
A write tool, such as issuing a voucher, times out after the request left, and the agent is about to call it again.
mediumpractitioner heuristic - Reserve human approval for the few high-impact actions, because constant prompts get rubber-stamped
A team makes an agent ask for approval on every tool call to be safe, and reviewers click approve out of habit.
mediumpractitioner heuristic - Search a large tool catalogue on demand instead of loading every definition
An assistant connected to several internal systems exposes hundreds of tools, and tool choice is wrong or slow.
mediumdocumented fact - Put a circuit breaker on each tool so a failing dependency stops being called
A pricing tool starts returning errors, and the agent keeps retrying it across many steps and many concurrent runs.
mediumdocumented fact - Record every tool request and response so a failed run can be replayed offline
An agent failed on a customer case last Tuesday, and the engineer cannot reproduce it because the live systems have changed.
mediumpractitioner heuristic - Give a sandboxed agent a domain allow-list for egress instead of all-or-nothing network access
A coding agent in a sandbox needs to install packages, but the team has either cut the network entirely or opened it fully.
mediumdocumented fact - Make cancel a first-class operation that stops the loop, reaches running tools and reports what already happened
A user presses stop on a long agent run that has already issued several writes and has a slow search in flight.
mediumpractitioner heuristic - Test the judge with planted bad answers before you let it gate releases
Judge-human agreement looks fine on a random sample, but the answers that matter are the rare fluent-but-wrong ones.
mediumpractitioner heuristic - Treat the judge model and prompt as part of the metric, and re-baseline when either changes
The provider updates the judge model, or someone edits the judge prompt, and eval scores shift without any change to your system.
mediumpractitioner heuristic - Tier regression gates by cost and cache model outputs keyed by their full input
The full LLM eval takes 40 minutes and real money per run, so engineers skip it, or it runs on every commit and the bill and queue grow.
mediumpractitioner heuristic - Shadow a candidate on mirrored production requests before it serves any user
A candidate prompt or model beats the baseline offline, but offline items differ from live traffic in length, language mix and context.
mediumpractitioner heuristic - Map red-team coverage as a grid of risk category by input channel and count empty cells
A red-team report lists attacks that succeeded and failed, but nobody can say what was never tried.
mediumpractitioner heuristic - Report attack success rate together with the number of attempts the attacker was allowed
A red-team summary says "the model resisted 98% of attacks" after one attempt per prompt.
mediumempirical finding - Strip invisible characters and inspect extracted text before documents reach the model
An assistant summarises uploaded PDFs, emails and web pages, and a reviewer approves its output after skimming the rendered source.
mediumpractitioner heuristic - Treat third-party tool descriptions as untrusted input and pin approved versions
An agent connects to tool servers run by other teams or vendors, and their tool names, descriptions and schemas are loaded into the prompt.
mediumpractitioner heuristic - Gate URL-fetching tools with an egress allow-list instead of the model's judgement
A research or support agent has a fetch-URL tool and runs inside your cloud network.
mediumpractitioner heuristic - Mint short-lived, narrowly scoped credentials per agent run
An agent's tools authenticate with a long-lived API key that sits in an environment variable or config file the agent runtime can read.
mediumpractitioner heuristic - Cap the volume of side effects per run and per hour in the tool layer
An agent can refund, email or modify records, and every individual call passes its authorisation check.
mediumpractitioner heuristic - Scan model output for secrets and personal identifiers before it leaves the trust zone
Answers can quote retrieved documents, tool results and logs, and they go to end users, other systems or chat channels.
mediumpractitioner heuristic - Install only allow-listed packages when a model suggests code dependencies
A coding assistant or code-writing agent proposes imports, and a developer or pipeline runs the install.
mediumpractitioner heuristic - Redact personal data before the prompt leaves your boundary and keep the mapping local
Support tickets and call transcripts with names, emails and booking references go to a hosted model for summarising or classifying.
mediumpractitioner heuristic - Keep prompt logs in a separate store with a short TTL, narrow access and access auditing
Engineers need prompts and responses to debug, and the content flows into the general observability platform with long retention and broad read access.
mediumpractitioner heuristic - Route data by classification at prompt assembly instead of trusting every caller
Several features share one LLM gateway, some documents are restricted, and the approved provider terms cover only less sensitive classes.
mediumpractitioner heuristic - Treat repeated refusals and guardrail hits per account as an abuse signal
A public or partner-facing assistant enforces per-user rate limits, and abusers stay under them while probing slowly.
mediumpractitioner heuristic - Decide in advance whether each action fails open or closed when a guardrail is down
Moderation or injection checks run as a separate service with its own latency and outages.
mediumpractitioner heuristic - Keep an evidence pack indexed to a risk framework, and label it evidence rather than compliance
Security, privacy or audit teams ask for proof that an LLM feature was assessed, tested and approved.
mediumpractitioner heuristic