Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Pair every target metric with a counter-metric that exposes gaming
A team will be rewarded on one KPI, such as on-time departure rate, forecast MAPE or average handling time.
easypractitioner heuristic - Write chart titles that state the finding and its scope, not the variables plotted
A slide is titled "Bookings by month, 2023-2024" and the presenter explains the point verbally.
easypractitioner heuristic - Keep rollback one alias change away and test it before you need it
A new model version misbehaves, and rollback requires rebuilding the old container and feature pipeline.
easydocumented fact - Instrument a model service with the golden signals plus model-specific signals
A model endpoint has a health check that returns OK, and users report slow and odd recommendations.
easydocumented fact - Check model and dataset licences before a prototype becomes a product
A prototype uses an open-weights model and a public dataset, and the business wants to launch it commercially.
easydocumented fact - Ship a model card with intended use and performance by relevant slice
A claims-triage model is handed to an operations team with only an overall accuracy figure.
easypractitioner heuristic - Track unit cost per decision, not just the monthly cloud bill
Monthly ML infrastructure cost rose 40%, and nobody can say whether it is growth, waste or a new model.
easypractitioner heuristic - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Feed raw logits to CrossEntropyLoss, never softmax outputs
A multi-class model ends in nn.Softmax and its output goes straight into nn.CrossEntropyLoss.
easydocumented fact - Overfit a tiny batch before launching full training
A new model or data pipeline is ready and you are about to start a multi-hour training run.
easypractitioner heuristic - Reproduce the pretrained model's preprocessing exactly before fine-tuning
You load a pretrained image or text backbone and feed it data prepared by your own pipeline.
easypractitioner heuristic - Start text classification with TF-IDF and a linear model
You need to route airline customer complaints into 8 queues and the first idea is to fine-tune a Transformer.
easyempirical finding - Report per-class recall and macro averages for long-tail intent classifiers
A banking chatbot intent model reports 94% accuracy over 60 intents, and most traffic falls into five intents.
easypractitioner heuristic - Score embeddings with the similarity function the model was trained for
You build semantic search over fare-rule documents and pick cosine similarity by habit for an embedding model you have not read about.
easydocumented fact - Do not treat a t-SNE or UMAP picture of embeddings as evidence of separation
A slide shows customer-message embeddings in 2D with neat colored clusters and claims the model separates segments.
easydocumented fact - Run the first training step on the largest batch and longest sequence you will see
A text model trains with dynamic padding, and batch memory depends on the longest document in each batch.
easypractitioner heuristic - Count tokens per language before sizing prompts, budgets and context limits
An airline assistant must answer customer messages in English, Turkish, Arabic and German, and the team sized its context and cost budget from English test prompts.
easyempirical finding - Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Match checkpoint type and chat template to the job before judging an open model
An engineer downloads open model weights, sends plain instructions and gets rambling, off-task text.
easydocumented fact - Set decoding parameters per task instead of one global temperature
One shared configuration sets a high temperature for every LLM call, from marketing copy to fare-rule extraction.
easyempirical finding - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Check the stop reason before parsing, and treat truncated output as a failure
A batch job extracts structured itinerary changes and occasionally receives JSON that ends mid-object.
easydocumented fact - Verify every identifier the model emits against the system of record
An assistant drafts rebooking notes that include flight numbers, fare basis codes and policy section numbers.
easypractitioner heuristic - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic - Use provider-enforced schemas instead of asking for JSON in the prompt
A pipeline extracts passenger name, flight and requested change from emails by asking the model to reply in JSON, then parses with a try/except.
easydocumented fact - Validate business rules after the schema passes, because schemas only check shape
A structured-output extractor for refund requests returns perfectly valid objects, and the team stops checking them.
easypractitioner heuristic - Give every extracted field an explicit unknown value so the model is not forced to invent
An extraction schema requires a booking reference, a fare class and a travel date for every complaint email.
easypractitioner heuristic - Repair failed outputs with the validator's error message and a retry cap
A structured extraction fails business-rule validation on a few percent of documents, and an engineer adds code that silently coerces bad values into defaults.
easypractitioner heuristic - Separate instructions from pasted data with clear delimiters, but do not call it security
A prompt mixes instructions with a pasted customer email, and the model sometimes follows sentences inside the email as if they were instructions.
easypractitioner heuristic - Supply context instead of personas, because role prompts do not add knowledge
A prompt starts by telling the model it is a world-class revenue-management expert, and the team expects better pricing analysis as a result.
easyempirical finding - Measure chain-of-thought prompting per task instead of switching it on everywhere
After reading that step-by-step prompting improves reasoning, a team adds it to every prompt, including simple classification.
easyempirical finding - Evaluate the exact artefact you will serve, not the training checkpoint
A LoRA adapter scores well during training, then is merged into the base weights and quantized for serving without re-evaluation.
easypractitioner heuristic - Choose models on a task-specific eval set, not on public leaderboard rank
A team picks the model at the top of a public leaderboard for a booking-change assistant.
easypractitioner heuristic - Check everything verifiable with code before reaching for an LLM judge
An eval pipeline sends every output to an LLM judge, including checks such as valid JSON, word limits and whether the booking reference matches the input.
easypractitioner heuristic - Turn every production failure into a regression test that gates changes
A model upgrade reintroduces a bug that was fixed months ago, quoting outdated pet-travel rules.
easypractitioner heuristic - Chunk along the document's own structure before tuning chunk size
You are ingesting manuals, policies or contracts that have headings, numbered clauses or tables into a RAG index.
easypractitioner heuristic - Keep a BM25 baseline when queries contain codes, identifiers or rare terms
Users search with fare basis codes, error codes, part numbers or policy clause ids, and the team plans a dense-only retriever.
easyempirical finding - Put the strongest evidence at the start of the context, not buried in the middle
Context assembly concatenates passages in arbitrary order, for example by document id or ingestion time.
easyempirical finding - Deduplicate near-identical chunks before they crowd out other evidence
The corpus contains copies of the same policy in several places: the PDF, an intranet page and an email announcement.
easypractitioner heuristic - Validate citation ids in code before showing a cited answer
Your RAG prompt asks the model to cite sources as [doc-id] markers.
easyempirical finding - Report recall@k at the k you actually pass to the model
A retrieval dashboard shows recall@100, while the generation prompt receives the top 5 chunks.
easydocumented fact - Use MRR when one right passage exists and how high it ranks matters
Each question has a single authoritative passage, such as one clause of a contract, and you want one number for ranking quality.
easydocumented fact - Label the evidence passages in your golden set, not only the expected answers
You are building the first evaluation set for a document assistant from real user questions.
easypractitioner heuristic - Write the workflow in code when the steps are known, and use the model only at fuzzy steps
A team proposes an autonomous agent for a process whose steps are already documented, such as refund eligibility checks or disruption rebooking.
easypractitioner heuristic - Give every agent run hard budgets and explicit stopping conditions
An agent loop runs "until the task is done" with no limit on iterations, tokens, spend or wall-clock time.
easypractitioner heuristic - Expose only the tools a task needs, as an explicit allow-list per route
One agent configuration registers every available tool (email, database, payments, file system) for all requests.
easydocumented fact - Return structured, classified tool errors so the agent can recover or stop
Tool failures reach the model as raw stack traces or a generic "error occurred".
easypractitioner heuristic - Report steps, tool calls and tokens per task alongside agent success rate
Two agent designs reach similar task success, and the comparison stops there.
easypractitioner heuristic - Write tool descriptions like an interface spec and test them with varied inputs
Tools are registered with one-line descriptions such as "search bookings", and the agent misuses them.
easypractitioner heuristic - Cap and shape tool results before they enter the context window
A tool returns a full API payload, a 2,000-row query result or an entire web page to the agent.
easypractitioner heuristic - Assume the system prompt will leak, so never put secrets or access rules only there
A system prompt contains API keys, internal URLs, discount codes, or the only statement of which users may see what.
easydocumented fact - Encode or parameterise model output before it reaches SQL, HTML or a shell
Generated text is inserted into a web page, a database query, a template or a command line.
easydocumented fact - Run GenAI design reviews against a shared threat checklist, not ad-hoc brainstorming
A GenAI feature reaches design review and the security section says "prompt injection mitigated by system prompt".
easydocumented fact - Set per-user quotas and spend limits as a security control, not only as finance hygiene
A public or partner-facing LLM endpoint has no per-user request, token or cost limits.
easydocumented fact - Branch error handling on the error type, because two 429s can need opposite responses
All non-200 responses from the model API go through one generic "wait and retry" handler.
easydocumented fact - Trace each LLM request as spans per stage with model, tokens and finish reason
A user reports a slow or wrong answer, and logs show only the final response and total latency.
easydocumented fact - Version prompt, retrieval settings and model together as one release
Prompts live in code, retrieval parameters in a config service and the model name in an environment variable, each changed independently.
easypractitioner heuristic - Pin dated model versions and schedule migrations before retirement dates
Production calls a moving model alias, or a pinned model that the provider has deprecated.
easydocumented fact - Send offline workloads such as eval runs and nightly enrichment through a batch API
Golden-set evaluations, document classification backfills or nightly summaries run through the real-time API with custom concurrency code.
easydocumented fact - Use character n-grams for short, noisy strings such as place names, codes and product titles
You classify or match short strings with typos and spelling variants, such as 'Antalya' against 'Antlya' or truncated product names.
easydocumented fact - Replace probability words with numeric ranges in written risk statements
A risk memo calls a forecast regression 'likely' and a launch delay 'possible', and three readers take away different odds (hypothetical).
easypractitioner heuristic - Quote an automation benefit as residual human workload, not as the share automated
A business case says a triage model will 'automate 80 percent' of 10,000 monthly cases (hypothetical).
easypractitioner heuristic - Walk stakeholders through twenty concrete predictions before debating the metric
A review meeting keeps stalling on whether an F1 of 0.71 is good (hypothetical).
easypractitioner heuristic - Write a time box and kill criteria into an exploratory modelling brief
An exploratory effort on call transcripts enters its fourth month with no go or no-go date (hypothetical).
easypractitioner heuristic - Measure pretrained word-vector coverage on your own tokens, by slice, before relying on static embeddings
GloVe or word2vec vectors feed a complaint classifier, and the text is full of flight codes, brand names and inflected Turkish forms.
easypractitioner heuristic - Do not use raw BERT [CLS] or mean-pooled outputs as sentence embeddings for similarity
A prototype embeds sentences with a base BERT checkpoint, takes the [CLS] vector or the mean of the last layer, and ranks by cosine.
easyempirical finding - Left-pad batched prompts for decoder-only generation and confirm batched outputs match single runs
You batch prompts of different lengths for a decoder-only model, using the tokenizer's default right padding.
easydocumented fact - Put long documents first and the question last, then test the reverse on your own task
A contract-review prompt places the instructions and question at the top and pastes 150 pages of terms underneath.
easyempirical finding - Diff output shape between model versions on a fixed prompt suite before switching traffic
A provider releases a successor model, and a JSON-extraction prompt for disruption notices is moved to it after a check on accuracy alone.
easyempirical finding - Apply argument defaults in code, because a schema default is only an annotation
A changeseat tool schema declares cabin with a default of economy and notify with a default of true, and the model omits both on some calls while the validator passes them.
easydocumented fact - Render prompt templates in strict mode and unit-test empty and missing variables
A prompt template inserts retrieved passages and a customer name, and a failed lookup leaves both empty.
easydocumented fact - Check which sampling parameters each model accepts before running a temperature study
An evaluation harness sweeps temperature from 0 to 1 across hosted models, including one released this quarter.
easydocumented fact - Downsample images to the resolution the task needs, because visual tokens scale with pixels
A baggage-claims assistant sends full-resolution phone photos and scanned forms with every request, and input cost and latency run well above the text-only estimate.
easydocumented fact - Normalise locale-specific numbers and dates before the model reads them
A Turkish fare notice lists 1.299,90 TL and 03.04.2025, and the model is asked to compare amounts and count days to departure.
easypractitioner heuristic - Resolve relative dates in code and pass the current date and timezone with each tool-calling request
A customer writes move me to the first flight tomorrow morning, and the rebooking tool takes a departure date argument.
easypractitioner heuristic - Enforce hard output limits in code, because length instructions in prompts are only targets
A delay push notification must fit 120 characters in the app, and the prompt says at most 120 characters.
easypractitioner heuristic - Follow the embedding model's own input format: query and passage prefixes, normalisation, similarity metric
You swap an open embedding model into a working pipeline using the same encode call, and recall drops without any error message.
easydocumented fact - Assert chunk token length against the embedding model's input limit at ingest, because truncation is silent
Chunks come from a splitter counting characters or a different tokenizer than the embedding model, and some policy sections are very long.
easydocumented fact - Price a full re-index from chunk size, overlap and header tokens before fixing those settings
A team picks 20% chunk overlap and a generated context header per chunk, and plans to re-embed the corpus whenever the model or chunker changes.
easypractitioner heuristic - Measure index freshness lag from source change to searchable, and show the as-of date in answers
A fare-conditions PDF is replaced at 10:00, and the support assistant quotes the old rule that afternoon.
easypractitioner heuristic - Multiply per-step reliability before promising an end-to-end agent success rate
A team demos an agent whose individual steps look reliable and plans to promise a task success rate to the business.
easypractitioner heuristic - Put a programmatic gate between chained model steps so a bad intermediate result stops early
A flow of fixed steps, such as extract a booking change, look up policy, draft a reply, has no agent but sometimes sends nonsense downstream.
easydocumented fact - Detect repeated identical tool calls and break the loop before the budget runs out
An agent searches, gets an unhelpful result, and issues the same call with the same arguments again and again.
easypractitioner heuristic - Hand a case to a human with a structured summary, evidence and attempted actions, not the transcript
An agent cannot resolve a baggage-delay claim and escalates to a human agent.
easypractitioner heuristic - Encode units, formats and allowed values in the tool schema so wrong calls cannot be expressed
A booking tool takes a free-text string for cabin class and a number for baggage weight with no unit stated.
easydocumented fact - Run independent read-only tool calls in parallel and serialise writes
An agent needs a flight status, a fare rule and a loyalty record before answering, and issues the lookups one after another.
easydocumented fact - Keep "no results" and "the call failed" as different tool outcomes
A lookup tool returns an empty list both when no record matches and when the backing service is down.
easypractitioner heuristic - Tie every golden item to its source document and effective date, then re-verify on a schedule
A golden set of policy or product questions was written months ago and the underlying rules, prices or documents have since been revised.
easypractitioner heuristic - Decompose a judge rubric into independent yes/no checks instead of one holistic score
A judge prompt asks "rate the answer's overall quality from 1 to 10" and scores cluster in a narrow band, or flip between runs.
easypractitioner heuristic - Score extraction field by field with typed comparators instead of whole-record exact match
An extraction pipeline is graded by exact string equality of the whole JSON output against gold records with eight fields.
easypractitioner heuristic - Blind and shuffle annotation tasks, and seed known-answer items to catch drift
Reviewers compare outputs from two systems, with system names visible and a fixed order, over several days of labelling.
easypractitioner heuristic - Publish an eval report with set version, judge version, runs, intervals and the number of variants tried
An internal eval result is shared as "new prompt: 84% vs 81%" with no further context.
easypractitioner heuristic - Plant unique canary strings so that a leaked prompt or document announces itself
System prompts, restricted documents and logs could leak through an output, a vendor or a screenshot, and you would have no signal when that happens.
easypractitioner heuristic - Record vendor data terms per endpoint, because retention differs inside one provider
Procurement approved a model provider, and teams now use its chat, batch, file, fine-tuning and embedding endpoints.
easypractitioner heuristic - Inventory every model call site so a deprecation notice answers what breaks and by when
A vendor announces that a model version will be retired, and prompts are spread across services, notebooks and no-code tools.
easypractitioner heuristic - Build a feature-level kill switch and a degraded mode before launch
A customer-facing LLM feature is about to ship and the incident plan is "redeploy".
easypractitioner heuristic - Rotate every credential that has entered a prompt, a tool result or a trace
An incident review finds an API key, session token or database URL in a prompt, a retrieved document, a tool response or a stored trace.
easypractitioner heuristic - Test the whole corpus in a cached prompt before building retrieval for a small stable knowledge base
A product assistant must answer from a policy handbook of a few hundred pages that changes a few times a year, and the plan starts with a vector index.
easypractitioner heuristic - Check which tokens carry loss before supervised fine-tuning, because the default depends on data format
An instruction-tuning set is loaded into a trainer and the same recipe is reused with a different dataset layout.
easypractitioner heuristic - Read the provider's terms before training a model on its outputs
A plan to distil a hosted model's answers into an in-house small model is approved on technical grounds.
easypractitioner heuristic - Decide per surface whether to stream or to buffer, because streamed text is shown before it is checked
Streaming is switched on for responsiveness, while the same endpoint feeds output checks such as schema validation, citation verification or policy filters.
easypractitioner heuristic - Tag every model call with feature, tenant tier and prompt version so spend can be attributed
The monthly model invoice jumps 40% and nobody can say which feature, customer segment or release caused it.
easypractitioner heuristic - Capture the human's edit of a model draft as feedback, because the diff shows what was wrong
Agents edit AI-drafted customer replies before sending, and the only feedback collected is a thumbs-up on the draft.
easypractitioner heuristic - Write to a response cache only when the generation ended naturally and passed validation
A response cache stores whatever came back from the model, keyed on the normalised request.
easypractitioner heuristic - Never let user text become the template, and render each prompt exactly once
A prompt is built by pasting a customer message into a template string and then passing the result through a second formatting call.
easypractitioner heuristic - Permute the options of every multiple-choice eval and report the spread, not one order
A model is scored on a four-option multiple-choice set, or asked to pick a label from a lettered list, and the options appear in one fixed order.
easyempirical finding - Send one request to write the prompt cache before fanning out parallel requests that share a prefix
A batch job sends 50 requests at once that all begin with the same 20,000-token policy document, with caching enabled.
easydocumented fact - Trim agent history by whole tool exchanges so a call and its result stay together
An agent loop drops the oldest messages to fit the context window, and the cut falls between a tool call and its result.
easypractitioner heuristic