Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Supply volatile facts at request time because parametric knowledge is frozen at training
Agents ask a chat model about this season's baggage allowance or a fare rule that changed last month.
easydocumented fact - Repeat evaluation runs because temperature 0 does not guarantee identical outputs
A prompt change is approved because a single evaluation run at temperature 0 scored two points higher than the old prompt.
easydocumented fact - Write success criteria and test cases before the first prompt revision
A team has rewritten a booking-change prompt eleven times, each time reacting to whichever example last went wrong.
easypractitioner heuristic - Supply context instead of personas, because role prompts do not add knowledge
A prompt starts by telling the model it is a world-class revenue-management expert, and the team expects better pricing analysis as a result.
easyempirical finding - Choose models on a task-specific eval set, not on public leaderboard rank
A team picks the model at the top of a public leaderboard for a booking-change assistant.
easypractitioner heuristic - Keep a BM25 baseline when queries contain codes, identifiers or rare terms
Users search with fare basis codes, error codes, part numbers or policy clause ids, and the team plans a dense-only retriever.
easyempirical finding - Report recall@k at the k you actually pass to the model
A retrieval dashboard shows recall@100, while the generation prompt receives the top 5 chunks.
easydocumented fact - Use MRR when one right passage exists and how high it ranks matters
Each question has a single authoritative passage, such as one clause of a contract, and you want one number for ranking quality.
easydocumented fact - Label the evidence passages in your golden set, not only the expected answers
You are building the first evaluation set for a document assistant from real user questions.
easypractitioner heuristic - Write the workflow in code when the steps are known, and use the model only at fuzzy steps
A team proposes an autonomous agent for a process whose steps are already documented, such as refund eligibility checks or disruption rebooking.
easypractitioner heuristic - Report steps, tool calls and tokens per task alongside agent success rate
Two agent designs reach similar task success, and the comparison stops there.
easypractitioner heuristic - Run GenAI design reviews against a shared threat checklist, not ad-hoc brainstorming
A GenAI feature reaches design review and the security section says "prompt injection mitigated by system prompt".
easydocumented fact - Version prompt, retrieval settings and model together as one release
Prompts live in code, retrieval parameters in a config service and the model name in an environment variable, each changed independently.
easypractitioner heuristic - Send offline workloads such as eval runs and nightly enrichment through a batch API
Golden-set evaluations, document classification backfills or nightly summaries run through the real-time API with custom concurrency code.
easydocumented fact - Check which sampling parameters each model accepts before running a temperature study
An evaluation harness sweeps temperature from 0 to 1 across hosted models, including one released this quarter.
easydocumented fact - Multiply per-step reliability before promising an end-to-end agent success rate
A team demos an agent whose individual steps look reliable and plans to promise a task success rate to the business.
easypractitioner heuristic - Score extraction field by field with typed comparators instead of whole-record exact match
An extraction pipeline is graded by exact string equality of the whole JSON output against gold records with eight fields.
easypractitioner heuristic - Blind and shuffle annotation tasks, and seed known-answer items to catch drift
Reviewers compare outputs from two systems, with system names visible and a fixed order, over several days of labelling.
easypractitioner heuristic - Publish an eval report with set version, judge version, runs, intervals and the number of variants tried
An internal eval result is shared as "new prompt: 84% vs 81%" with no further context.
easypractitioner heuristic - Record vendor data terms per endpoint, because retention differs inside one provider
Procurement approved a model provider, and teams now use its chat, batch, file, fine-tuning and embedding endpoints.
easypractitioner heuristic - Check which tokens carry loss before supervised fine-tuning, because the default depends on data format
An instruction-tuning set is loaded into a trainer and the same recipe is reused with a different dataset layout.
easypractitioner heuristic - Read the provider's terms before training a model on its outputs
A plan to distil a hosted model's answers into an in-house small model is approved on technical grounds.
easypractitioner heuristic - Capture the human's edit of a model draft as feedback, because the diff shows what was wrong
Agents edit AI-drafted customer replies before sending, and the only feedback collected is a thumbs-up on the draft.
easypractitioner heuristic - Permute the options of every multiple-choice eval and report the spread, not one order
A model is scored on a four-option multiple-choice set, or asked to pick a label from a lettered list, and the options appear in one fixed order.
easyempirical finding