Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Seed eval sets with questions that invite common misconceptions
A revenue-management help bot is evaluated only on questions written by the people who built it.
mediumempirical finding - Keep your hypothesis out of the prompt when asking an LLM to judge evidence
An analyst asks a model to confirm that a revenue drop on one route was caused by a competitor's new capacity.
mediumempirical finding - Give every extracted field an explicit unknown value so the model is not forced to invent
An extraction schema requires a booking reference, a fare class and a travel date for every complaint email.
easypractitioner heuristic - Extract from long documents section by section with provenance, then aggregate in code
An analyst asks a model to list every penalty clause from a long supplier contract in one call.
mediumpractitioner heuristic - Supply context instead of personas, because role prompts do not add knowledge
A prompt starts by telling the model it is a world-class revenue-management expert, and the team expects better pricing analysis as a result.
easyempirical finding - Enforce validity dates and scope with metadata filters, not with the prompt
The corpus holds several versions of the same policy, for example baggage rules effective before and after 2024-03-01, or tariffs per country.
mediumpractitioner heuristic - Deduplicate near-identical chunks before they crowd out other evidence
The corpus contains copies of the same policy in several places: the PDF, an intranet page and an email announcement.
easypractitioner heuristic - Label the evidence passages in your golden set, not only the expected answers
You are building the first evaluation set for a document assistant from real user questions.
easypractitioner heuristic - Slice retrieval metrics by query type before declaring a retriever better
A hybrid retriever raises average recall@5 by two points over the dense baseline.
mediumpractitioner heuristic - Monitor the ingestion pipeline, because parse failures drop evidence silently
A RAG index is rebuilt nightly from PDFs, HTML and spreadsheets owned by other teams.
mediumpractitioner heuristic - Route counting and aggregation questions to SQL, not to chunk retrieval
Users ask the document assistant "how many flights to FRA were cancelled last month?" or "average fare on IST-ADB in July".
mediumpractitioner heuristic - Budget GenAI features by cost per successful task, not price per token
A business case multiplies expected requests by the list price per million tokens and calls that the cost.
mediumpractitioner heuristic - Normalise locale-specific numbers and dates before the model reads them
A Turkish fare notice lists 1.299,90 TL and 03.04.2025, and the model is asked to compare amounts and count days to departure.
easypractitioner heuristic - Hand a case to a human with a structured summary, evidence and attempted actions, not the transcript
An agent cannot resolve a baggage-delay claim and escalates to a human agent.
easypractitioner heuristic - Read thumbs-up and thumbs-down rates as a selection-biased signal and sample the silent majority
A dashboard tracks the thumbs-down rate of an assistant and the team treats it as the quality metric.
mediumpractitioner heuristic - Tag every model call with feature, tenant tier and prompt version so spend can be attributed
The monthly model invoice jumps 40% and nobody can say which feature, customer segment or release caused it.
easypractitioner heuristic - Capture the human's edit of a model draft as feedback, because the diff shows what was wrong
Agents edit AI-drafted customer replies before sending, and the only feedback collected is a thumbs-up on the draft.
easypractitioner heuristic - Alert on shifts in output-side signals, because a degrading assistant raises no exceptions
Dashboards show request rate, error rate and latency, all green, while answers have become shorter, more hedged or ungrounded.
mediumpractitioner heuristic