Practical insights
Situation, action, reason and boundary — one decision per card. Filter by topic, role and difficulty; search covers the full text.
- 1000published
- 25in review
- 8topic families
Results
- Power guardrail metrics for the harm you must exclude, or the guardrail cannot protect you
A launch review says "no significant change in cancellations or page latency" based on a test sized only for the primary metric.
mediumdocumented fact - Run A/A tests on a new experimentation pipeline before trusting its first A/B result
A team has built its own assignment service and analysis notebook and is about to run its first real experiment.
mediumpractitioner heuristic - Refuse to answer a "what if we change X" question with a forecast model
An accurate demand forecast uses fare as a feature, and a manager asks it to predict bookings after a 5% fare cut.
mediumpractitioner heuristic - Agree the no-worse margin before a migration test and size it as an equivalence test
Moving the fare engine to a cheaper stack, the owner asks the test to show that revenue per search is "unchanged".
mediumdocumented fact - Evaluate an uplift-targeting rule on a randomised holdout, not by the outcome of the people it selected
A model ranks customers by predicted response to a discount, the top decile converts at 14%, and the slide says the rule "earns" 14%.
mediumpractitioner heuristic - Audit what is logged for each arm when a variant changes the tracking, not only the product
A new checkout variant replaces a page, and its event tracking was rebuilt by a different team.
mediumdocumented fact