Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Compute training and serving features from one shared definition
Training features are built in a SQL notebook, and the online service reimplements them in application code.
mediumpractitioner heuristic - Design a degraded mode for when upstream features are late or missing
The nightly feature pipeline fails, and the pricing model must still publish recommendations by 06:00.
mediumpractitioner heuristic - Write a data contract for every upstream table a model depends on
A model breaks because an upstream team renamed a status code and nobody told the data science team.
easypractitioner heuristic - Exclude immature labels when building training data for slow outcomes
A default model labels loans as 'not defaulted' if no default is recorded yet, including loans issued last month.
mediumpractitioner heuristic - Monitor leading indicators while labels are delayed, then confirm with matured labels
Fraud chargebacks arrive up to 90 days after the transaction, so model accuracy for this month is unknown.
mediumpractitioner heuristic - Confirm that drift hurts performance before retraining on it
A drift report shows the age distribution of applicants shifted, and the team schedules an urgent retrain.
mediumpractitioner heuristic - Use shadow mode to test plumbing and agreement, not to prove business impact
A new ranking model runs in shadow for two weeks, scores agree 92% with production, and the team claims a revenue uplift.
mediumpractitioner heuristic - Define reproducibility as recoverable inputs plus tolerances, not identical bits
An auditor asks you to reproduce last year's model, and retraining gives slightly different numbers.
mediumdocumented fact - Record why a model version was promoted, not only that it was
Six months after promotion, nobody can explain why version 14 replaced version 12 or what evidence was used.
easypractitioner heuristic - Make every automated decision reconstructable from a decision record
A customer disputes a declined credit-limit increase from eight months ago, and the team must explain it.
mediumpractitioner heuristic - Minimise personal data in features and give training snapshots a retention period
A churn model's training set copies full customer records, including fields the model never uses, and snapshots are kept indefinitely.
mediumdocumented fact - Write a datasheet for any dataset other teams will train on
A curated training table is published internally, and three teams start building models on it.
easypractitioner heuristic - Ship a model card with intended use and performance by relevant slice
A claims-triage model is handed to an operations team with only an overall accuracy figure.
easypractitioner heuristic - Scale governance effort to the risk tier of each model
Every model, from an internal dashboard forecast to a credit decision engine, goes through the same three-month review.
mediumpractitioner heuristic - Use batch scoring when decisions are made on a schedule
A team plans a low-latency online endpoint for a churn model whose scores are used in a weekly retention campaign.
easypractitioner heuristic - Set retraining cadence from a measured decay curve and the cost of each retrain
A model is retrained daily because the platform makes it easy, and nobody has checked whether daily retrains help.
mediumpractitioner heuristic - Declare model incidents early and give one person stakeholder communication
A pricing model has produced obviously wrong fares for an hour, and engineers are debugging while managers message them for updates.
easydocumented fact - Run blameless postmortems with triggers defined before incidents happen
After a model rollback, the team discusses who merged the bad change and moves on without writing anything.
easydocumented fact - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Open an executive memo with the decision, the evidence range and the main limit
A four-page analysis of an overbooking model starts with data sources and methods, and the recommendation is on page four.
easyopinion - Report a null result with its interval and detectable effect, not as 'no effect'
A test of a new recommendation model shows no statistically significant revenue change, and the summary says 'the model does not work'.
mediumdocumented fact - Present forecasts to decision makers as ranges tied to the actions they support
Leadership receives a single number for next season's demand and builds the capacity plan on it.
easypractitioner heuristic - Pair any model-driven KPI target with guardrail metrics to resist Goodhart effects
A call-centre routing model is rewarded on average handle time, and handle time falls while repeat calls rise.
mediumpractitioner heuristic - Refresh internal benchmark sets so teams stop tuning to a fixed test
Every quarter, teams report improvements on the same internal evaluation set used for two years.
mediumdocumented fact - Look for hidden feedback loops when a model influences its own future training data
A pricing model's recommendations shape the fares customers see, and next month's model trains on the resulting bookings.
harddocumented fact - Retire models whose lift over a simple rule no longer pays for their upkeep
A churn model built three years ago still runs, but recent evaluations show it barely beats a tenure-and-usage rule.
mediumopinion - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Limit and explain forecast churn between model versions
A new forecasting model version changes next month's plan by 15% for some routes, though inputs barely changed.
mediumpractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Version reference tables and mappings as model inputs, and freeze them until the new version is vetted
A model uses a lookup table, such as airport-to-region mapping or product-category codes, that another team refreshes whenever it likes.
mediumdocumented fact - Make lineage queryable from dataset to models so a bad table has a known blast radius
A data engineer finds that last month's customer table had a bad join, and nobody can say which models trained on it.
mediumpractitioner heuristic - Set canary duration from event volume and cycle length, not from a clock
A canary for a booking-conversion model runs for 30 minutes at 5% of traffic, and the rollout then goes to 100% on 'no issues'.
harddocumented fact - Re-derive decision thresholds whenever the model is retrained
A churn model's score cutoff of 0.35 was tuned for capacity in the first release and is hard-coded in the campaign job, while the model is retrained monthly.
mediumdocumented fact - Treat unobserved outcomes as censored by age, not as negatives, when evaluating on recent cohorts
Refund requests can arrive up to 45 days after a booking, and the weekly dashboard counts bookings without a refund as non-refunds.
hardpractitioner heuristic - Assign a prediction id at serving time and join delayed outcomes to it
Outcomes arrive weeks later keyed by customer and date, while the customer was scored several times by two model versions in the meantime.
mediumpractitioner heuristic - After an incident, replay the affected window with the corrected model into a separate table
A feature bug corrupted scores for 38 hours, and decisions were made on them before the fix.
mediumpractitioner heuristic - Agree the fairness criterion with business and risk owners before auditing groups
A default-risk model is audited by applicant group, and the report lists calibration, false-positive rate and approval rate for two groups whose default prevalence differs (hypothetical).
harddocumented fact - Test for proxies of a sensitive attribute instead of relying on having dropped it
Compliance asks for gender and age to be removed from a pricing model's features, and the model is then described as neutral (hypothetical).
mediumpractitioner heuristic - Evaluate against past decision-makers with a selective-labels method, not on approved cases alone
A credit model is scored on repaid versus defaulted loans, but outcomes exist only for applicants that underwriters approved (hypothetical).
hardpractitioner heuristic - Count rows that are unique on indirect identifiers before releasing a table
An extract with names and ids removed, but birth date, postcode and gender intact, is to be shared with a partner for research (hypothetical).
mediumdocumented fact - Treat hashed customer ids as pseudonymous personal data and keep the key apart
Emails in a training table are replaced by plain hashes, and the table is called anonymous and sent to a vendor (hypothetical).
mediumdocumented fact - Check each new model use against the purpose the data was collected for
Support-chat transcripts gathered to resolve tickets are proposed as training data for a sales-propensity model (hypothetical, EU customers).
mediumdocumented fact - Design a contest path into decisions that may fall under solely automated processing
A team plans to reject consumer loan applications straight from a score with no reviewer step (hypothetical, EU applicants).
mediumdocumented fact - Keep consent as a time-stamped event table and apply it when snapshots are built
A marketing-preference model is retrained monthly, and some customers withdrew consent after the previous snapshot (hypothetical).
mediumdocumented fact - Keep a model inventory that links each model to its data, owner and consumers
A source table is being deprecated and nobody can list which models, spreadsheets and vendor scores read it (hypothetical).
mediumpractitioner heuristic - Date-stamp the evaluation plan before looking at candidate results
Five variants and a new feature set are compared with last quarter's champion, with the primary metric, slices and exclusions still open (hypothetical).
mediumpractitioner heuristic - Look for bunching just past a KPI target to catch gaming
A branch bonus needs 95 percent of tickets resolved within the service target, and many branches report exactly 95 to 96 percent (hypothetical).
hardpractitioner heuristic - Version KPI definitions like code and restate history when they change
Finance changes 'active customer' from 30 to 90 days of activity in a dashboard, and the trend shows a jump in March (hypothetical).
easypractitioner heuristic - Seed review queues with known-answer cases to test whether reviewers really check
Analysts approve 99 percent of a fraud model's suggested actions (hypothetical).
mediumempirical finding - List who bears each error type before setting the threshold
A claims-flagging threshold is set from investigator hours alone, while flagged customers wait weeks for payment (hypothetical).
mediumpractitioner heuristic - Replace probability words with numeric ranges in written risk statements
A risk memo calls a forecast regression 'likely' and a launch delay 'possible', and three readers take away different odds (hypothetical).
easypractitioner heuristic - Use quantile dotplots, not only interval bars, when non-specialists decide under uncertainty
A gate-staffing planner receives delay forecasts as 80 percent interval bands and acts on the midpoint anyway (hypothetical).
mediumempirical finding - Quote an automation benefit as residual human workload, not as the share automated
A business case says a triage model will 'automate 80 percent' of 10,000 monthly cases (hypothetical).
easypractitioner heuristic - Walk stakeholders through twenty concrete predictions before debating the metric
A review meeting keeps stalling on whether an F1 of 0.71 is good (hypothetical).
easypractitioner heuristic - Gate project ranking on decision owner and label readiness before scoring value
A backlog of fourteen ideas is ranked by sponsor seniority and novelty (hypothetical).
mediumpractitioner heuristic - Write a time box and kill criteria into an exploratory modelling brief
An exploratory effort on call transcripts enters its fourth month with no go or no-go date (hypothetical).
easypractitioner heuristic - Keep a small permanent randomised control share to measure what the model earns
Six months after launch, finance asks what the retention model earned, and the only evidence is before-and-after revenue (hypothetical).
mediumpractitioner heuristic - Price the exit before buying a model or platform
A team picks a hosted risk-scoring API after a strong demo and a three-year contract is on the table (hypothetical).
mediumpractitioner heuristic - Hand engineers a specification with test vectors, not a notebook
A data scientist passes a notebook to engineers to reimplement in the serving stack (hypothetical).
mediumpractitioner heuristic - Rehearse a handover by letting the new owner run a full cycle while the builder watches
The builder of a monthly forecasting pipeline is moving teams, and a handover document has been written (hypothetical).
easypractitioner heuristic - Treat a pickled model file as executable code and control who can write the artifact path
A scoring service loads a model with pickle, joblib or an unrestricted torch.load from a shared bucket or registry path that more people can write to than can deploy.
mediumdocumented fact