Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Compute training and serving features from one shared definition
Training features are built in a SQL notebook, and the online service reimplements them in application code.
mediumpractitioner heuristic - Design a degraded mode for when upstream features are late or missing
The nightly feature pipeline fails, and the pricing model must still publish recommendations by 06:00.
mediumpractitioner heuristic - Exclude immature labels when building training data for slow outcomes
A default model labels loans as 'not defaulted' if no default is recorded yet, including loans issued last month.
mediumpractitioner heuristic - Monitor leading indicators while labels are delayed, then confirm with matured labels
Fraud chargebacks arrive up to 90 days after the transaction, so model accuracy for this month is unknown.
mediumpractitioner heuristic - Confirm that drift hurts performance before retraining on it
A drift report shows the age distribution of applicants shifted, and the team schedules an urgent retrain.
mediumpractitioner heuristic - Use shadow mode to test plumbing and agreement, not to prove business impact
A new ranking model runs in shadow for two weeks, scores agree 92% with production, and the team claims a revenue uplift.
mediumpractitioner heuristic - Define reproducibility as recoverable inputs plus tolerances, not identical bits
An auditor asks you to reproduce last year's model, and retraining gives slightly different numbers.
mediumdocumented fact - Make every automated decision reconstructable from a decision record
A customer disputes a declined credit-limit increase from eight months ago, and the team must explain it.
mediumpractitioner heuristic - Minimise personal data in features and give training snapshots a retention period
A churn model's training set copies full customer records, including fields the model never uses, and snapshots are kept indefinitely.
mediumdocumented fact - Scale governance effort to the risk tier of each model
Every model, from an internal dashboard forecast to a credit decision engine, goes through the same three-month review.
mediumpractitioner heuristic - Set retraining cadence from a measured decay curve and the cost of each retrain
A model is retrained daily because the platform makes it easy, and nobody has checked whether daily retrains help.
mediumpractitioner heuristic - Report a null result with its interval and detectable effect, not as 'no effect'
A test of a new recommendation model shows no statistically significant revenue change, and the summary says 'the model does not work'.
mediumdocumented fact - Pair any model-driven KPI target with guardrail metrics to resist Goodhart effects
A call-centre routing model is rewarded on average handle time, and handle time falls while repeat calls rise.
mediumpractitioner heuristic - Refresh internal benchmark sets so teams stop tuning to a fixed test
Every quarter, teams report improvements on the same internal evaluation set used for two years.
mediumdocumented fact - Retire models whose lift over a simple rule no longer pays for their upkeep
A churn model built three years ago still runs, but recent evaluations show it barely beats a tenure-and-usage rule.
mediumopinion - Limit and explain forecast churn between model versions
A new forecasting model version changes next month's plan by 15% for some routes, though inputs barely changed.
mediumpractitioner heuristic - Version reference tables and mappings as model inputs, and freeze them until the new version is vetted
A model uses a lookup table, such as airport-to-region mapping or product-category codes, that another team refreshes whenever it likes.
mediumdocumented fact - Make lineage queryable from dataset to models so a bad table has a known blast radius
A data engineer finds that last month's customer table had a bad join, and nobody can say which models trained on it.
mediumpractitioner heuristic - Re-derive decision thresholds whenever the model is retrained
A churn model's score cutoff of 0.35 was tuned for capacity in the first release and is hard-coded in the campaign job, while the model is retrained monthly.
mediumdocumented fact - Assign a prediction id at serving time and join delayed outcomes to it
Outcomes arrive weeks later keyed by customer and date, while the customer was scored several times by two model versions in the meantime.
mediumpractitioner heuristic - After an incident, replay the affected window with the corrected model into a separate table
A feature bug corrupted scores for 38 hours, and decisions were made on them before the fix.
mediumpractitioner heuristic - Test for proxies of a sensitive attribute instead of relying on having dropped it
Compliance asks for gender and age to be removed from a pricing model's features, and the model is then described as neutral (hypothetical).
mediumpractitioner heuristic - Count rows that are unique on indirect identifiers before releasing a table
An extract with names and ids removed, but birth date, postcode and gender intact, is to be shared with a partner for research (hypothetical).
mediumdocumented fact - Treat hashed customer ids as pseudonymous personal data and keep the key apart
Emails in a training table are replaced by plain hashes, and the table is called anonymous and sent to a vendor (hypothetical).
mediumdocumented fact - Check each new model use against the purpose the data was collected for
Support-chat transcripts gathered to resolve tickets are proposed as training data for a sales-propensity model (hypothetical, EU customers).
mediumdocumented fact - Design a contest path into decisions that may fall under solely automated processing
A team plans to reject consumer loan applications straight from a score with no reviewer step (hypothetical, EU applicants).
mediumdocumented fact - Keep consent as a time-stamped event table and apply it when snapshots are built
A marketing-preference model is retrained monthly, and some customers withdrew consent after the previous snapshot (hypothetical).
mediumdocumented fact - Keep a model inventory that links each model to its data, owner and consumers
A source table is being deprecated and nobody can list which models, spreadsheets and vendor scores read it (hypothetical).
mediumpractitioner heuristic - Date-stamp the evaluation plan before looking at candidate results
Five variants and a new feature set are compared with last quarter's champion, with the primary metric, slices and exclusions still open (hypothetical).
mediumpractitioner heuristic - Seed review queues with known-answer cases to test whether reviewers really check
Analysts approve 99 percent of a fraud model's suggested actions (hypothetical).
mediumempirical finding - List who bears each error type before setting the threshold
A claims-flagging threshold is set from investigator hours alone, while flagged customers wait weeks for payment (hypothetical).
mediumpractitioner heuristic - Use quantile dotplots, not only interval bars, when non-specialists decide under uncertainty
A gate-staffing planner receives delay forecasts as 80 percent interval bands and acts on the midpoint anyway (hypothetical).
mediumempirical finding - Gate project ranking on decision owner and label readiness before scoring value
A backlog of fourteen ideas is ranked by sponsor seniority and novelty (hypothetical).
mediumpractitioner heuristic - Keep a small permanent randomised control share to measure what the model earns
Six months after launch, finance asks what the retention model earned, and the only evidence is before-and-after revenue (hypothetical).
mediumpractitioner heuristic - Price the exit before buying a model or platform
A team picks a hosted risk-scoring API after a strong demo and a three-year contract is on the table (hypothetical).
mediumpractitioner heuristic - Hand engineers a specification with test vectors, not a notebook
A data scientist passes a notebook to engineers to reimplement in the serving stack (hypothetical).
mediumpractitioner heuristic - Treat a pickled model file as executable code and control who can write the artifact path
A scoring service loads a model with pickle, joblib or an unrestricted torch.load from a shared bucket or registry path that more people can write to than can deploy.
mediumdocumented fact