Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Plan how deletion requests propagate to snapshots, features and retraining
A customer's data is deleted from the operational database, but it remains in last quarter's training snapshot and feature tables.
hardpractitioner heuristic - Look for hidden feedback loops when a model influences its own future training data
A pricing model's recommendations shape the fares customers see, and next month's model trains on the resulting bookings.
harddocumented fact - Set canary duration from event volume and cycle length, not from a clock
A canary for a booking-conversion model runs for 30 minutes at 5% of traffic, and the rollout then goes to 100% on 'no issues'.
harddocumented fact - Roll back the model together with its preprocessing, schema and stored state
A model v7 is rolled back to v6 after a regression, but v7 introduced a new feature column, a new encoder file and updated entries in an online store.
hardpractitioner heuristic - Alert on error-budget burn rate over two windows for model service objectives
A model endpoint has a latency or valid-response objective, and the on-call is paged by a fixed threshold that either fires on every blip or misses a slow degradation.
harddocumented fact - Treat unobserved outcomes as censored by age, not as negatives, when evaluating on recent cohorts
Refund requests can arrive up to 45 days after a booking, and the weekly dashboard counts bookings without a refund as non-refunds.
hardpractitioner heuristic - Budget the tail latency of every dependency when one request fans out to many lookups
Each prediction needs 20 parallel feature lookups, and every store has a good median and a p99 of 80 ms.
hardpractitioner heuristic - Agree the fairness criterion with business and risk owners before auditing groups
A default-risk model is audited by applicant group, and the report lists calibration, false-positive rate and approval rate for two groups whose default prevalence differs (hypothetical).
harddocumented fact - Evaluate against past decision-makers with a selective-labels method, not on approved cases alone
A credit model is scored on repaid versus defaulted loans, but outcomes exist only for applicants that underwriters approved (hypothetical).
hardpractitioner heuristic - Look for bunching just past a KPI target to catch gaming
A branch bonus needs 95 percent of tickets resolved within the service target, and many branches report exactly 95 to 96 percent (hypothetical).
hardpractitioner heuristic