Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Compute training and serving features from one shared definition
Training features are built in a SQL notebook, and the online service reimplements them in application code.
mediumpractitioner heuristic - Log served features and replay them to measure skew, not just predictions
Offline AUC was 0.82, and live performance on matured labels is clearly lower.
mediumpractitioner heuristic - Give every feature a freshness SLA and enforce it at serving time
A fraud model's 'transactions in the last hour' feature comes from a stream that sometimes lags by 40 minutes.
mediumpractitioner heuristic - Design a degraded mode for when upstream features are late or missing
The nightly feature pipeline fails, and the pricing model must still publish recommendations by 06:00.
mediumpractitioner heuristic - Write a data contract for every upstream table a model depends on
A model breaks because an upstream team renamed a status code and nobody told the data science team.
easypractitioner heuristic - Enforce backward-compatible schema changes and upgrade consumers first
An event producer wants to add a field and drop an old one in the stream your features read from.
mediumdocumented fact - Monitor distributions to catch semantic changes that schemas cannot see
A price column switches from cents to currency units in a release; schema, types and null rates are unchanged.
easypractitioner heuristic - Make batch scoring writes idempotent so retries cannot duplicate decisions
A nightly scoring job times out halfway and the scheduler retries it.
easypractitioner heuristic - Deduplicate late-arriving corrections by business key and load time, explicitly
A flight-day feed sometimes sends the same row twice, the second with an updated revenue figure.
mediumpractitioner heuristic - Exclude immature labels when building training data for slow outcomes
A default model labels loans as 'not defaulted' if no default is recorded yet, including loans issued last month.
mediumpractitioner heuristic - Monitor leading indicators while labels are delayed, then confirm with matured labels
Fraud chargebacks arrive up to 90 days after the transaction, so model accuracy for this month is unknown.
mediumpractitioner heuristic - Page on model impact symptoms, and send per-feature drift tests to a dashboard
A drift monitor runs a statistical test on 300 features daily and pages on every p-value below 0.05.
mediumdocumented fact - Confirm that drift hurts performance before retraining on it
A drift report shows the age distribution of applicants shifted, and the team schedules an urgent retrain.
mediumpractitioner heuristic - Use shadow mode to test plumbing and agreement, not to prove business impact
A new ranking model runs in shadow for two weeks, scores agree 92% with production, and the team claims a revenue uplift.
mediumpractitioner heuristic - Canary a new model on a small traffic slice with pre-agreed rollback criteria
A retrained model is ready, and the plan is to switch all traffic at once on Monday.
mediumdocumented fact - Keep rollback one alias change away and test it before you need it
A new model version misbehaves, and rollback requires rebuilding the old container and feature pipeline.
easydocumented fact - Define reproducibility as recoverable inputs plus tolerances, not identical bits
An auditor asks you to reproduce last year's model, and retraining gives slightly different numbers.
mediumdocumented fact - Record why a model version was promoted, not only that it was
Six months after promotion, nobody can explain why version 14 replaced version 12 or what evidence was used.
easypractitioner heuristic - Put data, feature and model-quality tests in CI, not only unit tests for code
The ML repository's CI runs Python unit tests, and a broken join still reached production.
mediumdocumented fact - Gate releases on a frozen golden set with tolerances sized to run-to-run noise
A model-quality CI check fails randomly about one run in five, and engineers start rerunning until it passes.
mediumpractitioner heuristic - Instrument a model service with the golden signals plus model-specific signals
A model endpoint has a health check that returns OK, and users report slow and odd recommendations.
easydocumented fact - Make every automated decision reconstructable from a decision record
A customer disputes a declined credit-limit increase from eight months ago, and the team must explain it.
mediumpractitioner heuristic - Minimise personal data in features and give training snapshots a retention period
A churn model's training set copies full customer records, including fields the model never uses, and snapshots are kept indefinitely.
mediumdocumented fact - Plan how deletion requests propagate to snapshots, features and retraining
A customer's data is deleted from the operational database, but it remains in last quarter's training snapshot and feature tables.
hardpractitioner heuristic - Check model and dataset licences before a prototype becomes a product
A prototype uses an open-weights model and a public dataset, and the business wants to launch it commercially.
easydocumented fact - Write a datasheet for any dataset other teams will train on
A curated training table is published internally, and three teams start building models on it.
easypractitioner heuristic - Ship a model card with intended use and performance by relevant slice
A claims-triage model is handed to an operations team with only an overall accuracy figure.
easypractitioner heuristic - Scale governance effort to the risk tier of each model
Every model, from an internal dashboard forecast to a credit decision engine, goes through the same three-month review.
mediumpractitioner heuristic - Price inference per thousand predictions before choosing GPU serving
A team proposes GPU instances to serve a gradient-boosting model that handles 50 requests per second.
mediumpractitioner heuristic - Compare managed and self-hosted ML platforms on total cost including people
Self-hosting an open-source serving stack looks cheaper than a managed service on the cloud bill.
mediumpractitioner heuristic - Use batch scoring when decisions are made on a schedule
A team plans a low-latency online endpoint for a churn model whose scores are used in a weekly retention campaign.
easypractitioner heuristic - Track unit cost per decision, not just the monthly cloud bill
Monthly ML infrastructure cost rose 40%, and nobody can say whether it is growth, waste or a new model.
easypractitioner heuristic - Set retraining cadence from a measured decay curve and the cost of each retrain
A model is retrained daily because the platform makes it easy, and nobody has checked whether daily retrains help.
mediumpractitioner heuristic - Declare model incidents early and give one person stakeholder communication
A pricing model has produced obviously wrong fares for an hour, and engineers are debugging while managers message them for updates.
easydocumented fact - Run blameless postmortems with triggers defined before incidents happen
After a model rollback, the team discusses who merged the bad change and moves on without writing anything.
easydocumented fact - Monitor default and fallback rates to catch silent model failures
An upstream join breaks, a key feature becomes null, and the pipeline fills it with the training median.
easypractitioner heuristic - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Open an executive memo with the decision, the evidence range and the main limit
A four-page analysis of an overbooking model starts with data sources and methods, and the recommendation is on page four.
easyopinion - Report a null result with its interval and detectable effect, not as 'no effect'
A test of a new recommendation model shows no statistically significant revenue change, and the summary says 'the model does not work'.
mediumdocumented fact - Present forecasts to decision makers as ranges tied to the actions they support
Leadership receives a single number for next season's demand and builds the capacity plan on it.
easypractitioner heuristic - Pair any model-driven KPI target with guardrail metrics to resist Goodhart effects
A call-centre routing model is rewarded on average handle time, and handle time falls while repeat calls rise.
mediumpractitioner heuristic - Refresh internal benchmark sets so teams stop tuning to a fixed test
Every quarter, teams report improvements on the same internal evaluation set used for two years.
mediumdocumented fact - Look for hidden feedback loops when a model influences its own future training data
A pricing model's recommendations shape the fares customers see, and next month's model trains on the resulting bookings.
harddocumented fact - Retire models whose lift over a simple rule no longer pays for their upkeep
A churn model built three years ago still runs, but recent evaluations show it barely beats a tenure-and-usage rule.
mediumopinion - Give every production model a named owner, a runbook and a support path
A model alert fires at night, and the person who built it left the company months ago.
easypractitioner heuristic - Adopt a feature store when features are shared or served online, not by default
A team with two batch models plans to deploy a feature store as its first MLOps step.
mediumopinion - Classify data-quality checks into blocking and warning tiers before they fire
A new validation suite fails the nightly pipeline on a minor null-rate rise, and the business misses its morning report.
easypractitioner heuristic - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Limit and explain forecast churn between model versions
A new forecasting model version changes next month's plan by 15% for some routes, though inputs barely changed.
mediumpractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Validate requests at the serving boundary and refuse them instead of imputing silently
A scoring endpoint receives a request with a renamed field, a string where a number is expected, or an extra column, and the service quietly coerces it and returns a score.
mediumdocumented fact - Retire fields in an event or response schema; never reuse a name or number with a new meaning
A team wants to repurpose an unused field in the prediction log, for example turning a legacy score column into a calibrated probability.
mediumdocumented fact - Key cached predictions by entity, model version and feature version
A service caches scores per customer for 24 hours to save compute, and a new model version ships at noon.
mediumpractitioner heuristic - Version reference tables and mappings as model inputs, and freeze them until the new version is vetted
A model uses a lookup table, such as airport-to-region mapping or product-category codes, that another team refreshes whenever it likes.
mediumdocumented fact - Load the model artifact in a clean copy of the serving environment before release
A model is trained in a notebook environment with recent library versions and shipped as a pickle to a container with older ones.
easydocumented fact - Make lineage queryable from dataset to models so a bad table has a known blast radius
A data engineer finds that last month's customer table had a bad join, and nobody can say which models trained on it.
mediumpractitioner heuristic - Gate traffic on readiness that includes model load and warm-up, separate from liveness
A model container passes its health check as soon as the web process starts, but loading a large artifact and filling feature caches takes two minutes.
mediumdocumented fact - Test the trained artifact against the real serving binary, not just the training code path
Modeling code changes weekly, while the serving binary changes monthly, and a new model uses an operation or preprocessing step the deployed server does not know.
mediumdocumented fact - Set canary duration from event volume and cycle length, not from a clock
A canary for a booking-conversion model runs for 30 minutes at 5% of traffic, and the rollout then goes to 100% on 'no issues'.
harddocumented fact - Keep shadow deployments free of side effects, quotas and shared state
A candidate model is deployed in shadow mode and receives a copy of production requests, but it shares the production feature store, rate limits and an outbound notification client.
mediumpractitioner heuristic - Roll back the model together with its preprocessing, schema and stored state
A model v7 is rolled back to v6 after a regression, but v7 introduced a new feature column, a new encoder file and updated entries in an online store.
hardpractitioner heuristic - Alert on error-budget burn rate over two windows for model service objectives
A model endpoint has a latency or valid-response objective, and the on-call is paged by a fixed threshold that either fires on every blip or misses a slow degradation.
harddocumented fact - Aggregate latency from histogram buckets, never by averaging per-instance percentiles
A dashboard shows model p99 latency as the mean of the p99 values reported by 12 replicas, and it looks fine while users complain.
mediumdocumented fact - Re-derive decision thresholds whenever the model is retrained
A churn model's score cutoff of 0.35 was tuned for capacity in the first release and is hard-coded in the campaign job, while the model is retrained monthly.
mediumdocumented fact - Treat unobserved outcomes as censored by age, not as negatives, when evaluating on recent cohorts
Refund requests can arrive up to 45 days after a booking, and the weekly dashboard counts bookings without a refund as non-refunds.
hardpractitioner heuristic - Assign a prediction id at serving time and join delayed outcomes to it
Outcomes arrive weeks later keyed by customer and date, while the customer was scored several times by two model versions in the meantime.
mediumpractitioner heuristic - Add hysteresis and a cooldown to automated retraining triggers
A retraining job starts whenever a drift statistic crosses a threshold, and the statistic oscillates around the line for a week.
mediumpractitioner heuristic - Mitigate first with a kill switch, then diagnose, when a model-driven action is hurting users
A pricing model starts producing clearly wrong fares, and the team spends an hour in notebooks looking for the root cause while it keeps publishing.
mediumdocumented fact - After an incident, replay the affected window with the corrected model into a separate table
A feature bug corrupted scores for 38 hours, and decisions were made on them before the fix.
mediumpractitioner heuristic - Record time to detect and the detecting signal for every model incident
Three incidents in a quarter were each noticed by a business user, and the postmortems list different root causes.
mediumpractitioner heuristic - Add jitter, a cap and a retry budget when many batch workers call a model service
A nightly scoring job runs 200 workers against the model endpoint, which has a 30-second brownout, and every worker retries on the same fixed schedule.
mediumdocumented fact - Set the dynamic batching delay from the latency objective, then measure what it buys
A GPU-backed ranking model serves single requests at 15% utilisation, and a proposal is to batch requests for up to 100 ms.
mediumdocumented fact - Budget the tail latency of every dependency when one request fans out to many lookups
Each prediction needs 20 parallel feature lookups, and every store has a good median and a p99 of 80 ms.
hardpractitioner heuristic - Agree the fairness criterion with business and risk owners before auditing groups
A default-risk model is audited by applicant group, and the report lists calibration, false-positive rate and approval rate for two groups whose default prevalence differs (hypothetical).
harddocumented fact - Test for proxies of a sensitive attribute instead of relying on having dropped it
Compliance asks for gender and age to be removed from a pricing model's features, and the model is then described as neutral (hypothetical).
mediumpractitioner heuristic - Evaluate against past decision-makers with a selective-labels method, not on approved cases alone
A credit model is scored on repaid versus defaulted loans, but outcomes exist only for applicants that underwriters approved (hypothetical).
hardpractitioner heuristic - Count rows that are unique on indirect identifiers before releasing a table
An extract with names and ids removed, but birth date, postcode and gender intact, is to be shared with a partner for research (hypothetical).
mediumdocumented fact - Treat hashed customer ids as pseudonymous personal data and keep the key apart
Emails in a training table are replaced by plain hashes, and the table is called anonymous and sent to a vendor (hypothetical).
mediumdocumented fact - Check each new model use against the purpose the data was collected for
Support-chat transcripts gathered to resolve tickets are proposed as training data for a sales-propensity model (hypothetical, EU customers).
mediumdocumented fact - Design a contest path into decisions that may fall under solely automated processing
A team plans to reject consumer loan applications straight from a score with no reviewer step (hypothetical, EU applicants).
mediumdocumented fact - Keep consent as a time-stamped event table and apply it when snapshots are built
A marketing-preference model is retrained monthly, and some customers withdrew consent after the previous snapshot (hypothetical).
mediumdocumented fact - Keep a model inventory that links each model to its data, owner and consumers
A source table is being deprecated and nobody can list which models, spreadsheets and vendor scores read it (hypothetical).
mediumpractitioner heuristic - Date-stamp the evaluation plan before looking at candidate results
Five variants and a new feature set are compared with last quarter's champion, with the primary metric, slices and exclusions still open (hypothetical).
mediumpractitioner heuristic - Look for bunching just past a KPI target to catch gaming
A branch bonus needs 95 percent of tickets resolved within the service target, and many branches report exactly 95 to 96 percent (hypothetical).
hardpractitioner heuristic - Version KPI definitions like code and restate history when they change
Finance changes 'active customer' from 30 to 90 days of activity in a dashboard, and the trend shows a jump in March (hypothetical).
easypractitioner heuristic - Seed review queues with known-answer cases to test whether reviewers really check
Analysts approve 99 percent of a fraud model's suggested actions (hypothetical).
mediumempirical finding - List who bears each error type before setting the threshold
A claims-flagging threshold is set from investigator hours alone, while flagged customers wait weeks for payment (hypothetical).
mediumpractitioner heuristic - Replace probability words with numeric ranges in written risk statements
A risk memo calls a forecast regression 'likely' and a launch delay 'possible', and three readers take away different odds (hypothetical).
easypractitioner heuristic - Use quantile dotplots, not only interval bars, when non-specialists decide under uncertainty
A gate-staffing planner receives delay forecasts as 80 percent interval bands and acts on the midpoint anyway (hypothetical).
mediumempirical finding - Quote an automation benefit as residual human workload, not as the share automated
A business case says a triage model will 'automate 80 percent' of 10,000 monthly cases (hypothetical).
easypractitioner heuristic - Walk stakeholders through twenty concrete predictions before debating the metric
A review meeting keeps stalling on whether an F1 of 0.71 is good (hypothetical).
easypractitioner heuristic - Gate project ranking on decision owner and label readiness before scoring value
A backlog of fourteen ideas is ranked by sponsor seniority and novelty (hypothetical).
mediumpractitioner heuristic - Write a time box and kill criteria into an exploratory modelling brief
An exploratory effort on call transcripts enters its fourth month with no go or no-go date (hypothetical).
easypractitioner heuristic - Keep a small permanent randomised control share to measure what the model earns
Six months after launch, finance asks what the retention model earned, and the only evidence is before-and-after revenue (hypothetical).
mediumpractitioner heuristic - Price the exit before buying a model or platform
A team picks a hosted risk-scoring API after a strong demo and a three-year contract is on the table (hypothetical).
mediumpractitioner heuristic - Hand engineers a specification with test vectors, not a notebook
A data scientist passes a notebook to engineers to reimplement in the serving stack (hypothetical).
mediumpractitioner heuristic - Rehearse a handover by letting the new owner run a full cycle while the builder watches
The builder of a monthly forecasting pipeline is moving teams, and a handover document has been written (hypothetical).
easypractitioner heuristic - Generate load at a fixed arrival rate when load-testing a scoring service, not one request after another
A load test of a model endpoint uses fixed virtual users, each sending its next request after the previous response, and reports a clean p99.
mediumdocumented fact - Treat a pickled model file as executable code and control who can write the artifact path
A scoring service loads a model with pickle, joblib or an unrestricted torch.load from a shared bucket or registry path that more people can write to than can deploy.
mediumdocumented fact - Keep unbounded identifiers out of metric labels and log them as events instead
A team adds customer id or request id as a label on a model-monitoring metric to see one user's score trend.
easydocumented fact