Practical insights
- 1000published
- 25in review
- 8topic families
Results
- Canary a new model on a small traffic slice with pre-agreed rollback criteria
A retrained model is ready, and the plan is to switch all traffic at once on Monday.
mediumdocumented fact - Keep rollback one alias change away and test it before you need it
A new model version misbehaves, and rollback requires rebuilding the old container and feature pipeline.
easydocumented fact - Gate releases on a frozen golden set with tolerances sized to run-to-run noise
A model-quality CI check fails randomly about one run in five, and engineers start rerunning until it passes.
mediumpractitioner heuristic - Instrument a model service with the golden signals plus model-specific signals
A model endpoint has a health check that returns OK, and users report slow and odd recommendations.
easydocumented fact - Plan how deletion requests propagate to snapshots, features and retraining
A customer's data is deleted from the operational database, but it remains in last quarter's training snapshot and feature tables.
hardpractitioner heuristic - Check model and dataset licences before a prototype becomes a product
A prototype uses an open-weights model and a public dataset, and the business wants to launch it commercially.
easydocumented fact - Ship a model card with intended use and performance by relevant slice
A claims-triage model is handed to an operations team with only an overall accuracy figure.
easypractitioner heuristic - Scale governance effort to the risk tier of each model
Every model, from an internal dashboard forecast to a credit decision engine, goes through the same three-month review.
mediumpractitioner heuristic - Price inference per thousand predictions before choosing GPU serving
A team proposes GPU instances to serve a gradient-boosting model that handles 50 requests per second.
mediumpractitioner heuristic - Compare managed and self-hosted ML platforms on total cost including people
Self-hosting an open-source serving stack looks cheaper than a managed service on the cloud bill.
mediumpractitioner heuristic - Track unit cost per decision, not just the monthly cloud bill
Monthly ML infrastructure cost rose 40%, and nobody can say whether it is growth, waste or a new model.
easypractitioner heuristic - Negotiate an ambiguous AI request down to one decision, one metric and a baseline
A director asks the team to 'use AI to optimise the network' within a quarter.
easypractitioner heuristic - Refresh internal benchmark sets so teams stop tuning to a fixed test
Every quarter, teams report improvements on the same internal evaluation set used for two years.
mediumdocumented fact - Estimate the value of a model with a back-of-envelope calculation before building it
A team proposes a three-month project to improve a no-show prediction model.
easypractitioner heuristic - Keep a decision and assumption log so a model survives handover
A modelling project is handed to a new team, and the reasons for its target definition, exclusions and thresholds exist only in old chat threads.
easypractitioner heuristic - Agree the fairness criterion with business and risk owners before auditing groups
A default-risk model is audited by applicant group, and the report lists calibration, false-positive rate and approval rate for two groups whose default prevalence differs (hypothetical).
harddocumented fact - Test for proxies of a sensitive attribute instead of relying on having dropped it
Compliance asks for gender and age to be removed from a pricing model's features, and the model is then described as neutral (hypothetical).
mediumpractitioner heuristic - Check each new model use against the purpose the data was collected for
Support-chat transcripts gathered to resolve tickets are proposed as training data for a sales-propensity model (hypothetical, EU customers).
mediumdocumented fact - Design a contest path into decisions that may fall under solely automated processing
A team plans to reject consumer loan applications straight from a score with no reviewer step (hypothetical, EU applicants).
mediumdocumented fact - Keep a model inventory that links each model to its data, owner and consumers
A source table is being deprecated and nobody can list which models, spreadsheets and vendor scores read it (hypothetical).
mediumpractitioner heuristic - Date-stamp the evaluation plan before looking at candidate results
Five variants and a new feature set are compared with last quarter's champion, with the primary metric, slices and exclusions still open (hypothetical).
mediumpractitioner heuristic - Seed review queues with known-answer cases to test whether reviewers really check
Analysts approve 99 percent of a fraud model's suggested actions (hypothetical).
mediumempirical finding - List who bears each error type before setting the threshold
A claims-flagging threshold is set from investigator hours alone, while flagged customers wait weeks for payment (hypothetical).
mediumpractitioner heuristic - Replace probability words with numeric ranges in written risk statements
A risk memo calls a forecast regression 'likely' and a launch delay 'possible', and three readers take away different odds (hypothetical).
easypractitioner heuristic - Quote an automation benefit as residual human workload, not as the share automated
A business case says a triage model will 'automate 80 percent' of 10,000 monthly cases (hypothetical).
easypractitioner heuristic - Walk stakeholders through twenty concrete predictions before debating the metric
A review meeting keeps stalling on whether an F1 of 0.71 is good (hypothetical).
easypractitioner heuristic - Write a time box and kill criteria into an exploratory modelling brief
An exploratory effort on call transcripts enters its fourth month with no go or no-go date (hypothetical).
easypractitioner heuristic - Price the exit before buying a model or platform
A team picks a hosted risk-scoring API after a strong demo and a three-year contract is on the table (hypothetical).
mediumpractitioner heuristic