Engineering, production, governance, cost, communication · insight-1262
Gate traffic on readiness that includes model load and warm-up, separate from liveness
- Evidence type
- documented fact
- Difficulty
- medium
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A model container passes its health check as soon as the web process starts, but loading a large artifact and filling feature caches takes two minutes.
Action
Report ready only after the artifact is loaded and a warm-up request has passed, use a startup allowance for slow starts, keep liveness a cheap process check, and scale out ahead of the load.
Reason
Orchestrators route traffic by readiness and restart by liveness; mixing the two either sends traffic to unready pods or kills ones that are still starting.
Boundary
Containerised model servers, especially with large artifacts, GPU initialisation or in-memory feature caches.
Symptom or risk
Early requests time out or hit cold caches, autoscaling adds more cold instances during a spike, and a restart loop can follow if the liveness check is too strict.
Exceptions
Tiny models that load in milliseconds do not need a separate warm-up stage.
Trade-offs
A longer time to ready slows scale-out and deploys, so keep spare capacity during releases.
Pitfalls
A readiness check that calls an external dependency, so a feature-store blip removes every pod at once.
Lessons and concepts
Claims and their sources
Kubernetes readiness probes decide whether a pod receives traffic and do not restart the container, startup probes give slow-starting containers time before liveness checks apply, and liveness probes restart broken containers.