Engineering, production, governance, cost, communication · insight-0263
Make batch scoring writes idempotent so retries cannot duplicate decisions
- Evidence type
- practitioner heuristic
- Difficulty
- easy
- Roles
- ML engineer
- Industries
- cross-industry
Summary: situation, action, reason, boundary
Situation
A nightly scoring job times out halfway and the scheduler retries it.
Action
Key outputs by natural key plus run date, write with upsert or overwrite-partition semantics, and pass an idempotency key to any external side effect such as emails or API calls.
Reason
Retries are normal in distributed systems; idempotent writes make running a job twice produce the same result as running it once.
Boundary
Batch scoring, ETL, notification and messaging jobs, payment triggers.
Symptom or risk
Rows from the first attempt stay in the output table, so customers receive two offers or a downstream sum doubles.
Exceptions
Append-only audit logs should keep every attempt, labelled with an attempt id.
Trade-offs
Upserts are slower than appends on some warehouses.
Pitfalls
Using a generated UUID per attempt as the key, which defeats deduplication.
Lessons and concepts
Claims and their sources
Resend accepts an idempotency key so a repeated send request with the same key inside its retention window (24 hours when checked on 2026-10-11; it differs between services) returns the original response without sending the email again.