r16 results - scoring service contract, in-process

Recorded run of cd python && uv run python -m reference.r16_scoring_service_contract.run (full mode). Numbers are copied from outputs/results.json, outputs/contract_table.csv and the artifacts in outputs/artifacts/, written by that run. Data: synthetic bank-client contact log (30,000 rows, seed 11), temporal split by position 70/15/15 (train 21,000 / validation 4,500 / test 4,500).

Item	Value
Date	2026-10-12
Seeds	data 11; request stream 5; bootstrap 3; queue simulation 9 and 21
Python / numpy / scikit-learn	3.12.13 / 2.5.3 / 1.9.1 on Darwin arm64
Runtime of the recorded run	8.7 s wall, 8.7 s CPU; smoke run about 5 s; test file about 5 s
Hypothetical inputs	latency budget p99 50 ms; call budget 15% of the candidate list; simulated feature-store lookup (lognormal, median 2 ms, sigma 0.8)

1. Serving contract (from the code, outputs/contract_table.csv)

Field	Type	Required	Allowed / range	Known at	Source
age	int	yes	18..100	before the call	CRM core
job_group	cat	yes	admin, blue_collar, management, retired, student, other	before the call	CRM core
prev_outcome	cat	yes	none, failure, success	end of previous campaign	campaign history
contacts_before	int	yes	0..60	before this call (excludes this call)	campaign history
balance_eur	float	no (imputed)	-50000..5000000	month-end snapshot before the call	core banking
has_housing_loan	int	no (imputed)	0..1	before the call	CRM core
has_personal_loan	int	no (imputed)	0..1	before the call	CRM core
days_since_last_contact	float or null	no	0..4000	before the call; null = never contacted	campaign history
month	int	no	1..12	planning calendar	dialer
weekday	int	no	0..6	planning calendar	dialer
duration_last_call, subscribed, p_true	-	rejected (post_call_field)	-	after the call	-

Artifacts: v1 (fs1, 14 features, sha256 72cb74b0..., call threshold 0.1416) and v2 (fs2, 17 features, sha256 dfbca0f6..., threshold 0.1482).

2. Offline temporal test (4,500 rows, base rate 13.4%; train 9.8%, validation 14.1%)

Model	ROC-AUC	PR-AUC	Brier	ECE (10 bins)	Mean prediction	Precision at 15%	Recall at 15%
rule baseline (previous success, then recency)	0.657	0.347	0.166	0.206	0.340	0.385	0.430
v1 logistic regression (fs1)	0.755	0.417	0.0996	0.037	0.097	0.413	0.462
v2 logistic regression (fs2)	0.771	0.436	0.0979	0.037	0.098	0.424	0.474
v2 + duration_last_call (leaky, not servable)	0.862	0.647	0.0756	0.030	0.104	0.556	0.621

3. Feature parity (1,000 replayed test rows)

Online build_row vs offline build_matrix: maximum absolute feature difference 0.0, maximum absolute score difference 2.2e-16. Negative control (online path adding one to contacts_before): 1,000 of 1,000 rows differ.

4. Load generator, request outcomes and latency

Fixed stream of 20,000 requests (seed 5): 17,957 complete valid, 825 with a required feature absent or null, 603 invalid (post-call field, out-of-range age, wrong type), 496 identical replays (new request_id, same key), 119 key reuses with a changed payload. All 20,000 responses satisfied the response schema; 0 unexpected outcomes; turning the shadow on did not change one primary answer. The first 500 requests of each pass are warm-up and excluded (19,500 measured).

Path (ms, in-process, per request)	p50	p95	p99	p99.9	max	requests over 1 ms
shadow off	0.0210	0.0221	0.0400	0.350	57.4	2
shadow on (v2 also scored)	0.0259	0.0286	0.0416	0.265	69.8	1

By outcome kind, shadow on: complete valid p50 0.026 / p99 0.042 ms; missing required (fallback) 0.017 / 0.027; invalid (422) 0.005 / 0.007; replay 0.016 / 0.032; conflict (409) 0.010 / 0.017. Mean stage times (ms): validate 0.0032, idempotency 0.0046, route + feature build 0.0079, score 0.0015, shadow 0.0048, respond (store, log) 0.0076. Mean service time 0.0292 ms, i.e. about 34,000 requests per second for one worker. The first (cold) pass gave p99 0.038 ms (off) and 0.044 ms (on), within 0.005 ms of the second.

Each 19,500-request pass contained one to three requests above 1 ms, the largest 34-70 ms; their cause (garbage collection or OS scheduling) was not diagnosed. They do not move p99, but a single-worker queue turns one stall into a backlog:

Offered load (Poisson, one worker, measured service times)	Utilisation	p50 ms	p95 ms	p99 ms	p99.9 ms	Share over 50 ms
100 requests/s	0.003	0.026	0.029	0.042	0.271	0.01%
1,000 requests/s	0.03	0.026	0.032	0.049	48.9	0.09%
10,000 requests/s	0.29	0.026	0.73	55.8	68.2	1.4%
25,000 requests/s	0.73	0.051	56.9	67.2	69.4	6.9%
35,956 requests/s (utilisation 1.05)	1.05	0.65	68.3	69.7	70.0	39%

With a simulated remote feature-store lookup added to each service time (lognormal, median 2 ms; assumed, not measured): service-only p50 2.04 / p95 7.44 / p99 12.5 ms; in the queue at 100 / 200 / 300 requests/s (utilisation 0.28 / 0.56 / 0.83) p99 is 17.6 / 30.8 / 73.4 ms, and 0.015% / 0.07% / 6.1% of requests exceed 50 ms.

5. Idempotency (shadow-on pass)

496 replays returned the stored answer with replayed: true and caused no scoring: the model scored 17,957 requests, equal to the number of distinct complete valid requests; 119 key reuses with a changed payload returned 409; 603 invalid requests returned 422 and did not consume a key. A replay after the prod alias moved from v1 to v2 returned the original v1 answer. The LRU evicts the oldest key beyond its capacity (tested with capacity 3; 50,000 in the run, no eviction occurred).

6. Fallback rule under missing required features (first 4,000 test rows, v1 primary)

Each row, with the stated probability, has one required feature set to null.

Required feature missing	Share served by rule	Calls per 1,000	Precision of calls	Subscriptions captured per 1,000
0%	0.0%	143.0	0.428	61.3
5%	4.8%	141.5	0.428	60.5
20%	19.7%	132.5	0.442	58.5
50%	49.8%	118.0	0.434	51.3
rule for every row	100%	123.8	0.459	56.8

7. Shadow comparison, v1 primary vs v2 shadow (same 17,957 scored requests)

Quantity	Value
Spearman rank correlation of scores	0.913
Mean / 95th percentile absolute score difference	0.0177 / 0.0676
Same priority decision	97.8%; 188 clients v2 would call and v1 would not, 213 the reverse
Calls placed (v1 / v2)	2,564 / 2,539
Precision of called clients (v1 / v2)	0.424 / 0.435
AUC on these requests (v1 / v2)	0.7546 / 0.7697; paired bootstrap 95% interval of the difference +0.0110 to +0.0190 (200 resamples)
Failure injection (shadow artifact broken)	270 shadow errors counted; primary answers identical to the baseline for all 300 requests

8. Canary of two candidates (10% routed by client_id hash; 3,000 requests; rollback rule: canary calibration gap

differs from the concurrent control arm by more than 0.03)

Candidate	Share routed	Mean prediction - outcome rate, canary	Same, control	Decision
v2	10.4%	-0.053	-0.033	difference -0.020: keep
v3_bad (v2 with intercept +1.2, simulated bad build)	10.4%	+0.087	-0.033	difference +0.120: rollback

Rollback of v3_bad was an alias change measured at 1.3 microseconds in process; the next 500 requests were all served by v1.

Findings

1. The contract held under a fixed 20,000-request stream. Every response had the schema, each request kind ended in its expected outcome (200 model, 200 fallback, 422, replay, 409), and a model scoring happened exactly once per distinct valid request.
2. Feature parity is exact in code but only because both paths were written together and compared. The offline and online feature functions agree to 0 on 1,000 rows, and a one-line skew is caught on every row; parity is a property of a test that runs, not of the design.
3. The leaky variant looks far better and is not servable: AUC 0.862 against 0.771 and precision at budget 0.556 against 0.424. The post-call field is rejected at the door (post_call_field) and is refused in artifact verification.
4. Both models under-predict on the test period (mean prediction 0.097-0.098 against a 13.4% outcome rate) because the base rate drifted upward after training; ECE is 0.037 for both, and a threshold chosen on validation (14.1%) placed 14.3% of the first 4,000 test clients above it, not 15%. The calibration monitor in the canary sees the same gap in the control arm (-0.033); comparing arms, not absolute gaps, is what lets it separate a bad build from drift.
5. v2 is better than v1 in both the offline test and the shadow, by about 0.015 AUC (shadow interval +0.011 to +0.019), while changing the decision for about 2.2% of clients (401 of 17,957). That is large enough to matter at a call budget and small enough that a shadow phase of this size resolves it.
6. In-process latency is far below the 50 ms hypothetical budget (p99 0.042 ms), validation is the cheapest and response/storage the dearest stage, and shadow scoring adds about 0.003-0.005 ms mean when run inline. The budget is threatened by dependencies and by rare stalls, not by the model: a single stall in 19,500 requests puts the queue's p99 above 50 ms at 10,000 requests/s offered, and a plausible remote lookup does so at about 300 requests/s.
7. The fallback rule is competitive here, which is itself a finding. Rule-only decisions captured 56.8 subscriptions per 1,000 with 123.8 calls (precision 0.459); the model captured 61.3 with 143.0 calls (0.428). At a fixed 15% budget the model ranks better (precision 0.413-0.424 against 0.385), so the benefit of the model over the rule is about 0.03-0.04 in precision at budget, not an order of magnitude, and falling back for half the traffic cost 10 of 61 subscriptions per 1,000.

Success criteria of the project record

Criterion	Status
(1) Serving contract listing, per feature, when known and from which system; post-call fields excluded	Met in the table above and enforced by validation and artifact verification (tests). Owner column of the record is not populated per field.
(2) Parity test on a replayed sample	Met: 1,000 rows, max difference 0.0, with a negative control.
(3) Temporal test result with calibration and precision at the call budget	Met on synthetic data: ECE 0.037, precision at 15% 0.424; calibration is off by the base-rate drift (finding 4).
(4) Shadow-then-canary plan with explicit rollback thresholds	Partly: shadow comparison, canary routing and a calibration-gap rollback were exercised with a measured alias switch. The conversion-rate and latency thresholds of the plan are named below but not simulated.

Plan thresholds (all hypothetical, to be set by the owners): shadow for at least two weeks and until the AUC-difference interval excludes 0 and decision flips are signed off; canary 10% by client hash; roll back if the canary calibration gap differs from control by more than 0.03, conversion among called clients falls more than 20% relative to control over a matured window, the fallback share exceeds 5%, or p99 exceeds the latency budget for 5 consecutive minutes; rollback is an alias switch.

What this does not prove

• Nothing here is evidence about the real UCI Bank Marketing file, a real bank, a real network or real traffic; the data are simulated and the request stream is synthetic.
• Absolute latencies are single-process, single-machine wall-clock measurements without network or serialisation; the stall structure is specific to this laptop at the time of the run and the queue tails above are dominated by it.
• The lookup scenario is an assumed distribution, so its tail numbers show how a budget can be breached, not how it will be.
• The shadow and canary comparisons use labels for every request; in production labels arrive after calls mature, so decisions would lag.
• Idempotency is in-memory and single-process: no persistence, no concurrent duplicates, no clock-based expiry.
• Logistic regressions only; no claim about tree models, calibration methods or uplift (the model predicts response among contacted clients, not the effect of calling).

Reproduce

cd python && uv run python -m reference.r16_scoring_service_contract.run
cd python && uv run pytest tests/test_r16_scoring_service_contract.py