Skip to content
ML Academy

Week 11 · Saturday — build and investigate

Retrieval project: held-out ablation, failure tests, latency, cost and traces

Project thread rag · linked lessons day 51, day 52, day 53, day 54, day 55

Milestone R5: a held-out ablation comparison, automated failure and security tests, measured retrieval latency with a stated cost basis, and retry / trace checks with an explicit unavailable outcome.

  • 90 minsession length
  • 3timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 15′Recap and planning
  2. 60′Practical work
  3. 15′Error analysis and recording

Before you start

Before the session: corpus v1, evaluation set v1.1 with the still-closed held-out part, both retriever configurations runnable from one command, the trace schema and existing tests. Environment: the repository's Python project plus pytest (already listed). No new installation. A generator only if already configured in Week 10; otherwise the stub is used and every output from it is labelled stub.

Blocks3

  1. Block 1: Recap and planning

    15 min

    Freeze the two configurations to compare: the Week 9 baseline and the Week 10 improvement as finally chosen on the development set. Write down, before any run: the metrics (recall@5, MRR@10, nDCG@10 using your graded labels), the held-out query count, and how you will report a difference given that few queries (per-query wins / losses / ties, optionally a paired bootstrap interval; no significance claims from a handful of queries). List the failure tests you will add and the hardware you will measure latency on.

  2. Block 2: Practical work

    60 min
    1. Held-out ablation (15 min). Run both frozen configurations once on the held-out part. Report the three metrics overall and per query type with counts, and per-query wins / losses / ties. This is the only use of the held-out part for this comparison.
    2. Failure tests (20 min). Add automated tests that run in a few seconds:
      • Unanswerable queries: with a score threshold chosen on the development set, check how many held-out unanswerable queries are correctly marked "no supporting passage" and how many answerable ones are wrongly refused.
      • Untrusted content: add a clearly labelled synthetic test chunk to a test copy of the corpus that contains an instruction (for example, to reveal restricted records); assert that it is passed only as data with its trust label, that no tool call can be triggered from chunk text, and, in generated-answer mode, that the output follows the schema.
      • Access filter: tag a few test chunks restricted; assert they never reach the context for the learner role.
      • Input limits: empty, very long and non-text queries are rejected with a logged reason.
    3. Latency and cost (10 min). Time retrieval and context assembly for every held-out query (repeat each several times) and report p50 and p95 with the hardware and corpus size. Count context tokens or words per request. In generated-answer mode, record measured generation latency and the provider's billed or reported token counts; otherwise compute cost only from clearly labelled hypothetical prices.
    4. Retry and trace checks (15 min). Wrap the generator call (or a stub that simulates timeouts and rate limits, labelled as a stub) with a timeout, at most two retries with exponential backoff and jitter, and an idempotency key per request. Each attempt is written to the trace with request id, attempt number, outcome and duration. Write a test showing that after the retry budget is exhausted the system returns an explicit "service unavailable" result rather than an unsupported answer.

  3. Block 3: Error analysis and recording

    15 min

    Read every held-out loss of the improved configuration and every failed failure-test case. Write a 100-200 word note: held-out numbers with counts, whether the Week 10 decision survives, threshold trade-off, measured latency, cost basis (measured or hypothetical), and the next unresolved question. A held-out result that contradicts the development result is reported as found.

Tasks5

Sign in to save your work.

  1. Task 1

    code

    Evaluate the frozen baseline and improved configurations once on the held-out part with recall@5, MRR@10 and nDCG@10, overall, per query type and per query.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    code

    Add automated tests for unanswerable queries, an untrusted synthetic instruction chunk, a restricted-chunk access filter, and malformed input.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    analysis

    Measure p50 and p95 retrieval and assembly latency on stated hardware and compute cost per request with a stated basis.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    code

    Add a timeout, bounded retries with backoff and jitter, and idempotency keys around the generator or stub, and test the exhausted-budget behavior.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    writing

    Write the 100-200 word note on held-out results, failure tests, latency and cost basis.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Held-out comparisonBoth frozen configurations are evaluated once on the held-out part with counts and per-query outcomes.Interprets the result in light of the label-agreement rate and the small query count, without overclaiming.
Failure and security coverageTests cover unanswerable queries, untrusted content, access filtering and malformed input, and run automatically.Documents a failing case honestly and states the residual risk the tests do not cover.
Measurement honestyLatency is measured with repetitions on stated hardware; cost is labelled measured or hypothetical with the formula.Separates retrieval latency from generation latency and identifies the dominant term.
Reliability mechanicsRetries are bounded with backoff, requests carry idempotency keys, and exhaustion produces an explicit unavailable result recorded in the trace.Explains which failures should not be retried at all and enforces that in code.

Required output

The Saturday half of the Week 11 evidence: an evaluation comparison on the held-out part, failure and security tests, observability output (traces with attempts, timings and versions), measured latency, and a cost estimate whose basis (measured or hypothetical) is explicit. The production ADR follows on Sunday.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.