Skip to content
ML Academy

Week 11 · Sunday — consolidate and defend

Retrieval project: controlled incident and production ADR defense

Project thread rag · linked lessons day 51, day 52, day 53, day 54, day 55

Milestone R6: a controlled security/reliability incident with detection, fix and residual risk documented, and a defended production ADR with explicit budget assumptions.

  • 90 minsession length
  • 4timed blocks
  • 4tasks
  • —status

Session plan90 min

  1. 20′Spaced recall
  2. 40′Controlled incident
  3. 20′Teach-back and decision defense
  4. 10′Weekly review and planning

Before you start

Before the session: a test copy of the corpus and index (a copy command prepared and checked in advance), the Week 11 Saturday tests and trace output, the generator or labelled stub with a configurable failure rate, and an ADR template. No new installation. All injected content is synthetic and labelled as test data.

Blocks4

  1. Block 1: Spaced recall

    20 min

    Closed notes, then correct:

    • Day 51: compute recall@3, precision@3 and the reciprocal rank for one query with a ranked list and labels you write down; explain why faithfulness and answer relevance are different checks.
    • Day 52: which failures are safe to retry, which are not, and why an idempotency key matters.
    • Day 53: direct versus indirect prompt injection; three enforceable controls and one control that is not enforceable.
    • Day 54: a cost-per-1,000-requests formula from input tokens, output tokens and per-token prices (use hypothetical prices and label them).
    • From Weeks 7-10: one item you missed before, chosen from your review queue.

  2. Block 2: Controlled incident

    40 min

    Run a controlled incident on a test copy of your system, never on your frozen corpus or evaluation files.

    1. Inject (10 min). Add three clearly labelled synthetic chunks to the test copy: one with an indirect instruction (for example, to cite a restricted record or to call a tool), one near-duplicate of a popular chunk with one fact changed, and one restricted chunk on a frequently asked topic. Configure the generator or stub to fail with timeouts for a fraction of calls.
    2. Observe (10 min). Run the development queries. Using only traces and test output, find what went wrong: which queries retrieved the injected or altered chunks, whether any restricted chunk reached the context, whether any retry storm or unsupported answer occurred.
    3. Contain and fix (10 min). Apply the smallest enforceable fix (for example, provenance check at ingestion, duplicate detection, access filter before ranking, retry budget), re-run, and show the before/after test results.
    4. Incident note (10 min). 150-250 words: timeline, detection signal, impact on which queries, root cause, fix, residual risk, and the monitoring signal you would add.

  3. Block 3: Teach-back and decision defense

    20 min

    Write the production ADR (200-300 words) for serving the study assistant to a hypothetical cohort (state the users, requests per day and latency target as labelled assumptions): data flow, retriever choice backed by your held-out evidence, generation mode, security controls proven by tests, observability, budget per 1,000 requests with its basis, rollback plan, and revisit triggers. Then defend it aloud for three minutes against two challenges: "Use the largest available model; quality is all that matters" and "Your evaluation set is too small to justify anything."

  4. Block 4: Weekly review and planning

    10 min

    Log corrected recall misses, the incident note, ADR decisions and open risks. Week 12 priority: finalize the retrieval report and portfolio artefacts for all three projects, then the capstone defense. List each missing artefact by project.

Tasks4

Sign in to save your work.

  1. Task 1

    recall

    Answer the Days 51-54 recall set plus one older review item, including the ranked-list metrics and the cost formula with labelled hypothetical prices, and correct misses.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    code

    Inject the three synthetic chunks and the failure rate into a test copy, detect the impact from traces and tests, apply the smallest enforceable fix, and re-run.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    writing

    Write the 150-250 word incident note with timeline, detection signal, impact, root cause, fix, residual risk and a new monitoring signal.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    defense

    Write and defend the production ADR with labelled assumptions, evidence-backed retriever choice, generation mode, tested controls, observability, budget basis and rollback.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Incident detection and containmentEach injected problem is detected from traces or tests, the fix is enforceable, and before/after results are shown on a test copy.Identifies which detection signal would have caught the problem in production first and how quickly.
Incident communicationThe note has timeline, impact with ids, root cause, fix and residual risk in plain language.Separates contributing factors from the root cause and proposes an owner for each follow-up.
Production design and budgetThe ADR ties the retriever choice to held-out evidence, states the generation mode, and labels every budget number as measured or hypothetical.Shows the quality-cost trade-off of at least two options and the condition under which the cheaper option wins.
Defense under challengeBoth challenges are answered with evidence rather than assertion.Concedes a valid part of a challenge and states what additional evidence would resolve it.

Required output

Evaluation comparison, failure tests, observability output, and a production ADR, completed: corrected recall, the controlled incident with fix and before/after evidence, the incident note, the production ADR with labelled budget assumptions, and the Week 12 priority with a missing-artefact list.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.