Skip to content
ML Academy

Week 10 · Saturday — build and investigate

Retrieval project: context flow, one controlled improvement, honest generation mode

Project thread rag · linked lessons day 46, day 47, day 48, day 49, day 50

Milestone R3: a working retrieval and context flow with schema-validated evidence traces, an allow-listed search tool boundary with tests, one controlled retrieval improvement evaluated on the development set, and an explicitly labelled generation mode.

  • 90 minsession length
  • 3timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 15′Recap and planning
  2. 60′Practical work
  3. 15′Error analysis and recording

Before you start

Before the session: the Week 9 corpus v1, evaluation set v1.1 with its development / held-out split, the baseline run file and scoring script. Environment: the repository's Python project (numpy, scipy, scikit-learn); BM25 can be written in about 15 lines of numpy, so no new package is needed. Generator: only if one is already configured (local model or provider key set up outside the session, with its model name and version known); do not spend session time on installation or account setup. The held-out part stays closed this week.

Blocks3

  1. Block 1: Recap and planning

    15 min

    Re-read the Week 9 failure list and choose exactly one retrieval improvement that targets the most common fixable failure. Candidates (pick one): BM25 instead of TF-IDF weighting; hybrid lexical + latent-semantic (TruncatedSVD) retrieval merged with reciprocal rank fusion; a different chunk rule (for example heading-aware sections of bounded length); or adding the record title to each chunk's text. Write the hypothesis, the development-set metric you expect to move, and what result would count as no improvement.

    Then decide the generation mode and write it down: generated-answer mode only if a real local or provider model is configured and you can record its name and version; otherwise retrieval-evaluation mode, in which no answer quality is claimed.

  2. Block 2: Practical work

    60 min
    1. Context assembler and trace (20 min). Build assemble(query, k) that returns the top-k chunks with chunk id, source file, record status, score and a trust: corpus-content label, trimmed to a stated word budget. Every call writes an evidence trace record: query id, corpus and query-set versions, retriever config, ranked ids and scores, the ids that entered the context, the mode, and the generator name and version (or none). Validate each trace against a small JSON schema you write; a failing trace is an error, not a warning.
    2. Tool boundary (10 min). Expose retrieval only as search(query: str, k: int) with 1 <= k <= 10 and a maximum query length. Anything else (unknown tool name, out-of-range k, extra arguments) is rejected with a logged reason. Write two tests that prove the rejection.
    3. Controlled improvement (20 min). Implement the one chosen change, run it on the development part only, and compare with the baseline on recall@5 and MRR@10 overall and per query type. Count per-query wins, losses and ties.
    4. Generation (10 min). In generated-answer mode: generate answers for five development queries with a structured output (answer text plus a list of cited chunk ids), validate the schema, and check that every cited id was in the assembled context. In retrieval-evaluation mode: use these 10 minutes to write a two-line mode statement for the README and to inspect five traces by hand for context quality.

  3. Block 3: Error analysis and recording

    15 min

    For the queries that the improvement made worse, look at the before/after ranked lists and name the cause. Write a 100-200 word note: the change, the development-set numbers before and after with the query count, wins / losses / ties, the generation mode in one explicit sentence, and the next unresolved question. Do not claim answer quality in retrieval-evaluation mode. A change that does not help is a valid result.

Tasks5

Sign in to save your work.

  1. Task 1

    code

    Build the context assembler with a word budget and trust labels, and write a schema-validated evidence trace for every call.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    code

    Expose retrieval only through an allow-listed search(query, k) tool with bounded arguments and write tests for rejected calls.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    code

    Implement one pre-registered retrieval change and compare it with the baseline on the development set, overall, per query type and per query.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    analysis

    Either validate structured generated answers and their citations against the assembled context (generated-answer mode) or document retrieval-evaluation mode and inspect five traces by hand.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    writing

    Write the 100-200 word Saturday note with before/after numbers, mode statement and open question.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Trace and schema rigorEvery retrieval call produces a schema-valid trace with versions, config, ranked ids and context ids; invalid traces fail.Traces are sufficient to reproduce any reported metric from the files alone.
Boundary enforcementThe search tool is allow-listed with bounded arguments, and tests prove rejections are enforced in code rather than by instruction text.Explains which failure each check prevents and what a remaining gap is.
Controlled comparisonOne change, a pre-registered hypothesis, development set only, metrics with query counts, and per-query win / loss / tie counts.Diagnoses at least one regression from ranked lists and states whether the change should be kept.
Mode honestyThe generation mode is stated explicitly; no generated-answer claim is made without a real configured model and recorded version.In generated-answer mode, reports cited ids outside the context as failures; in retrieval-evaluation mode, lists exactly which competences remain unassessed.

Required output

The Saturday half of the Week 10 evidence: a retrieval prototype with grounded evidence traces, a tested tool boundary, one controlled improvement with development-set comparison, and an accurately labelled generation mode (generated-answer results only with a real configured generator; otherwise retrieval-evaluation mode).

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.