Skip to content
ML Academy

Week 12 · Saturday — build and investigate

Retrieval report, portfolio assembly and a new-condition transfer

Project thread rag · linked lessons day 56, day 57, day 58, day 59, day 60

Milestone R7 (thread close): the final retrieval report, a transfer result under a new corpus or question condition without re-tuning, and portfolio artefacts for all three projects with reproduction status recorded.

  • 90 minsession length
  • 3timed blocks
  • 4tasks
  • —status

Session plan90 min

  1. 15′Recap and planning
  2. 60′Practical work
  3. 15′Error analysis and recording

Before you start

Before the session: the frozen Week 11 configuration runnable from one command, corpus v1 manifest, evaluation set v1.1, all run files and tests, the Week 11 ADR and incident note, and the neural and forecasting project folders. For the transfer condition, prepare (outside the session, about 5 minutes) a second corpus snapshot from the current repository commit using the same chunk rule, and record its commit hash. No new installation.

Blocks3

  1. Block 1: Recap and planning

    15 min

    Open your missing-artefact list from Week 11 Sunday. For each of the three projects (forecasting, neural, retrieval) mark what exists, what is missing, and what will stay missing and be declared as a limitation. Plan the 60 minutes: about 25 for the transfer condition, about 35 for the retrieval report and portfolio checks. Decide which artefacts are private and must not be exported.

  2. Block 2: Practical work

    60 min
    1. Transfer condition (25 min, assessment). Complete the transfer task below with your frozen Week 11 configuration. Do not re-tune and do not open the reference implementation or any reference solution.
    2. Retrieval report (25 min). Write the final report of about 500-800 words plus computed tables: question and scope, corpus provenance and licence, evaluation-set construction and audit, baseline, the controlled improvement on development and held-out data, failure and security tests, latency and cost with their basis, generation mode, today's transfer result, limitations, and your decision with revisit triggers. In retrieval-evaluation mode, the report states that generated-answer competence was not assessed.
    3. Portfolio check (10 min). For each project, a README with the business question, data provenance and licence (synthetic data labelled synthetic), one reproduction command, the key result table, limitations and the decision record. Run each reproduction command once, or record honestly that it was not re-run. Nothing is published automatically; export stays a separate, deliberate step.

  3. Block 3: Error analysis and recording

    15 min

    Review the transfer result: which failures were caused by the new condition, which already existed, and which come from label drift rather than retrieval. Write a 100-200 word note with the numbers, and list the evidence you will bring to Sunday's capstone for each project (one strength, one failure you diagnosed, one decision you defended).

Tasks4

Sign in to save your work.

  1. Task 1

    analysis

    Apply the frozen retrieval system to the new corpus snapshot and new queries described in the transfer task, without re-tuning, and explain every change in results by cause.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    writing

    Write the final 500-800 word retrieval report with computed tables, limitations, generation-mode statement and decision.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    review

    Assemble README, provenance, reproduction command, key results, limitations and decision record for each of the three projects, and record whether each reproduction command was re-run today.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    writing

    List, per project, one strength, one diagnosed failure and one defended decision, each linked to an artefact.

    The expected evidence for this task is shown after you save your own output.

Transfer task

New corpus and question condition. Your frozen retrieval system was built and evaluated on corpus snapshot v1. The academy content has changed since then: records were added and some were edited. Using the second snapshot prepared in setup (the current repository commit, same chunk rule):

  1. Re-index the new snapshot with the frozen configuration. Do not change the retriever, chunk rule, threshold or k.
  2. Measure label drift: for every labelled chunk id in evaluation set v1.1, is it unchanged, edited (same id, different checksum), moved, or removed? Re-check the edited ones by hand and fix labels in a new version v1.2 with a change log.
  3. Write five new queries about records that exist only in the new snapshot, labelled before running retrieval, including at least one that the new snapshot cannot answer.
  4. Report results on v1.1 (old snapshot), v1.2 (new snapshot) and the five new queries, with counts.
  5. Attribute every changed outcome to a cause: corpus change, label change, retriever behavior, or threshold. State whether your Week 11 decision still holds and what you would monitor in production to detect this kind of drift.

If the repository has not changed since your v1 snapshot, use the fallback condition instead: split the three longest documents in the snapshot into two records each with new ids, re-chunk, and perform steps 2-5 on that modified copy.

Rubric4 criteria

CriterionMeetsExceeds
Transfer under a new conditionThe frozen system is applied without re-tuning; label drift is measured; result changes are attributed to causes with evidence.Proposes a maintenance rule for labels and evaluation when the corpus changes, and estimates its cost.
Report quality and honestyThe report covers provenance, evaluation design, comparison, failures, latency and cost basis, generation mode, limitations and a decision, with traceable numbers.States precisely what the evidence supports and what it does not, including small-sample limits and label-agreement rates.
Portfolio reproducibilityEach project has a README with provenance, a reproduction command, results and limitations; reproduction status is recorded honestly.A reviewer could reproduce the key table of each project from the README alone in under ten minutes.
Evidence selectionThe capstone list links a strength, a diagnosed failure and a defended decision to an artefact for each project.The chosen items map to the competencies of the target role and expose at least one genuine weakness.

Required output

Final project evidence and portfolio review: the final retrieval report, the transfer-condition result with label-drift analysis, portfolio READMEs for all three projects with reproduction status, and the capstone evidence list.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.