Skip to content
ML Academy

Week 9 · Sunday — consolidate and defend

Retrieval project: label audit, contamination check, adaptation ADR

Project thread rag · linked lessons day 41, day 42, day 43, day 44, day 45

Milestone R2: an audited, versioned evaluation set (v1.1) split into development and held-out parts, documented contamination decisions, baseline limits by query type, and the adaptation ADR.

  • 90 minsession length
  • 4timed blocks
  • 5tasks
  • —status

Session plan90 min

  1. 20′Spaced recall
  2. 40′Evaluation-set audit and baseline limits
  3. 20′Teach-back and decision defense
  4. 10′Weekly review and planning

Before you start

Before the session: Saturday's corpus/ and eval/ files, the baseline run file, and the scoring script that re-computes recall@5 and MRR@10 from a query file and a run file. No new installation or data. An ADR template with headings (context, decision, options, evidence, consequences, revisit trigger) in notes/adr/.

Blocks4

  1. Block 1: Spaced recall

    20 min

    Closed notes, then correct:

    • Day 41: what TF-IDF up-weights and down-weights, and one query type where it beats an embedding model.
    • Day 42: why a fluent answer is not evidence; what temperature and top-p change.
    • Day 43: the missing-capability question behind prompting vs retrieval vs fine-tuning, and why the evaluation set is built before adaptation.
    • Day 44: trainable parameters for a rank-8 LoRA update on a 4096 x 4096 weight matrix, compared with the full matrix.
    • Day 45: two ways a model-based judge can be biased, and what benchmark contamination means.
    • From Week 8: what your neural comparison did and did not prove.

    Tag each miss for review.

  2. Block 2: Evaluation-set audit and baseline limits

    40 min
    1. Label audit (15 min). Without looking at your labels, re-label a random sample of 10 answerable queries from the chunk file. Compare with the original labels: count agreements, list disagreements, and resolve each with a written reason. Save corrections as eval/queries_v1.1.jsonl with a change log; never edit v1 in place.
    2. Contamination check (10 min). For each query, compute the longest shared word sequence (or word n-gram overlap) with its labelled chunks. Flag queries that copy the source wording, and flag any query written after you saw retrieval output. Decide per flagged query: keep as a look-up item, rewrite as a new item, or exclude, and record the decision.
    3. Freeze the split (5 min). Split the audited set into a development part (about two thirds, used for all tuning in Weeks 10-11) and a held-out part (used once in Week 11 for the final comparison), stratified by query type. Record the hash of each file.
    4. What the baseline cannot answer (10 min). Re-score the baseline on v1.1 (development part only) and list the query types it handles badly, with query ids. Separate what retrieval cannot fix (the corpus lacks the answer) from what a better retriever might fix.

  3. Block 3: Teach-back and decision defense

    20 min

    Write the adaptation ADR (architecture decision record, 150-250 words) for a hypothetical academy study assistant built on this corpus: context, the missing capability, options considered (prompting only, retrieval, LoRA or full fine-tuning), decision, evidence used (your baseline results and audit), consequences, and the evidence that would make you revisit the decision. Then defend it aloud in two minutes against this challenge: "Just fine-tune a model on all the lessons; retrieval is unnecessary."

  4. Block 4: Weekly review and planning

    10 min

    Update the weekly log with corrected recall misses, the label-agreement result, contamination decisions and the dataset versions now frozen. Week 10 priority: the context flow and one controlled retrieval improvement chosen from today's failure list, evaluated on the development part only.

Tasks5

Sign in to save your work.

  1. Task 1

    recall

    Answer the Days 41-45 recall set plus one Week 8 item and correct misses; include the LoRA parameter arithmetic.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    review

    Blind re-label 10 random answerable queries, compare with the original labels, resolve disagreements in writing, and publish a new version with a change log.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    analysis

    Flag queries that copy source wording or were written after seeing retrieval output, decide what to do with each, and freeze a stratified development / held-out split.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    analysis

    Re-score the baseline on the development part of v1.1 and separate failures a retriever could fix from questions the corpus cannot answer.

    The expected evidence for this task is shown after you save your own output.

  5. Task 5

    defense

    Write and defend the adaptation ADR choosing among prompting, retrieval and fine-tuning for a study assistant on this corpus.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Label reliabilityA blind re-label sample is compared, disagreements are resolved in writing, and corrections go into a new version.Identifies a systematic labelling pattern (for example, partly relevant chunks graded as fully relevant) and fixes the guideline, not only the items.
Contamination and split disciplineCopied-wording and post-hoc queries are flagged with a stated measure and handled; the split is stratified and hashed.Explains how contamination would inflate the Week 10-11 comparison and how the held-out part protects it.
Adaptation decisionThe ADR names the missing capability, compares at least three options, and cites the learner's own evidence.States quantitatively what evidence would justify fine-tuning instead, and the cost of collecting it.
Recall and reviewRecall misses are corrected and tagged; dataset versions and the Week 10 priority are logged.Links a recall miss (for example, judge bias) to a concrete risk in the learner's own evaluation plan.

Required output

Versioned corpus/evaluation set, baseline results, and adaptation ADR: the audited evaluation set v1.1 with change log and frozen development / held-out split, contamination decisions, baseline limits by query type, corrected recall mistakes, the ADR, and the Week 10 priority.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.