Skip to content
ML Academy

Week 9 · Saturday — build and investigate

Retrieval project: versioned corpus, labelled queries, lexical baseline

Project thread rag · linked lessons day 41, day 42, day 43, day 44, day 45

Milestone R1 of the Weeks 9-12 retrieval thread: a versioned corpus snapshot with a manifest, a labelled query and evidence set written before any retrieval run, and executed TF-IDF baseline results with a first failure analysis.

  • 90 minsession length
  • 3timed blocks
  • 4tasks
  • —status

Session plan90 min

  1. 15′Recap and planning
  2. 60′Practical work
  3. 15′Error analysis and recording

Before you start

Before the session (about 10 minutes, outside the 90):

  • Licence and provenance: the corpus is the academy's own owner-authored text under /content (lessons, glossary, insights), so use and redistribution inside your portfolio are permitted by the owner. Exclude content/sources/ (bibliographic records describing third-party works) and do not add external documents. Note in the manifest that many records are still draft and may change; your snapshot is pinned to one commit.
  • Environment: the repository's Python project (numpy, pandas, scikit-learn, PyYAML). Check with uv run --project python python -c "import sklearn, yaml". No model downloads, no API keys.
  • Scaffold: a short loader that reads the YAML files into records (provided scaffold, or about 20 lines with PyYAML) kept in scaffold/. If no scaffold exists and writing one would take more than 10 minutes of the session, use one chunk per record for today and record that simplification.
  • Folders: a new retrieval-project/ repository or folder with corpus/, eval/, runs/, notes/, scaffold/ and src/.

Blocks3

  1. Block 1: Recap and planning

    15 min

    State the project question in one sentence: "Given a learner's question about academy topics, return the passages from the academy's own content that answer it, or say that none do." Then plan:

    • Corpus scope: which folders of the frozen snapshot you include (default: lessons, glossary and insights) and the size cap (if chunking yields more than about 2,000 chunks, restrict to Part II lessons plus the glossary and record that rule).
    • Chunk rule: one rule only, chosen now (default: one chunk per glossary term, per insight record, and per top-level lesson section such as concept intuition or mechanism), with stable chunk ids like lesson-048#concept.mechanism.
    • Evaluation boundary: queries are written and labelled before any retrieval run (Day 43: the evaluation set comes first), and the set will be split into development and held-out parts on Sunday.

  2. Block 2: Practical work

    60 min
    1. Freeze and chunk the corpus (15 min). Record the git commit hash of the snapshot and whether the working tree was clean (uncommitted edits are not in the hash), the included file list, and a SHA-256 per file in corpus/manifest_v1.json. Apply the chunk rule and save corpus/chunks_v1.jsonl (chunk id, source file, record id, field path, status of the source record, text). Print the chunk count and the length distribution in words.
    2. Write and label queries (25 min). Write about 30 queries in your own words in eval/queries_v1.jsonl before running any retrieval. Cover at least: direct look-ups, paraphrases that avoid the source wording, questions needing two chunks, airline-domain phrasing, and at least five questions the corpus cannot answer. For each answerable query, label the relevant chunk ids with a grade (2 = answers it, 1 = partly relevant) by searching the chunk file by hand. Add the author, date and a one-line rationale per label.
    3. Lexical baseline (20 min). TF-IDF vectors (scikit-learn) with cosine similarity, top 10 per query. Save every ranked list with scores to runs/baseline_tfidf_v1.jsonl. Compute recall@5 and MRR@10 on answerable queries, and print the top-1 score for answerable versus unanswerable queries side by side.

  3. Block 3: Error analysis and recording

    15 min

    Read the five worst answerable queries. For each, write one line naming the likely cause: vocabulary mismatch, chunk boundary, the relevant chunk is too long and diluted, a labelling mistake, or a genuinely ambiguous query. Check whether a top-1 score threshold could separate unanswerable queries, without choosing one yet. Write a 100-200 word note with the actual metrics, the failure-cause counts, and the next unresolved question.

Tasks4

Sign in to save your work.

  1. Task 1

    code

    Freeze the corpus at one commit, write the manifest with per-file checksums, apply one chunk rule, and save the chunk file with stable ids.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    writing

    Write about 30 queries before any retrieval run, spanning look-up, paraphrase, two-chunk, airline-domain and at least five unanswerable queries, and label graded relevant chunk ids with rationales.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    code

    Run a TF-IDF cosine baseline, save every ranked list with scores, and compute recall@5 and MRR@10 on answerable queries plus the top-1 score comparison for unanswerable queries.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    analysis

    Classify the five worst answerable queries by likely failure cause and write the 100-200 word Saturday note.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Corpus provenance and versioningCommit hash, file list, checksums, licence/provenance note and one documented chunk rule; chunk ids are stable.Records chunk statistics and a rule for what happens to labels when the corpus version changes.
Evaluation-set qualityQueries are written before retrieval, cover the required types including unanswerable ones, and every label has a grade and rationale.Avoids copying wording from the chunks for paraphrase queries and records which queries are likely to be ambiguous.
Baseline executionEvery ranked list is saved; metrics are computed on the stated number of answerable queries; unanswerable queries are analysed separately.Reports metrics per query type and notes how few queries each type has.
Failure analysisThe worst queries are labelled by cause with evidence from the ranked lists.Separates labelling mistakes from retrieval failures and fixes labels in a new version rather than silently.

Required output

The first part of the Week 9 evidence: a versioned corpus with manifest and chunk file, a versioned labelled query/evidence set (including unanswerable queries) created before retrieval, executed baseline results with recall@5 and MRR@10, and a short failure note. The adaptation ADR follows on Sunday.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.