Skip to content
ML Academy

Week 8 · Saturday — build and investigate

Neural project: one controlled experiment and the baseline comparison

Project thread neural · linked lessons day 36, day 37, day 38, day 39, day 40

Milestone N3 of the neural thread: one controlled regularization or learning-rate ablation with several seeds, a configuration chosen on validation, a single test comparison against the classical baseline, and an explained error and learning-curve comparison.

  • 90 minsession length
  • 3timed blocks
  • 4tasks
  • —status

Session plan90 min

  1. 15′Recap and planning
  2. 60′Practical work
  3. 15′Error analysis and recording

Before you start

Before the session: the Week 7 repository with data/split_v1.npz, a training function that accepts the factor as an argument and a seed, and a one-command run that finishes in under a minute per configuration on a laptop CPU. Same dataset as Week 7 (scikit-learn's bundled digits, or the synthetic airline tabular task if you chose it); no new data or installation. Optional: PyTorch only if it was already installed and used in Week 7.

Blocks3

  1. Block 1: Recap and planning

    15 min

    Re-read your Week 7 Sunday verdict and Week 8 priority. Write the experiment plan in notes/w08-sat-plan.md before touching code:

    • Factor: one regularization or learning-rate factor (for example weight decay, dropout rate, early-stopping patience, or learning rate) and three levels including your current setting.
    • Held fixed: split split_v1, architecture, epochs budget, metric, and data preprocessing.
    • Seeds: at least three per level (the digits network trains in seconds on a CPU, so this fits the budget).
    • Decision rule: how you will pick the final configuration from validation results only, and what difference you will treat as no difference.

    Also note in one line what Day 36 implies for this dataset: a fully connected network ignores that neighbouring pixels are related, which a convolution would exploit.

  2. Block 2: Practical work

    60 min
    1. Run the grid (25 min). Train every level x seed combination. Save per-run validation accuracy, log loss, and the final train loss to runs/w08-ablation.csv, plus the train/validation curves for one seed per level.
    2. Choose on validation (5 min). Apply your pre-written decision rule. Do not look at test results first.
    3. One-time test evaluation (10 min). Freeze the chosen MLP configuration and the logistic-regression baseline, retrain each on the training part with the frozen settings, and evaluate both once on the test part. Record that the test set was opened today and must not drive further changes.
    4. Error comparison (20 min). For both models on test: per-class recall, the confusion matrix, and a learning-curve check (validation metric when training on 25%, 50% and 100% of the training part) for the chosen MLP and the baseline. Save 8-12 misclassified images from each model and look at them.

  3. Block 3: Error analysis and recording

    15 min

    Write a 100-200 word result note: the ablation table summarized as mean and spread per level, the configuration chosen and why, the single test comparison, and which errors are shared by both models versus specific to one. State what the learning-curve check suggests (would more data likely help either model?) and the next unresolved question. Use only printed numbers. If the MLP does not beat the baseline, the note explains that outcome rather than hiding it.

Tasks4

Sign in to save your work.

  1. Task 1

    code

    Run one factor at three levels with at least three seeds each, holding split, architecture, preprocessing and metric fixed, and save every run.

    The expected evidence for this task is shown after you save your own output.

  2. Task 2

    analysis

    Choose the configuration with your pre-written validation rule, then evaluate the frozen MLP and the baseline once on the untouched test part.

    The expected evidence for this task is shown after you save your own output.

  3. Task 3

    analysis

    Compare per-class recall, confusion matrices, misclassified examples and a three-point learning curve for the MLP and the baseline.

    The expected evidence for this task is shown after you save your own output.

  4. Task 4

    writing

    Write the 100-200 word Saturday result note with the next unresolved question.

    The expected evidence for this task is shown after you save your own output.

Rubric4 criteria

CriterionMeetsExceeds
Experiment controlOne factor varies; everything else is fixed and stated; at least three seeds per level; the plan predates the runs.Reports seed spread alongside means and states in advance what size of difference would count as real.
Validation and test disciplineThe configuration is chosen on validation only; test is evaluated once for both models after freezing.Explains what would be lost if test results were used to choose between levels, in terms of the reported estimate.
Comparison qualitySame split, same metric and same training data for both models; per-class and confusion comparisons are shown and described.Connects a model-specific error pattern to model structure (for example, spatial structure that neither a linear model nor an MLP encodes) without claiming more than the evidence supports.
Honest conclusionStates whether the network beat the baseline, by how much, with the limitations of one dataset and few seeds.Says what additional evidence would change the conclusion and whether it is worth its cost.

Required output

The neural comparison and one ablation: the ablation CSV with seeds, the configuration chosen on validation, one test comparison of the frozen MLP against the classical baseline on the same split and metric, an error and learning-curve comparison, and a short note with limitations and the next question.

Studios are tracked separately from the 60 core lessons; completing one never completes a lesson.