Concept map: PEFT and LoRA

Taught and used in

LLM architecture and adaptation, MLOps and LLMOps

Insights 14

• Fine-tune for consistent behaviour that long prompts cannot hold
• Curate a small, consistent fine-tuning set before collecting a large noisy one
• Rerun safety and refusal tests after every fine-tune, even on benign data
• Do the LoRA parameter arithmetic before planning hardware
• Expect LoRA to learn less and forget less than full fine-tuning
• Tune learning rate and target modules before raising LoRA rank
• Use QLoRA to fit a larger base model, then evaluate the quantized result honestly
• Evaluate the exact artefact you will serve, not the training checkpoint

Projects 6

• Zero-shot time-series foundation models vs strong baselines on post-release monthly data (docs-only)
• LLM-as-judge reliability study: agreement with human pairwise labels, position bias and length bias
• Golden-set regression suite that gates prompt changes for a claims-intake triage assistant
• Summary faithfulness evaluation for legislative bills: human claim labels vs automatic checks
• LoRA/QLoRA adaptation plan for fine-grained banking intents (implementation-ready, docs-only, GPU optional)
• LLM question-answering service: versioned golden set, regression gate and request observability