LLMs, GenAI, RAG, agents, evaluation · insight-0544

Consider continued pretraining only when you hold a large unlabelled domain corpus

Situation
An insurer's claims text is full of domain jargon, and a general model handles it poorly even after prompt work.
Action
If a large unlabelled in-domain corpus exists, test domain-adaptive continued pretraining followed by task fine-tuning against the plain fine-tuned baseline on the same eval set.
Reason
Gururangan et al. found a second phase of pretraining on in-domain and task text improved downstream performance across several domains.
Boundary
Specialised domains with distinctive language and abundant unlabelled text.

Symptom or risk

The team jumps to supervised fine-tuning on a few labelled examples that cannot teach the vocabulary and style of the domain.

Small corpora or general-language tasks rarely justify it; retrieval or prompting may be enough.

Compute cost, data governance for the corpus, and risk of forgetting general skills.

Pretraining on documents that include eval-set content.

Gururangan et al. found that continued pretraining on in-domain text (DAPT) and on unlabelled task data (TAPT) improved downstream task performance.
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks