Language-model training and adaptation · MLD-0332
Domain-adaptive pretraining
Continued pretraining on text from a target domain.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Adapt language representations to technical writing.
Original caveat
Domain adaptation can reduce general-domain performance.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Evaluation.
Tags: DL · SLM · LLM · NLP.
Aliases: DAPT.
Related concepts
Concepts linking here
Original references
- Transformers glossary ↗ · Consulted
- PEFT: LoRA conceptual guide ↗ · Consulted
- TRL: supervised fine-tuning ↗ · Reference link
- TRL documentation ↗ · Consulted
- Original reference URL ↗