Distillation and model compression · MLD-0843
Online distillation
Distillation where teacher signals or teacher/student models are updated during the training process.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Train peers jointly with mutual prediction targets.
Original caveat
Online does not necessarily mean on-policy.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Deployment.
Tags: DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Original references
- Distilling the Knowledge in a Neural Network ↗ · Consulted
- PEFT: LoRA conceptual guide ↗ · Consulted
- PEFT: quantization guide ↗ · Consulted
- Original reference URL ↗