Regularization and generalization · MLD-1327
Weight tying
Using the same parameters in multiple model locations.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Share token input embeddings with output projection weights.
Original caveat
Sharing can constrain flexibility as well as save parameters.
Context and aliases
Level: Intermediate. Stages: Theory · Training · Evaluation.
Tags: ML · DL · SLM · LLM.
Aliases: Parameter sharing.
Related concepts
Concepts linking here
Original references
- Deep Learning: online textbook ↗ · Consulted
- Google Machine Learning Glossary ↗ · Consulted
- PyTorch neural-network layers ↗ · Reference link
- Original reference URL ↗