Distillation and model compression · MLD-0973
Quantization
Representing values with a smaller or constrained numerical set to reduce resource requirements.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Store weights using low-bit representations.
Original caveat
Memory savings do not guarantee speedups on all hardware.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Deployment.
Tags: DL · SLM · LLM.
Aliases: None recorded.
Related concepts
Concepts linking here
Technology connections
- Axolotl — Configuration-driven model fine-tuning toolkit.
- bitsandbytes — Low-bit quantization and memory-saving optimizer primitives.
- Core ML — Apple framework for integrating models into native applications.
- ExecuTorch — PyTorch edge-deployment runtime and tooling.
- Hugging Face PEFT — Parameter-efficient model adaptation methods.
- Hugging Face TRL — Post-training trainers for supervised and preference/RL-style methods.
- LiteRT — Google on-device inference runtime, formerly TensorFlow Lite.
- LLaMA-Factory — Unified training and fine-tuning toolkit for supported models.
- llama.cpp — Portable C/C++ inference engine for supported GGUF models.
- LM Studio — Desktop application for discovering and running local language models.
- MLC LLM — Compiler-based deployment stack for language models.
- NVIDIA Model Optimizer — Model optimization tools including supported quantization, pruning and distillation paths.
- Ollama — Local model-management and inference application with an API.
- OpenRLHF — Distributed RLHF and post-training framework.
- OpenVINO — Toolkit for optimizing and deploying inference across supported hardware.
- TorchAO — PyTorch architecture optimization and quantization tooling.
- torchtune — PyTorch-native post-training recipes; development has wound down.
- Transformers.js — JavaScript execution of supported pretrained models.
- Unsloth — Optimized fine-tuning tooling for supported model families.
- WebLLM — Browser LLM inference engine using WebGPU.
Original references
- Distilling the Knowledge in a Neural Network ↗ · Consulted
- PEFT: LoRA conceptual guide ↗ · Consulted
- PEFT: quantization guide ↗ · Consulted
- Original reference URL ↗