Model Distillation
モデル蒸留は、より小さく、複雑性の低いモデル（生徒）が、より大きく、より複雑なモデル（教師）の動作を模倣するように訓練される技術です。
モデル蒸留は、大規模で、多くの場合計算コストの高いモデルから、より小さなモデルに知識を転送することを目的としています。教師モデルは、データセットで事前に訓練されており、ハードラベルだけでなく、生徒モデルが学習するためのソフトラベルまたは確率を提供します。これにより、生徒モデルは教師の一般化能力を捉え、より少ないパラメータと少ない計算コストで同様のパフォーマンスを達成できる可能性があります。このプロセスには通常、元のトレーニングデータと教師の出力の組み合わせで生徒モデルをトレーニングすることが含まれ、多くの場合、生徒の予測が教師の予測と一致するように促す損失関数を使用します。
この技術は、エッジデバイスやモバイルアプリケーションなど、リソースの制約により完全な教師モデルをデプロイすることが非現実的なシナリオで特に役立ちます。より高速な推論とメモリフットプリントの削減を促進し、より大きな対応物のパフォーマンスを維持する効率的なモデルの作成を可能にします。
graph LR
Center["Model Distillation"]:::main
Pre_machine_learning["machine-learning"]:::pre --> Center
click Pre_machine_learning "/terms/machine-learning"
Pre_neural_network["neural-network"]:::pre --> Center
click Pre_neural_network "/terms/neural-network"
Pre_deep_learning["deep-learning"]:::pre --> Center
click Pre_deep_learning "/terms/deep-learning"
Rel_llm["llm"]:::related -.-> Center
click Rel_llm "/terms/llm"
classDef main fill:#7c3aed,stroke:#8b5cf6,stroke-width:2px,color:white,font-weight:bold,rx:5,ry:5;
classDef pre fill:#0f172a,stroke:#3b82f6,color:#94a3b8,rx:5,ry:5;
classDef child fill:#0f172a,stroke:#10b981,color:#94a3b8,rx:5,ry:5;
classDef related fill:#0f172a,stroke:#8b5cf6,stroke-dasharray: 5 5,color:#94a3b8,rx:5,ry:5;
linkStyle default stroke:#4b5563,stroke-width:2px;
🧠 理解度チェック
🧒 5歳でもわかるように説明
It's like a master chef teaching an apprentice. The apprentice learns not just the final recipe (the right answer) but also the subtle techniques and reasoning the master uses, so the apprentice can cook almost as well but much faster.
🤓 Expert Deep Dive
Model distillation, also known as knowledge distillation, is a form of model compression that leverages a teacher-student architecture. The objective function for the student model typically takes the form: L_total = α L_hard + (1 - α) L_soft, where L_hard is a standard cross-entropy loss against ground-truth labels, and L_soft is a loss (e.g., KL divergence or cross-entropy) comparing the student's softened outputs to the teacher's softened outputs. The temperature parameter T in the softmax function (softmax(z_i / T)) is crucial; a higher T produces a softer probability distribution, emphasizing inter-class similarities, while T=1 recovers the standard softmax. Variants include distilling intermediate feature representations (feature distillation) or attention maps, rather than just the final output probabilities. This can be particularly effective when the teacher and student architectures differ significantly. Offline distillation involves training the teacher first, then distilling it. Online distillation trains the teacher and student simultaneously. Self-distillation involves using a model of the same architecture as both teacher and student. Challenges include selecting the appropriate distillation loss, tuning the temperature and weighting parameters (α), and the potential for negative transfer if the teacher model is poorly suited or the student capacity is too low. The effectiveness often depends on the similarity between the teacher's learned function and the true underlying data distribution.