File size: 3,713 Bytes
e2ad4c8 e7e8fb7 3d56cd2 e2ad4c8 05554a6 e2ad4c8 3d56cd2 e2ad4c8 05554a6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
language: en
tags:
- text-generation
- knowledge-distillation
- llama
- causal-lm
- openwebtext
- wikitext
- transfer-learning
model_name: DistilLLaMA
license: apache-2.0
datasets:
- openwebtext
- wikitext
parameter_count: 80M
metrics:
- cosine-similarity
- exact-match
- rouge
library_name: transformers
base_model: meta-llama/LLaMA-2-7B
---
### Overview
This model is a distilled version of LLaMA 2, containing approximately 80 million parameters.
It was trained using a mix of OpenWebText and WikiText Raw V1 datasets.
Knowledge distillation was employed to transfer knowledge from a larger "teacher" model—Meta’s 7B LLaMA 2—to help this smaller model mimic the behavior of the teacher.
This version is the latest version of DistilLlama, which has gone through 5 days of training using two Nvidia A100 80G GPU.
### Update
30 out of 300 checkpoints were examined, and the one with the best performance in semantic and factual accuracy has now been updated in this repository.
### Model Architecture
The architecture is based on LLaMA 2, with the following parameters:
| Parameter | Value |
|-------------------------|-------|
| Hidden Dimension | 512 |
| Intermediate Dimension | 1536 |
| Max Positional Embeddings | 128 |
| Attention Heads | 8 |
| Transformer Layers | 16 |
### Evaluation Metrics
1. **Cosine Similarity using Word Embeddings**
- **Description**: Measures semantic similarity by mapping words/phrases to vectors.
- **Equation**: Cosine Similarity = ( A • B ) / ( ||A|| ||B|| )
- **Example**: "The dog chased the cat." vs. "A canine pursued a feline." (High similarity)
2. **Exact Match (EM)**
- **Description**: Checks if critical keywords are present.
- **Example**:
- Expected: "Paris"
- Response: "The capital of France is Paris." (EM = 1)
3. **ROUGE Score**
- **Description**: Measures the overlap of the longest common subsequences between reference and response texts.
- **Equation**:
- Precision = Precision = LCS(R, C) / Length of C
- Recall = Recall = LCS(R, C) / Length of R
### Model Evaluation Summary
| Model Name | Duration (s) | Emissions (kgCO₂e) | Avg. EM | Avg. Cosine Similarity | Avg. ROUGE Score |
|-----------------|--------------|--------------------|---------|------------------------|------------------|
| LLaMA-2-7B-HF | 18215.61 | 1.84e-01 | 0.715 | 0.7257 | 0.0821 |
| baby-llama-58m | 57.20 | 2.73e-06 | 0.025 | 0.6556 | 0.0097 |
| DistilLlama | 77.12 | 7.79e-04 | 0.02 | 0.6623 | 0.0115 |
| DistilLlamaV1 | 78.46 | 8.49e-04 | 0.065 | 0.6776 | 0.0135 |
*Note: CodeCarbon was used to track carbon emission. Allocated 80GB memory, 32 cores, Intel(R) Xeon(R) Gold 6448H for the evaluation*
### GitHub Repositories
- **Training Repo**: [DistilLlama Training Repository](https://github.com/HenryHuang2/DistilLlama)
- **Evaluation Repo**: [Knowledge Distillation Evaluation Repository](https://github.com/svarnim1805/Knowledge-Distillation)
### Reference
@misc{timiryasov2023babyllamaknowledgedistillation,
title={Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty},
author={Inar Timiryasov and Jean-Loup Tastet},
year={2023},
eprint={2308.02019},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2308.02019},
}
*Note: The repository will be updated as training progresses. Last update 2024-11-06* |