Update README.md

README.md

@@ -26,6 +26,7 @@ base_model: meta-llama/LLaMA-2-7B
 ### Overview
 
 This model is a distilled version of LLaMA 2, containing approximately 80 million parameters. It was trained using a mix of OpenWebText and WikiText Raw V1 datasets. Knowledge distillation was employed to transfer knowledge from a larger "teacher" model—Meta’s 7B LLaMA 2—to help this smaller model mimic the behavior of the teacher.
+This version is the latest version of DistilLlama, which has gone through 5 days of training using two Nvidia A100 80G GPU.
 
 ### Model Architecture
 
@@ -39,62 +40,6 @@ The architecture is based on LLaMA 2, with the following parameters:
 | Transformer Layers      | 16    |
 
 
-### Training Process
-
-During each training step, the input data \( X \) is fed to both the teacher and student models. The student model calculates output logits and loss with the true labels, while the teacher model only generates logits. The total loss combines task-specific loss and distillation loss:
-
-```python
-def distillation_loss(student_logits, teacher_logits, temperature=2.0):
-    return F.kl_div(
-        F.log_softmax(student_logits / temperature, dim=-1),
-        F.softmax(teacher_logits / temperature, dim=-1),
-        reduction='batchmean'
-    ) * (temperature ** 2)
-
-# Loss Calculation
-loss = (alpha * distill_loss) + ((1 - alpha) * task_loss)
-```
-
-### Training Configuration
-
-- **Batch Size**: 64
-- **Max Sequence Length**: 128
-- **Epochs**: 2
-- **Log Interval**: 3000
-- **Learning Rate**: 3e-4
-- **Warmup Steps**: 4000
-- **Accumulation Steps**: 8
-- **Load Model**: True
-- **Temperature**: 2.0
-- **Alpha**: 0.3
-
-### Performance Comparison
-
-The model’s performance is evaluated on 200 queries created in-house. For more details, visit the [GitHub repository](https://github.com/svarnim1805/Knowledge-Distillation/tree/main).
-
-#### Example Queries
-
-| **Query**                                    | **Keyword**       |
-|----------------------------------------------|-------------------|
-| The capital of France is                     | Paris             |
-| The author of *To Kill a Mockingbird* is     | Harper Lee        |
-| The planet known as the Red Planet is        | Mars              |
-| The smallest country in the world is         | Vatican City      |
-| The *Mona Lisa* was painted by               | Leonardo da Vinci |
-| The tallest mountain in the world is         | Mount Everest     |
-| The host country of the 2016 Summer Olympics | Brazil            |
-| The currency of Japan is                     | Yen               |
-| The "Father of Computers" is                 | Charles Babbage   |
-| The largest ocean on Earth is                | Pacific Ocean     |
-
-### Text Generation Examples
-
-1. **Input**: *The capital of France is*
-   - **Output**: "The capital of France is located in the southern province of Lyon, France. The capital is the main hub of the French capital, La Caillion, and the main hub of the French capital. A major part of the French capital is in the southern city of Paris."
-
-2. **Input**: *The tallest mountain in the world is*
-   - **Output**: "The tallest mountain in the world is Mount Everest, which is located in the northwest corner of the village of Kahuna. The mountain is about 1,000 feet (3,000 m) above sea level."
-
 ### Evaluation Metrics
 
 1. **Cosine Similarity using Word Embeddings**
@@ -121,14 +66,20 @@ The model’s performance is evaluated on 200 queries created in-house. For more
 | LLaMA-2-7B-HF   | 18215.61     | 1.84e-01           | 0.715   | 0.7257                 | 0.0821           |
 | baby-llama-58m  | 57.20        | 2.73e-06           | 0.025   | 0.6556                 | 0.0097           |
 | DistilLlama     | 77.12        | 7.79e-04           | 0.02    | 0.6623                 | 0.0115           |
+| DistilLlamaV1   | 78.46        | 8.49e-04           | 0.065   | 0.6776                 | 0.0135           |
 
 *Note: CodeCarbon was used to track carbon emission. Allocated 80GB memory, 32 cores, Intel(R) Xeon(R) Gold 6448H for the evaluation*
 
-### Acknowledgments
+### Example queries from our test set
+
+| Query                                      | Keyword          | Response                                                                                      | Exact Match | Cosine Similarity | ROUGE Score |
+|--------------------------------------------|------------------|-----------------------------------------------------------------------------------------------|-------------|-------------------|-------------|
+| The capital of France is                   | Paris           | The capital of France is the city of Paris, Paris is the capital and most populous city...    | 1           | 0.757961          | 0.0625      |
+| The currency of Japan is                   | Yen             | The currency of Japan is the Japanese currency called Yen.                                    | 1           | 0.774518          | 0.083333    |
+| The largest ocean on Earth is              | Pacific Ocean   | The largest ocean on Earth is Pacific Ocean.                               | 0           | 0.721646          | 0.222222    |
+| The continent known as the 'Dark Continent' is | Africa    | The continent known as the 'Dark Continent' is Africa.                                       | 1           | 0.725292          | 0.057143    |
+| The theory of relativity was developed by  | Albert Einstein | The theory of relativity was developed by Einstein, a famous physicist who...                 | 0           | 0.712056          | 0.055556    |
 
-- **University of Melbourne**
-- **AGL Energy**
-- **My teammates**: Svarnim and Mohit
 
 ### GitHub Repositories