End of training

README.md

@@ -0,0 +1,84 @@
+---
+base_model: roneneldan/TinyStories-33M
+library_name: Distily
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_obj_cross_v2
+  results: []
+---
+
+# distily_bench_obj_cross_v2
+
+This student model is distilled from the teacher model [roneneldan/TinyStories-33M](https://huggingface.co/roneneldan/TinyStories-33M) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 6023.2686
+- eval_frwikippl: 36635.6680
+- eval_zhwikippl: 63580.2773
+- eval_tinystoriesppl: 2521.4807
+- eval_loss: 5.0637
+- eval_runtime: 13.0817
+- eval_samples_per_second: 76.442
+- eval_steps_per_second: 9.555
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=2.0, loss_fn=mse, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
+- train_embeddings: True
+- learning_rate: 0.0004
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine_with_restarts
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 8.1729 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | tinystoriesppl | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 169.9865 | 47377.9414 |  |  |  |  | 3.9789 | 4998.1294 |
+| 0 | 0 | 35320.8594 | 74921.0938 | 6.5227 | 13.0719 | 76.5 | 9.562 | 24085.9062 | 74141.7188 |
+| 1000 | 0.0808 | 6057.8965 | 36584.1016 | 5.0635 | 13.0614 | 76.561 | 9.57 | 2544.0928 | 63818.1680 |
+| 2000 | 0.1616 | 6023.2686 | 36635.6680 | 5.0635 | 13.099 | 76.342 | 9.543 | 2523.1487 | 63580.2773 |
+| 3000 | 0.2424 | 6023.2686 | 36635.6680 | 5.0635 | 13.0902 | 76.393 | 9.549 | 2523.1487 | 63580.2773 |
+| 4000 | 0.3232 | 6021.4019 | 36635.6680 | 5.0637 | 13.0916 | 76.385 | 9.548 | 2519.3967 | 63580.2773 |
+| 5000 | 0.4040 | 6023.2686 | 36635.6680 | 5.0637 | 13.0968 | 76.354 | 9.544 | 2521.4807 | 63580.2773 |
+| 6000 | 0.4848 | 6023.2686 | 36635.6680 | 5.0637 | 13.0689 | 76.517 | 9.565 | 2521.4807 | 63580.2773 |
+| 7000 | 0.5657 | 6017.6704 | 36656.3242 | 5.0637 | 13.0875 | 76.409 | 9.551 | 2513.5720 | 63546.3320 |
+| 8000 | 0.6465 | 6023.2686 | 36635.6680 | 5.0637 | 13.1185 | 76.228 | 9.529 | 2521.4807 | 63580.2773 |
+| 9000 | 0.7273 | 6023.2686 | 36635.6680 | 5.0637 | 13.0817 | 76.442 | 9.555 | 2521.4807 | 63580.2773 |
+| 10000 | 0.8081 | 6017.6704 | 36635.6680 | 5.0637 | 13.0767 | 76.472 | 9.559 | 2515.2351 | 63546.3320 |
+| 11000 | 0.8889 | 6023.2686 | 36635.6680 | 5.0637 | 13.0841 | 76.429 | 9.554 | 2522.7314 | 63580.2773 |
+| 12000 | 0.9697 | 6023.2686 | 36635.6680 | 5.0637 | 13.0805 | 76.45 | 9.556 | 2521.0635 | 63580.2773 |
+| 12375 | 1.0 | 6023.2686 | 36635.6680 | 5.0637 | 13.0623 | 76.556 | 9.57 | 2521.0635 | 63580.2773 |
+
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.21.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

logs/attn_loss_fn=None, attn_weight=0, gradient_accumulation_steps=1, hs_loss_fn=mse, hs_weight=2.0, learning_rate=0.0004, lr_scheduler_kwargs=__num_cycles___4_, lr_scheduler_type=cosine_with_restarts, max/events.out.tfevents.1723834152.5f530b1cf724

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6beb75d2a80d967ca820553676715e1dcde8bf406ab9a1a956c20f6095df04de
+size 307

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d0fb2a2484cd2ddfc1ca74f378aecd493ed96dc95efbcd19968d8b21725ce360
+oid sha256:4dfea5361764269b261b7bed75573bc5f866f868a496b5eeb6a9ef5d9600cd1a
 size 137033984