End of training

README.md

@@ -78,13 +78,13 @@ GPT2LMHeadModel(
 
 # Resource Usage
 
-- Max Train VRAM Use: 15.7146 GB
-- Available VRAM: 23.6429 GB
+- Max Train VRAM Use: 15.7123 GB
+- Available VRAM: 23.4329 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
-- CPUs: 32
-- CPU Memory: 61.9353 GB
-- CPU Memory Bandwidth: 800 GB/s
+- CPUs: 64
+- CPU Memory: 251.7299 GB
+- CPU Memory Bandwidth: 1600 GB/s
 
 # Distillation (Teacher -> Student) Architecture Difference:
 
@@ -115,7 +115,7 @@ GPT2LMHeadModel(
 <br/>
 
 # Train Dataset
-Trained on 1,756,071,327 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 1,811,293,876 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
 - Num Samples: `3,996,000`
 - Subset: `20231101.en`
@@ -154,7 +154,7 @@ The following hyperparameters were used during training:
 - eval_batch_size: `8`
 - seed: `42`
 - optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08`
-- lr_scheduler_type: `constant_with_warmup`
+- lr_scheduler_type: `polynomial`
 - num_epochs: `1.0`
 - distillation_objective: `DistillationObjective(
     logits_loss_component=LossComponent(
@@ -172,14 +172,14 @@ The following hyperparameters were used during training:
         projector='orthogonal'
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7fbc60ceb9a0>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x7b4c4250be50>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `distilbert/distilgpt2`
 - student_model_config: `None`
 - reinitialize_weights: `None`
 - copy_teacher_modules: `[('lm_head', False)]`
 - student_model_as_bitnet: `False`
-- student_model_use_liger: `True`
+- student_model_use_liger: `False`
 - teacher_model_name_or_path: `gpt2`
 - teacher_load_in_8bit: `False`
 - teacher_load_in_4bit: `False`
@@ -189,7 +189,7 @@ The following hyperparameters were used during training:
 - dataset_column_name: `text`
 - dataset_sample_size: `4000000`
 - dataset_test_size: `0.001`
-- dataset_shuffle: `False`
+- dataset_shuffle: `True`
 - dataset_shuffle_seed: `42`
 - dataset_trust_remote_code: `False`
 - gradient_accumulation_steps: `1`

logs/learning_rate=0.0002, lr_scheduler_kwargs=__lr_end___0.0001_, lr_scheduler_type=polynomial, warmup_ratio=0/events.out.tfevents.1725885268.b57c76173204

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:70376c356ee5e84da705ede837e7dc5d448562e3ce3fb7d7f4bc7b1d01f334e3
-size 7849573
+oid sha256:265b82583edce459cc819e8faba0a26a6a20c42598c8c0812aa68fa32f3f7062
+size 8002593

logs/learning_rate=0.0002, lr_scheduler_kwargs=__lr_end___0.0001_, lr_scheduler_type=polynomial, warmup_ratio=0/events.out.tfevents.1725963114.b57c76173204

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07aca7a7f437cd35a4cd7d172ad1e05587a9a6173582bfa2b4530ce7853f2c59
+size 529

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54e6fc02cbfe08032db3b00688a74746cda4ac218ead8924a02dfb430b828520
+oid sha256:532afe9746621c9bdeb1ff0c099ddba6bf866c234175fb28c82d4acc3dc14e69
 size 163832792