End of training

README.md

@@ -77,7 +77,7 @@ LlamaForCausalLM(
 
 # Resource Usage
 
-- Max Train VRAM Use: 13.1110 GB
+- Max Train VRAM Use: 19.6182 GB
 - Available VRAM: 23.4329 GB
 - GPUs: 
   - 1x NVIDIA GeForce RTX 4090
@@ -114,9 +114,9 @@ LlamaForCausalLM(
 <br/>
 
 # Train Dataset
-Trained on 687,248,443 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
+Trained on 385,611,117 tokens from the [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) dataset.
 
-- Num Samples: `1,996,000`
+- Num Samples: `499,000`
 - Subset: `20231101.en`
 - Split: `train`
 
@@ -145,12 +145,11 @@ The following hyperparameters were used during training:
 <summary>Expand</summary>
 
 - learning_rate: `0.0002`
-- train_batch_size: `16`
+- train_batch_size: `4`
 - eval_batch_size: `2`
 - seed: `42`
 - optimizer: `Adam with betas=(0.9,0.999) and epsilon=1e-08`
 - lr_scheduler_type: `polynomial`
-- lr_scheduler_warmup_ratio: `0.1`
 - num_epochs: `1.0`
 - distillation_objective: `DistillationObjective(
     logits_loss_component=LossComponent(
@@ -164,7 +163,7 @@ The following hyperparameters were used during training:
         weight=0
     )
 )`
-- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x76ca0d527850>`
+- lr_scheduler: `<torch.optim.lr_scheduler.LambdaLR object at 0x76c84721bdc0>`
 - student_model_name_or_path: `None`
 - student_config_name_or_path: `None`
 - student_model_config: `{'num_hidden_layers': 15}`
@@ -179,8 +178,8 @@ The following hyperparameters were used during training:
 - dataset_subset: `20231101.en`
 - dataset_split: `train`
 - dataset_column_name: `text`
-- dataset_sample_size: `2000000`
-- dataset_max_seq_length: `512`
+- dataset_sample_size: `500000`
+- dataset_max_seq_length: `2048`
 - dataset_test_size: `0.002`
 - dataset_shuffle: `False`
 - dataset_shuffle_seed: `42`
@@ -188,7 +187,7 @@ The following hyperparameters were used during training:
 - gradient_accumulation_steps: `1`
 - weight_decay: `0.0`
 - max_grad_norm: `1.0`
-- warmup_ratio: `0.1`
+- warmup_ratio: `0.0`
 - warmup_steps: `0`
 - gradient_checkpointing: `True`
 

logs/dataset_max_seq_length=2048, dataset_sample_size=500000, per_device_train_batch_size=4/events.out.tfevents.1726453475.1c1a426a2fee

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65b94c02ab5a68b91f35d0c71ac0136bf01049efc17ec6e83ab5f5d2c03cdbdc
+size 529