| | --- |
| | language: en |
| | tags: |
| | - fill-mask |
| | --- |
| | |
| | ## Environmental Impact (CODE CARBON DEFAULT) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | Duration (in seconds) | [More Information Needed] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| | | CPU power (W) | [NO CPU] | |
| | | GPU power (W) | [No GPU] | |
| | | RAM power (W) | [More Information Needed] | |
| | | CPU energy (kWh) | [No CPU] | |
| | | GPU energy (kWh) | [No GPU] | |
| | | RAM energy (kWh) | [More Information Needed] | |
| | | Consumed energy (kWh) | [More Information Needed] | |
| | | Country name | [More Information Needed] | |
| | | Cloud provider | [No Cloud] | |
| | | Cloud region | [No Cloud] | |
| | | CPU count | [No CPU] | |
| | | CPU model | [No CPU] | |
| | | GPU count | [No GPU] | |
| | | GPU model | [No GPU] | |
| |
|
| | ## Environmental Impact (for one core) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | CPU energy (kWh) | [No CPU] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| |
|
| | ## Note |
| |
|
| | 5 juillet 2024 ! |
| |
|
| | ## My Config |
| |
|
| | | Config | Value | |
| | |--------------------------|-----------------| |
| | | checkpoint | albert-base-v2 | |
| | | model_name | fp_bs8_lr2e4_x2 | |
| | | sequence_length | 400 | |
| | | num_epoch | 6 | |
| | | learning_rate | 0.0002 | |
| | | batch_size | 8 | |
| | | weight_decay | 0.0 | |
| | | warm_up_prop | 0.0 | |
| | | drop_out_prob | 0.1 | |
| | | packing_length | 100 | |
| | | train_test_split | 0.2 | |
| | | num_steps | 83270 | |
| | |
| | ## Training and Testing steps |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | Epoch | Train Loss | Test Loss |
| | ---|---|--- |
| | | 0.0 | 18.083176 | 11.266975 | |
| | | 0.5 | 7.142726 | 7.066952 | |
| | | 1.0 | 7.035545 | 7.021196 | |
| | | 1.5 | 6.995318 | 7.026562 | |
| | | 2.0 | 6.978820 | 6.988426 | |
| | | 2.5 | 6.975003 | 6.972350 | |
| | | 3.0 | 6.974181 | 6.971103 | |
| | | 3.5 | 6.967837 | 6.964300 | |
| | | 4.0 | 6.965952 | 6.966316 | |
| | | 4.5 | 6.964535 | 6.966458 | |
| | | 5.0 | 6.956336 | 6.965864 | |
| | | 5.5 | 6.958633 | 6.957286 | |
| | |
