| | --- |
| | language: en |
| | tags: |
| | - fill-mask |
| | --- |
| | |
| | ## Environmental Impact (CODE CARBON DEFAULT) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | Duration (in seconds) | [More Information Needed] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| | | CPU power (W) | [NO CPU] | |
| | | GPU power (W) | [No GPU] | |
| | | RAM power (W) | [More Information Needed] | |
| | | CPU energy (kWh) | [No CPU] | |
| | | GPU energy (kWh) | [No GPU] | |
| | | RAM energy (kWh) | [More Information Needed] | |
| | | Consumed energy (kWh) | [More Information Needed] | |
| | | Country name | [More Information Needed] | |
| | | Cloud provider | [No Cloud] | |
| | | Cloud region | [No Cloud] | |
| | | CPU count | [No CPU] | |
| | | CPU model | [No CPU] | |
| | | GPU count | [No GPU] | |
| | | GPU model | [No GPU] | |
| |
|
| | ## Environmental Impact (for one core) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | CPU energy (kWh) | [No CPU] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| |
|
| | ## Note |
| |
|
| | 5 juillet 2024 ! |
| |
|
| | ## My Config |
| |
|
| | | Config | Value | |
| | |--------------------------|-----------------| |
| | | checkpoint | albert-base-v2 | |
| | | model_name | fp_bs8_lr3e4_x4 | |
| | | sequence_length | 400 | |
| | | num_epoch | 6 | |
| | | learning_rate | 0.0003 | |
| | | batch_size | 8 | |
| | | weight_decay | 0.0 | |
| | | warm_up_prop | 0.0 | |
| | | drop_out_prob | 0.1 | |
| | | packing_length | 100 | |
| | | train_test_split | 0.2 | |
| | | num_steps | 84763 | |
| | |
| | ## Training and Testing steps |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | Epoch | Train Loss | Test Loss |
| | ---|---|--- |
| | | 0.0 | 16.102894 | 12.949274 | |
| | | 0.5 | 7.149223 | 7.063690 | |
| | | 1.0 | 7.019910 | 7.019870 | |
| | | 1.5 | 7.000748 | 7.008951 | |
| | | 2.0 | 6.980634 | 6.991125 | |
| | | 2.5 | 6.975947 | 6.985942 | |
| | |
