| | --- |
| | language: en |
| | tags: |
| | - fill-mask |
| | --- |
| | |
| | ## Environmental Impact (CODE CARBON DEFAULT) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | Duration (in seconds) | [More Information Needed] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| | | CPU power (W) | [NO CPU] | |
| | | GPU power (W) | [No GPU] | |
| | | RAM power (W) | [More Information Needed] | |
| | | CPU energy (kWh) | [No CPU] | |
| | | GPU energy (kWh) | [No GPU] | |
| | | RAM energy (kWh) | [More Information Needed] | |
| | | Consumed energy (kWh) | [More Information Needed] | |
| | | Country name | [More Information Needed] | |
| | | Cloud provider | [No Cloud] | |
| | | Cloud region | [No Cloud] | |
| | | CPU count | [No CPU] | |
| | | CPU model | [No CPU] | |
| | | GPU count | [No GPU] | |
| | | GPU model | [No GPU] | |
| |
|
| | ## Environmental Impact (for one core) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | CPU energy (kWh) | [No CPU] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| |
|
| | ## Note |
| |
|
| | 5 juillet 2024 ! |
| |
|
| | ## My Config |
| |
|
| | | Config | Value | |
| | |--------------------------|-----------------| |
| | | checkpoint | albert-base-v2 | |
| | | model_name | fp_bs8_lr3e4_x1 | |
| | | sequence_length | 400 | |
| | | num_epoch | 6 | |
| | | learning_rate | 0.0003 | |
| | | batch_size | 8 | |
| | | weight_decay | 0.0 | |
| | | warm_up_prop | 0.0 | |
| | | drop_out_prob | 0.1 | |
| | | packing_length | 100 | |
| | | train_test_split | 0.2 | |
| | | num_steps | 82758 | |
| | |
| | ## Training and Testing steps |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | Epoch | Train Loss | Test Loss |
| | ---|---|--- |
| | | 0.0 | 9.285352 | 8.274062 | |
| | | 0.5 | 7.013156 | 7.051341 | |
| | | 1.0 | 7.033509 | 7.025020 | |
| | | 1.5 | 6.994916 | 6.988134 | |
| | | 2.0 | 6.981715 | 6.981714 | |
| | | 2.5 | 6.971288 | 6.973614 | |
| | |
