| | --- |
| | language: en |
| | tags: |
| | - fill-mask |
| | --- |
| | |
| | ## Environmental Impact (CODE CARBON DEFAULT) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | Duration (in seconds) | [More Information Needed] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| | | CPU power (W) | [NO CPU] | |
| | | GPU power (W) | [No GPU] | |
| | | RAM power (W) | [More Information Needed] | |
| | | CPU energy (kWh) | [No CPU] | |
| | | GPU energy (kWh) | [No GPU] | |
| | | RAM energy (kWh) | [More Information Needed] | |
| | | Consumed energy (kWh) | [More Information Needed] | |
| | | Country name | [More Information Needed] | |
| | | Cloud provider | [No Cloud] | |
| | | Cloud region | [No Cloud] | |
| | | CPU count | [No CPU] | |
| | | CPU model | [No CPU] | |
| | | GPU count | [No GPU] | |
| | | GPU model | [No GPU] | |
| |
|
| | ## Environmental Impact (for one core) |
| |
|
| | | Metric | Value | |
| | |--------------------------|---------------------------------| |
| | | CPU energy (kWh) | [No CPU] | |
| | | Emissions (Co2eq in kg) | [More Information Needed] | |
| |
|
| | ## Note |
| |
|
| | 5 juillet 2024 ! |
| |
|
| | ## My Config |
| |
|
| | | Config | Value | |
| | |--------------------------|-----------------| |
| | | checkpoint | albert-base-v2 | |
| | | model_name | fp_bs8_lr3e4_x8 | |
| | | sequence_length | 400 | |
| | | num_epoch | 6 | |
| | | learning_rate | 0.0003 | |
| | | batch_size | 8 | |
| | | weight_decay | 0.0 | |
| | | warm_up_prop | 0.0 | |
| | | drop_out_prob | 0.1 | |
| | | packing_length | 100 | |
| | | train_test_split | 0.2 | |
| | | num_steps | 83758 | |
| | |
| | ## Training and Testing steps |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | Epoch | Train Loss | Test Loss |
| | ---|---|--- |
| | | 0.0 | 16.981449 | 11.411258 | |
| | | 0.5 | 7.209574 | 7.074687 | |
| | | 1.0 | 7.037184 | 7.028060 | |
| | | 1.5 | 6.999856 | 6.993283 | |
| | | 2.0 | 6.979971 | 6.990040 | |
| | | 2.5 | 6.977837 | 6.976526 | |
| | |
