Upload folder using huggingface_hub

README.md

@@ -0,0 +1,46 @@
+# EsperBERTo Model Card
+
+## Model Description
+EsperBERTo is a RoBERTa-like model specifically trained from scratch on the Esperanto language using a large corpus from the OSCAR and Leipzig Corpora Collection. It is designed to perform masked language modeling and other text-based prediction tasks. This model is ideal for understanding and generating Esperanto text.
+
+### Datasets
+- **OSCAR Corpus (Esperanto)**: Extracted from Common Crawl dumps, filtered by language classification.
+- **Leipzig Corpora Collection (Esperanto)**: Includes texts from news, literature, and Wikipedia.
+
+### Preprocessing
+- Trained a byte-level Byte-pair encoding tokenizer with a vocabulary size of 52,000 tokens.
+
+### Hyperparameters
+- **Number of Epochs**: 1
+- **Batch Size per GPU**: 64
+- **Training Steps for Saving**: 10,000
+- **Limit of Saved Models**: 2
+- **Loss Calculation**: Prediction loss only
+
+### Software and Libraries
+- **Transformers Library Version**: [Transformers](https://github.com/huggingface/transformers)
+- **Training Script**: `run_language_modeling.py`
+
+```python
+from transformers import pipeline
+
+fill_mask = pipeline(
+    "fill-mask",
+    model="SamJoshua/EsperBERTo-small",
+    tokenizer="SamJoshua/EsperBERTo-small"
+)
+
+fill_mask("Jen la komenco de bela <mask>.")
+```
+
+## Evaluation Results
+The model has not yet been evaluated on a standardized test set. Future updates will include evaluation metrics such as perplexity and accuracy on a held-out validation set.
+
+## Intended Uses & Limitations
+**Intended Uses**: This model is intended for researchers, developers, and language enthusiasts who wish to explore Esperanto language processing for tasks like text generation, sentiment analysis, and more.
+
+**Limitations**: 
+- The model is trained only for one epoch due to computational constraints, which may affect its understanding of more complex language structures.
+- As the model is trained on public web text, it may inadvertently learn and replicate social biases present in the training data.
+
+Feel free to contribute to the model by fine-tuning on specific tasks or extending its training with more data or epochs. This model serves as a baseline for further research and development in Esperanto language modeling.

checkpoint-10000/config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.43.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 52000
+}

checkpoint-10000/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:64346a7029f05e9129b43c14bec6a90eba3b21374f55a75c256e50669c6c6c41
+size 334030264

checkpoint-10000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:534703ea7f111a5d936109d1687a96ca4ff44206bc581d7dd713f64b1bdab095
+size 668124218

checkpoint-10000/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27d2e95c87c6ddccfb180e047f9063359bb3c2a42535b516f982953ce9b79904
+size 14244

checkpoint-10000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e09a46302ae32307de3843894190190a4e30e2dca5bf8a1efb68f0cf10943996
+size 1064

checkpoint-10000/trainer_state.json

@@ -0,0 +1,173 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.6566850538481744,
+  "eval_steps": 500,
+  "global_step": 10000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03283425269240872,
+      "grad_norm": 2.148223638534546,
+      "learning_rate": 4.8358287365379566e-05,
+      "loss": 7.8532,
+      "step": 500
+    },
+    {
+      "epoch": 0.06566850538481744,
+      "grad_norm": 1.8999865055084229,
+      "learning_rate": 4.671657473075913e-05,
+      "loss": 7.2619,
+      "step": 1000
+    },
+    {
+      "epoch": 0.09850275807722617,
+      "grad_norm": 1.8169187307357788,
+      "learning_rate": 4.5074862096138694e-05,
+      "loss": 7.078,
+      "step": 1500
+    },
+    {
+      "epoch": 0.1313370107696349,
+      "grad_norm": 2.516470193862915,
+      "learning_rate": 4.343314946151826e-05,
+      "loss": 6.9502,
+      "step": 2000
+    },
+    {
+      "epoch": 0.1641712634620436,
+      "grad_norm": 2.3785781860351562,
+      "learning_rate": 4.179143682689782e-05,
+      "loss": 6.8573,
+      "step": 2500
+    },
+    {
+      "epoch": 0.19700551615445233,
+      "grad_norm": 2.1694188117980957,
+      "learning_rate": 4.0149724192277385e-05,
+      "loss": 6.8,
+      "step": 3000
+    },
+    {
+      "epoch": 0.22983976884686105,
+      "grad_norm": 2.474036455154419,
+      "learning_rate": 3.850801155765695e-05,
+      "loss": 6.744,
+      "step": 3500
+    },
+    {
+      "epoch": 0.2626740215392698,
+      "grad_norm": 2.19565749168396,
+      "learning_rate": 3.686629892303651e-05,
+      "loss": 6.6814,
+      "step": 4000
+    },
+    {
+      "epoch": 0.29550827423167847,
+      "grad_norm": 2.747317314147949,
+      "learning_rate": 3.522458628841608e-05,
+      "loss": 6.5869,
+      "step": 4500
+    },
+    {
+      "epoch": 0.3283425269240872,
+      "grad_norm": 3.399601936340332,
+      "learning_rate": 3.358287365379564e-05,
+      "loss": 6.4711,
+      "step": 5000
+    },
+    {
+      "epoch": 0.3611767796164959,
+      "grad_norm": 2.9853923320770264,
+      "learning_rate": 3.1941161019175205e-05,
+      "loss": 6.3167,
+      "step": 5500
+    },
+    {
+      "epoch": 0.39401103230890466,
+      "grad_norm": 3.57155704498291,
+      "learning_rate": 3.029944838455477e-05,
+      "loss": 6.2061,
+      "step": 6000
+    },
+    {
+      "epoch": 0.42684528500131336,
+      "grad_norm": 3.53355073928833,
+      "learning_rate": 2.8657735749934332e-05,
+      "loss": 6.0795,
+      "step": 6500
+    },
+    {
+      "epoch": 0.4596795376937221,
+      "grad_norm": 3.1891579627990723,
+      "learning_rate": 2.70160231153139e-05,
+      "loss": 5.9704,
+      "step": 7000
+    },
+    {
+      "epoch": 0.4925137903861308,
+      "grad_norm": 3.8728067874908447,
+      "learning_rate": 2.5374310480693457e-05,
+      "loss": 5.8494,
+      "step": 7500
+    },
+    {
+      "epoch": 0.5253480430785396,
+      "grad_norm": 3.752729654312134,
+      "learning_rate": 2.3732597846073024e-05,
+      "loss": 5.7467,
+      "step": 8000
+    },
+    {
+      "epoch": 0.5581822957709482,
+      "grad_norm": 3.093104124069214,
+      "learning_rate": 2.2090885211452588e-05,
+      "loss": 5.6585,
+      "step": 8500
+    },
+    {
+      "epoch": 0.5910165484633569,
+      "grad_norm": 3.5485284328460693,
+      "learning_rate": 2.0449172576832152e-05,
+      "loss": 5.5876,
+      "step": 9000
+    },
+    {
+      "epoch": 0.6238508011557657,
+      "grad_norm": 3.3730971813201904,
+      "learning_rate": 1.8807459942211716e-05,
+      "loss": 5.5165,
+      "step": 9500
+    },
+    {
+      "epoch": 0.6566850538481744,
+      "grad_norm": 3.767274856567383,
+      "learning_rate": 1.716574730759128e-05,
+      "loss": 5.4677,
+      "step": 10000
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 15228,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 10000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.122034184192e+16,
+  "train_batch_size": 64,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-10000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff57da9fde7b884e48db14f0049fe53a5e93321ff3fc5de4894a45f0aa892541
+size 5112

checkpoint-15228/config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.43.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 52000
+}

checkpoint-15228/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ec4c8484678948c3855c5f338a7b607fcb881e00eca1e7848d1fedc28b08300
+size 334030264

checkpoint-15228/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8fa5ab0ea7b92213009172ca6fb870666c9b4d2db45868bcceed46605971e23a
+size 668124218

checkpoint-15228/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fa9f26f4c75cba39c17dc2ffaaaf19772fec591f3112996eb055f5518a6f16a
+size 14244

checkpoint-15228/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8bad113ce9254ef8876588cf13f1a80562951be4985e148f8e1ce0f8a5305a3
+size 1064

checkpoint-15228/trainer_state.json

@@ -0,0 +1,243 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 500,
+  "global_step": 15228,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03283425269240872,
+      "grad_norm": 2.148223638534546,
+      "learning_rate": 4.8358287365379566e-05,
+      "loss": 7.8532,
+      "step": 500
+    },
+    {
+      "epoch": 0.06566850538481744,
+      "grad_norm": 1.8999865055084229,
+      "learning_rate": 4.671657473075913e-05,
+      "loss": 7.2619,
+      "step": 1000
+    },
+    {
+      "epoch": 0.09850275807722617,
+      "grad_norm": 1.8169187307357788,
+      "learning_rate": 4.5074862096138694e-05,
+      "loss": 7.078,
+      "step": 1500
+    },
+    {
+      "epoch": 0.1313370107696349,
+      "grad_norm": 2.516470193862915,
+      "learning_rate": 4.343314946151826e-05,
+      "loss": 6.9502,
+      "step": 2000
+    },
+    {
+      "epoch": 0.1641712634620436,
+      "grad_norm": 2.3785781860351562,
+      "learning_rate": 4.179143682689782e-05,
+      "loss": 6.8573,
+      "step": 2500
+    },
+    {
+      "epoch": 0.19700551615445233,
+      "grad_norm": 2.1694188117980957,
+      "learning_rate": 4.0149724192277385e-05,
+      "loss": 6.8,
+      "step": 3000
+    },
+    {
+      "epoch": 0.22983976884686105,
+      "grad_norm": 2.474036455154419,
+      "learning_rate": 3.850801155765695e-05,
+      "loss": 6.744,
+      "step": 3500
+    },
+    {
+      "epoch": 0.2626740215392698,
+      "grad_norm": 2.19565749168396,
+      "learning_rate": 3.686629892303651e-05,
+      "loss": 6.6814,
+      "step": 4000
+    },
+    {
+      "epoch": 0.29550827423167847,
+      "grad_norm": 2.747317314147949,
+      "learning_rate": 3.522458628841608e-05,
+      "loss": 6.5869,
+      "step": 4500
+    },
+    {
+      "epoch": 0.3283425269240872,
+      "grad_norm": 3.399601936340332,
+      "learning_rate": 3.358287365379564e-05,
+      "loss": 6.4711,
+      "step": 5000
+    },
+    {
+      "epoch": 0.3611767796164959,
+      "grad_norm": 2.9853923320770264,
+      "learning_rate": 3.1941161019175205e-05,
+      "loss": 6.3167,
+      "step": 5500
+    },
+    {
+      "epoch": 0.39401103230890466,
+      "grad_norm": 3.57155704498291,
+      "learning_rate": 3.029944838455477e-05,
+      "loss": 6.2061,
+      "step": 6000
+    },
+    {
+      "epoch": 0.42684528500131336,
+      "grad_norm": 3.53355073928833,
+      "learning_rate": 2.8657735749934332e-05,
+      "loss": 6.0795,
+      "step": 6500
+    },
+    {
+      "epoch": 0.4596795376937221,
+      "grad_norm": 3.1891579627990723,
+      "learning_rate": 2.70160231153139e-05,
+      "loss": 5.9704,
+      "step": 7000
+    },
+    {
+      "epoch": 0.4925137903861308,
+      "grad_norm": 3.8728067874908447,
+      "learning_rate": 2.5374310480693457e-05,
+      "loss": 5.8494,
+      "step": 7500
+    },
+    {
+      "epoch": 0.5253480430785396,
+      "grad_norm": 3.752729654312134,
+      "learning_rate": 2.3732597846073024e-05,
+      "loss": 5.7467,
+      "step": 8000
+    },
+    {
+      "epoch": 0.5581822957709482,
+      "grad_norm": 3.093104124069214,
+      "learning_rate": 2.2090885211452588e-05,
+      "loss": 5.6585,
+      "step": 8500
+    },
+    {
+      "epoch": 0.5910165484633569,
+      "grad_norm": 3.5485284328460693,
+      "learning_rate": 2.0449172576832152e-05,
+      "loss": 5.5876,
+      "step": 9000
+    },
+    {
+      "epoch": 0.6238508011557657,
+      "grad_norm": 3.3730971813201904,
+      "learning_rate": 1.8807459942211716e-05,
+      "loss": 5.5165,
+      "step": 9500
+    },
+    {
+      "epoch": 0.6566850538481744,
+      "grad_norm": 3.767274856567383,
+      "learning_rate": 1.716574730759128e-05,
+      "loss": 5.4677,
+      "step": 10000
+    },
+    {
+      "epoch": 0.6895193065405831,
+      "grad_norm": 3.7977945804595947,
+      "learning_rate": 1.5524034672970843e-05,
+      "loss": 5.4096,
+      "step": 10500
+    },
+    {
+      "epoch": 0.7223535592329918,
+      "grad_norm": 3.5391793251037598,
+      "learning_rate": 1.3882322038350407e-05,
+      "loss": 5.3555,
+      "step": 11000
+    },
+    {
+      "epoch": 0.7551878119254006,
+      "grad_norm": 3.8988144397735596,
+      "learning_rate": 1.2240609403729971e-05,
+      "loss": 5.3476,
+      "step": 11500
+    },
+    {
+      "epoch": 0.7880220646178093,
+      "grad_norm": 4.032622337341309,
+      "learning_rate": 1.0598896769109535e-05,
+      "loss": 5.2971,
+      "step": 12000
+    },
+    {
+      "epoch": 0.820856317310218,
+      "grad_norm": 3.8639235496520996,
+      "learning_rate": 8.957184134489099e-06,
+      "loss": 5.2853,
+      "step": 12500
+    },
+    {
+      "epoch": 0.8536905700026267,
+      "grad_norm": 4.275233268737793,
+      "learning_rate": 7.315471499868663e-06,
+      "loss": 5.2745,
+      "step": 13000
+    },
+    {
+      "epoch": 0.8865248226950354,
+      "grad_norm": 3.4154789447784424,
+      "learning_rate": 5.673758865248227e-06,
+      "loss": 5.2396,
+      "step": 13500
+    },
+    {
+      "epoch": 0.9193590753874442,
+      "grad_norm": 3.5738344192504883,
+      "learning_rate": 4.032046230627791e-06,
+      "loss": 5.2383,
+      "step": 14000
+    },
+    {
+      "epoch": 0.9521933280798529,
+      "grad_norm": 3.526001214981079,
+      "learning_rate": 2.390333596007355e-06,
+      "loss": 5.2186,
+      "step": 14500
+    },
+    {
+      "epoch": 0.9850275807722616,
+      "grad_norm": 3.600189208984375,
+      "learning_rate": 7.486209613869189e-07,
+      "loss": 5.2131,
+      "step": 15000
+    }
+  ],
+  "logging_steps": 500,
+  "max_steps": 15228,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 10000,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 3.231269529606144e+16,
+  "train_batch_size": 64,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-15228/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff57da9fde7b884e48db14f0049fe53a5e93321ff3fc5de4894a45f0aa892541
+size 5112

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "RobertaForMaskedLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "classifier_dropout": null,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 514,
+  "model_type": "roberta",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "pad_token_id": 1,
+  "position_embedding_type": "absolute",
+  "torch_dtype": "float32",
+  "transformers_version": "4.43.0.dev0",
+  "type_vocab_size": 1,
+  "use_cache": true,
+  "vocab_size": 52000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7ec4c8484678948c3855c5f338a7b607fcb881e00eca1e7848d1fedc28b08300
+size 334030264

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ff57da9fde7b884e48db14f0049fe53a5e93321ff3fc5de4894a45f0aa892541
+size 5112