Training in progress, step 2000

README.md

@@ -1,92 +1,42 @@
 ---
-library_name: transformers
+language:
+- en
+- lg
+- ach
 license: mit
-base_model: xlm-roberta-base
 tags:
-- generated_from_trainer
-metrics:
-- accuracy
-model-index:
-- name: xlm-roberta-uganda-languages
-  results: []
+- xlm-roberta
+- ugandan-languages
+- multilingual
+- masked-language-model
+datasets:
+- Sunbird/ug40
+- Sunbird/external-translation-data
+library_name: transformers
+pipeline_tag: fill-mask
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# xlm-roberta-uganda-languages
-
-This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the None dataset.
-It achieves the following results on the evaluation set:
-- Loss: 3.7352
-- Accuracy: 0.3547
-- Perplexity: 45.8048
-- Top 5 Accuracy: 0.5124
-- Num Masked Tokens: 3076
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
+# XLM-RoBERTa Fine-tuned on Ugandan Languages
 
-### Training hyperparameters
+This model is XLM-RoBERTa-base fine-tuned on a comprehensive dataset of Ugandan languages.
 
-The following hyperparameters were used during training:
-- learning_rate: 5e-05
-- train_batch_size: 8
-- eval_batch_size: 8
-- seed: 42
-- gradient_accumulation_steps: 4
-- total_train_batch_size: 32
-- optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: cosine
-- lr_scheduler_warmup_steps: 1000
-- num_epochs: 3
-- mixed_precision_training: Native AMP
+## Usage
 
-### Training results
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline
 
-| Training Loss | Epoch  | Step  | Validation Loss | Accuracy | Perplexity | Top 5 Accuracy | Num Masked Tokens |
-|:-------------:|:------:|:-----:|:---------------:|:--------:|:----------:|:--------------:|:-----------------:|
-| 3.0384        | 0.1198 | 1000  | 5.3648          | 0.2233   | 226.5299   | 0.3093         | 3113              |
-| 2.7463        | 0.2395 | 2000  | 4.9415          | 0.2577   | 148.3515   | 0.3629         | 3039              |
-| 2.5352        | 0.3593 | 3000  | 4.7300          | 0.2578   | 119.7309   | 0.3758         | 2994              |
-| 2.5007        | 0.4791 | 4000  | 4.8070          | 0.2517   | 124.9927   | 0.3736         | 3051              |
-| 2.4132        | 0.5989 | 5000  | 4.6717          | 0.2690   | 112.4399   | 0.3818         | 3059              |
-| 2.3463        | 0.7186 | 6000  | 4.4216          | 0.2790   | 89.3268    | 0.4012         | 3036              |
-| 2.243         | 0.8384 | 7000  | 4.3218          | 0.2997   | 81.7836    | 0.4243         | 3040              |
-| 2.2279        | 0.9582 | 8000  | 4.2193          | 0.3087   | 71.0217    | 0.4447         | 3022              |
-| 2.1695        | 1.0779 | 9000  | 4.2137          | 0.3013   | 73.8228    | 0.4487         | 3087              |
-| 2.1569        | 1.1976 | 10000 | 4.1268          | 0.3154   | 63.0148    | 0.4604         | 3034              |
-| 2.0617        | 1.3174 | 11000 | 4.1056          | 0.3088   | 66.3578    | 0.4554         | 3063              |
-| 2.0441        | 1.4372 | 12000 | 3.9889          | 0.3307   | 57.9487    | 0.4800         | 3106              |
-| 2.0425        | 1.5569 | 13000 | 4.0293          | 0.3244   | 59.6874    | 0.4706         | 3064              |
-| 2.001         | 1.6767 | 14000 | 4.0007          | 0.3170   | 57.7703    | 0.4634         | 3047              |
-| 1.9714        | 1.7965 | 15000 | 3.9024          | 0.3380   | 54.4641    | 0.4794         | 3112              |
-| 1.8897        | 1.9162 | 16000 | 3.8101          | 0.3377   | 49.3560    | 0.4960         | 3109              |
-| 1.8703        | 2.0359 | 17000 | 3.8102          | 0.3482   | 47.5697    | 0.5110         | 2990              |
-| 1.945         | 2.1557 | 18000 | 3.6760          | 0.3632   | 42.9140    | 0.5275         | 3020              |
-| 1.8542        | 2.2755 | 19000 | 3.7630          | 0.3638   | 44.9888    | 0.5137         | 3002              |
-| 1.8986        | 2.3952 | 20000 | 3.7119          | 0.3571   | 45.3950    | 0.5106         | 3016              |
-| 1.8528        | 2.5150 | 21000 | 3.7906          | 0.3584   | 45.3568    | 0.5072         | 3111              |
-| 1.8645        | 2.6348 | 22000 | 3.6920          | 0.3669   | 43.4224    | 0.5245         | 3001              |
-| 1.821         | 2.7546 | 23000 | 3.6817          | 0.3577   | 42.1523    | 0.5274         | 3053              |
-| 1.8161        | 2.8743 | 24000 | 3.7436          | 0.3606   | 45.0721    | 0.5251         | 2967              |
-| 1.8218        | 2.9941 | 25000 | 3.7352          | 0.3547   | 45.8048    | 0.5124         | 3076              |
+tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-uganda-languages")
+model = AutoModelForMaskedLM.from_pretrained("xlm-roberta-uganda-languages")
 
+fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)
+result = fill_mask("Abantu b'omubyalo tibatera kwikiriza [MASK] muyaaka.")
+print(result)
+```
 
-### Framework versions
+## Training Details
 
-- Transformers 4.52.3
-- Pytorch 2.5.1+cu121
-- Datasets 3.6.0
-- Tokenizers 0.21.1
+- Training Steps: N/A
+- Training Loss: 2.1567
+- Learning Rate: 5e-05
+- Batch Size: 8
+- Epochs: 3

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:32b8816766c0d24ee50e1c3d11121aae912515ac1c6b03ab00b5545315d0e935
+oid sha256:ff742c3a638961cc3bb47a8fb5b93938106be84a1dce338615c0a249a8522b83
 size 1113205088

tokenizer.json

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e63c7a5faf7f3257542d756311f64572a52913c2845c4a3849529e1ef87f8eef
+oid sha256:d0091a328b3441d754e481db5a390d7f3b8dabc6016869fd13ba350d23ddc4cd
 size 17082832

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1800ce548bb8e9c9458e6c849025d381e0fcfd0f8e6693b40e9a5da6a8f87e9d
+oid sha256:7440518be059315987d17e7285244ad9581f12f439b3ff112ce17224bdf01eed
 size 5368