Initial commit

README.md

@@ -1,3 +1,285 @@
----
-license: mit
----
+---
+license: mit
+base_model: openai-community/gpt2
+tags:
+- generated_from_trainer
+metrics:
+- accuracy
+model-index:
+- name: gpt2-lang-ident
+  results: []
+---
+
+# gpt2-lang-ident
+
+This model is a fine-tuned version of [openai-community/gpt2](https://huggingface.co/openai-community/gpt2) on sampled sentences from `stanford-oval/ccnews` and `qanastek/EMEA-V3` datasets.
+It achieves the following results on the evaluation set:
+- Loss: 0.1210
+- Accuracy: 0.9721
+
+## Model description
+
+This model is trained to predict the language of the input text.
+
+## Intended uses & limitations
+
+The model can predict the following languages:
+```
+["af", "am", "ar", "as", "az", "ba", "be", "bg", "bn", "ca", "ceb", "ckb", "cs", "cy", "da", "de", "dv", "el", "en", "eo", "es", "et", "eu", "fa", "fi", "fr", "fy", "ga", "gd", "gl", "gu", "he", "hi", "hr", "hu", "hy", "id", "is", "it", "ja", "ka", "kk", "kn", "ku", "ky", "la", "lb", "lt", "lv", "mg", "mk", "ml", "mn", "mr", "mt", "my", "nds", "ne", "nl", "nn", "no", "or", "pa", "pl", "ps", "pt", "ro", "ru", "sah", "sd", "si", "sk", "sl", "sq", "sr", "sv", "sw", "ta", "te", "tg", "th", "tk", "tl", "tr", "tt", "ug", "uk", "ur", "vi", "yi"]
+```
+
+How to use:
+
+```python
+from transformers import (AutoModelForSequenceClassification, AutoTokenizer,
+                          pipeline)
+
+checkpoint = f"nie3e/gpt2-lang-ident"
+model = AutoModelForSequenceClassification.from_pretrained(checkpoint)
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+
+pipe = pipeline(
+    task="text-classification",
+    model=model,
+    tokenizer=tokenizer,
+    top_k=5
+)
+
+result = pipe("To jest model służący do identyfikacji języka!")
+print(result)
+```
+```
+[[{'label': 'pl', 'score': 0.9999653100967407}, {'label': 'sr', 'score': 1.5228776646836195e-05}, {'label': 'hr', 'score': 1.057955432770541e-05}, {'label': 'bn', 'score': 1.590750912328076e-06}, {'label': 'cs', 'score': 1.3942196801508544e-06}]]
+```
+
+## Training and evaluation data
+
+<details><summary>Training data ([lang]: count)</summary>
+
+    [bn]: 1947
+    [ar]: 1947
+    [vi]: 1947
+    [uk]: 1947
+    [kn]: 1947
+    [mr]: 1947
+    [id]: 1947
+    [te]: 1947
+    [no]: 1947
+    [ru]: 1947
+    [he]: 1947
+    [az]: 1947
+    [ca]: 1946
+    [fa]: 1946
+    [hi]: 1946
+    [th]: 1946
+    [tr]: 1946
+    [mk]: 1946
+    [ta]: 1945
+    [sq]: 1945
+    [ur]: 1942
+    [gu]: 1939
+    [ml]: 1936
+    [is]: 1738
+    [de]: 1543
+    [da]: 1521
+    [fi]: 1461
+    [el]: 1431
+    [nl]: 1424
+    [fr]: 1408
+    [cs]: 1401
+    [es]: 1397
+    [en]: 1394
+    [lt]: 1392
+    [hu]: 1379
+    [pt]: 1375
+    [lv]: 1373
+    [it]: 1360
+    [pl]: 1355
+    [sk]: 1355
+    [et]: 1348
+    [sl]: 1328
+    [sv]: 1300
+    [bg]: 1278
+    [mt]: 1234
+    [ro]: 1218
+    [kk]: 1179
+    [hy]: 1176
+    [or]: 1112
+    [pa]: 780
+    [sr]: 744
+    [as]: 735
+    [hr]: 722
+    [ne]: 626
+    [gl]: 566
+    [ckb]: 563
+    [ka]: 560
+    [ug]: 485
+    [ky]: 453
+    [eu]: 351
+    [ps]: 311
+    [tl]: 307
+    [fy]: 290
+    [mn]: 289
+    [si]: 244
+    [cy]: 214
+    [nn]: 212
+    [ku]: 195
+    [tg]: 176
+    [am]: 141
+    [tt]: 121
+    [ja]: 104
+    [lb]: 93
+    [tk]: 72
+    [be]: 64
+    [sw]: 45
+    [af]: 44
+    [my]: 40
+    [ceb]: 35
+    [la]: 33
+    [dv]: 20
+    [ba]: 19
+    [ga]: 19
+    [eo]: 19
+    [gd]: 16
+    [mg]: 15
+    [yi]: 14
+    [sah]: 14
+    [sd]: 11
+    [nds]: 11
+</details>
+
+<details><summary>Eval data ([lang]: count)</summary>
+
+    [te]: 195
+    [mk]: 195
+    [bn]: 195
+    [uk]: 195
+    [hi]: 195
+    [ar]: 195
+    [sq]: 195
+    [kn]: 195
+    [tr]: 195
+    [ca]: 195
+    [az]: 195
+    [fa]: 195
+    [ru]: 195
+    [mr]: 195
+    [id]: 195
+    [no]: 195
+    [vi]: 195
+    [th]: 195
+    [he]: 195
+    [gu]: 194
+    [ml]: 194
+    [ta]: 194
+    [ur]: 194
+    [is]: 174
+    [de]: 154
+    [da]: 152
+    [fi]: 146
+    [el]: 143
+    [nl]: 142
+    [fr]: 141
+    [es]: 140
+    [cs]: 140
+    [en]: 139
+    [lt]: 139
+    [hu]: 138
+    [lv]: 137
+    [pt]: 137
+    [it]: 136
+    [et]: 135
+    [pl]: 135
+    [sk]: 135
+    [sl]: 133
+    [sv]: 130
+    [bg]: 128
+    [mt]: 123
+    [ro]: 122
+    [hy]: 118
+    [kk]: 118
+    [or]: 111
+    [pa]: 78
+    [sr]: 74
+    [as]: 74
+    [hr]: 72
+    [ne]: 63
+    [gl]: 57
+    [ckb]: 56
+    [ka]: 56
+    [ug]: 49
+    [ky]: 45
+    [eu]: 35
+    [ps]: 31
+    [tl]: 31
+    [mn]: 29
+    [fy]: 29
+    [si]: 24
+    [nn]: 21
+    [cy]: 21
+    [ku]: 19
+    [tg]: 18
+    [am]: 14
+    [tt]: 12
+    [ja]: 10
+    [lb]: 9
+    [tk]: 7
+    [be]: 6
+    [my]: 4
+    [sw]: 4
+    [af]: 4
+    [ceb]: 3
+    [la]: 3
+    [ba]: 2
+    [dv]: 2
+    [eo]: 2
+    [gd]: 2
+    [ga]: 2
+    [mg]: 1
+    [sd]: 1
+    [nds]: 1
+    [yi]: 1
+    [sah]: 1
+</details>
+
+### Training procedure
+
+GPU: RTX 3090 \
+Training time: 1h 53min
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-05
+- train_batch_size: 8
+- eval_batch_size: 4
+- seed: 42
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 10
+- mixed_precision_training: Native AMP
+
+### Training results
+
+| Training Loss | Epoch | Step  | Validation Loss | Accuracy |
+|:-------------:|:-----:|:-----:|:---------------:|:--------:|
+| 0.2833        | 1.0   | 2812  | 0.2004          | 0.94     |
+| 0.168         | 2.0   | 5625  | 0.1567          | 0.954    |
+| 0.1131        | 3.0   | 8437  | 0.1429          | 0.9586   |
+| 0.0832        | 4.0   | 11250 | 0.1257          | 0.967    |
+| 0.0635        | 5.0   | 14062 | 0.1222          | 0.9682   |
+| 0.0479        | 6.0   | 16875 | 0.1214          | 0.9704   |
+| 0.0361        | 7.0   | 19687 | 0.1255          | 0.9712   |
+| 0.0258        | 8.0   | 22500 | 0.1178          | 0.9712   |
+| 0.0243        | 9.0   | 25312 | 0.1223          | 0.9724   |
+| 0.0171        | 10.0  | 28120 | 0.1210          | 0.9721   |
+
+
+### Framework versions
+
+- Transformers 4.36.2
+- Pytorch 2.1.2+cu121
+- Datasets 2.16.1
+- Tokenizers 0.15.0

config.json

@@ -0,0 +1,223 @@
+{
+  "_name_or_path": "openai-community/gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2ForSequenceClassification"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "id2label": {"0": "af",
+ "1": "am",
+ "2": "ar",
+ "3": "as",
+ "4": "az",
+ "5": "ba",
+ "6": "be",
+ "7": "bg",
+ "8": "bn",
+ "9": "ca",
+ "10": "ceb",
+ "11": "ckb",
+ "12": "cs",
+ "13": "cy",
+ "14": "da",
+ "15": "de",
+ "16": "dv",
+ "17": "el",
+ "18": "en",
+ "19": "eo",
+ "20": "es",
+ "21": "et",
+ "22": "eu",
+ "23": "fa",
+ "24": "fi",
+ "25": "fr",
+ "26": "fy",
+ "27": "ga",
+ "28": "gd",
+ "29": "gl",
+ "30": "gu",
+ "31": "he",
+ "32": "hi",
+ "33": "hr",
+ "34": "hu",
+ "35": "hy",
+ "36": "id",
+ "37": "is",
+ "38": "it",
+ "39": "ja",
+ "40": "ka",
+ "41": "kk",
+ "42": "kn",
+ "43": "ku",
+ "44": "ky",
+ "45": "la",
+ "46": "lb",
+ "47": "lt",
+ "48": "lv",
+ "49": "mg",
+ "50": "mk",
+ "51": "ml",
+ "52": "mn",
+ "53": "mr",
+ "54": "mt",
+ "55": "my",
+ "56": "nds",
+ "57": "ne",
+ "58": "nl",
+ "59": "nn",
+ "60": "no",
+ "61": "or",
+ "62": "pa",
+ "63": "pl",
+ "64": "ps",
+ "65": "pt",
+ "66": "ro",
+ "67": "ru",
+ "68": "sah",
+ "69": "sd",
+ "70": "si",
+ "71": "sk",
+ "72": "sl",
+ "73": "sq",
+ "74": "sr",
+ "75": "sv",
+ "76": "sw",
+ "77": "ta",
+ "78": "te",
+ "79": "tg",
+ "80": "th",
+ "81": "tk",
+ "82": "tl",
+ "83": "tr",
+ "84": "tt",
+ "85": "ug",
+ "86": "uk",
+ "87": "ur",
+ "88": "vi",
+ "89": "yi"},
+  "initializer_range": 0.02,
+  "label2id": {
+    "af": 0,
+    "am": 1,
+    "ar": 2,
+    "as": 3,
+    "az": 4,
+    "ba": 5,
+    "be": 6,
+    "bg": 7,
+    "bn": 8,
+    "ca": 9,
+    "ceb": 10,
+    "ckb": 11,
+    "cs": 12,
+    "cy": 13,
+    "da": 14,
+    "de": 15,
+    "dv": 16,
+    "el": 17,
+    "en": 18,
+    "eo": 19,
+    "es": 20,
+    "et": 21,
+    "eu": 22,
+    "fa": 23,
+    "fi": 24,
+    "fr": 25,
+    "fy": 26,
+    "ga": 27,
+    "gd": 28,
+    "gl": 29,
+    "gu": 30,
+    "he": 31,
+    "hi": 32,
+    "hr": 33,
+    "hu": 34,
+    "hy": 35,
+    "id": 36,
+    "is": 37,
+    "it": 38,
+    "ja": 39,
+    "ka": 40,
+    "kk": 41,
+    "kn": 42,
+    "ku": 43,
+    "ky": 44,
+    "la": 45,
+    "lb": 46,
+    "lt": 47,
+    "lv": 48,
+    "mg": 49,
+    "mk": 50,
+    "ml": 51,
+    "mn": 52,
+    "mr": 53,
+    "mt": 54,
+    "my": 55,
+    "nds": 56,
+    "ne": 57,
+    "nl": 58,
+    "nn": 59,
+    "no": 60,
+    "or": 61,
+    "pa": 62,
+    "pl": 63,
+    "ps": 64,
+    "pt": 65,
+    "ro": 66,
+    "ru": 67,
+    "sah": 68,
+    "sd": 69,
+    "si": 70,
+    "sk": 71,
+    "sl": 72,
+    "sq": 73,
+    "sr": 74,
+    "sv": 75,
+    "sw": 76,
+    "ta": 77,
+    "te": 78,
+    "tg": 79,
+    "th": 80,
+    "tk": 81,
+    "tl": 82,
+    "tr": 83,
+    "tt": 84,
+    "ug": 85,
+    "uk": 86,
+    "ur": 87,
+    "vi": 88,
+    "yi": 89
+  },
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "pad_token_id": 50256,
+  "problem_type": "single_label_classification",
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "float32",
+  "transformers_version": "4.36.2",
+  "use_cache": true,
+  "vocab_size": 50257
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:041d6a825692c31bad97df5b268b6f8ab6976cd971c33366fbf0c738fa61546a
+size 498050768

special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}