DerivedFunction
/

polyglot-tagger-v2.2

@@ -162,12 +162,125 @@ factors were used to simulate messy text, and to reduce single character bias on
 - Random chance to change the casing of compatible language scripts, such as Latin and Cyrllic.
 - Low chance of simulating OCR and messy text with character mutation.
 To generalize well on both the target language and code switching a circulumn is provided:
 - Pure documents 55%: Single language to learn its vocabulary
 - Homogenous 25%: Single language + one foreign sentence to learn simple code switching
 - Spliced 10%: A foreign sentence is centered between two same-language sentence, with the first sentence's punctuation stripped, and second sentence's forced to be lowercased.
 - Mixed 10%: Generic mix of any languages.
 This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base).
 It achieves the following results on the evaluation set:
 - Loss: 0.0345

 - Random chance to change the casing of compatible language scripts, such as Latin and Cyrllic.
 - Low chance of simulating OCR and messy text with character mutation.
 To generalize well on both the target language and code switching a circulumn is provided:
 - Pure documents 55%: Single language to learn its vocabulary
 - Homogenous 25%: Single language + one foreign sentence to learn simple code switching
 - Spliced 10%: A foreign sentence is centered between two same-language sentence, with the first sentence's punctuation stripped, and second sentence's forced to be lowercased.
 - Mixed 10%: Generic mix of any languages.
+| lang | train | train % | eval | eval % | all_splits | all_splits % |
+| :--- | ---: | ---: | ---: | ---: | ---: | ---: |
+| en | 158088 | 2.45% | 2785 | 3.77% | 160873 | 2.46% |
+| ru | 125692 | 1.94% | 1915 | 2.59% | 127607 | 1.95% |
+| es | 125219 | 1.94% | 1809 | 2.45% | 127028 | 1.94% |
+| ja | 125212 | 1.94% | 1786 | 2.42% | 126998 | 1.94% |
+| fr | 123594 | 1.91% | 1803 | 2.44% | 125397 | 1.92% |
+| de | 121413 | 1.88% | 1714 | 2.32% | 123127 | 1.88% |
+| zh | 120667 | 1.87% | 1745 | 2.36% | 122412 | 1.87% |
+| pt | 119430 | 1.85% | 1749 | 2.37% | 121179 | 1.85% |
+| it | 117478 | 1.82% | 1596 | 2.16% | 119074 | 1.82% |
+| ar | 101539 | 1.57% | 1208 | 1.63% | 102747 | 1.57% |
+| fi | 100922 | 1.56% | 1490 | 2.02% | 102412 | 1.57% |
+| uk | 98233 | 1.52% | 1160 | 1.57% | 99393 | 1.52% |
+| pl | 96600 | 1.49% | 1144 | 1.55% | 97744 | 1.50% |
+| no | 93851 | 1.45% | 1471 | 1.99% | 95322 | 1.46% |
+| hu | 92941 | 1.44% | 1077 | 1.46% | 94018 | 1.44% |
+| tr | 92566 | 1.43% | 1053 | 1.42% | 93619 | 1.43% |
+| nl | 91431 | 1.41% | 1067 | 1.44% | 92498 | 1.41% |
+| he | 91496 | 1.42% | 832 | 1.13% | 92328 | 1.41% |
+| cs | 91275 | 1.41% | 989 | 1.34% | 92264 | 1.41% |
+| da | 87309 | 1.35% | 1223 | 1.65% | 88532 | 1.35% |
+| lt | 86897 | 1.34% | 935 | 1.26% | 87832 | 1.34% |
+| mk | 85658 | 1.33% | 824 | 1.11% | 86482 | 1.32% |
+| eo | 83778 | 1.30% | 794 | 1.07% | 84572 | 1.29% |
+| mr | 83264 | 1.29% | 744 | 1.01% | 84008 | 1.28% |
+| ko | 81829 | 1.27% | 1005 | 1.36% | 82834 | 1.27% |
+| hi | 81659 | 1.26% | 979 | 1.32% | 82638 | 1.26% |
+| tl | 79985 | 1.24% | 966 | 1.31% | 80951 | 1.24% |
+| hy | 76909 | 1.19% | 735 | 0.99% | 77644 | 1.19% |
+| el | 75369 | 1.17% | 728 | 0.98% | 76097 | 1.16% |
+| ro | 73559 | 1.14% | 763 | 1.03% | 74322 | 1.14% |
+| is | 72356 | 1.12% | 987 | 1.34% | 73343 | 1.12% |
+| sk | 71990 | 1.11% | 858 | 1.16% | 72848 | 1.11% |
+| la | 70651 | 1.09% | 745 | 1.01% | 71396 | 1.09% |
+| be | 70521 | 1.09% | 867 | 1.17% | 71388 | 1.09% |
+| fa | 70584 | 1.09% | 717 | 0.97% | 71301 | 1.09% |
+| bg | 69684 | 1.08% | 673 | 0.91% | 70357 | 1.08% |
+| lv | 67627 | 1.05% | 691 | 0.93% | 68318 | 1.04% |
+| ms | 66271 | 1.03% | 770 | 1.04% | 67041 | 1.03% |
+| af | 64699 | 1.00% | 982 | 1.33% | 65681 | 1.00% |
+| ckb | 64368 | 1.00% | 587 | 0.79% | 64955 | 0.99% |
+| kk | 63640 | 0.98% | 621 | 0.84% | 64261 | 0.98% |
+| eu | 63398 | 0.98% | 673 | 0.91% | 64071 | 0.98% |
+| ka | 63201 | 0.98% | 523 | 0.71% | 63724 | 0.97% |
+| mn | 62551 | 0.97% | 641 | 0.87% | 63192 | 0.97% |
+| hr | 62427 | 0.97% | 711 | 0.96% | 63138 | 0.97% |
+| oc | 62292 | 0.96% | 661 | 0.89% | 62953 | 0.96% |
+| id | 62134 | 0.96% | 732 | 0.99% | 62866 | 0.96% |
+| ky | 61634 | 0.95% | 637 | 0.86% | 62271 | 0.95% |
+| ba | 61637 | 0.95% | 584 | 0.79% | 62221 | 0.95% |
+| ur | 61550 | 0.95% | 578 | 0.78% | 62128 | 0.95% |
+| th | 60731 | 0.94% | 576 | 0.78% | 61307 | 0.94% |
+| bn | 60588 | 0.94% | 415 | 0.56% | 61003 | 0.93% |
+| ps | 60342 | 0.93% | 533 | 0.72% | 60875 | 0.93% |
+| sv | 59918 | 0.93% | 937 | 1.27% | 60855 | 0.93% |
+| tt | 60177 | 0.93% | 634 | 0.86% | 60811 | 0.93% |
+| pa | 60137 | 0.93% | 599 | 0.81% | 60736 | 0.93% |
+| sw | 60148 | 0.93% | 558 | 0.75% | 60706 | 0.93% |
+| kn | 60037 | 0.93% | 631 | 0.85% | 60668 | 0.93% |
+| as | 59839 | 0.93% | 374 | 0.51% | 60213 | 0.92% |
+| cy | 58188 | 0.90% | 655 | 0.89% | 58843 | 0.90% |
+| jv | 57805 | 0.89% | 508 | 0.69% | 58313 | 0.89% |
+| bs | 57399 | 0.89% | 655 | 0.89% | 58054 | 0.89% |
+| ga | 57233 | 0.89% | 672 | 0.91% | 57905 | 0.89% |
+| ca | 56547 | 0.87% | 606 | 0.82% | 57153 | 0.87% |
+| gl | 55312 | 0.86% | 577 | 0.78% | 55889 | 0.85% |
+| sl | 55017 | 0.85% | 598 | 0.81% | 55615 | 0.85% |
+| ku | 54674 | 0.85% | 537 | 0.73% | 55211 | 0.84% |
+| ne | 54102 | 0.84% | 440 | 0.60% | 54542 | 0.83% |
+| uz | 53777 | 0.83% | 507 | 0.69% | 54284 | 0.83% |
+| tg | 50762 | 0.79% | 502 | 0.68% | 51264 | 0.78% |
+| br | 49263 | 0.76% | 554 | 0.75% | 49817 | 0.76% |
+| et | 49249 | 0.76% | 511 | 0.69% | 49760 | 0.76% |
+| lb | 48192 | 0.75% | 492 | 0.67% | 48684 | 0.74% |
+| su | 48185 | 0.75% | 480 | 0.65% | 48665 | 0.74% |
+| mt | 47694 | 0.74% | 446 | 0.60% | 48140 | 0.74% |
+| sr | 47385 | 0.73% | 458 | 0.62% | 47843 | 0.73% |
+| sq | 45528 | 0.70% | 514 | 0.70% | 46042 | 0.70% |
+| ml | 43461 | 0.67% | 429 | 0.58% | 43890 | 0.67% |
+| or | 41301 | 0.64% | 413 | 0.56% | 41714 | 0.64% |
+| te | 40065 | 0.62% | 381 | 0.52% | 40446 | 0.62% |
+| yi | 38484 | 0.60% | 353 | 0.48% | 38837 | 0.59% |
+| ta | 35897 | 0.56% | 378 | 0.51% | 36275 | 0.55% |
+| mg | 35133 | 0.54% | 342 | 0.46% | 35475 | 0.54% |
+| si | 34611 | 0.54% | 343 | 0.46% | 34954 | 0.53% |
+| gu | 29347 | 0.45% | 298 | 0.40% | 29645 | 0.45% |
+| vi | 28448 | 0.44% | 329 | 0.45% | 28777 | 0.44% |
+| rm | 27668 | 0.43% | 252 | 0.34% | 27920 | 0.43% |
+| bo | 25636 | 0.40% | 217 | 0.29% | 25853 | 0.40% |
+| ug | 23932 | 0.37% | 213 | 0.29% | 24145 | 0.37% |
+| dv | 22580 | 0.35% | 204 | 0.28% | 22784 | 0.35% |
+| am | 22498 | 0.35% | 227 | 0.31% | 22725 | 0.35% |
+| yo | 22441 | 0.35% | 229 | 0.31% | 22670 | 0.35% |
+| my | 21832 | 0.34% | 210 | 0.28% | 22042 | 0.34% |
+| so | 21058 | 0.33% | 201 | 0.27% | 21259 | 0.33% |
+| km | 21064 | 0.33% | 187 | 0.25% | 21251 | 0.33% |
+| sd | 20471 | 0.32% | 199 | 0.27% | 20670 | 0.32% |
+| zu | 19688 | 0.30% | 186 | 0.25% | 19874 | 0.30% |
+| lo | 18555 | 0.29% | 188 | 0.25% | 18743 | 0.29% |
+| ti | 18116 | 0.28% | 193 | 0.26% | 18309 | 0.28% |
+| ce | 16789 | 0.26% | 181 | 0.24% | 16970 | 0.26% |
+| ny | 16544 | 0.26% | 159 | 0.22% | 16703 | 0.26% |
+| gd | 14012 | 0.22% | 142 | 0.19% | 14154 | 0.22% |
+| xh | 9373 | 0.15% | 96 | 0.13% | 9469 | 0.14% |
+| om | 6113 | 0.09% | 55 | 0.07% | 6168 | 0.09% |
+| sco | 3362 | 0.05% | 30 | 0.04% | 3392 | 0.05% |
+| **total** | 6463786 | 100.00% | 73931 | 100.00% | 6537717 | 100.00% |
 This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base).
 It achieves the following results on the evaluation set:
 - Loss: 0.0345