Upload all models and assets for an (latest)

README.md

@@ -36,7 +36,7 @@ metrics:
     value: 4.275
   - name: best_isotropy
     type: isotropy
-    value: 0.8230
+    value: 0.8232
   - name: vocabulary_size
     type: vocab
     value: 0
@@ -99,32 +99,32 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `CA Monzón puet estar: O Centro Atlético Monzón. O Club de Fútbol Atlético de Mon...`
+**Sample 1:** `Bobadilla puet estar: Bobadilla, un municipio de La Rioja. Bobadilla del Campo, ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ca ▁monzón ▁puet ▁estar : ▁o ▁centro ▁at lético ▁monzón ... (+10 more)` | 20 |
-| 16k | `▁ca ▁monzón ▁puet ▁estar : ▁o ▁centro ▁atlético ▁monzón . ... (+8 more)` | 18 |
-| 32k | `▁ca ▁monzón ▁puet ▁estar : ▁o ▁centro ▁atlético ▁monzón . ... (+8 more)` | 18 |
-| 64k | `▁ca ▁monzón ▁puet ▁estar : ▁o ▁centro ▁atlético ▁monzón . ... (+8 more)` | 18 |
+| 8k | `▁bob ad illa ▁puet ▁estar : ▁bob ad illa , ... (+17 more)` | 27 |
+| 16k | `▁bob ad illa ▁puet ▁estar : ▁bob ad illa , ... (+17 more)` | 27 |
+| 32k | `▁bob ad illa ▁puet ▁estar : ▁bob ad illa , ... (+17 more)` | 27 |
+| 64k | `▁bobadilla ▁puet ▁estar : ▁bobadilla , ▁un ▁municipio ▁de ▁la ... (+11 more)` | 21 |
 
-**Sample 2:** `En ista lista s'incluyen toz os presidents d'o Real Zaragoza dica hue: María Gay...`
+**Sample 2:** `Charleville-Mézières ye una localidat y comuna francesa, capital d'o departament...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁en ▁ista ▁lista ▁s ' incluy en ▁toz ▁os ▁presid ... (+24 more)` | 34 |
-| 16k | `▁en ▁ista ▁lista ▁s ' incluyen ▁toz ▁os ▁presidents ▁d ... (+22 more)` | 32 |
-| 32k | `▁en ▁ista ▁lista ▁s ' incluyen ▁toz ▁os ▁presidents ▁d ... (+22 more)` | 32 |
-| 64k | `▁en ▁ista ▁lista ▁s ' incluyen ▁toz ▁os ▁presidents ▁d ... (+22 more)` | 32 |
+| 8k | `▁char le ville - m é zi ères ▁ye ▁una ... (+26 more)` | 36 |
+| 16k | `▁char le ville - mé zi ères ▁ye ▁una ▁localidat ... (+25 more)` | 35 |
+| 32k | `▁char le ville - mé zi ères ▁ye ▁una ▁localidat ... (+23 more)` | 33 |
+| 64k | `▁charleville - mézières ▁ye ▁una ▁localidat ▁y ▁comuna ▁francesa , ... (+19 more)` | 29 |
 
-**Sample 3:** `Roitwalchen ye un lugar d'o municipio de Traunstein en o sud-este de Bavera, Ale...`
+**Sample 3:** `Schöngeising (en bavaro Scheegeising) ye un municipio de Bavera, Alemanya. Se tr...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ro it wal chen ▁ye ▁un ▁lugar ▁d ' o ... (+28 more)` | 38 |
-| 16k | `▁ro it wal chen ▁ye ▁un ▁lugar ▁d ' o ... (+28 more)` | 38 |
-| 32k | `▁ro it wal chen ▁ye ▁un ▁lugar ▁d ' o ... (+28 more)` | 38 |
-| 64k | `▁ro it walchen ▁ye ▁un ▁lugar ▁d ' o ▁municipio ... (+27 more)` | 37 |
+| 8k | `▁sch ön ge is ing ▁( en ▁bavaro ▁s che ... (+29 more)` | 39 |
+| 16k | `▁schön ge is ing ▁( en ▁bavaro ▁sche e ge ... (+25 more)` | 35 |
+| 32k | `▁schön ge ising ▁( en ▁bavaro ▁sche e ge ising ... (+20 more)` | 30 |
+| 64k | `▁schön ge ising ▁( en ▁bavaro ▁sche e ge ising ... (+20 more)` | 30 |
 
 
 ### Key Findings
@@ -274,27 +274,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `de gúdar s j simpson cyril cusack georgi parvanov con a comarca d o mesmo significau`
-2. `d a fundación d o comité d abril linear 8 d ixe paisache pocino de marzo`
-3. `a circuscripción de linkedin pachina web autualment no la provincia de bixuteria fina cappa producti...`
+1. `de las neveras y rfef aprebó a pachina web oficial d afers son asociadas con os`
+2. `d elba antiparte la provincia d as que se veiga torda collerada rafel vidaller tricas libro`
+3. `a rendición de sattler torna ta partecipar en ifriquiya y cariño homenage vasallage en aragonés vinc...`
 
 **Context Size 2:**
 
-1. `d a provincia de beneviento 2 071 m o río xalón y provincia de zaragoza y atros`
-2. `d o poro nucleyar afi komˈple k so di ˈi ˈpoɾo nuˈkliar ye per propio cualsiquier craba`
-3. `en a pachina web oficial la procedencia d os mes aptos os caracters comuns con atras posesions`
+1. `d a ciudat de zaragoza tomo i de castiella y leyón espanya o escritor de lausbubengeschichte ye`
+2. `d o reino se consolida la influyencia de l exercito estatounitesne en europa s extiende dende os`
+3. `en a provincia de teruel d o cual en fan parte 4 cantons y 129 comunas lista`
 
 **Context Size 3:**
 
-1. `a provincia de teruel comarca d a chacetania ye una d as prencipals actrices d o teatro y`
-2. `d a provincia de castellón de la plana alta la suya población ye de 651 habitants en germany`
-3. `una superficie de 62 99 km con una densidat de población de 8 hab km en ista localidat`
+1. `a provincia de zaragoza en a provincia de concepción y d as tres serols estando dimpués enamplato a`
+2. `d a provincia de guipuzcua ta atros usos se veiga carlos ix carlos ix 27 de chunio de`
+3. `una superficie de 158 60 km y una densidat de población de 346 35 hab km a suya`
 
 **Context Size 4:**
 
-1. `suya población ye de 100 habitants en una superficie de 9 77 km con una densidat de población de`
-2. `en una superficie de 13 70 km con una densidat de población de 29 15 hab km cheografía a`
-3. `d a provincia de cordoba ta atros usos se veiga o caire desambigación o caire u simplamnet caire الق...`
+1. `suya población ye de 81 habitants en una superficie de 194 49 km con una densidat de población de`
+2. `en una superficie de 64 16 km con una densidat de población de 43 44 hab km demografía administració...`
+3. `d a provincia de burgos ta atros usos se veiga fort yuma desambigación fort yuma títol orichinal en ...`
 
 
 ### Generated Text Samples (Subword-based)
@@ -303,27 +303,27 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_e_ra_ce_aren_pi`
-2. `al._iargo_rennye`
-3. `ebalobalo,_opo_c`
+1. `_un_der_dent_ckm`
+2. `as_en_as_2_tacla`
+3. `en_lern_don_vitr`
 
 **Context Size 2:**
 
-1. `a_cominelde_al_de`
-2. `_d'a_s'esmarroixe`
-3. `e_au_gottorioními`
+1. `a_saus_dabinascer`
+2. `_derfica_sublosti`
+3. `e_manaisitau_suyo`
 
 **Context Size 3:**
 
-1. `_de_papezina_creye`
-2. `de_heyemios_poblac`
-3. `_d'africa_de_mayo»`
+1. `_dens._val_novant,`
+2. `de_319_de_fuel,_qu`
+3. `_d'o_primetada_cic`
 
 **Context Size 4:**
 
-1. `_de_l'anyos._ye_reg`
-2. `_en_la_menclaude._v`
-3. `_d'africantón_de_se`
+1. `_de_jean-jose_(naix`
+2. `_en_sido_per_bueno,`
+3. `_d'anglés_jean_sabi`
 
 
 ### Key Findings
@@ -428,18 +428,18 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.8202 | 0.3403 | N/A | N/A |
-| **mono_64d** | 64 | 0.8230 🏆 | 0.2693 | N/A | N/A |
-| **mono_128d** | 128 | 0.8049 | 0.2129 | N/A | N/A |
-| **aligned_32d** | 32 | 0.8202 | 0.3495 | 0.1580 | 0.5000 |
-| **aligned_64d** | 64 | 0.8230 | 0.2690 | 0.2540 | 0.6280 |
-| **aligned_128d** | 128 | 0.8049 | 0.2090 | 0.3720 | 0.7380 |
+| **mono_32d** | 32 | 0.8168 | 0.3517 | N/A | N/A |
+| **mono_64d** | 64 | 0.8232 🏆 | 0.2779 | N/A | N/A |
+| **mono_128d** | 128 | 0.8044 | 0.2016 | N/A | N/A |
+| **aligned_32d** | 32 | 0.8168 | 0.3524 | 0.1520 | 0.4840 |
+| **aligned_64d** | 64 | 0.8232 | 0.2773 | 0.2480 | 0.6340 |
+| **aligned_128d** | 128 | 0.8044 | 0.2034 | 0.3740 | 0.7380 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_64d with 0.8230 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.2750. Lower values indicate better semantic separation.
-- **Alignment Quality:** Aligned models achieve up to 37.2% R@1 in cross-lingual retrieval.
+- **Best Isotropy:** mono_64d with 0.8232 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.2774. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 37.4% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
@@ -461,19 +461,20 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-co` | comuni, concebitas, complices |
-| `-ca` | cassovia, cazadors, castieillo |
-| `-ma` | macromutacions, mariahilfkirche, maree |
-| `-re` | restauro, reumert, recomendable |
+| `-co` | confrontatos, conchecturau, coluche |
+| `-ca` | casartelli, camprodón, canthus |
+| `-re` | reitzenstein, reformata, reinando |
+| `-de` | destruyir, denasalizadas, debucourt |
+| `-ma` | marktes, matosinhos, marciac |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-s` | viescas, tomus, biggs |
-| `-a` | abaurrepea, telangana, actuaba |
-| `-as` | viescas, medas, rallatas |
-| `-os` | vientos, rasos, tecnolochicos |
-| `-es` | phoenicopteriformes, gabes, bolcheviques |
+| `-s` | mourvilles, iliricas, mylonas |
+| `-a` | cingüenda, lecinyena, reformata |
+| `-as` | iliricas, mylonas, aeneas |
+| `-os` | confrontatos, agnatos, estranios |
+| `-es` | mourvilles, marktes, forbes |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -481,18 +482,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `ento` | 1.72x | 126 contexts | rento, gento, sento |
-| `rago` | 2.03x | 58 contexts | crago, drago, ragot |
-| `ranc` | 1.60x | 141 contexts | rancó, ranch, rance |
-| `enci` | 1.55x | 164 contexts | renci, encia, encies |
-| `obla` | 1.95x | 56 contexts | nobla, robla, pobla |
-| `renc` | 1.77x | 82 contexts | renci, arenc, wrench |
-| `ació` | 2.02x | 47 contexts | nació, fació, ación |
-| `ient` | 1.50x | 176 contexts | oient, cient, aient |
-| `nter` | 1.55x | 146 contexts | anter, enter, unter |
-| `cion` | 1.56x | 110 contexts | scion, accion, nacion |
-| `ncia` | 1.69x | 61 contexts | encia, uncia, oencia |
-| `mbre` | 1.55x | 75 contexts | mbret, ambre, umbre |
+| `ient` | 1.70x | 176 contexts | cient, oient, dient |
+| `ento` | 1.74x | 126 contexts | sento, bento, cento |
+| `rago` | 2.03x | 58 contexts | arago, trago, ragot |
+| `ranc` | 1.64x | 141 contexts | franc, rance, ranca |
+| `ació` | 2.09x | 47 contexts | nació, ación, fació |
+| `enci` | 1.53x | 164 contexts | encia, renci, oencia |
+| `obla` | 1.90x | 56 contexts | robla, pobla, nobla |
+| `nter` | 1.50x | 146 contexts | anter, enter, inter |
+| `ncia` | 1.72x | 61 contexts | encia, uncia, oencia |
+| `cion` | 1.50x | 110 contexts | scion, nacion, accion |
+| `idat` | 2.00x | 28 contexts | unidat, deidat, humidat |
+| `mbre` | 1.55x | 75 contexts | ambre, ombre, umbre |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -500,16 +501,16 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-ca` | `-s` | 64 words | cavens, cabaixos |
-| `-co` | `-s` | 62 words | conejares, concordias |
-| `-co` | `-a` | 57 words | condeixa, cogota |
-| `-ma` | `-s` | 52 words | manganés, manus |
-| `-ca` | `-a` | 44 words | camberra, carola |
-| `-re` | `-s` | 33 words | relationships, refusadas |
-| `-re` | `-a` | 26 words | representa, relacionada |
-| `-ma` | `-a` | 23 words | maganza, magalona |
-| `-co` | `-as` | 19 words | concordias, contadas |
-| `-ca` | `-os` | 16 words | cabaixos, calibos |
+| `-co` | `-s` | 71 words | concilios, comenges |
+| `-ca` | `-s` | 53 words | cabrinos, caracteres |
+| `-ca` | `-a` | 49 words | cafeína, caixera |
+| `-co` | `-a` | 49 words | cosida, conquiolina |
+| `-ma` | `-s` | 41 words | mauriscus, mandos |
+| `-ma` | `-a` | 36 words | mainila, mamma |
+| `-re` | `-s` | 34 words | reprimius, rechiradors |
+| `-re` | `-a` | 33 words | relochería, renacentista |
+| `-de` | `-a` | 30 words | desidia, dentada |
+| `-de` | `-s` | 30 words | demograficos, deverbativos |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -517,21 +518,21 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| lombardes | **`lombard-es`** | 4.5 | `lombard` |
-| fanaticos | **`fanatic-os`** | 4.5 | `fanatic` |
-| retransmite | **`re-transmite`** | 4.5 | `transmite` |
-| dimitrios | **`dimitri-os`** | 4.5 | `dimitri` |
-| terroristas | **`terrorist-as`** | 4.5 | `terrorist` |
-| normandas | **`normand-as`** | 4.5 | `normand` |
-| castellan | **`ca-stellan`** | 4.5 | `stellan` |
-| coorganización | **`co-organización`** | 4.5 | `organización` |
-| tortosinas | **`tortosin-as`** | 4.5 | `tortosin` |
-| reportaje | **`re-portaje`** | 4.5 | `portaje` |
-| requiestas | **`re-quiest-as`** | 3.0 | `quiest` |
-| consumitos | **`co-nsumit-os`** | 3.0 | `nsumit` |
-| califatos | **`ca-lifat-os`** | 3.0 | `lifat` |
-| conservamos | **`co-nservam-os`** | 3.0 | `nservam` |
-| colonizatos | **`co-lonizat-os`** | 3.0 | `lonizat` |
+| repoblatos | **`re-poblat-os`** | 6.0 | `poblat` |
+| altoaragonesas | **`altoaragon-es-as`** | 6.0 | `altoaragon` |
+| recullindo | **`re-cullindo`** | 4.5 | `cullindo` |
+| reorganizar | **`re-organizar`** | 4.5 | `organizar` |
+| romanticos | **`romantic-os`** | 4.5 | `romantic` |
+| casellato | **`ca-sellato`** | 4.5 | `sellato` |
+| discapacitatos | **`discapacitat-os`** | 4.5 | `discapacitat` |
+| lexicales | **`lexical-es`** | 4.5 | `lexical` |
+| monetarias | **`monetari-as`** | 4.5 | `monetari` |
+| reprodución | **`re-produción`** | 4.5 | `produción` |
+| deportaban | **`de-portaban`** | 4.5 | `portaban` |
+| desconoixitas | **`de-sconoixit-as`** | 3.0 | `sconoixit` |
+| caspolinas | **`ca-spolin-as`** | 3.0 | `spolin` |
+| conservaderas | **`co-nservader-as`** | 3.0 | `nservader` |
+| decimetros | **`de-cimetr-os`** | 3.0 | `cimetr` |
 
 ### 6.6 Linguistic Interpretation
 
@@ -763,4 +764,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 14:50:06*
+*Report Date: 2026-01-03 17:05:39*

models/embeddings/aligned/an_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5fd4098fa2ab7a020113c01a17200837ba8ac73638e61c8edb659c85aaef93fc
+oid sha256:ac84dbcff516b57e98d5bb2f38432f3dbb7fbcc5d376686ddcf84f5fe2c8ed90
 size 1149568062

models/embeddings/aligned/an_128d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cd99fd2982f86d44f39d6905f1bab4d510cd2a435654c9497853d449b81da158
+oid sha256:5c50c45cbc54bfd87537fecabc141a51b8cc24fab6a01562f7eab6f3f67380dd
 size 65664

models/embeddings/aligned/an_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8162f33f0e10366ed7d3fdfb8fa216d05dbd6d4a16b2198859372aa289d98b14
+oid sha256:044b1985e3203661353a60d5fc823ff5dcdf618f0d044f1c133ba2af374c3a98
 size 288983358

models/embeddings/aligned/an_32d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:353d511257321af48a3269657d78455f93120c5fcf17d70f5f3b6de72a95f856
+oid sha256:a78cf53ebf35034301a50aedca150893e531c05575b15e0fa980d7e00d38de78
 size 4224

models/embeddings/aligned/an_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95a4be660051d6fcd8c79a595fd6237cd7f54790bfab31f65b7a8f9c2533a8e2
+oid sha256:1125c90e30e6b620a58d0ab14de72ea4edb7f4ff2076ec610fe121f665628723
 size 575844926

models/embeddings/aligned/an_64d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0a69faaf2fde42a4fb0c85609a8ab9ff29cffe1093a3932f0aa3d238f32d74f9
+oid sha256:a1613a507b39a223bf7c8ea0a27567de19692f8961db6031ad1953e0aeac1b38
 size 16512

models/embeddings/monolingual/an_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5fd4098fa2ab7a020113c01a17200837ba8ac73638e61c8edb659c85aaef93fc
+oid sha256:ac84dbcff516b57e98d5bb2f38432f3dbb7fbcc5d376686ddcf84f5fe2c8ed90
 size 1149568062

models/embeddings/monolingual/an_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8162f33f0e10366ed7d3fdfb8fa216d05dbd6d4a16b2198859372aa289d98b14
+oid sha256:044b1985e3203661353a60d5fc823ff5dcdf618f0d044f1c133ba2af374c3a98
 size 288983358

models/embeddings/monolingual/an_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95a4be660051d6fcd8c79a595fd6237cd7f54790bfab31f65b7a8f9c2533a8e2
+oid sha256:1125c90e30e6b620a58d0ab14de72ea4edb7f4ff2076ec610fe121f665628723
 size 575844926

models/subword_markov/an_markov_ctx1_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:82737fda9661e7c5ecf12fa0a67a2d2bcc18af9b5e9e30a7b0158eef5c952909
-size 176131
+oid sha256:418a8dd35bc3f35f4c9a9f8bd9e49a75ef9f982d2a681c378c9b1ab291b79e05
+size 172370

models/subword_markov/an_markov_ctx2_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f0fc7fa0b6576fa1c527465494c2d88de956a333230c36d23818149db6ffd093
-size 938570
+oid sha256:64ec46893ee8e958fb992f8a0d6b1efbe094ad45c2bced5fab94ef6c4d5be0ae
+size 926386

models/subword_markov/an_markov_ctx3_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:002be2f1eb632f8e6574678f853e93f4bbca357d8fe282182dc11766db2955b4
-size 3816590
+oid sha256:c85ea7316bf72d0414be0d5c5a80ba3defafef47dd6564d73476a4f472ed3ccd
+size 3820524

models/subword_markov/an_markov_ctx4_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:1a8a70c3406dca6ea2afc3bd1a4133f81559c04cc64ebb35b40bd88bf0512e2e
-size 13281906
+oid sha256:47995cbd03e3f597500e069be7d15ea5b0f3b8dc1809809a35fff9125930aa43
+size 13273622

models/subword_ngram/an_2gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0f6ff5966a58dcbf3f78684a10d07bc43305bed96d2231b9a562c78ffe158f5e
-size 93856
+oid sha256:ea93eb082876f788ceac81f55884b7629479387260dac53859896c4df59aa158
+size 93694

models/subword_ngram/an_3gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:2b9410a959938ec2ee19d3d785fdbe7c03b9888b017ac84c0907cc567cc67382
-size 690365
+oid sha256:a4b994cdf96bfe61e18ebf7054adee26a9bed7de5843246f543ec78587c38b52
+size 689250

models/subword_ngram/an_4gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54766951ed2a769fc5227d30241529307ad56476f75a1c6ded8dfd17cfd005c6
-size 3361576
+oid sha256:c6577e2b6c3d47dd3140a98a011f6dced507fc1d8fedf05628f215cc75e72f0f
+size 3361849

models/subword_ngram/an_5gram_subword.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b2d2a42c46ff9f023d7a1e77f90126884ff7127984bdea7beb92f1a4ea9ba286
-size 10691226
+oid sha256:19eb72098a1a0f1357d659b77cb07df99d8c6d0d79d9a1a5a0d1e91653334361
+size 10680494

models/tokenizer/an_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:be4e4495dd91b35e24a5508fe0c746604c258fa5e24d680caa9984c3edca42a6
+oid sha256:00b5360b1c6c573f0a26c6c9caa8f8616ad8b3590c412e044215e61b09ae45bc
 size 511374

models/tokenizer/an_tokenizer_32k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5fa703683fed7866aee835ffdc13a44b7a714de834a38b62b70558897787c1fb
+oid sha256:af11636daefec89ab2ed283c1f92a1a80d6935eeb7b27eb28adc5ce1a6d61fc2
 size 791864

models/tokenizer/an_tokenizer_64k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f1989e3ff85462c98efc25880434afda45544dd0dff0bda857ccfabf9d8725dc
+oid sha256:0b59aef8468601ed51d2b619bff9f61ab6385da57f1f5a42cc720807d71c8da8
 size 1363420

models/tokenizer/an_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:24cd47460a4e12244cf3b7f7de3902be403e0466c77da2537d5c747ecd2e8c4d
+oid sha256:8826546ab7f635988b0fbd46cb99b29b0958ba8c0651006623da483a60e362bd
 size 374577

models/word_markov/an_markov_ctx1_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7b2b7d443bdb24601458b682628e19a693091fcc060312b8c29b8d0aeca42d42
-size 23695943
+oid sha256:30996f7f6a3f5ddb43fa633991dcaae0b81892f02fbee1a2f05770048ecdcf48
+size 23812223

models/word_markov/an_markov_ctx2_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f8a94e65ec2e1cbb3239d99357349f5de3f99c8f328da97ec36993f6bcaf9b2c
-size 65492964
+oid sha256:27f0b9160812153a28e0b058756dd952ab32f0301e008408a81a7b57503ce255
+size 65243685

models/word_markov/an_markov_ctx3_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:119d049347e9ea06e047d1c592bd25f448fb0db787357a8e2dde7fb81bdabf9a
-size 104857909
+oid sha256:df3fa39e60b02afcc490205e9561ff1be1ee90160d33ed2b69a5233b53751d46
+size 104894050

models/word_markov/an_markov_ctx4_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cf40f71c21fa590cd93a8da91c56eacb62e024b3a99a039bea79c763d3f480a5
-size 132108456
+oid sha256:c5116bdb5537880e6a2aace5c53eea87897150be7cb5c4b96eef2ccba19a2c81
+size 132167575

models/word_ngram/an_2gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c5767d5baec0276bb908cf29398dfa5499b6be1cc66006cd5708bac7b32e1a23
-size 3254179
+oid sha256:05063ada0a28c7b63bdccdb07799f7bc8b489337a2361d7b3a631dd2fd65d936
+size 3260032

models/word_ngram/an_3gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:32e5bc99c871366cc3acf00709b843ccb6ece85dff6f51fc4716cca234b59a2d
-size 6891480
+oid sha256:7948ddaa3f31659e4c68ba064ffff54008ef5201095f8ff9ea7f03e8f5c1e524
+size 6869267

models/word_ngram/an_4gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5783556da79bc42a443ee21b9fcee41427cdd2ae4de0f99a8ee08fd84e0bca87
-size 14120282
+oid sha256:643b3056a717c8bce2f91de9f804d96ea3c0aa62a9621b1cc8daeae7a498e00c
+size 14121384

models/word_ngram/an_5gram_word.parquet

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6470d74718860413111212c0cf7d05e0f6074fa1f9835d5e95d0ff11cd0c4f45
-size 12888396
+oid sha256:92e8eda0653cd749f66c469b4dfa2cb5c19637bb90b6defb0be21b38d2e6c5b7
+size 12899086