Upload all models and assets for ann (latest)

README.md

@@ -36,7 +36,7 @@ metrics:
     value: 4.353
   - name: best_isotropy
     type: isotropy
-    value: 0.1710
+    value: 0.1716
   - name: vocabulary_size
     type: vocab
     value: 0
@@ -97,26 +97,26 @@ We analyze tokenizers, n-gram models, Markov chains, vocabulary statistics, and
 
 Below are sample sentences tokenized with each vocabulary size:
 
-**Sample 1:** `Ọrọn môkọt ire: Ebi Ọrọn (ife) Ido Ọrọn (ama mè ere) Ọrọn (Mkpulu-ija) Usem Ọrọn...`
+**Sample 1:** `Ida Obolo ìre ikpa etip-usen eyi ebi Ogbo Ikwaan̄ Usem Obolo ekisan̄a isibi me e...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ọrọn ▁môkọt ▁ire : ▁ebi ▁ọrọn ▁( ife ) ▁ido ... (+17 more)` | 27 |
-| 16k | `▁ọrọn ▁môkọt ▁ire : ▁ebi ▁ọrọn ▁( ife ) ▁ido ... (+17 more)` | 27 |
+| 8k | `▁ida ▁obolo ▁ìre ▁ikpa ▁etip - usen ▁eyi ▁ebi ▁ogbo ... (+22 more)` | 32 |
+| 16k | `▁ida ▁obolo ▁ìre ▁ikpa ▁etip - usen ▁eyi ▁ebi ▁ogbo ... (+21 more)` | 31 |
 
-**Sample 2:** `Nde ìre oke mgbọ òsoso usen jaaba. Ekisa nde ifuk onyan̄ mè isa ifuk acha si. Nd...`
+**Sample 2:** `Jameni (òrere Deutschland me usem Jameni, mè ire Germany me usem Ebeke) ìre ido ...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁nde ▁ìre ▁oke ▁mgbọ ▁òsoso ▁usen ▁jaaba . ▁ekisa ▁nde ... (+27 more)` | 37 |
-| 16k | `▁nde ▁ìre ▁oke ▁mgbọ ▁òsoso ▁usen ▁jaaba . ▁ekisa ▁nde ... (+26 more)` | 36 |
+| 8k | `▁jameni ▁( òrere ▁de uts ch land ▁me ▁usem ▁jameni ... (+15 more)` | 25 |
+| 16k | `▁jameni ▁( òrere ▁deutschland ▁me ▁usem ▁jameni , ▁mè ▁ire ... (+12 more)` | 22 |
 
-**Sample 3:** `Ọngari (òrere Hungary me usem Ebeke, mè ire Magyarország me usem Ọn̄gari) ìre id...`
+**Sample 3:** `ìre ikọ ekisa ìjeen̄ uyok uyok ekiket kubọk nriki ònan̄a me inu ikeke ene chieen...`
 
 | Vocab | Tokens | Count |
 |-------|--------|-------|
-| 8k | `▁ọ n gari ▁( òrere ▁h ungary ▁me ▁usem ▁ebeke ... (+28 more)` | 38 |
-| 16k | `▁ọngari ▁( òrere ▁hungary ▁me ▁usem ▁ebeke , ▁mè ▁ire ... (+19 more)` | 29 |
+| 8k | `▁ìre ▁ikọ ▁ekisa ▁ìjeen̄ ▁uyok ▁uyok ▁ekiket ▁kubọk ▁nriki ▁ònan̄a ... (+17 more)` | 27 |
+| 16k | `▁ìre ▁ikọ ▁ekisa ▁ìjeen̄ ▁uyok ▁uyok ▁ekiket ▁kubọk ▁nriki ▁ònan̄a ... (+17 more)` | 27 |
 
 
 ### Key Findings
@@ -266,27 +266,27 @@ Below are text samples generated from each word-based Markov chain model:
 
 **Context Size 1:**
 
-1. `me lek kiban̄ ekitimbe akọn̄ kè ofirikosok gbọgbọ otu ifuk mè ikpọk ya ìre isilam inu`
-2. `mè ijọn̄ ido ya me naijiria agan̄ ichep ura sà ogwu òkitaak chieen̄ ikpọ mè mgbọ`
-3. `agan̄ mkpulu uwu usọ ifuk ene ewabe ichit me emen mgbọ etiopia ìkup me agọọk nkween̄`
+1. `me atasuk eyi akọp ìkigwat lek ogugo ijọn̄ afirika etete udun̄nde òrere dmitri mendeleev me esese`
+2. `mè anam ge me ere ònire agan̄ inyọn̄ me lek ijọn̄ sudan îgbuku igwook mè ikisa`
+3. `agan̄ erumfaka kiristien itap mè mbit pọtugalu ekisabe ibọp ekwu òkukup me usem obolo usini ekilọk`
 
 **Context Size 2:**
 
-1. `me lek adasi nkwukwuuk cha isa ikije mè isa me ikeya lesoto ìre ge me lek ijọn̄`
-2. `me agan̄ inyọn̄ mbum ura mè emen awaji atik me agan̄ inyọn̄ ichep ura mè agan̄ ichep`
-3. `me emen wire môkọtbe irọ inu due process odobe`
+1. `me lek èwê sayara me emen ido yi usem komoros furenchi mè usem afarì mè igwen okwaan̄`
+2. `me agan̄ inyọn̄ sabum mgbọ keyi ebi ene ewa ichit ebi un enyibe me acha ifofo belin`
+3. `me emen kan̄ mgbọ îkanabe ogwu biriten îsan̄a nchọi iba me emen utikpa ya okisibi igwook me`
 
 **Context Size 3:**
 
-1. `agan̄ ichep ura oniin̄ mè ikpọkpọk ikire ibot ikọ me ukpatu ebi uga ifuk ibot chereyi kperiọọn̄ owuw...`
-2. `me ido ya ìnire usem furenchi mèlek usem wolof sa me ebi otoko wolof erebe ebi ìwawa ichit`
-3. `me agan̄ osiki sà ruwanda burundi mè kongo kinshasa ekup me agan̄ inyọn̄ abia mè akwa ibom me`
+1. `agan̄ ichep ura otutuuk ekup me lek ogbọn̄ ikput lek ema ekitim akọn̄ me lek ido india me`
+2. `me ido ya efele oka agan̄ mkpulu gongola isa ichili taraba me 27 ọgọs me ukot mkpulu kè`
+3. `me agan̄ osiki ichep ura egwen agan̄ mkpulu rivas mè delita ijọ ijaw ore usem eikimalek itumu me`
 
 **Context Size 4:**
 
-1. `me agan̄ mbum ura kan̄ sà emen awaji atilantik otap ikana ọmọ me agan̄ inyọn̄ afirika agan̄ inyọn̄ ò...`
-2. `me agan̄ ichep ura isi ire iteke indus me agan̄ mbum ura me afirika mè ire si òso 20`
-3. `me ido ya ìre furench ire îre akọp irek me efit si re akọp mè irek go me efit`
+1. `me agan̄ mbum ura me agan̄ ichep ura îre ido ini ingilan skọtilan weelis mè ailan agan̄ inyọn̄ me`
+2. `me agan̄ ichep ura isi ire lek emen awaji pàsifik me agan̄ mbum ura sudan me agan̄ osiki mbum`
+3. `me ido ya ìre mamoudizou me grande terree acho eyi ilile usem mkpulu ìre furenchi eyi owuwa ene ekit...`
 
 
 ### Generated Text Samples (Subword-based)
@@ -295,27 +295,27 @@ Below are text samples generated from each subword-based Markov chain model:
 
 **Context Size 1:**
 
-1. `_idem_ijan̄_masee`
-2. `e_erelukp_mi_lup`
-3. `irit_enyikp_n_si`
+1. `_me_e_mọọn̄),_eki`
+2. `eranlatenctiko._`
+3. `ik_e_e_gbe_okire`
 
 **Context Size 2:**
 
-1. `e_obageeleki_[cor`
-2. `_ike_ubọ_erere_ik`
-3. `_me_ebi_mè_mem_ya`
+1. `e_nathe_in_iritim`
+2. `_igọọn̄_mîturusiny`
+3. `_me_ban̄_ge_ema_mf`
 
 **Context Size 3:**
 
-1. `_me_emire_ge,_mè_d`
-2. `me_jodan_ichechich`
-3. `re_ge,_ìkigwook,_ò`
+1. `_me_òbeluk_sọn_bro`
+2. `me_ere_<gdp>_òkike`
+3. `re_ere_òta_irọ_yi_`
 
 **Context Size 4:**
 
-1. `_me_si_inwàn_ikwaan̄`
-2. `_mè_ikikaan̄ge;_me_<`
-3. `lek_ebi_ìkike_eriọọ`
+1. `_me_emen_oka_akat._`
+2. `_mè_onan̄a_me_efit_e`
+3. `lek_ichit_me_agan̄_i`
 
 
 ### Key Findings
@@ -420,18 +420,18 @@ Below are text samples generated from each subword-based Markov chain model:
 
 | Model | Dimension | Isotropy | Semantic Density | Alignment R@1 | Alignment R@10 |
 |-------|-----------|----------|------------------|---------------|----------------|
-| **mono_32d** | 32 | 0.1710 🏆 | 0.5365 | N/A | N/A |
-| **mono_64d** | 64 | 0.0323 | 0.5579 | N/A | N/A |
-| **mono_128d** | 128 | 0.0059 | 0.5505 | N/A | N/A |
-| **aligned_32d** | 32 | 0.1710 | 0.5499 | 0.0111 | 0.1274 |
-| **aligned_64d** | 64 | 0.0323 | 0.5740 | 0.0222 | 0.1717 |
-| **aligned_128d** | 128 | 0.0059 | 0.5600 | 0.0139 | 0.1634 |
+| **mono_32d** | 32 | 0.1716 🏆 | 0.5548 | N/A | N/A |
+| **mono_64d** | 64 | 0.0315 | 0.5662 | N/A | N/A |
+| **mono_128d** | 128 | 0.0057 | 0.5736 | N/A | N/A |
+| **aligned_32d** | 32 | 0.1716 | 0.5361 | 0.0083 | 0.1330 |
+| **aligned_64d** | 64 | 0.0315 | 0.5580 | 0.0166 | 0.1745 |
+| **aligned_128d** | 128 | 0.0057 | 0.5602 | 0.0139 | 0.1717 |
 
 ### Key Findings
 
-- **Best Isotropy:** mono_32d with 0.1710 (more uniform distribution)
-- **Semantic Density:** Average pairwise similarity of 0.5548. Lower values indicate better semantic separation.
-- **Alignment Quality:** Aligned models achieve up to 2.2% R@1 in cross-lingual retrieval.
+- **Best Isotropy:** mono_32d with 0.1716 (more uniform distribution)
+- **Semantic Density:** Average pairwise similarity of 0.5582. Lower values indicate better semantic separation.
+- **Alignment Quality:** Aligned models achieve up to 1.7% R@1 in cross-lingual retrieval.
 - **Recommendation:** 128d aligned for best cross-lingual performance
 
 ---
@@ -453,14 +453,14 @@ These are the most productive prefixes and suffixes identified by sampling the v
 #### Productive Prefixes
 | Prefix | Examples |
 |--------|----------|
-| `-ek` | eket, ekpabe, ekibene |
-| `-ik` | ikpele, ikinen̄e, ikinyam |
+| `-ek` | ekijeje, ekpukpo, ekwukwu |
+| `-ik` | ikira, ikikween̄, ikpọkpọ |
 
 #### Productive Suffixes
 | Suffix | Examples |
 |--------|----------|
-| `-n̄` | ugwun̄, akwaan̄, esun̄ |
-| `-be` | ekpabe, îkwube, ejibibe |
+| `-n̄` | kpekaan̄, utọn̄, ikikween̄ |
+| `-be` | ojotbe, erọkọbe, egobobe |
 
 ### 6.3 Bound Stems (Lexical Roots)
 
@@ -468,18 +468,18 @@ Bound stems are high-frequency subword units that are semantically cohesive but
 
 | Stem | Cohesion | Substitutability | Examples |
 |------|----------|------------------|----------|
-| `gọọk` | 1.51x | 19 contexts | igọọk, agọọk, ogọọk |
-| `tumu` | 1.46x | 21 contexts | ntumu, etumu, itumu |
-| `kpul` | 1.58x | 16 contexts | ikpulu, îkpulu, òkpulu |
-| `sibi` | 1.51x | 18 contexts | nsibi, ìsibi, îsibi |
-| `kikp` | 1.44x | 19 contexts | òkikpa, òkikpọ, ekikpọ |
-| `kana` | 1.41x | 20 contexts | ekana, ìkana, nkana |
-| `chie` | 1.55x | 14 contexts | chief, ichieek, ìchieek |
-| `riọọ` | 1.63x | 12 contexts | nriọọk, riọọn̄, nriọọn̄ |
-| `kisa` | 1.43x | 17 contexts | òkisa, ìkisa, îkisa |
-| `gbaa` | 1.46x | 15 contexts | ògbaan̄, egbaan̄, ìgbaan̄ |
-| `ikaa` | 1.61x | 11 contexts | ikaan̄, enikaan̄, ebikaan̄ |
-| `kpọk` | 1.39x | 16 contexts | ukpọk, okpọk, ọkpọk |
+| `tumu` | 1.48x | 21 contexts | etumu, îtumu, itumu |
+| `gọọk` | 1.52x | 19 contexts | agọọk, egọọk, îgọọk |
+| `kpul` | 1.59x | 16 contexts | ìkpulu, îkpulu, òkpulu |
+| `sibi` | 1.51x | 18 contexts | ìsibi, îsibi, osibi |
+| `kikp` | 1.46x | 19 contexts | ìkikpa, òkikpọ, òkikpa |
+| `kana` | 1.42x | 20 contexts | nkana, îkana, ekana |
+| `kisa` | 1.45x | 17 contexts | îkisa, ikisa, ìkisa |
+| `chie` | 1.54x | 14 contexts | chief, nchieek, chieen̄ |
+| `riọọ` | 1.62x | 12 contexts | riọọn̄, nriọọk, iriọọn̄ |
+| `gbaa` | 1.47x | 15 contexts | igbaan̄, egbaan̄, ogbaan̄ |
+| `ikaa` | 1.61x | 11 contexts | ikaan̄, ìkikaan̄, ebikaan̄ |
+| `kpọk` | 1.39x | 16 contexts | okpọk, ukpọk, ikpọk |
 
 ### 6.4 Affix Compatibility (Co-occurrence)
 
@@ -487,9 +487,9 @@ This table shows which prefixes and suffixes most frequently co-occur on the sam
 
 | Prefix | Suffix | Frequency | Examples |
 |--------|--------|-----------|----------|
-| `-ik` | `-n̄` | 15 words | ikikpan̄, ikwaan̄ |
-| `-ek` | `-be` | 13 words | ekpabe, ekaan̄be |
-| `-ek` | `-n̄` | 10 words | ekijeen̄, ekitoon̄ |
+| `-ik` | `-n̄` | 15 words | ikikween̄, ikpan̄ |
+| `-ek` | `-be` | 13 words | ekpan̄be, ekpukbe |
+| `-ek` | `-n̄` | 10 words | ekigbaan̄, ekimun̄ |
 
 ### 6.5 Recursive Morpheme Segmentation
 
@@ -497,19 +497,19 @@ Using **Recursive Hierarchical Substitutability**, we decompose complex words in
 
 | Word | Suggested Split | Confidence | Stem |
 |------|-----------------|------------|------|
-| ekinyambe | **`ek-inyam-be`** | 6.0 | `inyam` |
 | ekitumube | **`ek-itumu-be`** | 6.0 | `itumu` |
+| ekinyambe | **`ek-inyam-be`** | 6.0 | `inyam` |
 | ekigwenbe | **`ek-igwen-be`** | 6.0 | `igwen` |
 | ikichieek | **`ik-ichieek`** | 4.5 | `ichieek` |
-| ekichichini | **`ek-ichichini`** | 4.5 | `ichichini` |
 | echichinibe | **`echichini-be`** | 4.5 | `echichini` |
-| ekiweweek | **`ek-iweweek`** | 4.5 | `iweweek` |
-| ekikpulube | **`ek-ik-pulu-be`** | 4.5 | `pulu` |
 | ekekikpulu | **`ek-ek-ik-pulu`** | 4.5 | `pulu` |
+| ekiweweek | **`ek-iweweek`** | 4.5 | `iweweek` |
 | echieekbe | **`echieek-be`** | 4.5 | `echieek` |
-| ekikpukpo | **`ek-ik-pukpo`** | 3.0 | `pukpo` |
-| ikpọchieen̄ | **`ik-pọchiee-n̄`** | 3.0 | `pọchiee` |
-| ikibieen̄ | **`ik-ibiee-n̄`** | 3.0 | `ibiee` |
+| ekikpulube | **`ek-ik-pulu-be`** | 4.5 | `pulu` |
+| ekichichini | **`ek-ichichini`** | 4.5 | `ichichini` |
+| ikikween̄ | **`ik-ik-ween̄`** | 3.0 | `ween̄` |
+| ekigbaan̄ | **`ek-igbaa-n̄`** | 3.0 | `igbaa` |
+| îriọọn̄be | **`îriọọ-n̄-be`** | 3.0 | `îriọọ` |
 | eriọọn̄be | **`eriọọ-n̄-be`** | 3.0 | `eriọọ` |
 | egbaan̄be | **`egbaa-n̄-be`** | 3.0 | `egbaa` |
 
@@ -745,4 +745,4 @@ MIT License - Free for academic and commercial use.
 ---
 *Generated by Wikilangs Models Pipeline*
 
-*Report Date: 2026-01-03 14:12:13*
+*Report Date: 2026-01-03 16:25:38*

models/embeddings/aligned/ann_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dc209d93d13b120c8c95cba7ed8ea3511e7f3a3be58c71d6b52f823aba7e3a58
+oid sha256:4ec3b02a29131159583e1802e66a0d3a117ff648330d804416e1080ffea1596e
 size 1025972707

models/embeddings/aligned/ann_128d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6ecbf9194a3110e9db3f94be28e4d970c89d4ae602cd76c5e4114d930138c369
+oid sha256:3873a81cc2944d94ca6990cc7557d6d55aea718409cedcb7289f53215b936c57
 size 65664

models/embeddings/aligned/ann_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:046109a4422a9a603a48c2abd0d192b758882631e72ac429557dc6f857e7b375
+oid sha256:6ef8cdd15f948d80be885263043c7688460fd182172f2154c015b80d3a886320
 size 256516579

models/embeddings/aligned/ann_32d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9ac69b84943611045a295f4bb05aa98d313c747e3e3ff20df0a86d90596c100c
+oid sha256:91d45dc35817a7af0f3e2baf4943e62863a7f004ecdf51c6c171ce0499b800eb
 size 4224

models/embeddings/aligned/ann_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b00d28e874e41679c615dce915575ad329279f02507e33677323ebb64d12a95e
+oid sha256:84916d177660159df0e2cd46c3284a2381f09279130834a1add5cb653d1676b5
 size 513001955

models/embeddings/aligned/ann_64d.projection.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8417b28498d4ded38028edb9ebd5965b99df3b52961d1191e017b63208828a47
+oid sha256:572e129219bb969b2237639b710483579e3d554be99bbf1f655d2d070f2f16a5
 size 16512

models/embeddings/monolingual/ann_128d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:dc209d93d13b120c8c95cba7ed8ea3511e7f3a3be58c71d6b52f823aba7e3a58
+oid sha256:4ec3b02a29131159583e1802e66a0d3a117ff648330d804416e1080ffea1596e
 size 1025972707

models/embeddings/monolingual/ann_32d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:046109a4422a9a603a48c2abd0d192b758882631e72ac429557dc6f857e7b375
+oid sha256:6ef8cdd15f948d80be885263043c7688460fd182172f2154c015b80d3a886320
 size 256516579

models/embeddings/monolingual/ann_64d.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b00d28e874e41679c615dce915575ad329279f02507e33677323ebb64d12a95e
+oid sha256:84916d177660159df0e2cd46c3284a2381f09279130834a1add5cb653d1676b5
 size 513001955

models/tokenizer/ann_tokenizer_16k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b74f59549b94c9bbb91cffef6bc44cacadb4f440a30004c5287afd482a9adcf2
+oid sha256:e4bb1ade5597ab4a46c633e58eb5a704bbb6e44b9af4b87b84cd7063c2be7a82
 size 510998

models/tokenizer/ann_tokenizer_8k.model

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:89d553eae70a0c59b46d72bb3b7a2e84791949a1b7ea431452fca72f093214e2
+oid sha256:677dae64939a27b2e6d16b59a170e54e602e80ae6e57143d5f85c4f79bfd5219
 size 374759