| --- |
| license: other |
| language: [hi, mr, ne, mai, sa, brx, kok, doi, gu, pa, or, ta, te, kn, ml, mni, bn, as, ur, sd, ks, sat] |
| tags: [transliteration, code-mixing, text-to-speech, indic] |
| --- |
| # indictrans: English words in Indian-language text, written in the request's script |
|
|
| Runtime assets of the code-mix transliteration layer of the Finesse TTS (`utils/codemix_xlit`): |
| English words inside Indic text are rewritten in the language's script, spelled the way an |
| Indian-English speaker says them, so an Indic voice reads them correctly; text typed entirely in |
| Latin is read word by word, the language's own words back into its script. 22 languages (Dogri and |
| Santali through the Hindi model). |
|
|
| Version 1.4.9+3df482785219. Built by `tools/codemix_xlit/build_assets.py`; `manifest.json` lists every file |
| with its size and SHA-256. |
|
|
| ## Contents |
| - `LICENSE`, `NOTICE`: the licences and attributions below, inside the bundle, so a copy of the folder |
| keeps them |
| - `model/`: the IndicXlit en->indic v1.0 transformer converted to CTranslate2 |
| - `lexicon/<code>.reviewed.json`, `lexicon/<code>.generated.json`: reviewed and generated spellings |
| - `rescore/`: pruned IndicXlit word-frequency dictionaries |
| - `cmudict.json`, `english_*`: English pronunciations, vocabulary and reviewed corrections |
| - `indic_words.json`: Indian words read natively; `ranker.json`: candidate ranker |
| - `roman/<code>.json`: how people type each language's words in Latin, how often, and the other words |
| a typing can be (`alts`) |
| - `native_freq/<token>.json`: word frequencies of the eleven languages without a `rescore/` list, for |
| the language's own words typed in Latin |
| - `roman_ctx/<code>.json`: for typings that are two different words, how often each is seen next to |
| the words that tell them apart (sentence context) |
| - `names/<code>.json`: Indian given names, surnames and place names typed in Latin and how the language |
| writes them (`tools/codemix_xlit/build_names.py`), read for a capitalised word (Sneha स्नेहा, not स्नेह) |
|
|
| ## Sources and licences |
| `license: other` because the files carry different licences; each is listed here with its notice. |
|
|
| - `model/`, `rescore/`, `native_freq/`: the IndicXlit model and word-frequency dictionaries, AI4Bharat, |
| MIT licence (https://github.com/AI4Bharat/IndicXlit). Notice: |
|
|
| > MIT License |
| > |
| > Copyright (c) 2022 AI4Bhārat |
| > |
| > Permission is hereby granted, free of charge, to any person obtaining a copy of this software and |
| > associated documentation files (the "Software"), to deal in the Software without restriction, |
| > including without limitation the rights to use, copy, modify, merge, publish, distribute, |
| > sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is |
| > furnished to do so, subject to the following conditions: |
| > |
| > The above copyright notice and this permission notice shall be included in all copies or |
| > substantial portions of the Software. |
| > |
| > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT |
| > NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND |
| > NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, |
| > DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT |
| > OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
|
|
| - `cmudict.json`: the CMU Pronouncing Dictionary (https://github.com/cmusphinx/cmudict), BSD-style |
| licence. Notice: |
|
|
| > Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved. |
| > |
| > Redistribution and use in source and binary forms, with or without modification, are permitted |
| > provided that the following conditions are met: |
| > |
| > 1. Redistributions of source code must retain the above copyright notice, this list of conditions |
| > and the following disclaimer. The contents of this file are deemed to be source code. |
| > 2. Redistributions in binary form must reproduce the above copyright notice, this list of |
| > conditions and the following disclaimer in the documentation and/or other materials provided |
| > with the distribution. |
| > |
| > This work was supported in part by funding from the Defense Advanced Research Projects Agency, the |
| > Office of Naval Research and the National Science Foundation of the United States of America, and |
| > by member companies of the Carnegie Mellon Sphinx Speech Consortium. We acknowledge the |
| > contributions of many volunteers to the expansion and improvement of this dictionary. |
| > |
| > THIS SOFTWARE IS PROVIDED BY CARNEGIE MELLON UNIVERSITY ``AS IS'' AND ANY EXPRESSED OR IMPLIED |
| > WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS |
| > FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY NOR ITS |
| > EMPLOYEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL |
| > DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
| > DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER |
| > IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT |
| > OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
|
|
| - `english_vocab.txt`, `english_pron_vocab.txt`: English word lists derived from wordfreq |
| (https://github.com/rspeer/wordfreq), whose data is CC BY-SA 4.0 |
| (https://creativecommons.org/licenses/by-sa/4.0/); these lists are shared under the same licence. |
| - `roman/` of bn, gu, hi, kn, ml, mr, pa, sd, ta, te, ur: counts derived from the romanized sentences |
| of the Dakshina dataset (Roark et al. 2020, https://github.com/google-research-datasets/dakshina), |
| CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these tables are shared under the |
| same licence. |
| - `roman_ctx/`: neighbour counts from AI4Bharat Sangraha, verified subset (Khan et al. 2024, |
| https://huggingface.co/datasets/ai4bharat/sangraha), CC BY 4.0 |
| (https://creativecommons.org/licenses/by/4.0/). |
| - `names/`: the English and native-script labels of people with Indian citizenship, of the name items |
| they carry and of India's states, union territories and places of 100,000 people, from Wikidata (https://www.wikidata.org), CC0 1.0 |
| (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above. |
| - Lexicons, corrections, ranker and the other romanized tables: derived from the above as described |
| below. |
|
|
| ## Who wrote the generated parts |
| - `lexicon/<code>.generated.json`: written offline by the layer's own pipeline (`build_assets.py |
| lexicon`: IndicXlit's candidates, spellings rendered from the pronunciation, the ranker). Its |
| entries that did not read back as the English word, among each language's 5,000 most frequent, |
| were reviewed by the LLM graders below and the errors corrected in the reviewed lexicon. |
| - `english_extra_prons.json` (Indian-English readings of words the CMU dictionary lacks: names, |
| brands, foreign words), `english_pron_overrides.json`, `indic_words.json`, the gold spellings in |
| `lexicon/<code>.reviewed.json`, and the everyday words in `roman/`: written by a large language |
| model, run as per-language writer and grader agents under fixed spelling conventions. They were |
| reviewed by further agents of the same model (a reading is read |
| back against the English pronunciation; a spelling is graded C/A/W blind against plain |
| IndicXlit's). Two independent graders agreed on 96.2-97.7% of labels. |
| - `ranker.json`: fitted on those graders' labels. |
| - The accuracy figures published for this bundle come from the same model graders. **No native |
| speaker has reviewed any of these files yet**; treat them as unverified for external claims. |
|
|
