model card 1.4.9+3df482785219

README.md

@@ -0,0 +1,117 @@
+---
+license: other
+language: [hi, mr, ne, mai, sa, brx, kok, doi, gu, pa, or, ta, te, kn, ml, mni, bn, as, ur, sd, ks, sat]
+tags: [transliteration, code-mixing, text-to-speech, indic]
+---
+# indictrans: English words in Indian-language text, written in the request's script
+
+Runtime assets of the code-mix transliteration layer of the Finesse TTS (`utils/codemix_xlit`):
+English words inside Indic text are rewritten in the language's script, spelled the way an
+Indian-English speaker says them, so an Indic voice reads them correctly; text typed entirely in
+Latin is read word by word, the language's own words back into its script. 22 languages (Dogri and
+Santali through the Hindi model).
+
+Version 1.4.9+3df482785219. Built by `tools/codemix_xlit/build_assets.py`; `manifest.json` lists every file
+with its size and SHA-256.
+
+## Contents
+- `LICENSE`, `NOTICE`: the licences and attributions below, inside the bundle, so a copy of the folder
+  keeps them
+- `model/`: the IndicXlit en->indic v1.0 transformer converted to CTranslate2
+- `lexicon/<code>.reviewed.json`, `lexicon/<code>.generated.json`: reviewed and generated spellings
+- `rescore/`: pruned IndicXlit word-frequency dictionaries
+- `cmudict.json`, `english_*`: English pronunciations, vocabulary and reviewed corrections
+- `indic_words.json`: Indian words read natively; `ranker.json`: candidate ranker
+- `roman/<code>.json`: how people type each language's words in Latin, how often, and the other words
+  a typing can be (`alts`)
+- `native_freq/<token>.json`: word frequencies of the eleven languages without a `rescore/` list, for
+  the language's own words typed in Latin
+- `roman_ctx/<code>.json`: for typings that are two different words, how often each is seen next to
+  the words that tell them apart (sentence context)
+- `names/<code>.json`: Indian given names, surnames and place names typed in Latin and how the language
+  writes them (`tools/codemix_xlit/build_names.py`), read for a capitalised word (Sneha स्नेहा, not स्नेह)
+
+## Sources and licences
+`license: other` because the files carry different licences; each is listed here with its notice.
+
+- `model/`, `rescore/`, `native_freq/`: the IndicXlit model and word-frequency dictionaries, AI4Bharat,
+  MIT licence (https://github.com/AI4Bharat/IndicXlit). Notice:
+
+  > MIT License
+  >
+  > Copyright (c) 2022 AI4Bhārat
+  >
+  > Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
+  > associated documentation files (the "Software"), to deal in the Software without restriction,
+  > including without limitation the rights to use, copy, modify, merge, publish, distribute,
+  > sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
+  > furnished to do so, subject to the following conditions:
+  >
+  > The above copyright notice and this permission notice shall be included in all copies or
+  > substantial portions of the Software.
+  >
+  > THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
+  > NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+  > NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+  > DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
+  > OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+
+- `cmudict.json`: the CMU Pronouncing Dictionary (https://github.com/cmusphinx/cmudict), BSD-style
+  licence. Notice:
+
+  > Copyright (C) 1993-2015 Carnegie Mellon University. All rights reserved.
+  >
+  > Redistribution and use in source and binary forms, with or without modification, are permitted
+  > provided that the following conditions are met:
+  >
+  > 1. Redistributions of source code must retain the above copyright notice, this list of conditions
+  >    and the following disclaimer. The contents of this file are deemed to be source code.
+  > 2. Redistributions in binary form must reproduce the above copyright notice, this list of
+  >    conditions and the following disclaimer in the documentation and/or other materials provided
+  >    with the distribution.
+  >
+  > This work was supported in part by funding from the Defense Advanced Research Projects Agency, the
+  > Office of Naval Research and the National Science Foundation of the United States of America, and
+  > by member companies of the Carnegie Mellon Sphinx Speech Consortium. We acknowledge the
+  > contributions of many volunteers to the expansion and improvement of this dictionary.
+  >
+  > THIS SOFTWARE IS PROVIDED BY CARNEGIE MELLON UNIVERSITY ``AS IS'' AND ANY EXPRESSED OR IMPLIED
+  > WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+  > FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY NOR ITS
+  > EMPLOYEES BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+  > DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+  > DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
+  > IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+  > OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+- `english_vocab.txt`, `english_pron_vocab.txt`: English word lists derived from wordfreq
+  (https://github.com/rspeer/wordfreq), whose data is CC BY-SA 4.0
+  (https://creativecommons.org/licenses/by-sa/4.0/); these lists are shared under the same licence.
+- `roman/` of bn, gu, hi, kn, ml, mr, pa, sd, ta, te, ur: counts derived from the romanized sentences
+  of the Dakshina dataset (Roark et al. 2020, https://github.com/google-research-datasets/dakshina),
+  CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/); these tables are shared under the
+  same licence.
+- `roman_ctx/`: neighbour counts from AI4Bharat Sangraha, verified subset (Khan et al. 2024,
+  https://huggingface.co/datasets/ai4bharat/sangraha), CC BY 4.0
+  (https://creativecommons.org/licenses/by/4.0/).
+- `names/`: the English and native-script labels of people with Indian citizenship, of the name items
+  they carry and of India's states, union territories and places of 100,000 people, from Wikidata (https://www.wikidata.org), CC0 1.0
+  (https://creativecommons.org/publicdomain/zero/1.0/), checked against the IndicXlit model above.
+- Lexicons, corrections, ranker and the other romanized tables: derived from the above as described
+  below.
+
+## Who wrote the generated parts
+- `lexicon/<code>.generated.json`: written offline by the layer's own pipeline (`build_assets.py
+  lexicon`: IndicXlit's candidates, spellings rendered from the pronunciation, the ranker). Its
+  entries that did not read back as the English word, among each language's 5,000 most frequent,
+  were reviewed by the LLM graders below and the errors corrected in the reviewed lexicon.
+- `english_extra_prons.json` (Indian-English readings of words the CMU dictionary lacks: names,
+  brands, foreign words), `english_pron_overrides.json`, `indic_words.json`, the gold spellings in
+  `lexicon/<code>.reviewed.json`, and the everyday words in `roman/`: written by a large language
+  model, run as per-language writer and grader agents under fixed spelling conventions. They were
+  reviewed by further agents of the same model (a reading is read
+  back against the English pronunciation; a spelling is graded C/A/W blind against plain
+  IndicXlit's). Two independent graders agreed on 96.2-97.7% of labels.
+- `ranker.json`: fitted on those graders' labels.
+- The accuracy figures published for this bundle come from the same model graders. **No native
+  speaker has reviewed any of these files yet**; treat them as unverified for external claims.