gector-bert-it / language-rules.json
Imperatore's picture
gector-bert-it: first public release (Apache-2.0)
a26008b verified
Raw History Blame Contribute Delete
829 Bytes
{
"version": 1,
"language": "it",
"_comment": "Per-bundle language facts (data-driven; see C# LanguageRules / ADR-004). stripTrailingTerminator: strip a single trailing . ! ? before the WordPiece model and re-attach it after (the Italian bundle is brittle to a trailing period). elision.nonElidingArticles: full (un-elided) articles the tokenizer treats as glued-elision prefixes (gli'avvocato -> gli avvocato); overrides ItalianElision.DefaultNonElidingArticles. Absent file => None (stripTerminator false, default article list).",
"stripTrailingTerminator": true,
"elision": {
"nonElidingArticles": ["gli", "le", "i", "il", "la", "lo", "una", "uno"]
},
"clauseConjunctions": ["ma", "e", "ed", "o", "oppure", "però", "mentre", "quindi", "poi", "perché", "anzi", "dunque", "infatti", "tuttavia", "perciò"]
}