MameLoshnLM

MameLoshnLM is an 8B-parameter language model for Yiddish, built by continued pretraining of meta-llama/Llama-3.1-8B on a curated native-Yiddish corpus combining contemporary web-native sources with an extensive collection of literary materials. It is, to our knowledge, the first large generative language model adapted specifically to Yiddish, and it accompanies our paper (reference below).

Model details

• Architecture LlamaForCausalLM (Llama 3.1 8B), unchanged tokenizer and vocabulary
• Training one epoch of continued pretraining on a large Yiddish corpus, with a small proportion of English data interleaved to mitigate catastrophic forgetting
• Precision bfloat16
• Context length during training 1,024 tokens (the architecture supports up to 128k)
• Hyperparameters learning rate 2e-5 with cosine schedule, warmup ratio 0.02, weight decay 0.01, 8-bit AdamW

Evaluation

We evaluated MameLoshnLM on a broad Yiddish benchmark suite spanning linguistic tasks (POS tagging, dependency parsing, lemmatization, transliteration on the UD Yiddish-YiTB treebank), understanding tasks (paraphrase detection, physical commonsense, question answering, all in Yiddish translation), named entity recognition (EHRI, WikiANN, newNLP), and machine translation (FLORES+ and Kashes-mt, a new translation task we created, in both directions). All results below are 5-shot. Bold marks the best model per row.

Task	Metric	MameLoshnLM	Llama 3.1 8B	Qwen3 8B
POS Tagging	Accuracy	88.6	86.9	85.9
Dependency Parsing	LAS	40.6	39.7	40.3
Lemmatization	Change Acc.	31.9	19.7	21.6
Transliteration	1 - CER	92.3	92.1	88.5
PAWS-Wiki	Accuracy	62.9	55.8	75.2
PIQA	Accuracy	47.1	45.0	50.3
Wiki QA	ROUGE-L	34.4	32.3	34.8
NER (EHRI)	Micro F1	41.3	34.2	20.8
NER (WikiANN)	Micro F1	59.7	58.1	54.6
NER (newNLP)	Micro F1	57.6	51.9	50.5
MT Eng-Yid (FLORES+)	COMET	78.5	64.8	46.5
MT Eng-Yid (Kashes-mt)	COMET	75.3	59.6	45.3
MT Yid-Eng (FLORES+)	COMET	87.2	82.2	79.1
MT Yid-Eng (Kashes-mt)	COMET	79.5	72.8	72.0
Average		62.6	56.8	54.7

The benchmark datasets will be released in this organization as well.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Yiddish-NLP/MameLoshnLM", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Yiddish-NLP/MameLoshnLM")

prompt = "די ייִדישע שפּראַך"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Note that this is a base (non-instruct) model. It is intended for completion-style prompting, few-shot in-context learning, and as a starting point for task-specific fine-tuning.

Limitations

The model inherits the limitations of its base model and training data. Yiddish is a low-resource language and parts of the training corpus are derived from OCR of historical printed material, so the model may reproduce OCR artifacts, historical orthographic variation, and the topical biases of its sources. It has not been instruction-tuned or safety-tuned.

License

MameLoshnLM is released for non-commercial use only, under the CC BY-NC 4.0 license. It is intended for research and educational purposes. For commercial licensing inquiries, please contact the authors.

Built with Llama. As a derivative of Llama 3.1, the model is additionally subject to the Llama 3.1 Community License; where the two licenses impose different conditions, the more restrictive applies.

Citation

The paper introducing MameLoshnLM has been accepted to COLM 2026. The full citation and BibTeX will be added here once the camera-ready version is published.

Acknowledgments

We are deeply grateful to the Yiddish Book Center, which granted us permission to train on the Yiddish Book Center's Digital Yiddish Library. This model and the accompanying research would not have been possible without their support and their decades of work digitizing and preserving Yiddish literature.