README.md

---
language:
- yi
license: cc-by-nc-4.0
license_link: https://creativecommons.org/licenses/by-nc/4.0/
base_model: meta-llama/Llama-3.1-8B
pipeline_tag: text-generation
tags:
- yiddish
- continued-pretraining
library_name: transformers
---

# MameLoshnLM

MameLoshnLM is an 8B-parameter language model for Yiddish, built by continued pretraining of [meta-llama/Llama-3.1-8B](https://huggingface.co/meta-llama/Llama-3.1-8B) on a curated native-Yiddish corpus combining contemporary web-native sources with an extensive collection of literary materials. It is, to our knowledge, the first large generative language model adapted specifically to Yiddish, and it accompanies our paper (reference below).

## Model details

- **Architecture** LlamaForCausalLM (Llama 3.1 8B), unchanged tokenizer and vocabulary
- **Training** one epoch of continued pretraining on a large Yiddish corpus, with a small proportion of English data interleaved to mitigate catastrophic forgetting
- **Precision** bfloat16
- **Context length during training** 1,024 tokens (the architecture supports up to 128k)
- **Hyperparameters** learning rate 2e-5 with cosine schedule, warmup ratio 0.02, weight decay 0.01, 8-bit AdamW

## Evaluation

We evaluated MameLoshnLM on a broad Yiddish benchmark suite spanning linguistic tasks (POS tagging, dependency parsing, lemmatization, transliteration on the UD Yiddish-YiTB treebank), understanding tasks (paraphrase detection, physical commonsense, question answering, all in Yiddish translation), named entity recognition (EHRI, WikiANN, newNLP), and machine translation (FLORES+ and Kashes-mt, a new translation task we created, in both directions). All results below are 5-shot. Bold marks the best model per row.

| Task | Metric | MameLoshnLM | Llama 3.1 8B | Qwen3 8B |
|---|---|---|---|---|
| POS Tagging | Accuracy | **88.6** | 86.9 | 85.9 |
| Dependency Parsing | LAS | **40.6** | 39.7 | 40.3 |
| Lemmatization | Change Acc. | **31.9** | 19.7 | 21.6 |
| Transliteration | 1 - CER | **92.3** | 92.1 | 88.5 |
| PAWS-Wiki | Accuracy | 62.9 | 55.8 | **75.2** |
| PIQA | Accuracy | 47.1 | 45.0 | **50.3** |
| Wiki QA | ROUGE-L | 34.4 | 32.3 | **34.8** |
| NER (EHRI) | Micro F1 | **41.3** | 34.2 | 20.8 |
| NER (WikiANN) | Micro F1 | **59.7** | 58.1 | 54.6 |
| NER (newNLP) | Micro F1 | **57.6** | 51.9 | 50.5 |
| MT Eng-Yid (FLORES+) | COMET | **78.5** | 64.8 | 46.5 |
| MT Eng-Yid (Kashes-mt) | COMET | **75.3** | 59.6 | 45.3 |
| MT Yid-Eng (FLORES+) | COMET | **87.2** | 82.2 | 79.1 |
| MT Yid-Eng (Kashes-mt) | COMET | **79.5** | 72.8 | 72.0 |
| **Average** | | **62.6** | 56.8 | 54.7 |

The benchmark datasets will be released in this organization as well.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Yiddish-NLP/MameLoshnLM", torch_dtype="bfloat16", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("Yiddish-NLP/MameLoshnLM")

prompt = "די ייִדישע שפּראַך"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

Note that this is a base (non-instruct) model. It is intended for completion-style prompting, few-shot in-context learning, and as a starting point for task-specific fine-tuning.

## Limitations

The model inherits the limitations of its base model and training data. Yiddish is a low-resource language and parts of the training corpus are derived from OCR of historical printed material, so the model may reproduce OCR artifacts, historical orthographic variation, and the topical biases of its sources. It has not been instruction-tuned or safety-tuned.

## License

MameLoshnLM is released for **non-commercial use only**, under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license. It is intended for research and educational purposes. For commercial licensing inquiries, please contact the authors.

Built with Llama. As a derivative of Llama 3.1, the model is additionally subject to the [Llama 3.1 Community License](https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/LICENSE); where the two licenses impose different conditions, the more restrictive applies.

## Citation

The paper introducing MameLoshnLM has been accepted to COLM 2026. The full citation and BibTeX will be added here once the camera-ready version is published.

## Acknowledgments

We are deeply grateful to the [Yiddish Book Center](https://www.yiddishbookcenter.org/), which granted us permission to train on the Yiddish Book Center's Digital Yiddish Library. This model and the accompanying research would not have been possible without their support and their decades of work digitizing and preserving Yiddish literature.