---
license: cc-by-4.0
language:
- he
inference: false
---
# Pre-BEREL: tbd

State-of-the-art language model for Rabbinic Hebrew, released [here] - add link. 

This model is the first ever Hebrew model fully pretrained on pre-segmented Hebrew texts. When inputting text to the model, the text is expected to be pre-segmented using [pre-BEREL](https://huggingface.co/dicta-il/pre-BEREL). Segmenting the text prior to training is the first step towards integrating morphological-aware-tokenization into language models. 

Sample usage:

```python
from transformers import AutoModel, AutoTokenizer, AutoModelForMaskedLM

sentence = 'וזה לשון הרמב״ן בפירושו על התורה, שהדבר ידוע ומפורסם לכל בעלי העיון שאין המקרא יוצא מידי פשוטו אף על פי שהדרש אמת.'

# First, load in the segmentation model, to preprocess the text
seg_tokenizer = AutoTokenizer.from_pretrained('dicta-il/BEREL-seg')
seg_model = AutoModel.from_pretrained('dicta-il/BEREL-seg', trust_remote_code=True).eval()

segmented_output = seg_model.predict([sentence], seg_tokenizer)[0] # sentence sent as a batch, pick the first one

# we mark the segmented tokens with a special separator, to distinguish them from regular work tokens. 
segmented_sentence = ' '.join('ףףף '.join(segmented_word) for segmented_word in segmented_output[1:-1]) # ignore cls/sep
print(segmented_sentence.replace('ףףף', '___'))
# ו___ זה לשון ה___ רמב ״ ן ב___ פירושו על ה___ תורה , שהד___ בר ידוע ו___ מפורסם ל___ כל בעלי ה___ עיון ש___ אין ה___ מקר
א יוצא מידי פשוטו אף על פי שהד___ רש אמת .

# we can mask out any word we want - in this case, the easiest is to just do a string replace. We could've masked in the original sentence, or anywhere in the pipeline.
segmented_sentence = segmented_sentence.replace("עיון", "[MASK]")

# Load in the new model
tokenizer = AutoTokenizer.from_pretrained('dicta-il/pre-BEREL')
model = AutoModelForMaskedLM.from_pretrained('dicta-il/pre-BEREL').eval()

output = model(tokenizer.encode(segmented_sentence, return_tensors='pt'))
# the [MASK] is the 24th token (including [CLS])
import torch
top_5 = torch.topk(output.logits[0, 23, :], 5)[1]
print('\n'.join(tokenizer.convert_ids_to_tokens(top_5))) # should print קבלה / פשט / דת / חכמה / גמרא
```

## Citation

If you use pre-BEREL in your research, please cite ```tbd```

**BibTeX:**

```bibtex
tbd
```

## License

Shield: [![CC BY 4.0][cc-by-shield]][cc-by]

This work is licensed under a
[Creative Commons Attribution 4.0 International License][cc-by].

[![CC BY 4.0][cc-by-image]][cc-by]

[cc-by]: http://creativecommons.org/licenses/by/4.0/
[cc-by-image]: https://i.creativecommons.org/l/by/4.0/88x31.png
[cc-by-shield]: https://img.shields.io/badge/License-CC%20BY%204.0-lightgrey.svg