metadata
language:
- he
- en
license: apache-2.0
tags:
- hebrew
- dint-transformer
- differential-attention
- pretrained
- text-generation
- from-scratch
Hebrew DINT Transformer
The first Hebrew LLM built from scratch using DINT (Differential + Integral) Transformer architecture.
Based on arxiv:2501.17486 (DINT) + arxiv:2410.05258 (DIFF Transformer).
Trained on HeDC4, Hebrew Wikipedia, Project Ben-Yehuda, and more Hebrew corpora.