---
language:
  - he
  - en
license: apache-2.0
tags:
  - hebrew
  - dint-transformer
  - differential-attention
  - pretrained
  - text-generation
  - from-scratch
---

# Hebrew DINT Transformer

The first Hebrew LLM built from scratch using DINT (Differential + Integral) Transformer architecture.

Based on arxiv:2501.17486 (DINT) + arxiv:2410.05258 (DIFF Transformer).

Trained on HeDC4, Hebrew Wikipedia, Project Ben-Yehuda, and more Hebrew corpora.