README.md

metadata

pipeline_tag: fill-mask
license: mit
base_model: microsoft/deberta-v2-xxlarge
library_name: kerasformers
tags:
  - keras
  - kerasformers
  - deberta
  - deberta-v2
  - fill-mask
  - text-encoder
  - arxiv:2006.03654
  - arxiv:2111.09543
  - pytorch
  - jax
  - tf

See our collection for all versions of DeBERTa (v1 / v2 / v3).

Run DeBERTa with Keras 3: JAX, PyTorch, or TensorFlow

kerasformers/deberta_v2_xxlarge

Papers: DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654) · DeBERTaV3 (arXiv:2111.09543) · HF Papers

DeBERTa is Microsoft's disentangled-attention text encoder (content + relative position). v1 uses byte-level BPE; v2/v3 use SentencePiece. v3 adds ELECTRA-style pretraining with gradient-disentangled embedding sharing. Import from deberta / deberta_v2 / deberta_v3 to match the generation.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of microsoft/deberta-v2-xxlarge for kerasformers. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a fill-mask / encoder checkpoint (DebertaV2MaskedLM, v2 xxlarge). Task heads (sequence/token classify, QA, …) load via hf: fine-tunes.

✨ Quick start (fill-mask)

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from kerasformers.models.deberta_v2 import (
    DebertaV2MaskedLM,
    DebertaV2Tokenizer,
)

mlm = DebertaV2MaskedLM.from_weights("kerasformers/deberta_v2_xxlarge")
tokenizer = DebertaV2Tokenizer.from_weights("kerasformers/deberta_v2_xxlarge")

inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs)  # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))

Load any DeBERTa variant the same way with from_weights("kerasformers/<variant>"):

Variant	Hub	Generation
deberta_base	kerasformers/deberta_base	v1
deberta_large	kerasformers/deberta_large	v1
deberta_v2_xlarge	kerasformers/deberta_v2_xlarge	v2
deberta_v2_xxlarge	kerasformers/deberta_v2_xxlarge	v2
deberta_v3_xsmall	kerasformers/deberta_v3_xsmall	v3
deberta_v3_small	kerasformers/deberta_v3_small	v3
deberta_v3_base	kerasformers/deberta_v3_base	v3
deberta_v3_large	kerasformers/deberta_v3_large	v3

Available classes

Load any of these from this repo with from_weights("kerasformers/deberta_v2_xxlarge") (or on the fly via the hf: prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a hf: fine-tune).

Class	Task
DebertaV2Model	Encoder backbone
DebertaV2MaskedLM	Masked language modeling (fill-mask)
DebertaV2SequenceClassify	Sequence classification
DebertaV2TokenClassify	Token classification (NER / POS)
DebertaV2QnA	Extractive question answering
DebertaV2MultipleChoice	Multiple choice

from kerasformers.models.deberta_v2 import DebertaV2SequenceClassify
model = DebertaV2SequenceClassify.from_weights("kerasformers/deberta_v2_xxlarge")

Tips

• Set KERAS_BACKEND before importing Keras / kerasformers.
• Prefer Tokenizer.from_weights(...) so vocab and mask token match.
• Do not mix packages across generations (v1 ≠ v2 ≠ v3).
• See DeBERTa docs and Loading Weights.
• Community / upstream safetensors still work via the hf: prefix, e.g. DebertaV2MaskedLM.from_weights("hf:microsoft/deberta-v2-xxlarge").

Special Thanks

A huge thank you to the Microsoft DeBERTa authors for creating and releasing these models.

License: MIT.