File size: 5,029 Bytes
54514e8 88479ae 54514e8 88479ae 54514e8 be9021c 54514e8 be9021c 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 88479ae 54514e8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 | ---
pipeline_tag: fill-mask
license: mit
base_model: microsoft/deberta-v2-xxlarge
library_name: zeromodels
tags:
- keras
- zeromodels
- deberta
- deberta-v2
- fill-mask
- text-encoder
- arxiv:2006.03654
- arxiv:2111.09543
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/deberta-v1-v2-v3-6a8eae49464403784b9d6cd0) for all versions of DeBERTa (v1 / v2 / v3).***
# Run DeBERTa with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/deberta/) [](https://huggingface.co/collections/zeromodels/deberta-v1-v2-v3-6a8eae49464403784b9d6cd0)
# zeromodels/deberta_v2_xxlarge
Papers: [DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654)](https://arxiv.org/abs/2006.03654) · [DeBERTaV3 (arXiv:2111.09543)](https://arxiv.org/abs/2111.09543) · [HF Papers](https://huggingface.co/papers/2006.03654)
DeBERTa is Microsoft's disentangled-attention text encoder (content + relative position). v1 uses byte-level BPE; v2/v3 use SentencePiece. v3 adds ELECTRA-style pretraining with gradient-disentangled embedding sharing. Import from `deberta` / `deberta_v2` / `deberta_v3` to match the generation.
For more details on the model, please go to the upstream [model card](https://huggingface.co/microsoft/deberta-v2-xxlarge).
Pure-**Keras 3** conversion of [`microsoft/deberta-v2-xxlarge`](https://huggingface.co/microsoft/deberta-v2-xxlarge) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is a **fill-mask / encoder** checkpoint (`DebertaV2MaskedLM`, v2 xxlarge). Task heads (sequence/token classify, QA, …) load via `hf:` fine-tunes.
## ✨ Quick start (fill-mask)
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.deberta_v2 import (
DebertaV2MaskedLM,
DebertaV2Tokenizer,
)
mlm = DebertaV2MaskedLM.from_weights("zeromodels/deberta_v2_xxlarge")
tokenizer = DebertaV2Tokenizer.from_weights("zeromodels/deberta_v2_xxlarge")
inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs) # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))
```
Load any DeBERTa variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Generation |
|---|---|---|
| `deberta_base` | [`zeromodels/deberta_base`](https://huggingface.co/zeromodels/deberta_base) | v1 |
| `deberta_large` | [`zeromodels/deberta_large`](https://huggingface.co/zeromodels/deberta_large) | v1 |
| `deberta_v2_xlarge` | [`zeromodels/deberta_v2_xlarge`](https://huggingface.co/zeromodels/deberta_v2_xlarge) | v2 |
| `deberta_v2_xxlarge` | [`zeromodels/deberta_v2_xxlarge`](https://huggingface.co/zeromodels/deberta_v2_xxlarge) | v2 |
| `deberta_v3_xsmall` | [`zeromodels/deberta_v3_xsmall`](https://huggingface.co/zeromodels/deberta_v3_xsmall) | v3 |
| `deberta_v3_small` | [`zeromodels/deberta_v3_small`](https://huggingface.co/zeromodels/deberta_v3_small) | v3 |
| `deberta_v3_base` | [`zeromodels/deberta_v3_base`](https://huggingface.co/zeromodels/deberta_v3_base) | v3 |
| `deberta_v3_large` | [`zeromodels/deberta_v3_large`](https://huggingface.co/zeromodels/deberta_v3_large) | v3 |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/deberta_v2_xxlarge")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `DebertaV2Model` | Encoder backbone |
| `DebertaV2MaskedLM` | Masked language modeling (fill-mask) |
| `DebertaV2SequenceClassify` | Sequence classification |
| `DebertaV2TokenClassify` | Token classification (NER / POS) |
| `DebertaV2QnA` | Extractive question answering |
| `DebertaV2MultipleChoice` | Multiple choice |
```python
from zeromodels.models.deberta_v2 import DebertaV2SequenceClassify
model = DebertaV2SequenceClassify.from_weights("zeromodels/deberta_v2_xxlarge")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `Tokenizer.from_weights(...)` so vocab and mask token match.
- Do not mix packages across generations (v1 ≠ v2 ≠ v3).
- See [DeBERTa docs](https://imvision12.github.io/ZeroModels/deberta/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `DebertaV2MaskedLM.from_weights("hf:microsoft/deberta-v2-xxlarge")`.
## Special Thanks
A huge thank you to the Microsoft DeBERTa authors for creating and releasing these models.
License: MIT.
|