File size: 4,709 Bytes
e69d31f 4f23b47 e69d31f 4f23b47 e69d31f c3dd379 e69d31f c3dd379 e69d31f 4f23b47 e69d31f 4f23b47 e69d31f 4f23b47 e69d31f 4f23b47 e69d31f 4f23b47 e69d31f 4f23b47 e69d31f aff6ad2 4f23b47 aff6ad2 4f23b47 aff6ad2 e69d31f 4f23b47 e69d31f 4f23b47 e69d31f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 | ---
pipeline_tag: feature-extraction
license: apache-2.0
base_model: google/electra-small-discriminator
library_name: zeromodels
tags:
- keras
- zeromodels
- electra
- discriminator
- text-encoder
- feature-extraction
- arxiv:2003.10555
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba) for all versions of ELECTRA.***
# Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/electra/) [](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba)
# zeromodels/electra_small_discriminator
Paper: [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555)](https://arxiv.org/abs/2003.10555) · [HF Papers](https://huggingface.co/papers/2003.10555)
ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token **discriminator** (with a smaller **generator** producing the corrupted tokens). This repo is the **encoder / downstream** checkpoint. WordPiece tokenizer; mask token `[MASK]`.
For more details on the model, please go to the upstream [model card](https://huggingface.co/google/electra-small-discriminator).
Pure-**Keras 3** conversion of [`google/electra-small-discriminator`](https://huggingface.co/google/electra-small-discriminator) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
## ✨ Quick start (encoder / downstream)
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.electra import ElectraModel, ElectraTokenizer
model = ElectraModel.from_weights("zeromodels/electra_small_discriminator")
tokenizer = ElectraTokenizer.from_weights("zeromodels/electra_small_discriminator")
out = model(tokenizer("The quick brown fox."))["last_hidden_state"] # (1, L, H)
```
The same repo also serves the task heads, loaded the same way: `ElectraSequenceClassify`, `ElectraTokenClassify`, `ElectraQnA`, `ElectraMultipleChoice` (each takes the pretrained encoder and a randomly-initialized head, ready for fine-tuning).
Load any ELECTRA variant the same way with `from_weights("zeromodels/<variant>")`:
| Size | Discriminator (encoder / downstream) | Generator (masked-LM) |
|---|---|---|
| small | [`zeromodels/electra_small_discriminator`](https://huggingface.co/zeromodels/electra_small_discriminator) | [`zeromodels/electra_small_generator`](https://huggingface.co/zeromodels/electra_small_generator) |
| base | [`zeromodels/electra_base_discriminator`](https://huggingface.co/zeromodels/electra_base_discriminator) | [`zeromodels/electra_base_generator`](https://huggingface.co/zeromodels/electra_base_generator) |
| large | [`zeromodels/electra_large_discriminator`](https://huggingface.co/zeromodels/electra_large_discriminator) | [`zeromodels/electra_large_generator`](https://huggingface.co/zeromodels/electra_large_generator) |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/electra_small_discriminator")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `ElectraModel` | Encoder backbone |
| `ElectraSequenceClassify` | Sequence classification |
| `ElectraTokenClassify` | Token classification (NER / POS) |
| `ElectraQnA` | Extractive question answering |
| `ElectraMultipleChoice` | Multiple choice |
```python
from zeromodels.models.electra import ElectraSequenceClassify
model = ElectraSequenceClassify.from_weights("zeromodels/electra_small_discriminator")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `ElectraTokenizer.from_weights(...)` so WordPiece tokenization matches.
- Downstream tasks (classification / QA / NER) use the **discriminator** repos; the **generator** repos are the masked-LM.
- See [ELECTRA docs](https://imvision12.github.io/ZeroModels/electra/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `ElectraModel.from_weights("hf:google/electra-small-discriminator")`.
## Special Thanks
A huge thank you to the Google ELECTRA authors for creating and releasing these models.
License: Apache 2.0.
|