IMvision12 commited on
Commit
8431fab
·
verified ·
1 Parent(s): 460104b

Update model card (BERT-style tags + license)

Browse files
Files changed (1) hide show
  1. README.md +72 -22
README.md CHANGED
@@ -1,22 +1,72 @@
1
- ---
2
- library_name: kerasformers
3
- license: apache-2.0
4
- pipeline_tag: fill-mask
5
- tags:
6
- - keras
7
- - electra
8
- base_model: google/electra-base-generator
9
- ---
10
-
11
- # electra_base_generator
12
-
13
- ELECTRA masked-LM checkpoint converted to [kerasformers](https://github.com/IMvision12/KerasFormers) (pure Keras 3, runnable on JAX / PyTorch / TensorFlow). Converted from [`google/electra-base-generator`](https://huggingface.co/google/electra-base-generator).
14
-
15
- ```python
16
- from kerasformers.models.electra import ElectraMaskedLM, ElectraTokenizer
17
-
18
- model = ElectraMaskedLM.from_weights("kerasformers/electra_base_generator")
19
- tokenizer = ElectraTokenizer.from_weights("kerasformers/electra_base_generator")
20
- ```
21
-
22
- embed_dim 256 | layers 12 | heads 4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: fill-mask
3
+ license: apache-2.0
4
+ base_model: google/electra-base-generator
5
+ library_name: kerasformers
6
+ tags:
7
+ - keras
8
+ - kerasformers
9
+ - electra
10
+ - generator
11
+ - text-encoder
12
+ - fill-mask
13
+ - arxiv:2003.10555
14
+ - pytorch
15
+ - jax
16
+ - tf
17
+ ---
18
+
19
+ ## ***See [our collection](https://huggingface.co/collections/kerasformers/electra-6a8540d1f5831e07dc89d8d1) for all versions of ELECTRA.***
20
+
21
+ # Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow
22
+
23
+ [![GitHub](https://img.shields.io/badge/GitHub-KerasFormers-black?logo=github)](https://github.com/IMvision12/KerasFormers) [![Docs](https://img.shields.io/badge/Docs-ELECTRA-blue)](https://imvision12.github.io/KerasFormers/electra/) [![Collection](https://img.shields.io/badge/HF-ELECTRA%20collection-yellow)](https://huggingface.co/collections/kerasformers/electra-6a8540d1f5831e07dc89d8d1)
24
+
25
+ # kerasformers/electra_base_generator
26
+
27
+ Paper: [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555)](https://arxiv.org/abs/2003.10555) · [HF Papers](https://huggingface.co/papers/2003.10555)
28
+
29
+ ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token **discriminator** (with a smaller **generator** producing the corrupted tokens). This repo is the **masked-LM (fill-mask)** checkpoint. WordPiece tokenizer; mask token `[MASK]`.
30
+
31
+ For more details on the model, please go to the upstream [model card](https://huggingface.co/google/electra-base-generator).
32
+
33
+ Pure-**Keras 3** conversion of [`google/electra-base-generator`](https://huggingface.co/google/electra-base-generator) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
34
+
35
+ ## ✨ Quick start (masked-LM (fill-mask))
36
+
37
+ ```python
38
+ import os
39
+ os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
40
+
41
+ from kerasformers.models.electra import ElectraMaskedLM, ElectraTokenizer
42
+
43
+ mlm = ElectraMaskedLM.from_weights("kerasformers/electra_base_generator")
44
+ tokenizer = ElectraTokenizer.from_weights("kerasformers/electra_base_generator")
45
+
46
+ inputs = tokenizer("The capital of France is [MASK].")
47
+ logits = mlm(inputs) # (1, L, vocab_size)
48
+ mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
49
+ print(tokenizer.decode([int(logits[0, mask].argmax())]))
50
+ ```
51
+
52
+ Load any ELECTRA variant the same way with `from_weights("kerasformers/<variant>")`:
53
+
54
+ | Size | Discriminator (encoder / downstream) | Generator (masked-LM) |
55
+ |---|---|---|
56
+ | small | [`kerasformers/electra_small_discriminator`](https://huggingface.co/kerasformers/electra_small_discriminator) | [`kerasformers/electra_small_generator`](https://huggingface.co/kerasformers/electra_small_generator) |
57
+ | base | [`kerasformers/electra_base_discriminator`](https://huggingface.co/kerasformers/electra_base_discriminator) | [`kerasformers/electra_base_generator`](https://huggingface.co/kerasformers/electra_base_generator) |
58
+ | large | [`kerasformers/electra_large_discriminator`](https://huggingface.co/kerasformers/electra_large_discriminator) | [`kerasformers/electra_large_generator`](https://huggingface.co/kerasformers/electra_large_generator) |
59
+
60
+ ## Tips
61
+
62
+ - Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
63
+ - Prefer `ElectraTokenizer.from_weights(...)` so WordPiece tokenization matches.
64
+ - Downstream tasks (classification / QA / NER) use the **discriminator** repos; the **generator** repos are the masked-LM.
65
+ - See [ELECTRA docs](https://imvision12.github.io/KerasFormers/electra/) and [Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
66
+ - Community / upstream safetensors still work via the `hf:` prefix, e.g. `ElectraModel.from_weights("hf:google/electra-base-generator")`.
67
+
68
+ ## Special Thanks
69
+
70
+ A huge thank you to the Google ELECTRA authors for creating and releasing these models.
71
+
72
+ License: Apache 2.0.