IMvision12 commited on
Commit
e69d31f
·
verified ·
1 Parent(s): 3cfcc70

Update model card (BERT-style tags + license)

Browse files
Files changed (1) hide show
  1. README.md +71 -22
README.md CHANGED
@@ -1,22 +1,71 @@
1
- ---
2
- library_name: kerasformers
3
- license: apache-2.0
4
- pipeline_tag: fill-mask
5
- tags:
6
- - keras
7
- - electra
8
- base_model: google/electra-small-discriminator
9
- ---
10
-
11
- # electra_small_discriminator
12
-
13
- ELECTRA encoder / downstream checkpoint converted to [kerasformers](https://github.com/IMvision12/KerasFormers) (pure Keras 3, runnable on JAX / PyTorch / TensorFlow). Converted from [`google/electra-small-discriminator`](https://huggingface.co/google/electra-small-discriminator).
14
-
15
- ```python
16
- from kerasformers.models.electra import ElectraModel, ElectraTokenizer
17
-
18
- model = ElectraModel.from_weights("kerasformers/electra_small_discriminator")
19
- tokenizer = ElectraTokenizer.from_weights("kerasformers/electra_small_discriminator")
20
- ```
21
-
22
- embed_dim 256 | layers 12 | heads 4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ pipeline_tag: feature-extraction
3
+ license: apache-2.0
4
+ base_model: google/electra-small-discriminator
5
+ library_name: kerasformers
6
+ tags:
7
+ - keras
8
+ - kerasformers
9
+ - electra
10
+ - discriminator
11
+ - text-encoder
12
+ - feature-extraction
13
+ - arxiv:2003.10555
14
+ - pytorch
15
+ - jax
16
+ - tf
17
+ ---
18
+
19
+ ## ***See [our collection](https://huggingface.co/collections/kerasformers/electra-6a8540d1f5831e07dc89d8d1) for all versions of ELECTRA.***
20
+
21
+ # Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow
22
+
23
+ [![GitHub](https://img.shields.io/badge/GitHub-KerasFormers-black?logo=github)](https://github.com/IMvision12/KerasFormers) [![Docs](https://img.shields.io/badge/Docs-ELECTRA-blue)](https://imvision12.github.io/KerasFormers/electra/) [![Collection](https://img.shields.io/badge/HF-ELECTRA%20collection-yellow)](https://huggingface.co/collections/kerasformers/electra-6a8540d1f5831e07dc89d8d1)
24
+
25
+ # kerasformers/electra_small_discriminator
26
+
27
+ Paper: [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555)](https://arxiv.org/abs/2003.10555) · [HF Papers](https://huggingface.co/papers/2003.10555)
28
+
29
+ ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token **discriminator** (with a smaller **generator** producing the corrupted tokens). This repo is the **encoder / downstream** checkpoint. WordPiece tokenizer; mask token `[MASK]`.
30
+
31
+ For more details on the model, please go to the upstream [model card](https://huggingface.co/google/electra-small-discriminator).
32
+
33
+ Pure-**Keras 3** conversion of [`google/electra-small-discriminator`](https://huggingface.co/google/electra-small-discriminator) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
34
+
35
+ ## ✨ Quick start (encoder / downstream)
36
+
37
+ ```python
38
+ import os
39
+ os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
40
+
41
+ from kerasformers.models.electra import ElectraModel, ElectraTokenizer
42
+
43
+ model = ElectraModel.from_weights("kerasformers/electra_small_discriminator")
44
+ tokenizer = ElectraTokenizer.from_weights("kerasformers/electra_small_discriminator")
45
+
46
+ out = model(tokenizer("The quick brown fox."))["last_hidden_state"] # (1, L, H)
47
+ ```
48
+
49
+ The same repo also serves the task heads, loaded the same way: `ElectraSequenceClassify`, `ElectraTokenClassify`, `ElectraQnA`, `ElectraMultipleChoice` (each takes the pretrained encoder and a randomly-initialized head, ready for fine-tuning).
50
+
51
+ Load any ELECTRA variant the same way with `from_weights("kerasformers/<variant>")`:
52
+
53
+ | Size | Discriminator (encoder / downstream) | Generator (masked-LM) |
54
+ |---|---|---|
55
+ | small | [`kerasformers/electra_small_discriminator`](https://huggingface.co/kerasformers/electra_small_discriminator) | [`kerasformers/electra_small_generator`](https://huggingface.co/kerasformers/electra_small_generator) |
56
+ | base | [`kerasformers/electra_base_discriminator`](https://huggingface.co/kerasformers/electra_base_discriminator) | [`kerasformers/electra_base_generator`](https://huggingface.co/kerasformers/electra_base_generator) |
57
+ | large | [`kerasformers/electra_large_discriminator`](https://huggingface.co/kerasformers/electra_large_discriminator) | [`kerasformers/electra_large_generator`](https://huggingface.co/kerasformers/electra_large_generator) |
58
+
59
+ ## Tips
60
+
61
+ - Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
62
+ - Prefer `ElectraTokenizer.from_weights(...)` so WordPiece tokenization matches.
63
+ - Downstream tasks (classification / QA / NER) use the **discriminator** repos; the **generator** repos are the masked-LM.
64
+ - See [ELECTRA docs](https://imvision12.github.io/KerasFormers/electra/) and [Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
65
+ - Community / upstream safetensors still work via the `hf:` prefix, e.g. `ElectraModel.from_weights("hf:google/electra-small-discriminator")`.
66
+
67
+ ## Special Thanks
68
+
69
+ A huge thank you to the Google ELECTRA authors for creating and releasing these models.
70
+
71
+ License: Apache 2.0.