IMvision12 commited on
Commit
86c0f24
·
verified ·
1 Parent(s): 26467e8

Migrate to zeromodels (rename kf_*.json -> zm_*.json, fix refs in config + README, ensure tag + badge)

Browse files
Files changed (2) hide show
  1. README.md +16 -16
  2. kf_config.json → zm_config.json +25 -25
README.md CHANGED
@@ -2,10 +2,10 @@
2
  pipeline_tag: fill-mask
3
  license: apache-2.0
4
  base_model: answerdotai/ModernBERT-base
5
- library_name: kerasformers
6
  tags:
7
  - keras
8
- - kerasformers
9
  - modernbert
10
  - fill-mask
11
  - text-encoder
@@ -17,9 +17,9 @@ tags:
17
 
18
  # Run ModernBERT with Keras 3: JAX, PyTorch, or TensorFlow
19
 
20
- [![GitHub](https://img.shields.io/badge/GitHub-KerasFormers-black?logo=github)](https://github.com/IMvision12/KerasFormers) [![Docs](https://img.shields.io/badge/Docs-ModernBERT-blue)](https://imvision12.github.io/KerasFormers/modernbert/)
21
 
22
- # kerasformers/modernbert_base
23
 
24
  Paper: [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder (arXiv:2412.13663)](https://arxiv.org/abs/2412.13663) · [HF Papers](https://huggingface.co/papers/2412.13663)
25
 
@@ -27,7 +27,7 @@ ModernBERT is Answer.AI / LightOn's modernized bidirectional transformer text en
27
 
28
  For more details on the model, please go to the upstream [model card](https://huggingface.co/answerdotai/ModernBERT-base).
29
 
30
- Pure-**Keras 3** conversion of [`answerdotai/ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base) for [kerasformers](https://github.com/IMvision12/KerasFormers). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
31
 
32
  This is a **fill-mask / encoder** checkpoint (`ModernBertMaskedLM`, modernbert_base). Task heads (sequence / token classify, QA, multiple choice) load via `hf:` fine-tunes.
33
 
@@ -37,10 +37,10 @@ This is a **fill-mask / encoder** checkpoint (`ModernBertMaskedLM`, modernbert_b
37
  import os
38
  os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
39
 
40
- from kerasformers.models.modernbert import ModernBertMaskedLM, ModernBertTokenizer
41
 
42
- mlm = ModernBertMaskedLM.from_weights("kerasformers/modernbert_base")
43
- tokenizer = ModernBertTokenizer.from_weights("kerasformers/modernbert_base")
44
 
45
  inputs = tokenizer("The capital of France is [MASK].")
46
  logits = mlm(inputs) # (1, L, vocab_size)
@@ -48,16 +48,16 @@ mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
48
  print(tokenizer.decode([int(logits[0, mask].argmax())]))
49
  ```
50
 
51
- Load any ModernBERT variant the same way with `from_weights("kerasformers/<variant>")`:
52
 
53
  | Variant | Hub | layers | embed_dim |
54
  |---|---|---|---|
55
- | `modernbert_base` | [`kerasformers/modernbert_base`](https://huggingface.co/kerasformers/modernbert_base) | 22 | 768 |
56
- | `modernbert_large` | [`kerasformers/modernbert_large`](https://huggingface.co/kerasformers/modernbert_large) | 28 | 1024 |
57
 
58
  ## Available classes
59
 
60
- Load any of these from this repo with `from_weights("kerasformers/modernbert_base")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
61
 
62
  | Class | Task |
63
  |---|---|
@@ -69,17 +69,17 @@ Load any of these from this repo with `from_weights("kerasformers/modernbert_bas
69
  | `ModernBertMultipleChoice` | Multiple choice |
70
 
71
  ```python
72
- from kerasformers.models.modernbert import ModernBertSequenceClassify
73
- model = ModernBertSequenceClassify.from_weights("kerasformers/modernbert_base")
74
  ```
75
 
76
  ## Tips
77
 
78
- - Set `KERAS_BACKEND` **before** importing Keras / kerasformers.
79
  - Prefer `ModernBertTokenizer.from_weights(...)` so tokenization matches.
80
  - Use `[MASK]` (not `<mask>`).
81
  - ModernBERT has no token-type ids; the tokenizer emits only `input_ids` / `attention_mask`.
82
- - See [ModernBERT docs](https://imvision12.github.io/KerasFormers/modernbert/) and [Loading Weights](https://imvision12.github.io/KerasFormers/loading_weights/).
83
  - Community / upstream safetensors still work via the `hf:` prefix, e.g. `ModernBertMaskedLM.from_weights("hf:answerdotai/ModernBERT-base")`.
84
 
85
  ## Special Thanks
 
2
  pipeline_tag: fill-mask
3
  license: apache-2.0
4
  base_model: answerdotai/ModernBERT-base
5
+ library_name: zeromodels
6
  tags:
7
  - keras
8
+ - zeromodels
9
  - modernbert
10
  - fill-mask
11
  - text-encoder
 
17
 
18
  # Run ModernBERT with Keras 3: JAX, PyTorch, or TensorFlow
19
 
20
+ [![GitHub](https://img.shields.io/badge/GitHub-ZeroModels-black?logo=github)](https://github.com/IMvision12/ZeroModels) [![Docs](https://img.shields.io/badge/Docs-ModernBERT-blue)](https://imvision12.github.io/ZeroModels/modernbert/)
21
 
22
+ # zeromodels/modernbert_base
23
 
24
  Paper: [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder (arXiv:2412.13663)](https://arxiv.org/abs/2412.13663) · [HF Papers](https://huggingface.co/papers/2412.13663)
25
 
 
27
 
28
  For more details on the model, please go to the upstream [model card](https://huggingface.co/answerdotai/ModernBERT-base).
29
 
30
+ Pure-**Keras 3** conversion of [`answerdotai/ModernBERT-base`](https://huggingface.co/answerdotai/ModernBERT-base) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
31
 
32
  This is a **fill-mask / encoder** checkpoint (`ModernBertMaskedLM`, modernbert_base). Task heads (sequence / token classify, QA, multiple choice) load via `hf:` fine-tunes.
33
 
 
37
  import os
38
  os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
39
 
40
+ from zeromodels.models.modernbert import ModernBertMaskedLM, ModernBertTokenizer
41
 
42
+ mlm = ModernBertMaskedLM.from_weights("zeromodels/modernbert_base")
43
+ tokenizer = ModernBertTokenizer.from_weights("zeromodels/modernbert_base")
44
 
45
  inputs = tokenizer("The capital of France is [MASK].")
46
  logits = mlm(inputs) # (1, L, vocab_size)
 
48
  print(tokenizer.decode([int(logits[0, mask].argmax())]))
49
  ```
50
 
51
+ Load any ModernBERT variant the same way with `from_weights("zeromodels/<variant>")`:
52
 
53
  | Variant | Hub | layers | embed_dim |
54
  |---|---|---|---|
55
+ | `modernbert_base` | [`zeromodels/modernbert_base`](https://huggingface.co/zeromodels/modernbert_base) | 22 | 768 |
56
+ | `modernbert_large` | [`zeromodels/modernbert_large`](https://huggingface.co/zeromodels/modernbert_large) | 28 | 1024 |
57
 
58
  ## Available classes
59
 
60
+ Load any of these from this repo with `from_weights("zeromodels/modernbert_base")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
61
 
62
  | Class | Task |
63
  |---|---|
 
69
  | `ModernBertMultipleChoice` | Multiple choice |
70
 
71
  ```python
72
+ from zeromodels.models.modernbert import ModernBertSequenceClassify
73
+ model = ModernBertSequenceClassify.from_weights("zeromodels/modernbert_base")
74
  ```
75
 
76
  ## Tips
77
 
78
+ - Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
79
  - Prefer `ModernBertTokenizer.from_weights(...)` so tokenization matches.
80
  - Use `[MASK]` (not `<mask>`).
81
  - ModernBERT has no token-type ids; the tokenizer emits only `input_ids` / `attention_mask`.
82
+ - See [ModernBERT docs](https://imvision12.github.io/ZeroModels/modernbert/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
83
  - Community / upstream safetensors still work via the `hf:` prefix, e.g. `ModernBertMaskedLM.from_weights("hf:answerdotai/ModernBERT-base")`.
84
 
85
  ## Special Thanks
kf_config.json → zm_config.json RENAMED
@@ -1,26 +1,26 @@
1
- {
2
- "library_name": "kerasformers",
3
- "kerasformers_version": "1.2.3",
4
- "model_module": "kerasformers.models.modernbert",
5
- "model_class": "ModernBertModel",
6
- "variant": "modernbert_base",
7
- "weights": "model.weights.h5",
8
- "schema_version": 2,
9
- "weight_dtype": "float32",
10
- "model_type": "modernbert",
11
- "text_config": {
12
- "vocab_size": 50368,
13
- "embed_dim": 768,
14
- "num_layers": 22,
15
- "num_heads": 12,
16
- "mlp_dim": 1152,
17
- "max_position_embeddings": 8192,
18
- "hidden_act": "gelu",
19
- "norm_eps": 1e-05,
20
- "local_attention": 128,
21
- "global_attn_every_n_layers": 3,
22
- "global_rope_theta": 160000.0,
23
- "local_rope_theta": 10000.0,
24
- "pad_token_id": 50283
25
- }
26
  }
 
1
+ {
2
+ "library_name": "zeromodels",
3
+ "zeromodels_version": "1.2.3",
4
+ "model_module": "zeromodels.models.modernbert",
5
+ "model_class": "ModernBertModel",
6
+ "variant": "modernbert_base",
7
+ "weights": "model.weights.h5",
8
+ "schema_version": 2,
9
+ "weight_dtype": "float32",
10
+ "model_type": "modernbert",
11
+ "text_config": {
12
+ "vocab_size": 50368,
13
+ "embed_dim": 768,
14
+ "num_layers": 22,
15
+ "num_heads": 12,
16
+ "mlp_dim": 1152,
17
+ "max_position_embeddings": 8192,
18
+ "hidden_act": "gelu",
19
+ "norm_eps": 1e-05,
20
+ "local_attention": 128,
21
+ "global_attn_every_n_layers": 3,
22
+ "global_rope_theta": 160000.0,
23
+ "local_rope_theta": 10000.0,
24
+ "pad_token_id": 50283
25
+ }
26
  }