Model card: the paper's result, library versions, where to train your own model
Browse files
README.md
CHANGED
|
@@ -23,6 +23,8 @@ LoRA adapters for [Falcon3-7B-Instruct](https://huggingface.co/tiiuae/Falcon3-7B
|
|
| 23 |
|
| 24 |
Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as *safety*, *protection* and *security*, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.
|
| 25 |
|
|
|
|
|
|
|
| 26 |
## Usage
|
| 27 |
|
| 28 |
```python
|
|
@@ -35,11 +37,11 @@ model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-7B-Instruct", dtype
|
|
| 35 |
model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-7B-Instruct-Conceptizer")
|
| 36 |
```
|
| 37 |
|
| 38 |
-
The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with `subfolder="seed37"` or `subfolder="seed42"`.
|
| 39 |
|
| 40 |
## Training
|
| 41 |
|
| 42 |
-
Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model
|
| 43 |
|
| 44 |
## License
|
| 45 |
|
|
|
|
| 23 |
|
| 24 |
Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as *safety*, *protection* and *security*, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.
|
| 25 |
|
| 26 |
+
In the paper, Conceptizer gives the lowest held-out perplexity at concept-target positions against all three baselines and improves knowledge and sentiment tasks, while staying comparable on commonsense.
|
| 27 |
+
|
| 28 |
## Usage
|
| 29 |
|
| 30 |
```python
|
|
|
|
| 37 |
model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-7B-Instruct-Conceptizer")
|
| 38 |
```
|
| 39 |
|
| 40 |
+
The paper used transformers 5.0 and peft 0.19. The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with `subfolder="seed37"` or `subfolder="seed42"`.
|
| 41 |
|
| 42 |
## Training
|
| 43 |
|
| 44 |
+
Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model; the paper has the details. To train your own model with Conceptizer, the [Conceptizer repository](https://github.com/HaoranZhengRaul/conceptizer-code) has the code, the concept vocabulary, the training data and the commands.
|
| 45 |
|
| 46 |
## License
|
| 47 |
|