---
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
base_model: tiiuae/Falcon3-7B-Instruct
base_model_relation: adapter
library_name: peft
pipeline_tag: text-generation
language:
- en
datasets:
- HuggingFaceFW/fineweb-edu
tags:
- lora
- conceptizer
---

# Falcon3-7B-Instruct-Conceptizer

**Falcon3-7B-Instruct-Conceptizer is built using artificial intelligence technology from the Technology Innovation Institute.**

LoRA adapters for [Falcon3-7B-Instruct](https://huggingface.co/tiiuae/Falcon3-7B-Instruct) from the paper [*Conceptizer: Next-Token Prediction Is Not All You Need*](https://arxiv.org/abs/2610.33027) (Haoran Zheng and Chen Shani).

Conceptizer adds a fixed vocabulary of units of meaning alongside the model's tokenizer, so post-training rewards different wordings of the same meaning, such as *safety*, *protection* and *security*, not only the exact next word. The concepts enter only the training loss; the adapters generate ordinary text.

In the paper, Conceptizer gives the lowest held-out perplexity at concept-target positions against all three baselines and improves knowledge and sentiment tasks, while staying comparable on commonsense.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon3-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-7B-Instruct", dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "FlyinGodzilla/Falcon3-7B-Instruct-Conceptizer")
```

The paper used transformers 5.0 and peft 0.19. The repository holds the paper's three training seeds: seed 0 loads by default, and seeds 37 and 42 load with `subfolder="seed37"` or `subfolder="seed42"`.

## Training

Rank-16 LoRA post-training on FineWeb-Edu text for 50,000 steps, with next-token prediction, the concept loss and a retention loss that keeps the model close to the base model; the paper has the details. To train your own model with Conceptizer, the [Conceptizer repository](https://github.com/HaoranZhengRaul/conceptizer-code) has the code, the concept vocabulary, the training data and the commands.

## License

Distributed under the [TII Falcon License](LICENSE); use must comply with TII's [Acceptable Use Policy](https://falconllm.tii.ae/falcon-acceptable-use-policy.html).