Instructions to use tiiuae/Falcon3-Mamba-7B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tiiuae/Falcon3-Mamba-7B-Instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="tiiuae/Falcon3-Mamba-7B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("tiiuae/Falcon3-Mamba-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("tiiuae/Falcon3-Mamba-7B-Instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tiiuae/Falcon3-Mamba-7B-Instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tiiuae/Falcon3-Mamba-7B-Instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon3-Mamba-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/tiiuae/Falcon3-Mamba-7B-Instruct

SGLang

How to use tiiuae/Falcon3-Mamba-7B-Instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tiiuae/Falcon3-Mamba-7B-Instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon3-Mamba-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tiiuae/Falcon3-Mamba-7B-Instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tiiuae/Falcon3-Mamba-7B-Instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use tiiuae/Falcon3-Mamba-7B-Instruct with Docker Model Runner:
```
docker model run hf.co/tiiuae/Falcon3-Mamba-7B-Instruct
```

Request for Guidance on Fine-Tuning Falcon3-Mamba-7B-Instruct + Technical Questions

#10

by DeltaWhiplash - opened Feb 3, 2025

Discussion

DeltaWhiplash

Feb 3, 2025

•

edited Feb 3, 2025

Subject: Request for Guidance on Fine-Tuning Falcon3-Mamba-7B-Instruct + Technical Questions

Hello Falcon3-Mamba team,

First, congratulations on this groundbreaking work with Mamba architecture - truly inspiring to see Transformer alternatives pushing LLM boundaries! 🚀

As a student exploring SSM-based models, I'd appreciate your insights for my fine-tuning project (budget ~$200). Could you help address these technical questions?

Core Technical Queries

Fine-Tuning Infrastructure
- What's the minimum VRAM required for full-parameter vs LoRA/QLoRA fine-tuning?
- Do you recommend gradient checkpointing or activation recomputation for Mamba backprop?
- Have you tested 8-bit/4-bit AdamW optimizers with this architecture?
Architecture-Specific Training
- How do optimal learning rates (LR) for Mamba compare to Transformer-based Falcon variants?
- What's your recommended LR scheduler (linear vs cosine) and warmup ratio?
- Any sequence length limitations during fine-tuning vs pretraining?
- Is there anything you learned while training this model
Knowledge Distillation Challenges
- How critical is architectural alignment when distilling Transformer-based Qwen2.5-72B into SSM?
- Would you recommend freezing specific layers (e.g., SSM blocks) during distillation?
- Any successful prior attempts at cross-architecture distillation?
DPO Implementation
- Have you tested synthetic preference data from judge models (e.g., DeepSeek-R1-70B)?
- Does Mamba's recurrent nature impact pairwise comparison during DPO?
- Preferred reward normalization techniques for SSM models?
Dataset Optimization
- Maximum recommended batch size for 24GB VRAM (Colab Pro) with 2k token sequences?
- Any tokenization mismatches observed with multilingual datasets like Aya?
- Experience with curriculum learning for conversational fine-tuning?

Project-Specific Questions

Approach A (Distillation + SFT + DPO):

Would layer-wise distillation (e.g., attention outputs → SSM states) be feasible?
How to handle dimensional mismatches in projection layers during transfer?

Approach B (Conversational Focus):

Optimal context window configuration for multi-turn dialogues?
Recommended techniques for maintaining Mamba's throughput advantage during long chats?

Resource Requests

Could you share a Colab-compatible fine-tuning template (PEFT/LoRA preferred)?
Any known issues with Hugging Face Trainer vs custom training loops?
Recommended monitoring tools for SSM-specific metrics (hidden states evolution, memory throughput)?

Architecture Curiosity

How does Mamba handle positional information compared to RoPE in Transformers?
Any plans to release ablation studies on SSM parameter initialization?
Maximum effective context length observed in practice for instruction tasks?

Thank you for advancing open-source LLM innovation - your expertise would be invaluable for exploring Mamba's full potential!

Best regards,

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment