Instructions to use Akicou/INTELLECT-3-REAP-50-FP8-Dynamic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Akicou/INTELLECT-3-REAP-50-FP8-Dynamic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Akicou/INTELLECT-3-REAP-50-FP8-Dynamic")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Akicou/INTELLECT-3-REAP-50-FP8-Dynamic")
model = AutoModelForCausalLM.from_pretrained("Akicou/INTELLECT-3-REAP-50-FP8-Dynamic")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use Akicou/INTELLECT-3-REAP-50-FP8-Dynamic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Akicou/INTELLECT-3-REAP-50-FP8-Dynamic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Akicou/INTELLECT-3-REAP-50-FP8-Dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Akicou/INTELLECT-3-REAP-50-FP8-Dynamic

SGLang

How to use Akicou/INTELLECT-3-REAP-50-FP8-Dynamic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Akicou/INTELLECT-3-REAP-50-FP8-Dynamic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Akicou/INTELLECT-3-REAP-50-FP8-Dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Akicou/INTELLECT-3-REAP-50-FP8-Dynamic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Akicou/INTELLECT-3-REAP-50-FP8-Dynamic",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Akicou/INTELLECT-3-REAP-50-FP8-Dynamic with Docker Model Runner:
```
docker model run hf.co/Akicou/INTELLECT-3-REAP-50-FP8-Dynamic
```

INTELLECT-3-REAP-50-FP8-Dynamic

Model Overview

This is a quantized version of INTELLECT-3-REAP-50, a Router Expert Activation Pruned (REAP) Mixture of Experts (MoE) model. This version has been compressed to FP8-Dynamic precision using the llmcompressor library to optimize it for high-performance inference with a reduced memory footprint.

Key Features

Quantization: FP8-Dynamic (activations and weights).
Architecture: REAP-optimized MoE based on GLM-4.
Efficiency: Designed to run on modern GPUs (NVIDIA Ada Lovelace and Hopper architectures) with significant VRAM savings.
Algorithm: One-Shot Post-Training Quantization (PTQ).

REAP Optimization

REAP (Router Expert Activation Pruning) enhances MoE efficiency by pruning the activation of experts through a specialized routing mechanism. By combining this architecture with FP8-Dynamic quantization, the model achieves a balance between the high parameter count of MoE and the low latency required for production environments.

Installation

To run this model, ensure you have the latest transformers and torch versions installed:

pip install torch torchvision transformers typing_extensions llmcompressor

Usage Example

from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "Akicou/INTELLECT-3-REAP-50-FP8-Dynamic"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    device_map="auto",
    torch_dtype="auto",
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

prompt = "Write a technical summary of how FP8 quantization improves LLM inference."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

output = model.generate(**inputs, max_new_tokens=150)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Usage on Runpod

Quantization Details

The model was quantized using the following llmcompressor configuration:

Targets: Linear layers.
Scheme: FP8_DYNAMIC.
Ignored Layers: lm_head.
Calibration: Performed with oneshot algorithm.

Limitations

Hardware: Native FP8 support requires NVIDIA Blackwell, Hopper, or Ada Lovelace GPUs.
Precision: While dynamic scaling minimizes loss, slight accuracy deviations may occur compared to the original BF16 weights in highly niche benchmarks.