Instructions to use Jaidchen/Focus-Red-Int4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Jaidchen/Focus-Red-Int4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Jaidchen/Focus-Red-Int4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Jaidchen/Focus-Red-Int4")
model = AutoModelForCausalLM.from_pretrained("Jaidchen/Focus-Red-Int4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Jaidchen/Focus-Red-Int4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Jaidchen/Focus-Red-Int4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jaidchen/Focus-Red-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Jaidchen/Focus-Red-Int4

SGLang

How to use Jaidchen/Focus-Red-Int4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Jaidchen/Focus-Red-Int4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jaidchen/Focus-Red-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Jaidchen/Focus-Red-Int4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Jaidchen/Focus-Red-Int4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Jaidchen/Focus-Red-Int4 with Docker Model Runner:
```
docker model run hf.co/Jaidchen/Focus-Red-Int4
```

Focus-Red-Int4 / README.md

Jaidchen

Fix model card repository name

4220f89 5 days ago

preview code

Raw

History Blame Contribute Delete

6.98 kB

	---
	library_name: transformers
	license: apache-2.0
	license_link: https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE
	base_model: Jaidchen/Focus-Red
	pipeline_tag: text-generation
	tags:
	- dense
	- coding
	- agentic
	- unimodal
	- repackaged
	- quantized
	- compressed-tensors
	- int4
	---

	<center>
	<img style='height: 16em' src='https://cdn-uploads.huggingface.co/production/uploads/63e9a1c4ccae1fe5c622b9f6/zOXMkdhMfeQWxDq6JHj0B.webp'/>
	</center>

	# Focus-Red-Int4

	4-bit quantization of [Focus-Red](https://huggingface.co/Jaidchen/Focus-Red), itself a text-only repackaging of [Qwen 3.8 27B](https://huggingface.co/Qwen/Qwen3.8-27B).

	- W4A16, group size 32, asymmetric weights
	- `compressed-tensors` / `pack-quantized` format
	- selected linear-attention projections and `lm_head` remain BF16
	- single `model.safetensors` file

	## comparison

	<table>
	<tr>
	<th></th>
	<th>Qwen 3.8 27B</th>
	<th>Focus-Red-Int4</th>
	</tr>
	<tr>
	<td>author</td>
	<td><a href='https://huggingface.co/Qwen'>Alibaba Qwen</a></td>
	<td><a href='https://huggingface.co/Jaidchen'>Jaid</a></td>
	</tr>
	<tr>
	<td>repository</td>
	<td><a href='https://huggingface.co/Qwen/Qwen3.8-27B'>Qwen/Qwen3.8-27B</a></td>
	<td><a href='https://huggingface.co/Jaidchen/Focus-Red-Int4'>Jaidchen/Focus-Red-Int4</a></td>
	</tr>
	<tr>
	<td>architecture</td>
	<td><code>qwen3_5</code></td>
	<td><code>qwen3_5_text</code></td>
	</tr>
	<tr>
	<td>Transformers handler</td>
	<td>
	<a href='https://huggingface.co/docs/transformers/model_doc/qwen3_5#transformers.Qwen3_5ForConditionalGeneration'><code>Qwen3_5ForConditionalGeneration</code></a>
	</td>
	<td>
	<a href='https://huggingface.co/docs/transformers/model_doc/qwen3_5#transformers.Qwen3_5ForCausalLM'><code>Qwen3_5ForCausalLM</code></a>
	</td>
	</tr>
	<tr>
	<td>tensor entries</td>
	<td>1199</td>
	<td>2051</td>
	</tr>
	<tr style='opacity: 50%'>
	<td>tensor type</td>
	<td>bf16</td>
	<td>W4A16 G32 asymmetric + selected BF16</td>
	</tr>
	<tr>
	<td>parameters</td>
	<td>27 781 427 952</td>
	<td>26 895 998 464</td>
	</tr>
	<tr style='opacity: 50%'>
	<td>active</td>
	<td>100%</td>
	<td>100%</td>
	</tr>
	<tr style='opacity: 50%'>
	<td>vocabulary size</td>
	<td>248 320</td>
	<td>248 320</td>
	</tr>
	<tr style='opacity: 50%'>
	<td>context size</td>
	<td>262 144</td>
	<td>262 144</td>
	</tr>
	<tr style='opacity: 50%'>
	<td>MTP</td>
	<td>integrated</td>
	<td>detached → <a href='https://huggingface.co/Jaidchen/Focus-Red-MTP'>Focus-Red-MTP</a></td>
	</tr>
	<tr>
	<td>sampling strategy</td>
	<td>random sampling</td>
	<td>greedy/deterministic</td>
	</tr>
	<tr>
	<td>sampling parameters</td>
	<td>
	<div style='font-family: Jaidevka Code, JetBrains Mono, monospace; line-height: initial'>
	<span style='color: hsl(from currentColor 0 100% l)'>do_sample</span>: <span style='color: hsl(from currentColor 50 80% l)'>true</span><br>
	<span style='color: hsl(from currentColor 0 100% l)'>temperature</span>: <span style='color: hsl(from currentColor 50 80% l)'>1.0</span><br>
	<span style='color: hsl(from currentColor 0 100% l)'>top_k</span>: <span style='color: hsl(from currentColor 50 80% l)'>20</span><br>
	<span style='color: hsl(from currentColor 0 100% l)'>top_p</span>: <span style='color: hsl(from currentColor 50 80% l)'>0.95</span></div>
	</td>
	<td>
	<div style='font-family: Jaidevka Code, JetBrains Mono, monospace; line-height: initial'>
	<span style='color: hsl(from currentColor 0 100% l)'>do_sample</span>: <span style='color: hsl(from currentColor 50 80% l)'>false</span><br>
	<span style='color: hsl(from currentColor 0 100% l)'>temperature</span>: <span style='color: hsl(from currentColor 50 80% l)'>0</span><br>
	<span style='color: hsl(from currentColor 0 100% l)'>top_k</span>: <span style='color: hsl(from currentColor 50 80% l)'>1</span><br>
	<span style='color: hsl(from currentColor 0 100% l)'>top_p</span>: <span style='color: hsl(from currentColor 50 80% l)'>1</span></div>
	</div>
	</td>
	</tr>
	<tr>
	<td>input modality</td>
	<td>text, image, video</td>
	<td>text</td>
	</tr>
	<tr>
	<td>model size</td>
	<td>55 562 855 904</td>
	<td>19 202 352 336 bytes on disk</td>
	</tr>
	<tr>
	<td>splits</td>
	<td>18</td>
	<td>none</td>
	</tr>
	<tr>
	<td>Jinja template</td>
	<td><a href='https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/chat_template.jinja'>Qwen original</a></td>
	<td><a href='https://github.com/jaidlab/focus-chat-template'>focus-chat-template</a> dist build</td>
	</tr>
	</table>

	## quantization

	- method: `compressed-tensors`
	- format: `pack-quantized`
	- weights: 4-bit integer, group size 32, asymmetric
	- activations: unquantized / BF16
	- quantization status: `compressed`
	- model file: 19,202,352,336 bytes

	## pros

	- reduced storage needs
	- reduced loading time
	- reduced VRAM occupancy, thus more room for context
	- increased inference speed
	- simplified architecture, unlocking some further potential for optimizing low-level procedures

	## cons

	- legally blind
	- Pictures and video frames can still be present in the context without crashing, but their contents are no longer interpreted by the model and won’t do anything else than waste space.
	- If you occasionally rely on those capabilities, I suggest adding a `consult` tool to your harness that calls a vision-enabled subagent model like [Gemini Flash](https://openrouter.ai/~google/gemini-flash-latest) or [GPT](https://openrouter.ai/~openai/gpt-latest).
	- reduced compatibility
	- The simplified architecture is handled by the `Qwen3_5ForCausalLM` class which may not be included in your inference engine. In this case you would need to ask your agent or integrate it yourself.
	- Inference engines confirmed to have native support:
	- [SGLang](https://github.com/sgl-project/sglang) ≥ v0.5.17 (PR [#32401](https://github.com/sgl-project/sglang/pull/32401))
	- [vLLM](https://github.com/vllm-project/vllm) ≥ v0.26.0 (PR [#50210](https://github.com/vllm-project/vllm/pull/50210))
	- The applied coercions may confuse your inference engine in case it has fixed expectations about the model’s architecture and thus lead to unpredictable behavior.

	## caveats

	- model file not split, possibly causing issues if intended to be stored on an HDD from the previous century
	- random sampling disabled by default, less suitable for long-form writing, entertainment and casual chat

	## Jinja template

	This build uses `dist/chat_template.jinja` from [jaidlab/focus-chat-template](https://github.com/jaidlab/focus-chat-template). The template is reproducibly built from Qwen/Qwen3.8-27B's pinned upstream template at commit `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` plus the repository's ordered patch stack.

	- template SHA-256: `5c381ca45e9538c7a2406331b554ee7d62cf3d0b8c115f17687d4fdd5590a239`
	- template size: 9,710 bytes

	## license

	Apache 2.0 – inherited from [Qwen 3.8 27B](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/LICENSE)