Instructions to use tcclaviger/Tess-27B-RFA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tcclaviger/Tess-27B-RFA with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tcclaviger/Tess-27B-RFA")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tcclaviger/Tess-27B-RFA")
model = AutoModelForMultimodalLM.from_pretrained("tcclaviger/Tess-27B-RFA")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tcclaviger/Tess-27B-RFA with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tcclaviger/Tess-27B-RFA"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-27B-RFA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tcclaviger/Tess-27B-RFA

SGLang

How to use tcclaviger/Tess-27B-RFA with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tcclaviger/Tess-27B-RFA" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-27B-RFA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tcclaviger/Tess-27B-RFA" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-27B-RFA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tcclaviger/Tess-27B-RFA with Docker Model Runner:
```
docker model run hf.co/tcclaviger/Tess-27B-RFA
```

Tess-27B-RFA / README.md

tcclaviger

Rebaseline base Qwen3.6-27B column to comparable 2026-07-12 re-measurement

7c65698 3 days ago

preview code

Raw

History Blame Contribute Delete

7.4 kB

	---
	license: apache-2.0
	library_name: transformers
	pipeline_tag: image-text-to-text
	language:
	- en
	tags:
	- quantized
	- rfa
	- 4-bit
	base_model:
	- migtissera/Tess-4-27B
	base_model_relation: quantized
	---

	> [!IMPORTANT]
	> ## All-RFA 4-bit quant of [migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)
	>
	> Runtime: requires [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) — an RDNA 4 (gfx12xx) vLLM image and the only build with the RFA kernels; no other vLLM build loads these weights. Not validated on any other hardware at this time.

	# Tess-27B-RFA

	Aggressive all-RFA 4-bit quantization of Tess-4-27B by Migel Tissera — an agentic,
	thinking-native finetune of Qwen3.6-27B. Unlike the sibling
	[Tess-27B-RFI](https://huggingface.co/tcclaviger/Tess-27B-RFI) (int8 attention),
	this build runs every attention path at 4-bit: full attention, GDN linear
	attention, and even the MTP speculative head's attention. All credit for the model
	to its author; this repo only changes the numerics.

	## Quantization by component

	- All attention (full self_attn + GDN linear attention) — 4-bit float weights (RFA): IQ4_NL non-linear grid, group size 16, asymmetric, Hadamard-16 rotation, block-float scales stored as int8 mantissa + int8 exponent.
	- MLP layers — same RFA 4-bit scheme.
	- MTP speculative-decode head — its attention projections are RFA 4-bit too; its fc and MLP stay bf16.
	- Kept in bf16 (not quantized) — vision encoder, GDN `in_proj_a`/`in_proj_b`, embeddings, norms, and the lm_head.

	Result: 20.5 GB (vs 55.6 GB bf16, 28.9 GB RFI) at essentially zero measured quality cost.

	## Serving context — 512K via YaRN

	All evaluation below was run while serving at `--max-model-len 524288` (512K tokens),
	extended from the native 256K window with YaRN via `--hf-overrides`:

	```json
	{"text_config": {"rope_parameters": {"rope_type": "yarn", "factor": 2.0,
	"original_max_position_embeddings": 262144, "mrope_interleaved": true,
	"mrope_section": [11, 11, 10], "partial_rotary_factor": 0.25,
	"rope_theta": 10000000}}}
	```

	## Evaluation results

	Six builds measured with identical methodology, each against its own live vLLM endpoint (July 2026):
	Qwen3.6-27B (bf16 base) → Tess-4-27B (the tune, bf16) → Tess-27B-RFI (int8+4-bit hybrid) →
	Tess-27B-RFA (this build, all-4-bit) → Tess-FP8 (W8A8 block-128 FP8 sibling),
	with Qwen3.6-35B-A3B (MoE, bf16) as a comparative reference.
	Bold marks the best score in each row (ties all bolded).

	### Quantization cost (all-RFA 4-bit) — Tess-4-27B → Tess-27B-RFA

	\| Metric \| Change \|
	\|---\|---\|
	\| Checkpoint size \| 55.6 → 20.5 GB (−63%) \|
	\| WikiText-2 perplexity \| 6.669 → 6.629 (−0.6%, better) \|
	\| Codeneedle overall recall \| 97.7% → 98.1% (best of family) \|
	\| MC accuracy (4 tasks) \| ≈ flat (−0.6 to +1.3 pp) \|
	\| Tool-eval / GSM8K / MMLU / IFEval \| 86 / 98% / 80% / 95% \|
	\| MTP acceptance length \| ~2.9 → ~2.8 (≈ flat) \|
	\| Decode @ conc 1 (ISL 128) \| 59.9 → 67.9 tok/s (+13%) \|
	\| Decode @ conc 50 (ISL 128) \| 528 → 423 tok/s (−20%) \|

	### Quality

	\| Metric \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| WikiText-2 PPL (n_ctx 2048, lower is better) \| 7.0559 \| 6.6691 \| 6.6632 \| 6.6292 \| 6.6627 \| 6.5092 \|
	\| ARC-Challenge (acc_norm) \| 59.30% \| 60.84% \| 60.41% \| 60.32% \| 60.49% \| 55.20% \|
	\| ARC-Easy (acc_norm) \| 75.93% \| 77.53% \| 77.40% \| 78.87% \| 77.82% \| 71.13% \|
	\| Winogrande (acc) \| 77.51% \| 77.43% \| 77.51% \| 76.80% \| 77.66% \| 73.40% \|
	\| HellaSwag (acc_norm) \| 84.12% \| 84.21% \| 84.27% \| 84.05% \| 84.13% \| 82.95% \|

	Multiple-choice accuracy is lm-eval loglikelihood scoring, 0-shot.

	### Long-context positional recall (codeneedle)

	Verbatim function recall under 10K–80K-token contexts.

	\| Corpus \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| Python \| 100% \| 100% \| 100% \| 100% \| 99.55% \| 99.09% \|
	\| C++ \| 98.12% \| 98.12% \| 98.44% \| 98.75% \| 98.75% \| 98.44% \|
	\| Rust \| 99.69% \| 99.69% \| 99.69% \| 99.69% \| 99.69% \| 99.38% \|
	\| JS (~80K tokens) \| 93.44% \| 93.13% \| 93.75% \| 93.75% \| 93.44% \| 92.19% \|
	\| Tools \| 98.26% \| 99.57% \| 99.57% \| 99.57% \| 99.57% \| 93.48% \|
	\| Overall recall \| 97.81% \| 97.73% \| 97.97% \| 98.05% \| 97.86% \| 97.28% \|

	### Tool calling & accuracy benches

	\| Bench \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| tool-eval final (full 69, TC-61 excl) \| 86 \| 85 \| 87 \| 86 \| 87 \| 90 \|
	\| GSM8K (50q) \| 98.0% \| 94.0% \| 98.0% \| 98.0% \| 98.0% \| 96.0% \|
	\| MMLU (50q) \| 74.0% \| 76.0% \| 82.0% \| 80.0% \| 76.0% \| 64.0% \|
	\| IFEval (20 prompts, prompt-level) \| 90.0% \| 90.0% \| 90.0% \| 95.0% \| 90.0% \| 90.0% \|

	### Decode throughput — tok/s output (ISL 128 / ISL 512)

	`vllm bench serve`, random dataset, OSL 128, saturation, 4× R9700 (gfx1201), TP 4.

	\| Concurrency \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 1 \| 57.0 / 61.4 \| 59.9 / 63.8 \| 75.3 / 69.3 \| 67.9 / 58.2 \| 86.0 / 85.1 \| 91.9 / 114.4 \|
	\| 10 \| 304.5 / 233.6 \| 289.6 / 227.8 \| 281.5 / 219.2 \| 292.2 / 194.8 \| 280.6 / 269.2 \| 434.9 / 440.3 \|
	\| 25 \| 424.4 / 321.7 \| 492.0 / 341.1 \| 429.1 / 284.0 \| 369.0 / 260.3 \| 533.2 / 355.4 \| 688.9 / 563.7 \|
	\| 50 \| 556.0 / 349.8 \| 527.9 / 345.6 \| 452.8 / 295.0 \| 422.6 / 278.4 \| 560.3 / 394.1 \| 889.8 / 702.9 \|

	### MTP draft acceptance by work category

	Measured from live serving logs, k=5 draft tokens, drafted-token-weighted aggregation.

	\| Work category \| Overall acceptance \| Pos 1 \| Pos 2 \| Pos 3 \| Pos 4 \| Pos 5 \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| JSON generation \| 87.3% \| 96.4% \| 92.2% \| 86.8% \| 82.9% \| 78.0% \|
	\| Math \| 78.2% \| 95.6% \| 87.7% \| 78.6% \| 68.8% \| 60.2% \|
	\| Code \| 63.4% \| 89.4% \| 74.9% \| 60.7% \| 49.8% \| 41.9% \|
	\| Creative English \| 63.1% \| 88.0% \| 73.3% \| 61.6% \| 50.1% \| 42.3% \|

	## Notes

	All builds serve on the [tcclaviger/vllm:latest](https://hub.docker.com/r/tcclaviger/vllm) image, which has kernel tunes baked in.
	TunableOp is untuned — GEMMs run on default heuristic-determined values.
	Base Qwen3.6-27B figures are the 2026-07-12 re-measurement on the same tcclaviger/vllm:latest image and thinking-OFF methodology as every other build, replacing an earlier non-comparable run.

	## Credits

	- Tess-4-27B by Migel Tissera ([migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)) — the model quantized here:

	```bibtex
	@misc{tissera2026tess4,
	title = {Tess-4-27B},
	author = {Migel Tissera},
	year = {2026},
	howpublished = {\url{https://huggingface.co/migtissera/Tess-4-27B}}
	}
	```

	- codeneedle (positional recall) originally by Alexander Ziskind, expanded test suite by tcclaviger (Rob Smith).
	- Tool-calling scenarios (incl. TC-61) run on tool-eval-bench by SeraphimSerapis (Tim Messerschmidt), scenario methodology adapted from ToolCall-15 by stevibe.
	- GSM8K, MMLU, and IFEval run via tool-eval-bench's built-in accuracy benchmarks at their defaults: GSM8K 8-shot CoT, MMLU 5-shot, IFEval zero-shot.