Instructions to use tcclaviger/Tess-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tcclaviger/Tess-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tcclaviger/Tess-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tcclaviger/Tess-FP8")
model = AutoModelForMultimodalLM.from_pretrained("tcclaviger/Tess-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tcclaviger/Tess-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tcclaviger/Tess-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tcclaviger/Tess-FP8

SGLang

How to use tcclaviger/Tess-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tcclaviger/Tess-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tcclaviger/Tess-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tcclaviger/Tess-FP8 with Docker Model Runner:
```
docker model run hf.co/tcclaviger/Tess-FP8
```

Tess-FP8 / README.md

tcclaviger

Tess-FP8: W8A8 block-128 FP8 quant of Tess-4-27B

fa52a89 3 days ago

preview code

Raw

History Blame Contribute Delete

7.43 kB

	---
	license: apache-2.0
	library_name: transformers
	pipeline_tag: image-text-to-text
	language:
	- en
	tags:
	- quantized
	- fp8
	- w8a8
	base_model:
	- migtissera/Tess-4-27B
	base_model_relation: quantized
	---

	> [!IMPORTANT]
	> ## W8A8 block-128 FP8 quant of [migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)
	>
	> Runtime: standard vLLM `fp8` quant_method (e4m3, `weight_block_size [128, 128]`, dynamic activations). All evaluation below was run on [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) on RDNA 4 (gfx12xx). Not validated on any other hardware at this time.

	# Tess-FP8

	Block-wise W8A8 FP8 (e4m3fn) quantization of Tess-4-27B by Migel Tissera — an
	agentic, thinking-native finetune of Qwen3.6-27B. Sibling quants:
	[Tess-27B-RFI](https://huggingface.co/tcclaviger/Tess-27B-RFI) (int8+4-bit hybrid) and
	[Tess-27B-RFA](https://huggingface.co/tcclaviger/Tess-27B-RFA) (all-attention 4-bit).
	All credit for the model to its author; this repo only changes the numerics.

	## Quantization by component

	- All attention and MLP linear weights (self_attn q/k/v/o, MLP gate/up/down, GDN linear-attention in_proj_qkv / in_proj_z / out_proj) — FP8 e4m3fn, block-wise 128×128, per-block bf16 dequant scales (`weight_scale_inv`), dynamic activation scheme (W8A8).
	- MTP speculative-decode head — its MLP and attention projections are FP8 by the same scheme; its fc and norms stay bf16.
	- Kept in bf16 (not quantized) — vision encoder, GDN `A_log`/`conv1d`/`dt_bias`/`in_proj_a`/`in_proj_b`, embeddings, norms, and the lm_head.

	Tensor layout and naming match the Qwen3.6-27B-FP8 production reference exactly.

	## Serving context — 512K via YaRN

	All evaluation below was run while serving at `--max-model-len 524288` (512K tokens),
	extended from the native 256K window with YaRN via `--hf-overrides`:

	```json
	{"text_config": {"rope_parameters": {"rope_type": "yarn", "factor": 2.0,
	"original_max_position_embeddings": 262144, "mrope_interleaved": true,
	"mrope_section": [11, 11, 10], "partial_rotary_factor": 0.25,
	"rope_theta": 10000000}}}
	```

	## Evaluation results

	Six builds measured with identical methodology, each against its own live vLLM endpoint (July 2026):
	Qwen3.6-27B (bf16 base) → Tess-4-27B (the tune, bf16) →
	[Tess-27B-RFI](https://huggingface.co/tcclaviger/Tess-27B-RFI) (int8+4-bit hybrid) →
	[Tess-27B-RFA](https://huggingface.co/tcclaviger/Tess-27B-RFA) (all-4-bit) →
	Tess-FP8 (this build, W8A8 block-128 FP8),
	with Qwen3.6-35B-A3B (MoE, bf16) as a comparative reference.
	Bold marks the best score in each row (ties all bolded).

	### Quantization cost (W8A8 FP8 block-128) — Tess-4-27B → Tess-FP8

	\| Metric \| Change \|
	\|---\|---\|
	\| Checkpoint size \| 55.6 → 29.0 GB (−48%) \|
	\| WikiText-2 perplexity \| 6.669 → 6.663 (≈ flat) \|
	\| Codeneedle overall recall \| 97.7% → 97.9% (≈ flat) \|
	\| MC accuracy (4 tasks) \| ≈ flat (−0.4 to +0.3 pp) \|
	\| Tool-eval / GSM8K / MMLU / IFEval \| 87 / 98% / 76% / 90% \|
	\| Decode @ conc 1 (ISL 128) \| 59.9 → 86.0 tok/s (+44%) \|
	\| Decode @ conc 50 (ISL 128) \| 528 → 560 tok/s (+6%) \|

	### Quality

	\| Metric \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| WikiText-2 PPL (n_ctx 2048, lower is better) \| 7.0559 \| 6.6691 \| 6.6632 \| 6.6292 \| 6.6627 \| 6.5092 \|
	\| ARC-Challenge (acc_norm) \| 59.30% \| 60.84% \| 60.41% \| 60.32% \| 60.49% \| 55.20% \|
	\| ARC-Easy (acc_norm) \| 75.93% \| 77.53% \| 77.40% \| 78.87% \| 77.82% \| 71.13% \|
	\| Winogrande (acc) \| 77.51% \| 77.43% \| 77.51% \| 76.80% \| 77.66% \| 73.40% \|
	\| HellaSwag (acc_norm) \| 84.12% \| 84.21% \| 84.27% \| 84.05% \| 84.13% \| 82.95% \|

	Multiple-choice accuracy is lm-eval loglikelihood scoring, 0-shot.

	### Long-context positional recall (codeneedle)

	Verbatim function recall under 10K–80K-token contexts.

	\| Corpus \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| Python \| 100% \| 100% \| 100% \| 100% \| 99.55% \| 99.09% \|
	\| C++ \| 98.12% \| 98.12% \| 98.44% \| 98.75% \| 98.75% \| 98.44% \|
	\| Rust \| 99.69% \| 99.69% \| 99.69% \| 99.69% \| 99.69% \| 99.38% \|
	\| JS (~80K tokens) \| 93.44% \| 93.13% \| 93.75% \| 93.75% \| 93.44% \| 92.19% \|
	\| Tools \| 98.26% \| 99.57% \| 99.57% \| 99.57% \| 99.57% \| 93.48% \|
	\| Overall recall \| 97.81% \| 97.73% \| 97.97% \| 98.05% \| 97.86% \| 97.28% \|

	### Tool calling & accuracy benches

	\| Bench \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| tool-eval final (full 69, TC-61 excl) \| 86 \| 85 \| 87 \| 86 \| 87 \| 90 \|
	\| GSM8K (50q) \| 98.0% \| 94.0% \| 98.0% \| 98.0% \| 98.0% \| 96.0% \|
	\| MMLU (50q) \| 74.0% \| 76.0% \| 82.0% \| 80.0% \| 76.0% \| 64.0% \|
	\| IFEval (20 prompts, prompt-level) \| 90.0% \| 90.0% \| 90.0% \| 95.0% \| 90.0% \| 90.0% \|

	### Decode throughput — tok/s output (ISL 128 / ISL 512)

	`vllm bench serve`, random dataset, OSL 128, saturation, 4× R9700 (gfx1201), TP 4.

	\| Concurrency \| Qwen3.6-27B (base) \| Tess-4-27B \| Tess-27B-RFI \| Tess-27B-RFA \| Tess-FP8 \| Qwen3.6-35B-A3B \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| 1 \| 57.0 / 61.4 \| 59.9 / 63.8 \| 75.3 / 69.3 \| 67.9 / 58.2 \| 86.0 / 85.1 \| 91.9 / 114.4 \|
	\| 10 \| 304.5 / 233.6 \| 289.6 / 227.8 \| 281.5 / 219.2 \| 292.2 / 194.8 \| 280.6 / 269.2 \| 434.9 / 440.3 \|
	\| 25 \| 424.4 / 321.7 \| 492.0 / 341.1 \| 429.1 / 284.0 \| 369.0 / 260.3 \| 533.2 / 355.4 \| 688.9 / 563.7 \|
	\| 50 \| 556.0 / 349.8 \| 527.9 / 345.6 \| 452.8 / 295.0 \| 422.6 / 278.4 \| 560.3 / 394.1 \| 889.8 / 702.9 \|

	### MTP draft acceptance by work category

	Measured from live serving logs, k=5 draft tokens, drafted-token-weighted aggregation.

	\| Work category \| Overall acceptance \| Pos 1 \| Pos 2 \| Pos 3 \| Pos 4 \| Pos 5 \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| JSON generation \| 96.7% \| 99.4% \| 98.8% \| 98.0% \| 94.9% \| 92.5% \|
	\| Math \| 79.9% \| 96.2% \| 88.5% \| 79.9% \| 70.9% \| 64.2% \|
	\| Code \| 63.8% \| 89.5% \| 74.4% \| 62.0% \| 51.2% \| 42.0% \|
	\| Creative English \| 62.1% \| 88.7% \| 72.9% \| 59.8% \| 48.5% \| 40.7% \|

	## Notes

	All builds serve on the [tcclaviger/vllm:latest](https://hub.docker.com/r/tcclaviger/vllm) image, which has kernel tunes baked in.
	TunableOp is untuned — GEMMs run on default heuristic-determined values.
	Base Qwen3.6-27B figures are the 2026-07-12 re-measurement on the same tcclaviger/vllm:latest image and thinking-OFF methodology as every other build, replacing an earlier non-comparable run.

	## Credits

	- Tess-4-27B by Migel Tissera ([migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)) — the model quantized here:

	```bibtex
	@misc{tissera2026tess4,
	title = {Tess-4-27B},
	author = {Migel Tissera},
	year = {2026},
	howpublished = {\url{https://huggingface.co/migtissera/Tess-4-27B}}
	}
	```

	- codeneedle (positional recall) originally by Alexander Ziskind, expanded test suite by tcclaviger (Rob Smith).
	- Tool-calling scenarios (incl. TC-61) run on tool-eval-bench by SeraphimSerapis (Tim Messerschmidt), scenario methodology adapted from ToolCall-15 by stevibe.
	- GSM8K, MMLU, and IFEval run via tool-eval-bench's built-in accuracy benchmarks at their defaults: GSM8K 8-shot CoT, MMLU 5-shot, IFEval zero-shot.