KIEFERSA/Sophea-Titan-1-FP8

Sophea-Titan-1-FP8

FP8_DYNAMIC (W8A8) quantization of Sophea-Titan-1

FP8_DYNAMIC (W8A8) quantization of KIEFERSA/Sophea-Titan-1 — the 27B Greek fine-tuned chat model built on Qwen3.6-27B. Same behaviour as the bf16 release, ≈0.55× the disk / VRAM, and verified lossless (numbers below).

• Base model: KIEFERSA/Sophea-Titan-1 (bf16)
• Scheme: FP8_DYNAMIC — per-channel FP8 (E4M3) weights + dynamic per-token FP8 activations (W8A8), compressed-tensors float-quantized
• Size: 29.4 GB (bf16 base ≈ 54 GB → 0.55×)
• Creator: Kiefer SA
• Decoding: non-thinking (enable_thinking=false) — see recommended sampling under Usage

What is / isn't quantized. Data-free one-shot PTQ via llm-compressor (FP8_DYNAMIC, no calibration). Quantized: the language-model Linear layers (attention q/k/v/o, MLP, GDN in/out projections). Kept bf16: lm_head. GDN conv1d/A_log/dt_bias/norms are non-Linear and untouched. This quantized build is text-only — the vision tower of the multimodal base (Sophea-Titan-1) is not included in this FP8 release.

Serve non-thinking. Set enable_thinking=false in the chat template. With thinking left on, Greek output quality degrades.

Verified lossless (this FP8 build)

Measured on this checkpoint through the vLLM decode path (greedy, non-thinking), vs the bf16 base:

Metric	bf16 (published)	FP8 (this build)	Δ
General Greek benchmarks macro (9, logprob)	0.7369	0.7373	+0.0004
English retention macro (5)	0.8783	0.8767	−0.0016
GreekMMLU (30-subj)	0.854	0.853	−0.001

Every axis is within measurement noise of bf16 → lossless. FP8 also loads in HF transformers.

Per-benchmark detail (measured on this FP8 build)

General Greek benchmarks (9, accuracy)

benchmark	FP8	bf16
arc_challenge	0.9486	0.950
arc_easy	0.9731	0.973
belebele	0.9511	0.950
greekmmlu	0.8531	0.854
hellaswag	0.6794	0.680
medical_mcqa	0.3912	0.387
truthfulqa	0.4125	0.415
winogrande	0.6290	0.626
mmlu_greek	0.7980	0.797
MACRO	0.7373	0.7369

English retention — 5 benchmarks (accuracy)

benchmark	FP8	bf16
arc_challenge	0.9770	0.979
arc_easy	0.9920	0.992
hellaswag	0.8030	0.805
mmlu	0.8554	0.858
winogrande	0.7561	0.758
MACRO	0.8767	0.8783

greekmmlu — per-subject (accuracy)

subject	FP8	subject	FP8
Accounting	0.859	Greek Traditions	0.886
Agriculture	0.875	Law	0.700
Art	0.753	Management	0.838
Biology	0.868	Maritime Safety & Rescue	0.669
Chemistry	0.802	Mathematics	0.917
Civil Engineering	0.822	Medicine	0.901
Clinical Knowledge	0.820	Modern Greek Language	0.920
Computer Networks & Security	0.762	Physics	0.849
Computer Science	0.880	Prehistory	1.000
Driving Rules	0.821	World History	0.950
Economics	0.922	World Religions	0.774
Education	0.867	Geography	0.964
Electrical Engineering	0.817	Government and Politics	0.932
General Knowledge	0.789	Greek History	0.874
Greek Literature	0.714	Greek Mythology	0.845
OVERALL	0.853

Usage

Serve with vLLM (compressed-tensors FP8 auto-detected; Cutlass FP8 kernels on Blackwell/Hopper):

vllm serve KIEFERSA/Sophea-Titan-1-FP8 --served-model-name sophea-titan-1-fp8 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

Serve with vision enabled — this is a multimodal checkpoint (Qwen3_5ForConditionalGeneration). The vision tower is excluded from quantization and kept in bf16, so image input works unchanged:

vllm serve KIEFERSA/Sophea-Titan-1-FP8 --served-model-name sophea-titan-1-fp8 --trust-remote-code \
  --limit-mm-per-prompt '{"image": 4}' \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3

For text-only serving, add --language-model-only to skip loading the vision tower entirely.

resp = client.chat.completions.create(
    model="sophea-titan-1-fp8",
    messages=[{"role": "user", "content": [
        {"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
        {"type": "text", "text": "Περίγραψε την εικόνα στα ελληνικά."},
    ]}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},   # required: non-thinking
)
print(resp.choices[0].message.content)

Recommended sampling (instruct / non-thinking): temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0.

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="sophea-titan-1-fp8",
    messages=[{"role": "user", "content": "Ποια είναι η πρωτεύουσα της Ελλάδας;"}],
    temperature=0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},   # required: non-thinking
)
print(resp.choices[0].message.content)

Speculative decoding (MTP)

This build ships the model's multi-token-prediction head — 15 mtp.* tensors (~0.85 GB, bf16) in model-mtp.safetensors. This is the single-layer draft stack that config.json has always declared through mtp_num_hidden_layers: 1.

Earlier revisions of this repo did not contain it. The LoRA merge loaded the base through AutoModelForCausalLM, and transformers declares _keys_to_ignore_on_load_unexpected = [r"^mtp.*"] for this architecture, so the head was discarded at load and never written back — and llm-compressor dropped it again during quantization. The config therefore advertised a module the weights did not contain. To pin the previous bytes, use revision="11380ae677053cff0deab6cb8ae17d192611924a".

The MTP Linears are held in bf16 and listed in quantization_config.ignore, so the compressed-tensors loader treats that block as unquantized rather than expecting packed weights.

Provenance. These are the base Qwen3.6-27B MTP weights. LoRA never targeted mtp.*, so no fine-tuned draft head exists. This cannot affect output quality: speculative decoding verifies every drafted token against the main model, so a stale drafter changes throughput only, never the output distribution.

Enable it with vLLM (≥ 0.23.0):

vllm serve KIEFERSA/Sophea-Titan-1-FP8 --served-model-name sophea-titan-1-fp8 --trust-remote-code \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'

The head is loaded but stays inactive unless --speculative-config is passed, so existing serve commands are unaffected.

License

Inherits the Qwen3.6-27B base-model license. Verify base-model terms before use.