fable-traces-NVFP4

This is a quantization of a joke model. The upstream model card explicitly says: "This is a joke. This is not an actual model." Do not use this checkpoint for real reasoning, Japanese QA, coding, evaluation, production, or any task where correctness matters.

NVFP4 W4A4 compressed-tensors quantization of AliesTaha/fable-traces, a dense Qwen3 4B causal language model.

This repository is meant only as a Blackwell/vLLM/NVFP4 format experiment. It preserves the upstream joke behavior and only changes the tensor format. Quick Japanese reasoning checks failed badly, often drifting into the same unrelated Egypt/Australia line, so treat the benchmark numbers below as throughput tests, not evidence of model quality.

Practical Warning

• Not a serious instruction model
• Not reliable for Japanese reasoning
• Not reliable for arithmetic, logic, coding, or factual answers
• Useful only for testing NVFP4 loading, serving, KV-cache behavior, and raw TPS

Quantization

• Source precision: BF16 safetensors
• Output format: compressed-tensors
• NVFP4 format: nvfp4-pack-quantized
• Scheme: NVFP4 W4A4
• Targets: Linear
• Ignored modules: lm_head
• Calibration: neuralmagic/calibration, LLM, 512 samples, max sequence 4096
• Tooling: llmcompressor, compressed-tensors

The resulting model.safetensors is about 2.82 GB.

vLLM Usage

For a 128K context run with FP8 KV cache:

vllm serve sakamakismile/fable-traces-NVFP4 \
  --max-model-len 131072 \
  --kv-cache-dtype fp8 \
  --gpu-memory-utilization 0.92

For Python:

from vllm import LLM, SamplingParams

llm = LLM(
    model="sakamakismile/fable-traces-NVFP4",
    max_model_len=131072,
    kv_cache_dtype="fp8",
    gpu_memory_utilization=0.92,
)

messages = [[{"role": "user", "content": "Tell me something interesting."}]]
outputs = llm.chat(messages, SamplingParams(max_tokens=128, temperature=0.0))
print(outputs[0].outputs[0].text)

Benchmark

Measured locally with vLLM 0.21.0+cu129 on NVIDIA RTX PRO 2000 Blackwell 16 GB GPUs, driver 595.71.05.

Common settings:

• max_model_len=131072
• kv_cache_dtype=fp8
• max_num_seqs=8
• max_tokens=256
• repeats=3, warmup=1
• enforce_eager=True

TP=1

vLLM reported 156,576 GPU KV-cache tokens with FP8 KV cache.

TP	Parallel	Mean output tok/s	Best output tok/s	Mean total tok/s
1	1	69.03	71.45	118.07
1	2	129.30	131.40	219.80
1	4	238.04	244.81	404.32
1	8	449.17	461.41	770.48

TP=2

vLLM reported 359,488 GPU KV-cache tokens with FP8 KV cache.

TP	Parallel	Mean output tok/s	Best output tok/s	Mean total tok/s
2	1	60.37	60.72	103.26
2	2	100.86	122.85	166.75
2	4	234.43	236.21	388.02
2	8	360.84	415.11	613.43

For this short-output benchmark, TP=1 is faster; TP=2 mainly buys more KV-cache headroom for long-context use.

Raw benchmark files are included under benchmarks/.