P4pps3n/BTL-4-EXL3-8bpw

BTL-4 EXL3 8.00 bpw

A quality-first EXL3 quantization of badtheorylabs/BTL-4, retaining the complete text architecture and multimodal vision component.

This release targets ExLlamaV3 and TabbyAPI. It was converted at the maximum nominal EXL3 bitrate supported by the converter, with the input embedding, output head, and vision tower protected at 16-bit precision.

Model details

• Architecture: Qwen3.5 MoE multimodal conditional generation
• Logical parameters: 35,107,181,936
• Active parameters: approximately 3B per token
• Hidden size: 2,048
• Vocabulary size: 248,320
• Transformer layers: 40
• Full-attention layers: 10
• Recurrent Gated DeltaNet layers: 30
• Maximum context: 262,144 tokens
• Vision parameters: 446,571,248
• Native MTP layers: 0
• Upstream revision: b46d88b3fc26062086e303e4f9d34f593c9470b4
• License: Apache-2.0, inherited from the upstream model

The Hugging Face generic tensor viewer may miscount this checkpoint because EXL3 uses packed custom tensors. The logical parameter total above comes from the complete upstream tensor inventory; no language or vision layers were removed.

Quantization details

Property	Value
EXL3 nominal bitrate	8.00 bpw
Codebook	mul1
Output scales	Always enabled
Input embedding	BF16
Output head	Unquantized FP16 via -hb 16
Vision tensors	223 BF16 and 110 F32 tensors
Calibration rows	768
Calibration columns	4,096
Calibration tokens	3,145,728
Calibration seed	20260810
ExLlamaV3 format version	1.4.1
Indexed tensor bytes	36,757,084,872
Package size	34.32 GiB

Calibration emphasized code, debugging, tests, tool calls, structured data, technical material, reasoning and mathematics, multilingual technical text, general instruction data, and reference material. Construction used fixed revisions, document-hash deduplication, deterministic packing, and the actual BTL-4 tokenizer and chat template.

Measured benchmark

These are measured deployment results, not vendor estimates.

Hardware and runtime:

• 2x NVIDIA RTX 3090 24 GB
• No active NVLink
• ExLlamaV3 1.4.1 through TabbyAPI
• Layer autosplit, tensor parallelism disabled
• 262,144-token FP16 KV cache
• Four request slots
• Vision enabled
• CPU MoE offload disabled
• Speculative decoding disabled

Workload	Result
Four concurrent generations, 512 output tokens each	18.925 s wall time
Aggregate four-slot decode throughput	108.216 tokens/s
Approximate per-request throughput at four slots	27.211 tokens/s
Minimum free VRAM during four-slot test	3,029 / 4,565 MiB
214,680-token prefill	1,470.31 tokens/s
248,013-token prefill	1,349.22 tokens/s
Decode after 248K prefill	63.46 tokens/s
Minimum free VRAM during 248K test	2,767 / 4,271 MiB
Vision test generation	104.76 tokens/s
Reasoning test generation	107.43 tokens/s

Functional acceptance:

• Four concurrent request slots: pass
• Exact structured JSON: pass
• Parsed Qwen tool call: pass
• Coding smoke test: 4/4 cases passed
• Arithmetic reasoning smoke test: pass
• 248K middle-needle retrieval: exact match
• Vision shape recognition and OCR: exact match

Throughput varies with prompt length, output length, sampler, slot occupancy, GPU topology, driver, and TabbyAPI/ExLlamaV3 versions. The functional checks above are focused deployment smoke tests, not a replacement for standardized academic evaluation suites.

Download

hf download P4pps3n/BTL-4-EXL3-8bpw \
  --local-dir /path/to/models/BTL-4-EXL3-8bpw

TabbyAPI setup

Use a current TabbyAPI checkout with ExLlamaV3 1.4.1 or a compatible newer release. A tested example is included as tabby_config.yml.

Edit only the model path and network/auth settings for your environment:

model:
  model_dir: /path/to/models
  model_name: BTL-4-EXL3-8bpw
  backend: exllamav3

  max_seq_len: 262144
  cache_size: 262144
  cache_mode: FP16
  max_batch_size: 4

  tensor_parallel: false
  gpu_split_auto: true
  autosplit_reserve: [2048, 2048]
  cpu_moe_offload_layers: 0

  vision: true
  reasoning: true
  tool_format: qwen3_coder

Select GPUs before starting TabbyAPI. UUIDs are safer than unstable numeric indices:

export CUDA_DEVICE_ORDER=FASTEST_FIRST
export CUDA_VISIBLE_DEVICES="GPU_UUID_1,GPU_UUID_2"
python /path/to/tabbyAPI/main.py --config /path/to/tabby_config.yml

Check the server:

curl -fsS http://127.0.0.1:5004/health
curl -fsS http://127.0.0.1:5004/v1/model/list

The tested 256K FP16, four-slot profile uses roughly 41 GB across both cards. For systems with less aggregate VRAM, reduce max_batch_size before reducing model bitrate, context length, or cache precision. If necessary, test Q8 cache as a separate capacity profile.

OpenAI-compatible usage

Text example:

curl http://127.0.0.1:5004/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "BTL-4-EXL3-8bpw",
    "messages": [{"role": "user", "content": "Explain delta networks."}],
    "template_vars": {"enable_thinking": false},
    "temperature": 0.2,
    "max_tokens": 512
  }'

Thinking can be enabled per request:

"template_vars": {"enable_thinking": true}

For tool calling, provide OpenAI-compatible tools and use TabbyAPI's qwen3_coder tool parser. The model's chat template emits XML tool blocks; TabbyAPI converts them into standard tool_calls responses.

Vision example in Python:

import base64
import requests

with open("image.png", "rb") as f:
    image = base64.b64encode(f.read()).decode("ascii")

payload = {
    "model": "BTL-4-EXL3-8bpw",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image."},
            {
                "type": "image_url",
                "image_url": {"url": "data:image/png;base64," + image},
            },
        ],
    }],
    "template_vars": {"enable_thinking": False},
    "max_tokens": 512,
}

response = requests.post(
    "http://127.0.0.1:5004/v1/chat/completions",
    json=payload,
    timeout=300,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"])

Image embeddings consume tokens from the same shared context pool.

Notes and limitations

• This is an EXL3 checkpoint and requires an ExLlamaV3-compatible runtime.
• The full vision component is retained; preprocessor_config.json is required.
• There is no native MTP component in BTL-4.
• No speculative draft model is included.
• Native tensor parallelism was not selected because the tested GPUs had no active NVLink; layer autosplit produced strong throughput and stable output.
• -hb 16 is stored as unquantized FP16 by this ExLlamaV3 converter, not BF16.
• The 256K FP16 profile was validated on two 24 GB GPUs. Smaller systems must tune cache precision, slot count, context, or model placement.

Acknowledgements

• Base model: badtheorylabs/BTL-4
• Quantization/runtime: ExLlamaV3
• Serving: TabbyAPI