Add BaseRT Q4 + Q8 builds

d13290f verified 13 days ago

711 Bytes

license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: baseRT
tags:
  - baseRT
  - basecompute
  - apple-silicon

Qwen3-4B-Instruct-2507 (BaseRT)

BaseRT .base builds of Qwen/Qwen3-4B-Instruct-2507 for fast local inference on Apple Silicon.

File	Quant
`Qwen3-4B-Instruct-2507-Q4.base`	4-bit (default)
`Qwen3-4B-Instruct-2507-Q8.base`	8-bit

basert serve basecompute/Qwen3-4B-Instruct-2507                       # Q4 (default)
basert serve basecompute/Qwen3-4B-Instruct-2507 --variant default-q8     # Q8

Documentation: https://docs.basecompute.co