Instructions to use QuantTrio/Kimi-K3-Cubic-2.5Bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="QuantTrio/Kimi-K3-Cubic-2.5Bit", trust_remote_code=True)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("QuantTrio/Kimi-K3-Cubic-2.5Bit", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/Kimi-K3-Cubic-2.5Bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/QuantTrio/Kimi-K3-Cubic-2.5Bit

SGLang

How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/Kimi-K3-Cubic-2.5Bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/Kimi-K3-Cubic-2.5Bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Kimi-K3-Cubic-2.5Bit",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use QuantTrio/Kimi-K3-Cubic-2.5Bit with Docker Model Runner:
```
docker model run hf.co/QuantTrio/Kimi-K3-Cubic-2.5Bit
```

Kimi-K3-Cubic-2.5Bit / cubic_quantization_report.json

JunHowie

Upload folder using huggingface_hub

fd703d8 verified 5 days ago

Raw

History Blame Contribute Delete

3.34 kB

	{
	"manifest": {
	"source": "moonshotai/Kimi-K3",
	"output": "Kimi-K3-Cubic-2.5Bit",
	"moe_schedule": "1-3:3@256,4-32:3@512,33-91:2@512,92:4@512",
	"linear_schedule": "",
	"conversion_hardware": {
	"device_type": "cuda",
	"device_count": 8
	},
	"shard_size_gib": 3.0,
	"a8_carrier_aware": true,
	"a8_correction_default": "enabled",
	"fitting_objective": "groupwise-least-squares",
	"reported_loss": "NRMSE = sqrt(joint SSE / source weight SSE)",
	"row_chunk_size": -1,
	"source_shards": 96,
	"expert_payload_bits": 2.369565217391304,
	"expert_effective_bits": 2.498641304347826,
	"converted_tensor_effective_bits": 2.498641304347826,
	"worker_scheduling": "dynamic_source_shard_queue",
	"output_partitioning": "deterministic_source_shard_index",
	"script": "quantize_k3.py",
	"script_sha256": "89fe8836bb407a027a2663637bc0380bd1adb29ffb2778f0879e606dc8ed31f3",
	"source_config_sha256": "9710e121a58d03ac92c8d6da287a19541994319afbbe6d6202af001ffd379213",
	"source_index_sha256": "a1c5210650ce71d2d3ae9ec5a101ac4afd3cf4b10091be589853437eb967febd",
	"output_shards": 341,
	"output_total_bytes": 964943224704,
	"combined_report": "cubic_quantization_report.json"
	},
	"loss_statistics": {
	"objective": "mean(continuous MSE, rounded-A8-carrier MSE)",
	"normalization": "loss is sqrt(joint SSE / source weight SSE), i.e. NRMSE; the fitting objective itself remains least squares.",
	"metadata_precision": "FP32 scale and FP16 a/b before recomputation",
	"by_bit": {
	"2": {
	"bits": 2,
	"loss": 0.4512510057287302,
	"clipped_percent": 12.230740958337455,
	"a8_correction_loss": 0.4512510057287302
	},
	"3": {
	"bits": 3,
	"loss": 0.18088323398973324,
	"clipped_percent": 2.966870055819044,
	"a8_correction_loss": 0.18091672638061443
	},
	"4": {
	"bits": 4,
	"loss": 0.0430471542191854,
	"clipped_percent": 0.5635432713156114,
	"a8_correction_loss": 0.04324462046872603
	}
	},
	"by_bit_and_group_size": {
	"2@512": {
	"bits": 2,
	"loss": 0.4512510057287302,
	"clipped_percent": 12.230740958337455,
	"a8_correction_loss": 0.4512510057287302,
	"group_size": 512
	},
	"3@256": {
	"bits": 3,
	"loss": 0.17960835023454785,
	"clipped_percent": 2.6970932295353056,
	"a8_correction_loss": 0.1795795260456478,
	"group_size": 256
	},
	"3@512": {
	"bits": 3,
	"loss": 0.18100745927884193,
	"clipped_percent": 2.994778003365638,
	"a8_correction_loss": 0.18104699922459405,
	"group_size": 512
	},
	"4@512": {
	"bits": 4,
	"loss": 0.0430471542191854,
	"clipped_percent": 0.5635432713156114,
	"a8_correction_loss": 0.04324462046872603,
	"group_size": 512
	}
	}
	},
	"audit": {
	"checkpoint": "Kimi-K3-Cubic-2.5Bit",
	"shards": 341,
	"tensors": 991812,
	"total_size": 964943224704,
	"max_shard_bytes": 3221225472,
	"widths_present": [
	2,
	3,
	4
	],
	"converted_tensor_effective_bits": 2.498641304347826,
	"dtype_counts": {
	"BF16": 2122,
	"F32": 247802,
	"F16": 494592,
	"U8": 247296
	}
	}
	}