Instructions to use axteam/GLM-4.7-Flash-NVFP4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use axteam/GLM-4.7-Flash-NVFP4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="axteam/GLM-4.7-Flash-NVFP4A16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("axteam/GLM-4.7-Flash-NVFP4A16")
model = AutoModelForCausalLM.from_pretrained("axteam/GLM-4.7-Flash-NVFP4A16", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use axteam/GLM-4.7-Flash-NVFP4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "axteam/GLM-4.7-Flash-NVFP4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/axteam/GLM-4.7-Flash-NVFP4A16

SGLang

How to use axteam/GLM-4.7-Flash-NVFP4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "axteam/GLM-4.7-Flash-NVFP4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "axteam/GLM-4.7-Flash-NVFP4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "axteam/GLM-4.7-Flash-NVFP4A16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use axteam/GLM-4.7-Flash-NVFP4A16 with Docker Model Runner:
```
docker model run hf.co/axteam/GLM-4.7-Flash-NVFP4A16
```

GLM-4.7-Flash · NVFP4A16 (W4A16, FP4 weights)

4-bit quantization of zai-org/GLM-4.7-Flash for vLLM serving. Quantized by axteam.


Base model	`zai-org/GLM-4.7-Flash`
Method	NVFP4A16 (W4A16, FP4 weights)
Format	compressed-tensors `nvfp4-pack-quantized`
Weights / activations	W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations
Architecture	MoE (GLM-4.7 Lite · Glm4MoeLite) · 31.2B · 202K context · text-only
Size on disk	~18.9 GB
Calibration	None (data-free, name-based `model_free_ptq`)
License	mit

Quantization details

Scheme — W4A16 — FP4 (E2M1) weights group 16 + FP8 scale, BF16 activations, stored as compressed-tensors nvfp4-pack-quantized.
Kept in BF16 — token embeddings, the LM head, and all RMSNorms. Only attention & MLP projections are quantized. In addition, the MoE router (mlp.gate, incl. e_score_correction_bias) is kept in BF16. The MTP layer (model.layers.47: eh_proj, shared_head.head) is quantized like the other layers.
Post-training quantization (no retraining). Tokenizer / chat_template identical to the original.
FP4 global scales are shared across the pairs vLLM fuses — q_a_proj + kv_a_proj_with_mqa (MLA) and gate_proj + up_proj of every routed and shared expert.

Revision 2026-10-31: the first upload of this repo (same day) quantized those pairs with separate global scales. vLLM runs each pair as one GEMM with one scale, so that build produced corrupted text. The weights were re-quantized and replaced; if you downloaded this repo before the fix, please download it again.

Benchmarks (measured)

Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.

Benchmark	BF16 (original)	FP8-dynamic	NVFP4A16 (this repo)
MMLU (5-shot)	72.19	71.78 (99.4%)	69.97 (96.9%)
GSM8K (5-shot, strict)	83.70	82.71 (98.8%)	79.83 (95.4%)
ARC-Challenge (0-shot)	57.51	57.51 (100.0%)	55.03 (95.7%)
Winogrande (5-shot)	74.51	73.32 (98.4%)	73.40 (98.5%)
KMMLU (5-shot, direct)	50.04	50.15 (100.2%)	47.38 (94.7%)
HAE-RAE (0-shot)	64.89	63.89 (98.4%)	59.67 (91.9%)
Average	67.14	66.56 (99.1%)	64.21 (95.6%)

⚠️ Note: average recovery is 95.6% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.

Requirements


Runtime	vLLM — the architecture is in vLLM ≥ 0.15.0; tested here with vLLM 0.32.0.
vLLM install	vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately).
GPU	NVIDIA Turing (sm75) or newer — weight-only FP4, no FP4 tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.)
Memory	Weights: 18.9 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU.
Transformers (without vLLM)	Not loadable correctly by Transformers (checked up to 5.22.1) — use vLLM.
trust_remote_code	not needed

Usage (vLLM)

Start an OpenAI-compatible server:

vllm serve axteam/GLM-4.7-Flash-NVFP4A16 --max-model-len 32768

Query it:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/GLM-4.7-Flash-NVFP4A16",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "max_tokens": 512
}'

Offline inference in Python:

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/GLM-4.7-Flash-NVFP4A16", max_model_len=32768)
params = SamplingParams(temperature=1.0, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)

Recommended sampling (from the original generation_config.json): temperature=1.0. Thinking mode is on by default in the chat template; to turn it off pass chat_template_kwargs={"enable_thinking": False} (Python llm.chat(..., chat_template_kwargs=...), or "chat_template_kwargs": {"enable_thinking": false} in the request body).

Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

vllm serve axteam/GLM-4.7-Flash-NVFP4A16 --kv-cache-dtype fp8

License

Quantized derivative of zai-org/GLM-4.7-Flash (mit).

한국어 요약

zai-org/GLM-4.7-Flash의 NVFP4A16 (W4A16) 양자화 빌드 · 제작 axteam

개요

항목	내용
원본 모델	`zai-org/GLM-4.7-Flash`
양자화 방식	NVFP4A16 (W4A16, FP4 가중치)
저장 형식	compressed-tensors `nvfp4-pack-quantized`
모델 구조	MoE (GLM-4.7 Lite, `Glm4MoeLite`) · 31.2B · 컨텍스트 202K · 텍스트 전용
용량	약 18.9 GB
보정 데이터	없음 (가중치만으로 스케일 계산, 이름 기반 `model_free_ptq`)

양자화 상세

가중치: FP4(E2M1), 그룹 크기 16 + 그룹별 FP8 스케일 (아래 결합 쌍은 전역 스케일 공유)
활성화: BF16 (양자화하지 않음)
양자화 절차: PTQ(학습 후 양자화, 재학습 없음), 보정 데이터 없이 가중치만으로 스케일을 정해 반올림(RTN)
BF16 유지: 토큰 임베딩, LM 헤드, 모든 RMSNorm, MoE 라우터(mlp.gate, e_score_correction_bias 포함)
양자화 대상: 어텐션·MLP 선형층(projection) (MTP 층 model.layers.47도 다른 층과 같이 양자화)
전역 스케일 공유: vLLM이 한 GEMM으로 합쳐 계산하는 쌍(MLA q_a_proj+kv_a_proj_with_mqa, 모든 라우팅 대상 전문가(routed expert)·공유 전문가의 gate_proj+up_proj)은 FP4 전역 스케일 하나를 공유
원본 파일: 토크나이저·chat template은 원본과 동일

품질 (실측)

벤치마크 평균: 64.21 (원본 67.14, 95.6%) — MMLU·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 few-shot·greedy, 모든 빌드를 같은 설정으로 측정
한국어 벤치마크: KMMLU 47.38 (원본 50.04, 94.7%) · HAE-RAE 59.67 (원본 64.89, 91.9%)
주의: ⚠️ 이 정밀도에서 품질 저하가 뚜렷함 (벤치마크 평균 원본 대비 95.6%) — 품질이 중요하면 8비트(FP8·AWQ-8bit) 빌드 권장

요구사항

실행 엔진: vLLM 0.15.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인
vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
GPU: NVIDIA Turing(sm75) 이상, V100(sm70) 미지원 — 가중치 전용 FP4라 FP4 전용 텐서코어 불필요 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
메모리: 가중치 18.9 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면 --tensor-parallel-size N으로 분산
Transformers로 직접 사용: 불가 — 5.22.1까지 올바르게 로드되지 않음 확인, vLLM 사용
trust_remote_code: 불필요

사용법

서버 실행: vllm serve axteam/GLM-4.7-Flash-NVFP4A16 --max-model-len 32768 — OpenAI 호환 API
Python: LLM(model="axteam/GLM-4.7-Flash-NVFP4A16", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
권장 샘플링: temperature=1.0 (원본 generation_config.json 기준)
thinking 모드: 기본 켜짐 — 끄려면 chat_template_kwargs={"enable_thinking": False} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": false})
KV 캐시 절감(선택): vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)

참고

수정 이력 (2026-10-31): 처음 올린 빌드는 vLLM이 한 GEMM으로 계산하는 가중치 쌍의 FP4 전역 스케일이 서로 달라 출력이 깨졌음 — 다시 양자화해 교체, 이전에 받았다면 다시 받을 것

라이선스

라이선스: MIT — 원본 모델의 라이선스를 그대로 따름
권리: 원본 모델의 권리는 Z.ai에게 있음, 이 리포는 양자화 파생물

Downloads last month: 736

Safetensors

Model size

31B params

Tensor type

F32

BF16

Model tree for axteam/GLM-4.7-Flash-NVFP4A16

Base model

zai-org/GLM-4.7-Flash

Quantized

(111)

this model