Instructions to use RedHatAI/GLM-5.3-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use RedHatAI/GLM-5.3-NVFP4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="RedHatAI/GLM-5.3-NVFP4") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("RedHatAI/GLM-5.3-NVFP4") model = AutoModelForCausalLM.from_pretrained("RedHatAI/GLM-5.3-NVFP4", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use RedHatAI/GLM-5.3-NVFP4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "RedHatAI/GLM-5.3-NVFP4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/GLM-5.3-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/RedHatAI/GLM-5.3-NVFP4
- SGLang
How to use RedHatAI/GLM-5.3-NVFP4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "RedHatAI/GLM-5.3-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/GLM-5.3-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "RedHatAI/GLM-5.3-NVFP4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "RedHatAI/GLM-5.3-NVFP4", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use RedHatAI/GLM-5.3-NVFP4 with Docker Model Runner:
docker model run hf.co/RedHatAI/GLM-5.3-NVFP4
Dont use this model
We deployed RedHatAI/GLM-5.3-NVFP4 on vLLM (B300, 262K context) and found it unusable for production. On basic tasks the model often never reaches an answer: it keeps reasoning until the entire token budget is used up and returns no final output. The same tasks are routine for the base model, so this doesn’t look like normal GLM-5.3 behavior.
The model card only evaluates short-context math/QA benchmarks, and even there IFEval drops to 95.5% recovery. There is no evaluation of agentic, tool-calling, or long-context use, which is where we saw the failures.
Has anyone else seen runaway reasoning with this checkpoint? Would be good to see an evaluation of reasoning termination and agentic tasks against zai-org/GLM-5.3 FP8.
Hi @danbrima ,
Sorry to hear that this model is underperforming. What kind of basic tasks have you found that this model suffers on? We have found that the base model also has some wierd behavior sometimes, but regardless we are going to test this model in long-context and agentic evals to investigate.
Even on basic tasks, like making an edit in a moderately sized codebase, the model got stuck reasoning until it hit context compaction, several times in a row, without making a single tool call.