Instructions to use Nanbeige/Nanbeige4.2-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nanbeige/Nanbeige4.2-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nanbeige/Nanbeige4.2-3B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Nanbeige/Nanbeige4.2-3B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nanbeige/Nanbeige4.2-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.2-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nanbeige/Nanbeige4.2-3B

SGLang

How to use Nanbeige/Nanbeige4.2-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.2-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nanbeige/Nanbeige4.2-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nanbeige/Nanbeige4.2-3B with Docker Model Runner:
```
docker model run hf.co/Nanbeige/Nanbeige4.2-3B
```

Issue: Grammar-constrained decoding (`response_format: json_schema`) fails at sampler init — but `--no-jinja` fixes it

by dmhagar - opened 9 days ago

Discussion

dmhagar

9 days ago

Grammar-constrained decoding (`response_format: json_schema`) fails at sampler init — but `--no-jinja` fixes it

First, thanks for shipping this — the looped-transformer 3B is genuinely nice to run locally, and once I got past the issue below it produces clean, sensible structured output.

The problem. With the default jinja chat path, any request that uses a grammar / JSON-schema response_format fails at sampler initialization, before a single token is generated:

HTTP 400
{"error":{"code":400,"type":"invalid_request_error",
 "message":"Failed to initialize samplers: Unexpected empty grammar stack after accepting piece:  assistant (13886)"}}

Note the accepted piece is the assistant-turn marker token " assistant" (id 13886) — the grammar sampler is being fed a trigger derived from the chat template's assistant marker, and the grammar stack empties on it.

Reproduction (built from the nanbeige42 branch, git log -1 = "support nanbeige4.2 model"):

llama-server -m Nanbeige4.2-3B-Q4_K_M.gguf -c 4096 --port 8096   # jinja is default

curl -s http://127.0.0.1:8096/v1/chat/completions -H "content-type: application/json" -d '{
  "model":"nanbeige-4.2-3b",
  "messages":[{"role":"system","content":"Extract entities as strict JSON."},
              {"role":"user","content":"Kael the smith greets you at the Iron Forge in Dunhold."}],
  "response_format":{"type":"json_schema","json_schema":{"name":"extraction","strict":false,
    "schema":{"type":"object","properties":{"entities":{"type":"array","items":{"type":"object",
      "properties":{"name":{"type":"string"},"type":{"type":"string"}},"required":["name","type"]}}},
      "required":["entities"]}}},
  "max_tokens":300,"temperature":0
}'
# -> HTTP 400, "empty grammar stack after accepting piece:  assistant (13886)"

Plain (non-grammar) generation is unaffected — the model answers normally, and enable_thinking:false gives clean non-reasoning output.

The fix / workaround. Adding --no-jinja bypasses it completely. The identical grammar request then returns HTTP 200 with valid, correct JSON:

{"entities":[
  {"name":"Kael","type":"person"},
  {"name":"smith","type":"occupation"},
  {"name":"Iron Forge","type":"location"},
  {"name":"Dunhold","type":"location"}
]}

Because --no-jinja fixes it, the bug is isolated to the jinja chat path — specifically the auto-parser / grammar-trigger construction (common/chat-auto-parser-generator.cpp and the surrounding common/chat.cpp trigger handling). Something on that path installs a grammar trigger that resolves to the assistant marker token even for a plain json_schema request (no tools, tool_choice unset), and common_sampler_init then can't build the grammar. Overriding the surface chat template (--chat-template chatml) does not help — only disabling jinja does — which points at the trigger-generation logic rather than the template text itself.

Why it matters. This blocks the model as a backend for anything that relies on constrained decoding — JSON extraction, tool/function calling, or any grammar-guided output — under the default configuration. --no-jinja is a viable stopgap for pure schema-constrained JSON, but it also disables the reasoning/tool-call template handling, so it isn't a fix for the tool-calling case.

Happy to share the full server logs or test more configurations if useful. Thanks again for the model.

IingHu

4 days ago

Disclaimer: this investigation and write-up were carried out by an AI coding agent
working on my machine, under my direction. Findings are from real runs against a real
deployment, but wording, tables and analysis are the agent's. Sample sizes are small and
noted as such. Please treat the reasoning as a starting point rather than a verified
root-cause analysis.

I hit the same issue and can add a few data points that may help narrow it down.

Environment

llama.cpp d28da86 (MSVC, Windows AMD64), router/preset mode
Nanbeige4.2-3B-Q8_0.gguf, general.architecture=nanbeige (community quant — repo publishes no official GGUF)
16 GB GPU, -ngl 99, flash-attn on, kv-unified on, ctx-size 65536
Workload: JSON fact extraction, response_format: json_schema on every request

1. Reproduction

HTTP 400
Failed to initialize samplers: Unexpected empty grammar stack
after accepting piece:  assistant (13886)

curl -s http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model":"Nanbeige4.2-3B",
  "messages":[{"role":"user","content":"Extract facts from: A fixed the gateway."}],
  "response_format":{"type":"json_schema","json_schema":{"name":"f","strict":false,
    "schema":{"type":"object","properties":{"facts":{"type":"array","items":{"type":"string"}}},
    "required":["facts"]}}},
  "max_tokens":200}'

The failing token is assistant (13886) — the generation-prompt prefix, not <think>.

2. Does not help (all still 400)

attempt	result
`"strict": false` instead of `true`	400 — llama grammar-enforces on the presence of `json_schema`, the flag is irrelevant
`--chat-template chatml` (jinja still on)	400
`chat_template_kwargs: {"enable_thinking": false}`	400
`--reasoning off`	400
`--override-kv tokenizer.ggml.add_bos_token=bool:false`	400

3. Does work — and isolates the fault

attempt	result
`/completion` + `json_schema`, raw prompt, no chat template	✅ `{"facts":["A fixed the gateway."]}`
`/v1/chat/completions` + `--no-jinja`	✅ valid, schema-conformant JSON
plain chat, no `response_format`	✅ normal

The /completion result seems the most diagnostic: the model and the GBNF grammar are each
fine in isolation. Same schema compiled to a grammar, raw prompt, no template → clean
conformant output. The failure is confined to the chat-completions path, which is consistent
with the grammar-trigger construction hypothesis rather than anything in the template text
(swapping the template to chatml did not help; only leaving the jinja path did).

Possible link to discussion #17 (whitespace in trigger markers)

#17 reports dropped tool
calls caused by whitespace handling in trigger-marker construction in
common/chat-auto-parser-generator.cpp — until() looks for the exact literal
<tool_call>\n and misses it when a space precedes the newline.

That may be the same underlying defect as this issue. The token we fail on is
assistant — with a leading space (note the two spaces after piece: in the error, and
token 13886 in this vocab is space-prefixed). If the grammar trigger is built from an
unpadded literal while the tokenizer yields a space-prefixed piece, the trigger never
matches and the grammar stack empties — which is what the error reports.

Both symptoms would then be one cause — trigger markers constructed without accounting for
leading/trailing whitespace — differing only in what fails downstream: dropped tool calls in
#17, sampler-init failure here. #17 also notes the affected code path exists upstream and
touches other models, which would fit.

I have not verified this against the source; it's offered as a lead for whoever looks at the
trigger-generation code, since #17 already identifies a concrete two-line fix in that area.

Possibly related, same class:
llama.cpp#21600 (Qwen 3.5, fails on <think>, closed not planned),
#23775 (json_schema + chatml + router mode),
#19051 and
#21228 (grammar failures that fail open silently).

4. Cost of the `--no-jinja` workaround

Worth flagging, since it's the only workaround and it isn't free. Config:
--no-jinja --chat-template chatml (the native format is chatml, so message framing should be
equivalent). All of the following disappeared when the model ran with its own template on
non-grammar requests.

Language locking breaks. The extraction prompt contains a capitalised instruction:
"Detect the language of the input text and produce ALL output in that EXACT same language.
You are STRICTLY FORBIDDEN from translating." Given English input:

in : "Handbag analysis confirmed silver studs, suede material, zipper closure,
      removable adjustable strap."
out: "銀色裝飾配件確認於手提包上，材料為超絨皮革，結構包含拉鏈閉合與可移除調節鍋扣。"

可移除調節鍋扣 back-translates to "removable adjustable pot buckle". This was
intermittent — the same config produced correct English on other inputs minutes earlier.

Instruction text leaks into output fields. With ~1.4 KB of task instructions on top of a
6.3 KB base prompt, the model emitted its instructions back inside a grammar-valid string:

"... | When: on July 22, </think># Fatti estratti dai testi (Chunk: 1/1) — lingua: **inglese**
 ... ### 1. what= | ..."

Note the stray </think>. Shortening the instruction block to ~460 chars removed this
completely, so it reads as an adherence/capacity effect rather than corruption. Worth noting
generally: grammar constrains JSON shape, not the contents of a string field.

Exact-string reproduction becomes unreliable. Observed at temperature 0.0–0.6:

expected	produced
`394381c6-7d14-47c6-8747-261b1840d1cf`	`394381c6-7d14-47c6-8747-261b1840dcf` (one char dropped)
`...\llama\config.ini`	`...\llamauntime.ini`
a valid UUID	`""` and `"null"` in separate runs

Recommended sampling doesn't rescue it. At the card's temperature 0.6, top_p 0.95, top_k 20, on a fixed structured task, 4 runs: 3 of 4 produced malformed identifiers, 0 of 4
applied a required update. One earlier run at 0.6 was clean and I initially read that as a
fix — it did not reproduce.

5. `json_object` is not a usable fallback either

Returns valid JSON wrapped in a markdown fence (json …), which fails a strict parser at
char 0. More importantly, given a ~3.4 KB schema in the prompt the model often echoed the
schema definition back instead of instantiating it:

{"type":"object","properties":{"facts":{"type":"array", ...

and in another run nested the extracted data under properties. Output shape varied across
identical calls, so it can't be reliably normalised downstream.

Summary

For grammar-constrained/structured output on llama.cpp today:

jinja on (default): unusable — every json_schema request 400s
--no-jinja: runs, but without the native template, and with it goes reliable language
locking, instruction adherence at longer prompts, and verbatim identifier copying
json_object: shape not dependable

I've moved this workload to another model for now. This is not a claim that the model is
weak — the published benchmarks are strong, and everything in §4 was measured in a
configuration that strips the chat template, which is exactly what those benchmarks depend on.
The blocker is the template ↔ grammar-trigger interaction; every workaround for it removes the
template.

A fix in the trigger-construction path would likely resolve §1 and, by removing the need for
--no-jinja, most of §4 as well. Happy to re-test against a patched build or a revised
chat template.

Caveats

Single machine, single community Q8_0 quant. Q8_0 is near-lossless so quantization is an
unlikely cause of §4, but I could not rule it out — testing BF16 needs a stack that runs
modeling_nanbeige.py directly, which I don't have set up.
The §1 400 is fully deterministic. The §4 quality observations are not, and come from
single-digit run counts per configuration. They should be read as indicative.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment