Instructions to use Nanbeige/Nanbeige4.2-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nanbeige/Nanbeige4.2-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nanbeige/Nanbeige4.2-3B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Nanbeige/Nanbeige4.2-3B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nanbeige/Nanbeige4.2-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.2-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nanbeige/Nanbeige4.2-3B

SGLang

How to use Nanbeige/Nanbeige4.2-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.2-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nanbeige/Nanbeige4.2-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nanbeige/Nanbeige4.2-3B with Docker Model Runner:
```
docker model run hf.co/Nanbeige/Nanbeige4.2-3B
```

Pretrain recipe questions for small-model compute budgeting (not data)

#26

by Infatoshi - opened 5 days ago

Discussion

Infatoshi

5 days ago

Hi Nanbeige team — thanks for the report, modeling code, and the detailed answers on loops / LoopSplit in #14, #16, #20.

Context: we are pretraining Nanbeige-like dense models at ~3–4B on a small B300 cluster and running short architecture ablations (dense R=1/R=2, hybrid attention + mHC stubs, tiny MoE / LatentMoE). Goal is not to copy your data mix. Goal is to avoid burning GPU-months on architecture paths you already found unattractive at this scale, and to match the training regime (precision, seq schedule, hardware class) enough that our kernel work is not wasted.

Anything you can share at “yes/no + rough order of magnitude” is more useful than silence. Feel free to decline anything proprietary.

1. Hardware / system class

What accelerator generation did you (or do you) primarily pretrain on for 4.x (e.g. H100 / H200 / B200 / Ascend / other)? Was pretrain mostly dense DP/FSDP, or pipeline/tensor parallel from the start? Knowing the class of chip matters more than exact cluster size — we care which kernels and precisions are first-class.

2. Low-precision training path

For pretrain (not serve): did 4.2 stay mostly BF16/FP16, or did you use FP8 / block-scaled FP4 / TE-style recipes? Any hard failures that made you drop a precision path at 3B (unstable loss, bad expert GEMMs, attention kernels, etc.)? We have seen TE NVFP4 work on dense FA2 but fall over on some MoE expert shapes on Blackwell.

3. Dense + loops vs MoE at ~3B (what not to waste compute on)

In #3 you said MoE / linear-sparse attention are on the roadmap after 4.5. For this size, was dense+loop preferred mainly because:

(a) tok/s and system simplicity win at small scale,
(b) MoE quality did not beat looped dense at matched wall-time,
(c) routing / load-balance / expert kernel immaturity,
or something else?

If you ran early MoE ablations and killed them: what failed first (quality, thr, stability)? Any “do not bother until X is fixed” advice for small MoE pretrain?

4. Relation to DeepSeek-V4-style and Kimi-K3-style ideas

Config already previews mHC / depth attention / LoopSplit. Separately, we are looking at whether to invest in:

hybrid / compressed-sparse attention (CSA/HCA-like),
residual / attention residual variants,
LatentMoE / shared+routed experts,
KDA / NoPE-style attention kernels.

Question: for Nanbeige 4.5 / 5, which of these are actively in the training stack, which are research-only, and which did you deprioritize for small dense models because thr tax > quality gain? Even a coarse “in / out / later” list would save us a lot of dead ends. We are not asking for implementation details.

5. Sequence length and batch philosophy (pretrain only)

Without data details: did you train mostly at a fixed mid seq (e.g. 4k/8k) then extend, or long-seq early? Any rule of thumb you trust for tokens/day vs model quality at 3B — e.g. “at this size, architecture novelty rarely beats another 50–100B tokens on a stable dense spine”?

6. Distillation vs from-scratch (optional)

If you have a public stance: for a small looped dense student, do you see value in distilling larger MoE teachers (Kimi / DeepSeek class), or do you treat strong from-scratch + post-train as the main path? Again, no data recipe needed — just whether distillation is load-bearing in your thinking.

Happy to share our B300 thr/loss grid numbers (dense vs unfused hybrid vs BF16-fallback MoE) if that is useful context for your answers. Thanks again for the unusually open technical engagement on this model.

leran1995

Nanbeige LLM Lab org 5 days ago

Hi! Thanks for your interest, and for the thoughtful set of questions. Although we can't go into everything, here's what we can share at the level you asked for. 🤝

Hardware / System class
At the 4k stage we train with ZeRO-1 only; for the 32k and 128k stages we combine ZeRO-1 with tensor and pipeline parallelism.
Dense + loops at ~3B
For the small-model part of the Nanbeige4 series, our focus is on how far we can push the capability ceiling under a fixed, tight parameter budget, by co-optimizing architecture, data, and training method. That's the through-line from 4 → 4.1 → 4.2, and into 4.5. MoE is very much on our roadmap; you'll see some of that land in Nanbeige5.
Relation to other model structure ideas
Of the things previewed in the config: mHC with depth attention, LoopSplit, and concatenated n-gram embeddings are confirmed to ship in Nanbeige4.5. The other directions you list are still under active exploration on ourside, which means nothing there has been deprioritized or ruled out. We just haven't committed them to a release yet.
Sequence length
We train in a fixed 4k → 32k → 128k progression during pre-training, with the majority of tokens at 4k.

Thanks again for engaging so openly with the model, really glad you're finding it useful. ❤️

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment