Instructions to use inference-optimization/NemotronH-3.4B-A3.4B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use inference-optimization/NemotronH-3.4B-A3.4B-small with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="inference-optimization/NemotronH-3.4B-A3.4B-small")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("inference-optimization/NemotronH-3.4B-A3.4B-small")
model = AutoModelForCausalLM.from_pretrained("inference-optimization/NemotronH-3.4B-A3.4B-small", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use inference-optimization/NemotronH-3.4B-A3.4B-small with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "inference-optimization/NemotronH-3.4B-A3.4B-small"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/NemotronH-3.4B-A3.4B-small",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/inference-optimization/NemotronH-3.4B-A3.4B-small

SGLang

How to use inference-optimization/NemotronH-3.4B-A3.4B-small with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "inference-optimization/NemotronH-3.4B-A3.4B-small" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/NemotronH-3.4B-A3.4B-small",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "inference-optimization/NemotronH-3.4B-A3.4B-small" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "inference-optimization/NemotronH-3.4B-A3.4B-small",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use inference-optimization/NemotronH-3.4B-A3.4B-small with Docker Model Runner:
```
docker model run hf.co/inference-optimization/NemotronH-3.4B-A3.4B-small
```

NemotronH-3.4B-A3.4B-small

This is a small version of nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-Base-BF16 created for testing and development. It keeps the base model's hidden dimensions and all hyperparameters identical, reducing only the layer count and the number of MoE experts, so that it exercises the same NemotronH hybrid Mamba-2 / attention / latent-MoE architecture in vLLM and transformers.

Model Details

Base Model: nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-Base-BF16
Architecture: nemotron_h (NemotronHForCausalLM)
Total Parameters: 3.41B (3,412,404,992)
Activated Parameters: 3.37B (3,370,461,952)

Configuration Changes

Only the following config fields were changed relative to the base model (all other hyperparameters — hidden size 8192, vocab 131072, mamba settings (256 heads x 64 head dim, n_groups 8, ssm_state 128, conv_kernel 4, chunk_size 128), attention settings (64 heads / 2 KV heads / head_dim 128, rope_theta 10000), MoE settings (moe_intermediate_size 5120, moe_latent_size 2048, shared expert 10240, num_experts_per_tok 22, routed_scaling_factor 5.0, norm_topk_prob), tie_word_embeddings: false, num_nextn_predict_layers: 1, etc. — are identical):

Parameter	Base (NVIDIA-Nemotron-3-Ultra-550B-A55B-Base-BF16)	Small
`layers_block_type`	108 layers: 48x `linear_attention` (Mamba-2) + 48x `moe` + 12x `full_attention` (repeating 7-layer hybrid unit)	3 layers: 1x `linear_attention`, 1x `full_attention`, 1x `moe`
`n_routed_experts`	512	24

The minimal layer set is a contiguous slice of the base's hybrid pattern (base layers 7-9) and covers every distinct block type of the base: a Mamba-2 linear-attention block, a GQA full-attention block, and a latent-projected MoE block. The expert count (24) is >= the base top-k (22, unchanged) so routing is still exercised. The untied 131072x8192 input/output embeddings (~2.15B params) dominate the parameter count; with hidden size fixed at 8192 this is the minimal size covering all block types.

Checkpoint Structure

Single model.safetensors file (bf16, ~6.8 GB). Tensor naming follows the base checkpoint exactly, including the backbone. prefix (instead of model.) and per-expert (ungrouped) expert tensors: backbone.embeddings.weight, backbone.layers.{i}.norm.weight, backbone.layers.{i}.mixer.{in_proj,conv1d,dt_bias,A_log,D,norm,out_proj} (mamba), backbone.layers.{i}.mixer.{q,k,v,o}_proj (attention), backbone.layers.{i}.mixer.experts.{e}.{up,down}_proj.weight, backbone.layers.{i}.mixer.shared_experts.*, backbone.layers.{i}.mixer.{fc1,fc2}_latent_proj.weight, backbone.layers.{i}.mixer.gate.*, backbone.norm_f.weight, lm_head.weight. Verified against the base model.safetensors.index.json — all 24 tensor-name patterns match modulo layer/expert count. MTP layers of the base (mtp.*) are intentionally not materialized (see Notes).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("inference-optimization/NemotronH-3.4B-A3.4B-small", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/NemotronH-3.4B-A3.4B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

The base AutoConfig was loaded and only the allowed changes above were applied.
The model was built without downloading base weights, weights were initialized (init_weights + NaN/extreme-value fixup), and saved in bf16 via the nemotron_h weight-conversion mapping (backbone prefix + per-expert checkpoint format).
The model was fine-tuned on a small copypasta dataset until perplexity reached ~1.0 (well below the 3.0 target; training loss ~0.02). The Mamba-2 layers trained fine with the reference (non-fused) kernels; no layer freezing or attention-implementation fallback was needed.
Checkpoint structure was validated against the base model.safetensors.index.json.

Validation

$ python validate_tiny_model.py NemotronH-3.4B-A3.4B-small
Success: 1.0000312328338623 <= 10.0

==================================================
Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly.
==================================================

Notes

Backbone-only checkpoint: num_nextn_predict_layers is kept at the base value (1) and mtp_layers_block_type at ["full_attention", "moe"], but transformers does not materialize MTP weights on init/save, so no mtp.* tensors are present (matching the convention for these small test models).
The base is unquantized BF16, so no quantization_config was present or stripped.
Activated parameters are computed per token: full embeddings/lm_head + mamba + attention + shared expert + latent projections + top-22 of 24 routed experts.
use_cache was restored to true (the base value) after training; Trainer saves it as false.