litert-community/Nemotron-H-4B-Instruct-128K

Nemotron-H-4B-Instruct — LiteRT-LM

nvidia/Nemotron-H-4B-Instruct-128K converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15. To our knowledge this is the first Nemotron-H in LiteRT form — a three-kind hybrid: 24 Mamba2 selective-scan layers + 24 plain MLP layers + 4 grouped-query attention layers (52 layers total), running fully delegated on the GPU.

The mamba layers carry constant-size conv + SSM recurrent state, only the 4 attention layers keep KV (4096-token budget here), and the MLP layers carry no state at all — memory stays nearly flat with context length.

File	Recipe	Size
Nemotron-H-4B-Instruct-128K_int8.litertlm	int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU	4.67 GB

Correctness

• Logits parity vs PyTorch: the float export matches the HF model teacher-forced across 8 decode positions — max|logit diff| 5.8e-05, correlation 1.000000, top-1 and top-5 identical at every position.
• 8-question sanity gate: 8/8 on CPU and 8/8 on GPU (Mac, litert-lm 0.16.0), answers near-verbatim identical across backends.
• Prompt-length robustness: hermetic prefill-chunk sweep (fresh engine per length) — all lengths clean at the ship shape.
• iPhone 17 Pro (Metal): runs on GPU and CPU with identical answers on the composite probe (7/8-class; one arithmetic near-miss appears identically on BOTH backends — a quantization-level composite-prompt effect, not a backend bug).

Usage

litert-lm run ./Nemotron-H-4B-Instruct-128K_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Nemotron-H-4B-Instruct-128K_int8.litertlm --backend gpu --cache no --prompt "..."

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock Nemotron-H chat template.

Performance

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	724 tok/s	75.0 tok/s	0.37 s
CPU	99 tok/s	20.1 tok/s	2.63 s

On device (cold start, single runs, 131-token composite prompt, quality harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	59.8 tok/s	10.7 tok/s	2.48 s	4.02 GB

Honest notes:

• A 4B does not fit an 8 GB Android phone: on a Pixel 8a, engine creation aborts on both backends (4.67 GB weights plus the multi-signature arena exceed the ~3.8 GB available). Android needs higher-RAM devices; the iPhone rows above used the increased-memory entitlement.
• GPU runs with fp32 activations (declared in the bundle).
• On composite many-question prompts, int8 costs borderline arithmetic items (identically on every backend). Per-question use is clean (8/8).

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm nemotron_h_work/):

• Folded selective scan: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math) — this is what makes the graph fully delegable on GPU.
• Cache-less MLP layer type: NemotronH interleaves plain MLP blocks; a dedicated no-state cache layer keeps absolute layer indexing without phantom KV buffers (24 of them would otherwise be allocated and paid for in RAM).
• Min-only dt clamp handled exactly: NemotronH clamps dt at time_step_min with no upper bound; engine pad steps are forced to exact identity (dt = 0 post-clamp) so partially-filled prefill chunks cannot decay the state.
• Class-registry patching: NemotronH constructs its mixers from an import-time class registry — the export patch swaps the registry entry (module-attribute swapping alone silently exports the unrewritten reference scan; a loud guard now prevents that).
• Quantization: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.

License and changes

Distributed under the NVIDIA Open Model License (inherited from the base model — see the license link). Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified. This repository is a community conversion and is not affiliated with NVIDIA.