kel-jp/llm-jp-4-8b-instruct-NVFP4

llm-jp-4-8b-instruct — NVFP4 (Japanese-calibrated)

NVFP4 quantization of llm-jp/llm-jp-4-8b-instruct for native serving on NVIDIA Blackwell GPUs (SM120/SM121) with vLLM — 8.4 GB (0.49× of BF16), 1.3–1.5× decode speedup, ~98.5% quality recovery, calibrated on Japanese chat/instruction data through the model's own Harmony chat template.

Unlike existing community NVFP4 recipes (calibrated on English data), this checkpoint was calibrated on a Japanese-heavy mix matched to the model's deployment distribution. The full recipe, calibration datasheet, and every evaluation number (including the unflattering ones) are published here.

Quantization recipe

• Tool: NVIDIA Model Optimizer (nvidia-modelopt==0.44.0), modelopt-format export → vLLM's native fast path (FlashInferCutlassNvFp4LinearKernel)
• Mixed precision (following NVIDIA's own NVFP4 model releases):
  • MLP linears (gate/up/down): NVFP4 (FP4 block-16, FP8 block scales)
  • Attention projections (q/k/v/o): FP8 (E4M3 per-tensor)
  • First 2 and last 2 decoder layers, embeddings, lm_head, norms: BF16 (Nemotron-style exclusion of the most quantization-sensitive layers)
• Calibration: 512 conversations (~1.4M tokens), all passed through the model's Harmony chat template so activation ranges see deployment-realistic token streams. Mix: 40% JA multi-turn chat (oasst2-ja), 20% JA knowledge (Wikipedia-ja), 15% JA math, 10% JA-instructed code, 15% EN chat (UltraChat). Full datasheet: quantization_recipe/calibration_datasheet.json.
• Reproduction scripts in quantization_recipe/.

Evaluation (all measured through served vLLM kernels, not simulation)

Baselines measured on the same harness, same settings, same machine (RTX PRO 6000 Blackwell, vLLM 0.19.2 nightly). llm-jp-eval v2.1.5, 100 samples/dataset, temperature 0, 59 datasets + xlsum supplement.

Metric	BF16	this checkpoint	recovery
llm-jp-eval AVG (14 categories)	0.5401	0.5319	98.5%
— excluding jsem (see limitations)	0.5427	0.5420	99.9%
ja MT-Bench (3-round mean, fixed local judge)	7.38	7.11	−0.27
JA held-out perplexity (200k tokens)	11.82	11.97	+1.3%
mgsm (n=250, paired)	0.224	0.212	within noise
gsm8k (n=1319, paired)	0.223	0.208	−1.6pp (z=2.1)
mawps (n=500, paired)	0.852	0.854	within noise

Category detail (llm-jp-eval): NLI 0.81→0.69 (entirely the jsem dataset — see limitations), QA 0.54→0.53, RC 0.72→0.71, CR 0.81→0.80, HE-JA 0.38→0.37, HE-EN 0.47→0.47, EL 0.62→0.62, FA 0.33→0.33, MR 0.21→0.22, MT 0.84→0.84, SUM 0.10→0.10, IF 0.34→0.36, BBH 0.60→0.60, LM 0.80→0.80.

MT-Bench note: judged with a fixed local judge (Gemma-4-31B-IT-NVFP4, greedy), all rounds of both models in one batch. Scores are relative comparisons only — not comparable to LLM-jp's published GPT-judge numbers.

Throughput (RTX PRO 6000, pp=2048/tg=256, llama-benchy)

concurrency	BF16 decode tok/s	NVFP4 decode tok/s	speedup	TTFT
1	84.5	129.1	1.53×	149→99 ms
4	303	463	1.53×	433→254 ms
8	468	613	1.31×	876→438 ms
16	681	987	1.45×	1408→913 ms

KV-cache capacity at --gpu-memory-utilization 0.9 grows from ~547K to ~627K tokens thanks to the smaller weights.

Serving (IMPORTANT — read before vllm serve)

Requires vLLM with modelopt NVFP4 + mixed-precision support (nightly ≥ 0.19, tested on 0.19.2rc1) on Blackwell. Validated end-to-end on SM120 (RTX PRO 6000); SM121 (DGX Spark / GB10) validation is pending — the same vLLM modelopt path is reported to work there, and this card will be updated once we've verified it ourselves. Two llm-jp-4-specific requirements, both inherited from the base model's OpenAI-Harmony chat format:

1. Reasoning parser: install the bundled vLLM plugin first, or the server cannot split Harmony channels (<|channel|>analysis/final):

   pip install ./vllm_plugin   # registers the "llmjp4" reasoning parser
   vllm serve <this-repo> --trust-remote-code --reasoning-parser llmjp4
   

2. Non-streaming chat completions additionally require "skip_special_tokens": false in the request body (vLLM strips the Harmony structure tokens otherwise, and content comes back null). Streaming is unaffected. If your client can't set this per-request, run the bundled transparent proxy: python3 vllm_plugin/harmony_proxy.py (set UPSTREAM_BASE_URL), which injects it for you.

The plugin includes a fix over the upstream llm-jp-4-cookbook parser for a SentencePiece boundary-space artifact in vLLM's incremental detokenizer that otherwise breaks non-streaming channel extraction.

Known limitations (measured, not hypothetical)

• jsem format brittleness: on the jsem NLI dataset's zero-shot format, the quantized model frequently emits the correct label without the expected <answer>...</answer> tags (e.g. <yes>), scoring 0.13 vs BF16's 0.66 on exact-match. The other four NLI datasets are unaffected. If your application depends on strict output-format adherence, verify against your own formats.
• Small math regression: paired per-item analysis on the full gsm8k test set shows a net −21/1319 items (−1.6pp, z=2.1). mgsm and mawps show no significant change.
• MT-Bench −0.27 (7.38→7.11) under a fixed local judge whose own round-to-round spread is ±0.15 at 3 rounds.
• Concurrency ≥16: rare "Already borrowed" HTTP 500s when the reasoning parser is active (a vLLM/tokenizers interaction present with the BF16 model too; clean at ≤8 concurrent; retry-safe).
• Base-model quirk (not quantization-related): the model may identify itself as "ChatGPT" when asked who it is.

License & attribution

Apache-2.0, same as the base model. Base model by LLM-jp. Calibration data: llm-jp/oasst2-33k-ja (Apache-2.0), wikimedia/wikipedia 20231101.ja (CC-BY-SA-4.0), Kendamarron/magpie-japanese-math-instruction-17k (Apache-2.0), ronantakizawa/python-code-instructions-japanese (MIT), HuggingFaceH4/ultrachat_200k (MIT). The CC-BY-NC xlsum_ja dataset was used ONLY for evaluation scoring, never for calibration.

---

Original model card: llm-jp-4-8b-instruct

llm-jp-4-8b-instruct

LLM-jp-4 is a series of large language models developed by the Research and Development Center for Large Language Models at the National Institute of Informatics.

This repository provides the llm-jp-4-8b-instruct For an overview of the LLM-jp-4 models across different parameter sizes, please refer to:

• LLM-jp-4 Models

Base models are trained with pre-training and mid-training only. Post-trained models are aligned using supervised fine-tuning (SFT) and direct preference optimization (DPO), without reinforcement learning.

While the thinking variants are trained with both SFT and DPO, this instruct model is trained using SFT only, without DPO.

For practical usage examples and detailed instructions on how to use the models, please also refer to our cookbook.

To support the continued development of LLM-jp, we would greatly appreciate it if you could share how you utilize LLM-jp outcomes via the survey form.

Usage

Please refer to our cookbook for practical usage examples and detailed instructions on how to use the models.

Model Details

• Model type: Transformer-based Language Model
• Architectures:

Dense model:

Params	Layers	Hidden size	Heads	Context length	Embedding parameters	Non-embedding parameters	Total parameters
8B	32	4,096	32	65,536	805,306,368	7,784,894,464	8,590,200,832

MoE model:

Params	Layers	Hidden size	Heads	Routed Experts	Activated Experts	Context length	Embedding parameters	Non-embedding parameters	Activated parameters	Total parameters
32B-A3B	32	2,560	40	128	8	65,536	503,316,480	31,635,712,512	3,827,476,992	32,139,028,992

Tokenizer

The tokenizer of this model is based on huggingface/tokenizers Unigram byte-fallback model. The vocabulary entries were converted from llm-jp-tokenizer v4.0. Please refer to README.md of llm-jp-tokenizer for details on the vocabulary construction procedure (the pure SentencePiece training does not reproduce our vocabulary).

The chat template of this model is designed to be compatible with the OpenAI Harmony response format. However, the tokenizer differs from the one assumed by the openai-harmony library, and therefore direct tokenization with openai-harmony is not supported. For correct behavior, please use the tokenizer provided with this model. For detailed usage, please refer to our cookbook.

Training

Pre-training

This model is trained through a multi-stage pipeline consisting of pre-training and mid-training phases, using a total of 11.7T tokens.

The corpora used for pre-training and mid-training are publicly available at the following links:

• Pre-training
• Mid-training

Although most of the corpora have been released, some portions are excluded from public release due to licensing constraints.

Post-training

We have fine-tuned the pre-trained checkpoint using SFT and further aligned it with DPO.

The datasets used for post-training are also publicly available at the following links:

• SFT
• DPO (for llm-jp-4-8b-thinking model)
• DPO (for llm-jp-4-32b-a3b-thinking model)

Evaluation

llm-jp-judge

We evaluated the model on a variety of tasks using an LLM-as-a-Judge framework. The descriptions of each task are as follows.

• MT-Bench (JA/EN): A benchmark for measuring multi-turn conversational task-solving ability.
• AnswerCarefully: A benchmark for evaluating safety in Japanese. We used 336 questions from the v2.0 test set.
• llm-jp-instructions: A set of human-created single-turn question–answer pairs. We used 400 questions from the test set.

We evaluated the models using gpt-5.4-2026-03-05.

Note: In earlier evaluations of the llm-jp-3 series, we used gpt-4o-2024-08-06. The newer evaluator gpt-5.4-2026-03-05 provides a stricter and more reliable assessment, which results in lower scores on benchmarks such as MT-Bench compared to those reported for the llm-jp-3 series.

The scores represent the average values obtained from three rounds of inference and evaluation. For more details, please refer to the codes.

Model Name	MT-Bench (JA)	MT-Bench (EN)	AnswerCarefully	llm-jp-instructions
gpt-4o-2024-08-06	7.29	7.69	4.00	4.07
gpt-5.4-2026-03-05 (reasoning_effort = low)	8.87	8.76	4.38	4.79
gpt-5.4-2026-03-05 (reasoning_effort = medium)	8.87	8.89	4.43	4.82
gpt-5.4-2026-03-05 (reasoning_effort = high)	8.98	8.85	4.41	4.83
gpt-oss-20b (reasoning_effort = low)	7.21	7.95	3.39	3.08
gpt-oss-20b (reasoning_effort = medium)	7.33	7.85	3.55	3.16
llm-jp-4-8b-thinking (reasoning_effort = low)	7.23	7.54	3.58	3.50
llm-jp-4-8b-thinking (reasoning_effort = medium)	7.54	7.79	3.69	3.54
llm-jp-4-32b-a3b-thinking (reasoning_effort = low)	7.57	7.70	3.61	3.61
llm-jp-4-32b-a3b-thinking (reasoning_effort = medium)	7.82	7.86	3.70	3.61

Risks and Limitations

The models released here are in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.

Send Questions to

llm-jp(at)nii.ac.jp

License

Apache License, Version 2.0

Acknowledgement

To develop this model, we used the NINJAL Web Japanese Corpus (whole-NWJC) from the National Institute for Japanese Language and Linguistics (NINJAL).

Model Card Authors

The names are listed in alphabetical order.

Hirokazu Kiyomaru and Takashi Kodama.