GatekeeperZA/InternLM2-1.8B-w8a8-RKLLM-v1.2.3

InternLM2-Chat-1.8B — RKLLM v1.2.3 (w8a8, RK3588)

RKLLM conversion of internlm/internlm2-chat-1_8b for Rockchip RK3588 NPU inference.

Converted with RKLLM Toolkit v1.2.3. This model provides a different architecture option alongside Qwen3 models on the RK3588, offering strong multilingual support (English + Chinese) and good general-purpose chat capability at ~15.6 tokens/sec.

Key Details

Base Model	internlm/internlm2-chat-1_8b
Parameters	1.8B
Toolkit Version	RKLLM Toolkit v1.2.3
Runtime Version	RKLLM Runtime ≥ v1.2.0 (v1.2.3 recommended)
Quantization	w8a8 (8-bit weights, 8-bit activations)
Quantization Algorithm	normal
Target Platform	RK3588
NPU Cores	3
Max Context Length	4,096 tokens
Optimization Level	1
Thinking Mode	❌ Not supported (standard instruct model)
Languages	English, Chinese

Performance (RK3588 Official Benchmark)

From the RKLLM v1.2.3 benchmark (w8a8, SeqLen=128, New_tokens=64):

Metric	Value
Decode Speed	15.58 tokens/sec
Prefill (TTFT)	374 ms
Memory Usage	~1,766 MB

Why InternLM2-1.8B?

InternLM2 brings architectural diversity to an RK3588 model lineup. If you already run Qwen3 models, adding InternLM2 gives you a different model family with its own strengths:

• Strong bilingual capability — trained extensively on both English and Chinese data
• Good instruction following — RLHF-aligned for chat applications
• Efficient memory usage — 1,766 MB is significantly less than 3-4B models (3.7-4.3 GB)
• Fast inference — 15.58 tok/s is solidly in the "responsive chat" bracket
• 200K native context — the base model supports ultra-long contexts (RKLLM conversion caps at 4K for NPU efficiency, but the architecture handles long dependencies well)

Benchmarks (Base Model)

Benchmark	InternLM2-Chat-1.8B	InternLM2-1.8B (base)
MMLU	47.1	46.9
AGIEval	38.8	33.4
BBH	35.2	37.5
GSM8K	39.7	31.2
MATH	11.8	5.6
HumanEval	32.9	25.0
MBPP (Sanitized)	23.2	22.2

Source: OpenCompass

Hardware Tested

• Orange Pi 5 Plus — RK3588, 16 GB RAM, Armbian Linux
• RKNPU driver 0.9.8
• RKLLM Runtime v1.2.3

Usage

1. Download

Place the .rkllm file in a model directory on your RK3588 board:

mkdir -p ~/models/InternLM2-1.8B
cd ~/models/InternLM2-1.8B
# Copy the .rkllm file into this directory

2. Run with the official RKLLM API demo

# Clone the runtime
git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm/examples/rkllm_api_demo

# Run (aarch64)
./build/rkllm_api_demo /path/to/InternLM2-1.8B-w8a8-rk3588.rkllm 2048 4096

3. Chat template

InternLM2 uses the following chat format:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
How does photosynthesis work?<|im_end|>
<|im_start|>assistant

The RKLLM runtime handles this automatically — no manual template needed.

4. With a custom OpenAI-compatible server

Any server that wraps the RKLLM binary/library will work. The model responds to standard chat completion requests. See the RKLLM API Server project for a full OpenAI-compatible implementation with multi-model support.

Conversion Script

from rkllm.api import RKLLM

model_path = "internlm/internlm2-chat-1_8b"  # or local path
output_path = "./InternLM2-1.8B-w8a8-rk3588.rkllm"
dataset_path = "./data_quant.json"  # calibration data

# Load
llm = RKLLM()
llm.load_huggingface(model=model_path, model_lora=None, device="cpu")

# Build
llm.build(
    do_quantization=True,
    optimization_level=1,
    quantized_dtype="w8a8",
    quantized_algorithm="normal",
    target_platform="rk3588",
    num_npu_core=3,
    extra_qparams=None,
    dataset=dataset_path,
    max_context=4096,
)

# Export
llm.export_rkllm(output_path)

Calibration dataset: 21 diverse prompt/completion pairs generated with generate_data_quant.py from the rknn-llm examples.

File Listing

File	Description
InternLM2-1.8B-w8a8-rk3588.rkllm	Quantized model for RK3588 NPU

Compatibility Notes

• Minimum runtime: RKLLM Runtime v1.2.0. v1.2.3 recommended.
• RKNPU driver: ≥ 0.9.6
• SoCs: RK3588 / RK3588S (3 NPU cores). Not compatible with RK3576 (2 cores) without reconversion.
• RAM: ~1.8 GB loaded. Runs comfortably on 8 GB+ boards.
• No thinking mode: InternLM2 is a standard instruct/chat model — it does not produce <think>…</think> reasoning blocks. For thinking mode, use Qwen3-1.7B-RKLLM-v1.2.3.

Known Issues

• The folder name containing the model must not include dots (e.g., InternLM2-1.8B not InternLM2.1.8B) due to Python module import issues during conversion.
• InternLM2 uses a custom tokenizer (trust_remote_code=True required during conversion).

Acknowledgements

• InternLM Team (Shanghai AI Laboratory) for the base model
• Rockchip / airockchip for the RKLLM toolkit and runtime
• Converted by GatekeeperZA

Citation

@misc{cai2024internlm2,
      title={InternLM2 Technical Report},
      author={Zheng Cai and Maosong Cao and Haojiong Chen and Kai Chen and others},
      year={2024},
      eprint={2403.17297},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}