README.md

---
license: other
license_name: internlm-license
license_link: https://huggingface.co/internlm/internlm2-chat-1_8b/blob/main/LICENSE
base_model: internlm/internlm2-chat-1_8b
tags:
  - internlm2
  - rk3588
  - npu
  - rockchip
  - quantized
  - w8a8
  - rkllm
  - edge
language:
  - en
  - zh
pipeline_tag: text-generation
library_name: rkllm
---

# InternLM2-Chat-1.8B — RKLLM v1.2.3 (w8a8, RK3588)

RKLLM conversion of [internlm/internlm2-chat-1_8b](https://huggingface.co/internlm/internlm2-chat-1_8b) for Rockchip RK3588 NPU inference.

Converted with **RKLLM Toolkit v1.2.3**. This model provides a different architecture option alongside Qwen3 models on the RK3588, offering strong multilingual support (English + Chinese) and good general-purpose chat capability at ~15.6 tokens/sec.

## Key Details

| | |
|---|---|
| **Base Model** | internlm/internlm2-chat-1_8b |
| **Parameters** | 1.8B |
| **Toolkit Version** | RKLLM Toolkit v1.2.3 |
| **Runtime Version** | RKLLM Runtime ≥ v1.2.0 (v1.2.3 recommended) |
| **Quantization** | w8a8 (8-bit weights, 8-bit activations) |
| **Quantization Algorithm** | normal |
| **Target Platform** | RK3588 |
| **NPU Cores** | 3 |
| **Max Context Length** | 4,096 tokens |
| **Optimization Level** | 1 |
| **Thinking Mode** | ❌ Not supported (standard instruct model) |
| **Languages** | English, Chinese |

## Performance (RK3588 Official Benchmark)

From the [RKLLM v1.2.3 benchmark](https://github.com/airockchip/rknn-llm/blob/main/benchmark.md) (w8a8, SeqLen=128, New_tokens=64):

| Metric | Value |
|--------|-------|
| **Decode Speed** | 15.58 tokens/sec |
| **Prefill (TTFT)** | 374 ms |
| **Memory Usage** | ~1,766 MB |

## Why InternLM2-1.8B?

InternLM2 brings **architectural diversity** to an RK3588 model lineup. If you already run Qwen3 models, adding InternLM2 gives you a different model family with its own strengths:

- **Strong bilingual capability** — trained extensively on both English and Chinese data
- **Good instruction following** — RLHF-aligned for chat applications
- **Efficient memory usage** — ~1,766 MB is significantly less than 3-4B models (~3.7-4.3 GB)
- **Fast inference** — 15.58 tok/s is solidly in the "responsive chat" bracket
- **200K native context** — the base model supports ultra-long contexts (RKLLM conversion caps at 4K for NPU efficiency, but the architecture handles long dependencies well)

### Benchmarks (Base Model)

| Benchmark | InternLM2-Chat-1.8B | InternLM2-1.8B (base) |
|-----------|---------------------|----------------------|
| MMLU | 47.1 | 46.9 |
| AGIEval | 38.8 | 33.4 |
| BBH | 35.2 | 37.5 |
| GSM8K | 39.7 | 31.2 |
| MATH | 11.8 | 5.6 |
| HumanEval | 32.9 | 25.0 |
| MBPP (Sanitized) | 23.2 | 22.2 |

Source: [OpenCompass](https://github.com/open-compass/opencompass)

## Hardware Tested

- **Orange Pi 5 Plus** — RK3588, 16 GB RAM, Armbian Linux
- RKNPU driver 0.9.8
- RKLLM Runtime v1.2.3

## Usage

### 1. Download

Place the `.rkllm` file in a model directory on your RK3588 board:

```bash
mkdir -p ~/models/InternLM2-1.8B
cd ~/models/InternLM2-1.8B
# Copy the .rkllm file into this directory
```

### 2. Run with the official RKLLM API demo

```bash
# Clone the runtime
git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm/examples/rkllm_api_demo

# Run (aarch64)
./build/rkllm_api_demo /path/to/InternLM2-1.8B-w8a8-rk3588.rkllm 2048 4096
```

### 3. Chat template

InternLM2 uses the following chat format:

```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
How does photosynthesis work?<|im_end|>
<|im_start|>assistant
```

The RKLLM runtime handles this automatically — no manual template needed.

### 4. With a custom OpenAI-compatible server

Any server that wraps the RKLLM binary/library will work. The model responds to standard chat completion requests. See the [RKLLM API Server](https://github.com/GatekeeperZA/RKLLM-API-Server) project for a full OpenAI-compatible implementation with multi-model support.

## Conversion Script

```python
from rkllm.api import RKLLM

model_path = "internlm/internlm2-chat-1_8b"  # or local path
output_path = "./InternLM2-1.8B-w8a8-rk3588.rkllm"
dataset_path = "./data_quant.json"  # calibration data

# Load
llm = RKLLM()
llm.load_huggingface(model=model_path, model_lora=None, device="cpu")

# Build
llm.build(
    do_quantization=True,
    optimization_level=1,
    quantized_dtype="w8a8",
    quantized_algorithm="normal",
    target_platform="rk3588",
    num_npu_core=3,
    extra_qparams=None,
    dataset=dataset_path,
    max_context=4096,
)

# Export
llm.export_rkllm(output_path)
```

Calibration dataset: 21 diverse prompt/completion pairs generated with `generate_data_quant.py` from the [rknn-llm examples](https://github.com/airockchip/rknn-llm/tree/main/examples/rkllm_api_demo/export).

## File Listing

| File | Description |
|------|-------------|
| `InternLM2-1.8B-w8a8-rk3588.rkllm` | Quantized model for RK3588 NPU |

## Compatibility Notes

- **Minimum runtime:** RKLLM Runtime v1.2.0. v1.2.3 recommended.
- **RKNPU driver:** ≥ 0.9.6
- **SoCs:** RK3588 / RK3588S (3 NPU cores). Not compatible with RK3576 (2 cores) without reconversion.
- **RAM:** ~1.8 GB loaded. Runs comfortably on 8 GB+ boards.
- **No thinking mode:** InternLM2 is a standard instruct/chat model — it does not produce `<think>…</think>` reasoning blocks. For thinking mode, use [Qwen3-1.7B-RKLLM-v1.2.3](https://huggingface.co/GatekeeperZA/Qwen3-1.7B-RKLLM-v1.2.3).

## Known Issues

- The folder name containing the model must **not** include dots (e.g., `InternLM2-1.8B` not `InternLM2.1.8B`) due to Python module import issues during conversion.
- InternLM2 uses a custom tokenizer (`trust_remote_code=True` required during conversion).

## Acknowledgements

- [InternLM Team (Shanghai AI Laboratory)](https://huggingface.co/internlm) for the base model
- [Rockchip / airockchip](https://github.com/airockchip/rknn-llm) for the RKLLM toolkit and runtime
- Converted by [GatekeeperZA](https://huggingface.co/GatekeeperZA)

## Citation

```bibtex
@misc{cai2024internlm2,
      title={InternLM2 Technical Report},
      author={Zheng Cai and Maosong Cao and Haojiong Chen and Kai Chen and others},
      year={2024},
      eprint={2403.17297},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```