Update README.md

README.md

@@ -1,3 +1,137 @@
----
-license: mit
----
+---
+license: mit
+language:
+- en
+base_model:
+- openbmb/MiniCPM4-0.5B
+pipeline_tag: text2text-generation
+tags:
+- minicpm4
+- int8
+---
+
+
+# MiniCPM4-0.5B-Int8
+
+This version of MiniCPM4-0.5B has been converted to run on the Axera NPU using **w8a16** quantization.
+
+This model has been optimized with the following LoRA: 
+
+Compatible with Pulsar2 version: 4.1(Not released yet)
+
+## Convert tools links:
+
+For those who are interested in model conversion, you can try to export axmodel through the original repo :
+https://huggingface.co/openbmb/MiniCPM4-0.5B
+
+[Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) 
+
+[AXera NPU LLM Runtime](https://github.com/AXERA-TECH/ax-llm) 
+
+
+## Support Platform
+
+- AX650
+  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
+  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
+- AX630C
+  - *developing*
+ 
+|Chips|w8a16|w4a16|
+|--|--|--|
+|AX650| 36 tokens/sec|TBD|
+
+## How to use
+
+Download all files from this repository to the device
+
+```
+root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx# tree -L 1
+.
+|-- main_ax650
+|-- main_axcl_aarch64
+|-- main_axcl_x86
+|-- minicpm4-0.5b-int8-ctx-ax650
+|-- minicpm4_tokenizer
+|-- minicpm4_tokenizer_uid.py
+|-- post_config.json
+|-- run_minicpm4_0.5b_int8_ctx_ax650.sh
+`-- run_minicpm4_0.5b_int8_ctx_axcl_x86.sh
+2 directories, 7 files
+```
+
+#### Start the Tokenizer service
+
+Install requirement
+
+```
+pip install transformers jinja2
+```
+
+```
+root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx# python3 minicpm4_tokenizer_uid.py
+Server running at http://0.0.0.0:12345
+```
+
+#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board
+
+Open another terminal and run `run_minicpm4_0.5b_int8_ctx_ax650.sh`
+
+```
+root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx# ./run_minicpm4_0.5b_int8_ctx_ax650.sh
+[I][                            Init][ 110]: LLM init start
+[I][                            Init][  34]: connect http://127.0.0.1:12345 ok
+[I][                            Init][  57]: uid: c779ded0-ff14-4877-869b-1aacc948f2d8
+bos_id: 1, eos_id: 73440
+100% | ████████████████████████████████ |  27 /  27 [2.53s<2.53s, 10.67 count/s] init post axmodel ok,remain_cmm(4244 MB)
+[I][                            Init][ 188]: max_token_len : 1023
+[I][                            Init][ 193]: kv_cache_size : 128, kv_cache_num: 1023
+[I][                            Init][ 201]: prefill_token_num : 128
+[I][                            Init][ 205]: grp: 1, prefill_max_token_num : 1
+[I][                            Init][ 205]: grp: 2, prefill_max_token_num : 128
+[I][                            Init][ 205]: grp: 3, prefill_max_token_num : 512
+[I][                            Init][ 209]: prefill_max_token_num : 512
+[I][                     load_config][ 282]: load config:
+{
+    "enable_repetition_penalty": false,
+    "enable_temperature": false,
+    "enable_top_k_sampling": true,
+    "enable_top_p_sampling": false,
+    "penalty_window": 20,
+    "repetition_penalty": 1.2,
+    "temperature": 0.9,
+    "top_k": 1,
+    "top_p": 0.8
+}
+
+[I][                            Init][ 218]: LLM init ok
+Type "q" to exit, Ctrl+c to stop current running
+[I][          GenerateKVCachePrefill][ 271]: input token num : 25, prefill_split_num : 1 prefill_grpid : 2
+[I][          GenerateKVCachePrefill][ 308]: input_num_token:25
+[I][                            main][ 230]: precompute_len: 25
+[I][                            main][ 231]: system_prompt: You are MiniCPM4, created by ModelBest. You are a helpful assistant.
+prompt >> 你是谁?
+[I][                      SetKVCache][ 531]: prefill_grpid:2 kv_cache_num:128 precompute_len:25 input_num_token:12
+[I][                      SetKVCache][ 534]: current prefill_max_token_num:384
+[I][                             Run][ 660]: input token num : 12, prefill_split_num : 1
+[I][                             Run][ 686]: input_num_token:12
+[I][                             Run][ 829]: ttft: 147.65 ms
+你好，我是MiniCPM系列模型，由面壁智能和OpenBMB开源社区开发。详细信息请访问https://github.com/OpenBMB/
+
+[N][                             Run][ 943]: hit eos,avg 35.75 token/s
+
+[I][                      GetKVCache][ 500]: precompute_len:162, remaining:350
+prompt >> 9.9与9.11
+[I][                      SetKVCache][ 531]: prefill_grpid:3 kv_cache_num:512 precompute_len:162 input_num_token:17
+[I][                      SetKVCache][ 534]: current prefill_max_token_num:256
+[I][                             Run][ 660]: input token num : 17, prefill_split_num : 1
+[I][                             Run][ 686]: input_num_token:17
+[I][                             Run][ 829]: ttft: 274.38 ms
+9.9比9.11大。
+
+[N][                             Run][ 943]: hit eos,avg 35.44 token/s
+
+[I][                      GetKVCache][ 500]: precompute_len:189, remaining:323
+prompt >> q
+root@ax650:/mnt/qtang/llm-test/minicpm4-0.5b-ctx#
+```