Update README.md

README.md

@@ -1,3 +1,194 @@
----
-license: mit
----
+---
+library_name: transformers
+license: apache-2.0
+base_model:
+- HuggingFaceTB/SmolLM2-360M-Instruct
+tags:
+- HuggingFaceTB
+- SmolLM2
+- SmolLM2-360M-Instruct
+- Int8
+- M5Stack
+- RaspberryPi 5
+language:
+- en
+---
+
+# SmolLM2-360M-Instruct
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/oWWfzW4RbWkVIo7f-5444.png)
+
+This version of SmolLM2-360M-Instruct has been converted to run on the Axera NPU using **w8a16** quantization.
+
+This model has been optimized with the following LoRA: 
+
+Compatible with Pulsar2 version: 3.4(Not released yet)
+
+## Convert tools links:
+
+For those who are interested in model conversion, you can try to export axmodel through the original repo
+https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct
+
+[Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) 
+
+[AXera NPU HOST LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/internvl2) 
+
+[AXera NPU AXCL LLM Runtime](https://github.com/AXERA-TECH/ax-llm/tree/axcl-llm-internvl)
+
+## Support Platform
+
+- AX650
+  - AX650N DEMO Board
+  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
+  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
+- AX630C
+  - [爱芯派2](https://axera-pi-2-docs-cn.readthedocs.io/zh-cn/latest/index.html)
+  - [Module-LLM](https://docs.m5stack.com/zh_CN/module/Module-LLM)
+  - [LLM630 Compute Kit](https://docs.m5stack.com/zh_CN/core/LLM630%20Compute%20Kit)
+ 
+|Chips|w8a16|w4a16|
+|--|--|--|
+|AX650| 39 tokens/sec|todo|
+|AX630C| 14 tokens/sec|todo|
+
+## How to use
+
+Download all files from this repository to the device
+
+```
+root@ax650:/mnt/qtang/llm-test/smollm2-360m# tree -L 1
+.
+|-- main_axcl_aarch64
+|-- main_axcl_x86
+|-- main_prefill
+|-- post_config.json
+|-- run_smollm2_360m_ax630c.sh
+|-- run_smollm2_360m_ax650.sh
+|-- run_smollm2_360m_axcl_aarch64.sh
+|-- run_smollm2_360m_axcl_x86.sh
+|-- smollm2-360m-ax630c
+|-- smollm2-360m-ax650
+|-- smollm2_tokenizer
+`-- smollm2_tokenizer.py
+```
+
+### Install transformer
+
+```
+pip install transformers==4.41.1
+```
+
+### Start the Tokenizer service
+
+```
+root@ax650:/mnt/qtang/llm-test/smollm2-360m$ python smollm2_tokenizer.py --port 12345
+1 <|im_start|> 2 <|im_end|>
+<|im_start|>system
+You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
+<|im_start|>user
+hello world<|im_end|>
+<|im_start|>assistant
+
+[1, 9690, 198, 2683, 359, 253, 5356, 5646, 11173, 3365, 3511, 308, 34519, 28, 7018, 411, 407, 19712, 8182, 2, 198, 1, 4093, 198, 28120, 905, 2, 198, 1, 520, 9531, 198]
+http://localhost:12345
+```
+
+### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board
+
+Open another terminal and run `run_smollm2_360m_ax650.sh`
+
+```
+root@ax650:/mnt/qtang/llm-test/smollm2-360m# ./run_smollm2_360m_ax650.sh
+[I][                            Init][ 125]: LLM init start
+bos_id: 1, eos_id: 2
+  2% | █                                 |   1 /  35 [0.00s<0.14s, 250.00 count/s] tokenizer init ok
+[I][                            Init][  26]: LLaMaEmbedSelector use mmap
+100% | ████████████████████████████████ |  35 /  35 [0.81s<0.81s, 43.37 count/s] init post axmodel ok,remain_cmm(3339 MB)
+[I][                            Init][ 241]: max_token_len : 1023
+[I][                            Init][ 246]: kv_cache_size : 320, kv_cache_num: 1023
+[I][                            Init][ 254]: prefill_token_num : 128
+[I][                     load_config][ 281]: load config:
+{
+    "enable_repetition_penalty": false,
+    "enable_temperature": true,
+    "enable_top_k_sampling": true,
+    "enable_top_p_sampling": false,
+    "penalty_window": 20,
+    "repetition_penalty": 1.2,
+    "temperature": 0.9,
+    "top_k": 10,
+    "top_p": 0.8
+}
+
+[I][                            Init][ 268]: LLM init ok
+Type "q" to exit, Ctrl+c to stop current running
+>> who are you?
+[I][                             Run][ 466]: ttft: 156.63 ms
+I'm a chatbot developed by the Artificial Intelligence Research and Development Lab (AI R&D Lab) at Hugging Face Labs,
+specifically designed to facilitate and augment human-AI conversations. My role is to provide assistance in understanding
+ and responding to natural language queries, using advanced language models and AI algorithms to understand context and intent.
+
+[N][                             Run][ 605]: hit eos,avg 38.70 token/s
+
+>> q
+
+```
+
+### Inference with M.2 Accelerator card
+
+[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5.
+
+```
+(base) axera@raspberrypi:~/samples/smollm2-360m $ ./run_smollm2_360m_axcl_aarch64.sh
+build time: Feb 13 2025 15:44:57
+[I][                            Init][ 111]: LLM init start
+bos_id: 1, eos_id: 2
+100% | ██████���█████████████████████████ |  35 /  35 [18.07s<18.07s, 1.94 count/s] init post axmodel okremain_cmm(6621 MB)
+[I][                            Init][ 226]: max_token_len : 1023
+[I][                            Init][ 231]: kv_cache_size : 320, kv_cache_num: 1023
+[I][                     load_config][ 282]: load config:
+{
+    "enable_repetition_penalty": false,
+    "enable_temperature": true,
+    "enable_top_k_sampling": true,
+    "enable_top_p_sampling": false,
+    "penalty_window": 20,
+    "repetition_penalty": 1.2,
+    "temperature": 0.9,
+    "top_k": 10,
+    "top_p": 0.8
+}
+
+[I][                            Init][ 288]: LLM init ok
+Type "q" to exit, Ctrl+c to stop current running
+>> who are you?
+
+I'm a virtual AI assistant, designed to support users with their questions and tasks.
+I was trained on a vast dataset of text, including text from various sources and
+conversations. This extensive training allows me to understand and respond to a wide range of queries.
+I'm here to be helpful and provide answers to your questions.
+
+[N][                             Run][ 610]: hit eos,avg 20.81 token/s
+
+
+>> ^Cq
+
+(base) axera@raspberrypi:~ $ axcl-smi
++------------------------------------------------------------------------------------------------+
+| AXCL-SMI  V2.26.0_20250205130139                                Driver  V2.26.0_20250205130139 |
++-----------------------------------------+--------------+---------------------------------------+
+| Card  Name                     Firmware | Bus-Id       |                          Memory-Usage |
+| Fan   Temp                Pwr:Usage/Cap | CPU      NPU |                             CMM-Usage |
+|=========================================+==============+=======================================|
+|    0  AX650N                    V2.26.0 | 0000:01:00.0 |                171 MiB /      945 MiB |
+|   --   39C                      -- / -- | 2%        0% |                468 MiB /     7040 MiB |
++-----------------------------------------+--------------+---------------------------------------+
+
++------------------------------------------------------------------------------------------------+
+| Processes:                                                                                     |
+| Card      PID  Process Name                                                   NPU Memory Usage |
+|================================================================================================|
+|    0    18636  /home/axera/qtang/llm-test/smollm2-360m/main_axcl_aarch64            418580 KiB |
++------------------------------------------------------------------------------------------------+
+(base) axera@raspberrypi:~ $
+```