Rahspide commited on
Commit
c252ce0
·
verified ·
1 Parent(s): b9bc2eb

docs: Q4_1 build, MNN sources, epochs and parameter count

Browse files
Files changed (1) hide show
  1. README.md +5 -3
README.md CHANGED
@@ -22,14 +22,15 @@ tags:
22
 
23
  ![MNN-Pocket-0.3B-RU-Chat — a Russian assistant in 0.3B](assets/cover_chat.png)
24
 
25
- ![params](https://img.shields.io/badge/params-361M-E2543F?style=flat-square)
26
  ![sft](https://img.shields.io/badge/sft-50M%20tokens-5FD3B4?style=flat-square)
27
  ![data](https://img.shields.io/badge/data-18.8M%20ru%20%2B%206M%20en-6FA8FF?style=flat-square)
28
  ![gguf](https://img.shields.io/badge/gguf-Q8__0%20%C2%B7%20Q4__K__M-E2543F?style=flat-square)
29
  ![license](https://img.shields.io/badge/license-apache--2.0-8B95A4?style=flat-square)
30
  ![status](https://img.shields.io/badge/status-v1%20released-5FD3B4?style=flat-square)
31
 
32
- > **Status: v1 released.** Instruction-tuned on 50 M tokens (18.8 M Russian + 6 M English replay) on one Kaggle
 
33
  > T4 in ~6.5 h. Weights and GGUF builds are in this repository.
34
 
35
  Russian instruction-following assistant, built by SFT on top of
@@ -101,13 +102,14 @@ The model answers instead of continuing the prompt. That is all this table prove
101
  |---|---|---|---|
102
  | Q8_0 | 310 MB | [`MNN-Pocket-0.3B-RU-Chat-Q8_0.gguf`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat/blob/main/gguf/MNN-Pocket-0.3B-RU-Chat-Q8_0.gguf) | near-lossless |
103
  | Q4_K_M | 223 MB | [`MNN-Pocket-0.3B-RU-Chat-Q4_K_M.gguf`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat/blob/main/gguf/MNN-Pocket-0.3B-RU-Chat-Q4_K_M.gguf) | phone size |
 
104
  | MNN INT4 | 165 MB | [mnn/MNN-Pocket-0.3B-RU-Chat-INT4](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat/tree/main/mnn/MNN-Pocket-0.3B-RU-Chat-INT4) | Android, MNN runtime |
105
 
106
  ```bash
107
  llama-cli -hf Rahspide/MNN-Pocket-0.3B-RU-Chat:Q4_K_M --jinja -p "Объясни, что такое инфляция" -n 60
108
  ```
109
 
110
- The MNN build runs on MNN's runtime (`llm_demo`, built with `-DMNN_BUILD_LLM=ON`). Its `llm_config.json` keeps
111
  the ChatML template, so it answers out of the box:
112
 
113
  ```bash
 
22
 
23
  ![MNN-Pocket-0.3B-RU-Chat — a Russian assistant in 0.3B](assets/cover_chat.png)
24
 
25
+ ![params](https://img.shields.io/badge/params-303M-E2543F?style=flat-square)
26
  ![sft](https://img.shields.io/badge/sft-50M%20tokens-5FD3B4?style=flat-square)
27
  ![data](https://img.shields.io/badge/data-18.8M%20ru%20%2B%206M%20en-6FA8FF?style=flat-square)
28
  ![gguf](https://img.shields.io/badge/gguf-Q8__0%20%C2%B7%20Q4__K__M-E2543F?style=flat-square)
29
  ![license](https://img.shields.io/badge/license-apache--2.0-8B95A4?style=flat-square)
30
  ![status](https://img.shields.io/badge/status-v1%20released-5FD3B4?style=flat-square)
31
 
32
+ > **Status: v1 released.** Instruction-tuned for two epochs over 24.8 M unique tokens (18.8 M Russian + 6 M English
33
+ > replay), 50 M tokens consumed in total, on one Kaggle
34
  > T4 in ~6.5 h. Weights and GGUF builds are in this repository.
35
 
36
  Russian instruction-following assistant, built by SFT on top of
 
102
  |---|---|---|---|
103
  | Q8_0 | 310 MB | [`MNN-Pocket-0.3B-RU-Chat-Q8_0.gguf`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat/blob/main/gguf/MNN-Pocket-0.3B-RU-Chat-Q8_0.gguf) | near-lossless |
104
  | Q4_K_M | 223 MB | [`MNN-Pocket-0.3B-RU-Chat-Q4_K_M.gguf`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat/blob/main/gguf/MNN-Pocket-0.3B-RU-Chat-Q4_K_M.gguf) | phone size |
105
+ | Q4_1 | 208 MB | [`MNN-Pocket-0.3B-RU-Chat-Q4_1.gguf`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat-GGUF/blob/main/MNN-Pocket-0.3B-RU-Chat-Q4_1.gguf) | fastest generation on CPU |
106
  | MNN INT4 | 165 MB | [mnn/MNN-Pocket-0.3B-RU-Chat-INT4](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat/tree/main/mnn/MNN-Pocket-0.3B-RU-Chat-INT4) | Android, MNN runtime |
107
 
108
  ```bash
109
  llama-cli -hf Rahspide/MNN-Pocket-0.3B-RU-Chat:Q4_K_M --jinja -p "Объясни, что такое инфляция" -n 60
110
  ```
111
 
112
+ The MNN build runs on MNN's runtime (`llm_demo`, built with `-DMNN_BUILD_LLM=ON`). Runtime sources: [alibaba/MNN](https://github.com/alibaba/MNN), the [LLM export and inference guide](https://github.com/alibaba/MNN/blob/master/transformers/README.md), and the official Android app [`apps/Android/MnnLlmChat`](https://github.com/alibaba/MNN/tree/master/apps/Android/MnnLlmChat). Its `llm_config.json` keeps
113
  the ChatML template, so it answers out of the box:
114
 
115
  ```bash