Add MNN Q4 conversion for TokForge mobile inference

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+llm.mnn filter=lfs diff=lfs merge=lfs -text
+llm.mnn.weight filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+language:
+  - en
+pipeline_tag: text-generation
+base_model: deepseek-ai/DeepSeek-R1-Distill-Llama-8B
+tags:
+  - mnn
+  - llama
+  - mobile
+  - on-device
+  - tokforge
+  - uncensored
+  - abliterated
+---
+
+# DeepSeek-R1-Distill-Llama-8B-MNN
+
+Pre-converted [DeepSeek R1 Distill Llama 8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B) in MNN format for on-device inference with [TokForge](https://tokforge.ai).
+
+> **Original model by [DeepSeek](https://huggingface.co/DeepSeek)** — converted to MNN Q4 for mobile deployment.
+
+## Model Details
+
+| | |
+|---|---|
+| **Architecture** | Llama 3.1 (standard attention, 32 layers, GQA — distilled from DeepSeek-R1) |
+| **Parameters** | 8B (4-bit quantized) |
+| **Format** | MNN (Alibaba Mobile Neural Network) |
+| **Quantization** | W4A16 (4-bit weights, block size 128) |
+| **Vocab** | 128,256 tokens |
+| **Source** | [deepseek-ai/DeepSeek-R1-Distill-Llama-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B) |
+
+## Description
+
+DeepSeek's R1 reasoning capability distilled into a Llama 3.1 8B body. Brings chain-of-thought reasoning to mobile devices. Shows its thinking process step-by-step, making it excellent for math, logic puzzles, coding, and complex analysis. Performance comparable to OpenAI o1 on reasoning tasks.
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `llm.mnn` | Model computation graph |
+| `llm.mnn.weight` | Quantized weight data (Q4, block=128) |
+| `llm_config.json` | Model config with Jinja chat template |
+| `tokenizer.txt` | Tokenizer vocabulary |
+| `config.json` | MNN runtime config |
+
+## Usage with TokForge
+
+This model is optimized for **[TokForge](https://tokforge.ai)** — a free Android app for private, on-device LLM inference.
+
+1. Download [TokForge from the Play Store](https://tokforge.ai)
+2. Open the app → Models → Download this model
+3. Start chatting — runs 100% locally, no internet required
+
+### Recommended Settings
+
+| Setting | Value |
+|---------|-------|
+| Backend | OpenCL (Qualcomm) / Vulkan (MediaTek) / CPU (fallback) |
+| Precision | Low |
+| Threads | 4 |
+| Thinking | Off (or On for thinking-capable models) |
+
+
+
+## Performance
+
+Actual speed varies by device, thermal state, and generation length. Typical ranges for this model size:
+
+| Device | SoC | Backend | tok/s |
+|---|---|---|---|
+| RedMagic 11 Pro | SM8850 | OpenCL | ~14 tok/s |
+| Lenovo TB520FU | SM8650 | OpenCL | ~10 tok/s |
+
+## Attribution
+
+This is an MNN conversion of **[DeepSeek R1 Distill Llama 8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B)** by **[DeepSeek](https://huggingface.co/DeepSeek)**. All credit for the model architecture, training, and fine-tuning goes to the original author(s). This conversion only changes the runtime format for mobile deployment.
+
+## Limitations
+
+- Intended for TokForge / MNN on-device inference on Android
+- This is a runtime bundle, not a standard Transformers training checkpoint
+- Quantization (Q4) may slightly reduce quality compared to the full-precision original
+- Abliterated/uncensored models have had safety filters removed — **use responsibly**
+
+## Community
+
+- **Website:** [tokforge.ai](https://tokforge.ai)
+- **Discord:** [Join our Discord](https://discord.gg/Acv3CBtfVm)
+- **GitHub:** [TokForge on GitHub](https://github.com/darkmaniac7/Elysium)
+
+## Export Details
+
+Converted using MNN's `llmexport` pipeline:
+```bash
+python llmexport.py --path deepseek-ai/DeepSeek-R1-Distill-Llama-8B --export mnn --quant_bit 4 --quant_block 128
+```

config.json

@@ -0,0 +1,10 @@
+{
+    "llm_model": "llm.mnn",
+    "llm_weight": "llm.mnn.weight",
+    "backend_type": "cpu",
+    "thread_num": 4,
+    "precision": "low",
+    "memory": "low",
+    "sampler_type": "penalty",
+    "penalty": 1.1
+}

embeddings_bf16.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8491b384e310f1a5e986ed878f43caae53af8ec494abeb388643f43cf37248b
+size 1050673152

export_args.json

@@ -0,0 +1,42 @@
+{
+    "path": "/root/models/hf_convert_queue/DeepSeek-R1-Distill-Llama-8B",
+    "type": null,
+    "tokenizer_path": "/root/models/hf_convert_queue/DeepSeek-R1-Distill-Llama-8B",
+    "eagle_path": null,
+    "lora_path": null,
+    "gptq_path": null,
+    "dst_path": "/root/models/hf_uploads/DeepSeek-R1-Distill-Llama-8B-MNN",
+    "verbose": false,
+    "test": null,
+    "export": "mnn",
+    "onnx_slim": false,
+    "quant_bit": 4,
+    "quant_block": 128,
+    "visual_quant_bit": null,
+    "visual_quant_block": null,
+    "lm_quant_bit": 4,
+    "lm_quant_block": 128,
+    "mnnconvert": "../../../build/MNNConvert",
+    "ppl": false,
+    "awq": false,
+    "hqq": false,
+    "omni": false,
+    "transformer_fuse": false,
+    "group_conv_native": false,
+    "smooth": false,
+    "sym": false,
+    "visual_sym": false,
+    "seperate_embed": true,
+    "lora_split": false,
+    "calib_data": null,
+    "act_bit": 16,
+    "embed_bit": 16,
+    "act_sym": false,
+    "quant_config": null,
+    "generate_for_npu": false,
+    "skip_weight": false,
+    "omni_epochs": 20,
+    "omni_lr": 0.005,
+    "omni_wd": 0.0001,
+    "tie_word_embeddings": false
+}

llm.mnn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:47003d840b362f0e90e26b5ae1be4c103448b21453a11d3a42de57632b044904
+size 557656

llm.mnn.weight

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1085472c8100f39b439d49ee50b6cccf616d3480c43c42d42d424244ea6d6325
+size 4223505242

llm_config.json

@@ -0,0 +1,12 @@
+{
+    "model_type": "llama",
+    "hidden_size": 4096,
+    "attention_mask": "float",
+    "attention_type": "full",
+    "is_mrope": false,
+    "jinja": {
+        "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set ns = namespace(is_first=false, is_tool=false, is_output_first=true, system_prompt='') %}{%- for message in messages %}{%- if message['role'] == 'system' %}{% set ns.system_prompt = message['content'] %}{%- endif %}{%- endfor %}{{bos_token}}{{ns.system_prompt}}{%- for message in messages %}{%- if message['role'] == 'user' %}{%- set ns.is_tool = false -%}{{'<｜User｜>' + message['content']}}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is none %}{%- set ns.is_tool = false -%}{%- for tool in message['tool_calls']%}{%- if not ns.is_first %}{{'<｜Assistant｜><｜tool▁calls▁begin｜><｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{%- set ns.is_first = true -%}{%- else %}{{'\\n' + '<｜tool▁call▁begin｜>' + tool['type'] + '<｜tool▁sep｜>' + tool['function']['name'] + '\\n' + '```json' + '\\n' + tool['function']['arguments'] + '\\n' + '```' + '<｜tool▁call▁end｜>'}}{{'<｜tool▁calls▁end｜><｜end▁of▁sentence｜>'}}{%- endif %}{%- endfor %}{%- endif %}{%- if message['role'] == 'assistant' and message['content'] is not none %}{%- if ns.is_tool %}{{'<｜tool▁outputs▁end｜>' + message['content'] + '<｜end▁of▁sentence｜>'}}{%- set ns.is_tool = false -%}{%- else %}{% set content = message['content'] %}{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}{% endif %}{{'<｜Assistant｜>' + content + '<｜end▁of▁sentence｜>'}}{%- endif %}{%- endif %}{%- if message['role'] == 'tool' %}{%- set ns.is_tool = true -%}{%- if ns.is_output_first %}{{'<｜tool▁outputs▁begin｜><｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- set ns.is_output_first = false %}{%- else %}{{'\\n<｜tool▁output▁begin｜>' + message['content'] + '<｜tool▁output▁end｜>'}}{%- endif %}{%- endif %}{%- endfor -%}{% if ns.is_tool %}{{'<｜tool▁outputs▁end｜>'}}{% endif %}{% if add_generation_prompt and not ns.is_tool %}{{'<｜Assistant｜><think>\\n'}}{% endif %}",
+        "bos": "<｜begin▁of▁sentence｜>",
+        "eos": "<｜end▁of▁sentence｜>"
+    }
+}