Add MNN Q4 conversion for TokForge mobile inference

.gitattributes

@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+llm.mnn filter=lfs diff=lfs merge=lfs -text
+llm.mnn.weight filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,99 @@
+---
+license: apache-2.0
+language:
+  - en
+pipeline_tag: text-generation
+base_model: mistralai/Mistral-Small-24B-Instruct-2501
+tags:
+  - mnn
+  - mistral
+  - mobile
+  - on-device
+  - tokforge
+  - uncensored
+  - abliterated
+---
+
+# Mistral-Small-24B-Instruct-MNN
+
+Pre-converted [Mistral Small 24B Instruct](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) in MNN format for on-device inference with [TokForge](https://tokforge.ai).
+
+> **Original model by [Mistral AI](https://huggingface.co/Mistral AI)** — converted to MNN Q4 for mobile deployment.
+
+## Model Details
+
+| | |
+|---|---|
+| **Architecture** | Mistral (sliding window attention, 40 layers) |
+| **Parameters** | 24B (4-bit quantized) |
+| **Format** | MNN (Alibaba Mobile Neural Network) |
+| **Quantization** | W4A16 (4-bit weights, block size 128) |
+| **Vocab** | 32,768 tokens |
+| **Source** | [mistralai/Mistral-Small-24B-Instruct-2501](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501) |
+
+## Description
+
+Mistral AI's Small 24B — a knowledge-dense 24B model that fits on a single high-end GPU. Excellent for complex reasoning, function calling, and multi-step tasks. Runs on flagship phones with 24GB RAM. The most capable model in this collection.
+
+## Files
+
+| File | Description |
+|------|-------------|
+| `llm.mnn` | Model computation graph |
+| `llm.mnn.weight` | Quantized weight data (Q4, block=128) |
+| `llm_config.json` | Model config with Jinja chat template |
+| `tokenizer.txt` | Tokenizer vocabulary |
+| `config.json` | MNN runtime config |
+
+## Usage with TokForge
+
+This model is optimized for **[TokForge](https://tokforge.ai)** — a free Android app for private, on-device LLM inference.
+
+1. Download [TokForge from the Play Store](https://tokforge.ai)
+2. Open the app → Models → Download this model
+3. Start chatting — runs 100% locally, no internet required
+
+### Recommended Settings
+
+| Setting | Value |
+|---------|-------|
+| Backend | OpenCL (Qualcomm) / Vulkan (MediaTek) / CPU (fallback) |
+| Precision | Low |
+| Threads | 4 |
+| Thinking | Off (or On for thinking-capable models) |
+
+
+
+## Performance
+
+Actual speed varies by device, thermal state, and generation length. Typical ranges for this model size:
+
+| Device | SoC | Backend | tok/s |
+|---|---|---|---|
+| RedMagic 11 Pro (24GB) | SM8850 | OpenCL | ~5-6 tok/s |
+
+> **Note:** Requires 24GB+ RAM. Best for flagship phones with 24GB RAM and minimal background apps.
+
+## Attribution
+
+This is an MNN conversion of **[Mistral Small 24B Instruct](https://huggingface.co/mistralai/Mistral-Small-24B-Instruct-2501)** by **[Mistral AI](https://huggingface.co/Mistral AI)**. All credit for the model architecture, training, and fine-tuning goes to the original author(s). This conversion only changes the runtime format for mobile deployment.
+
+## Limitations
+
+- Intended for TokForge / MNN on-device inference on Android
+- This is a runtime bundle, not a standard Transformers training checkpoint
+- Quantization (Q4) may slightly reduce quality compared to the full-precision original
+- Abliterated/uncensored models have had safety filters removed — **use responsibly**
+
+## Community
+
+- **Website:** [tokforge.ai](https://tokforge.ai)
+- **Discord:** [Join our Discord](https://discord.gg/Acv3CBtfVm)
+- **GitHub:** [TokForge on GitHub](https://github.com/darkmaniac7/Elysium)
+
+## Export Details
+
+Converted using MNN's `llmexport` pipeline:
+```bash
+python llmexport.py --path mistralai/Mistral-Small-24B-Instruct-2501 --export mnn --quant_bit 4 --quant_block 128
+```

config.json

@@ -0,0 +1,10 @@
+{
+    "llm_model": "llm.mnn",
+    "llm_weight": "llm.mnn.weight",
+    "backend_type": "cpu",
+    "thread_num": 4,
+    "precision": "low",
+    "memory": "low",
+    "sampler_type": "penalty",
+    "penalty": 1.1
+}

embeddings_bf16.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:be028c1aa62cb6cb18d5c0cfdb938061c95f816f50460103dc7f17633092515a
+size 1342177280

export_args.json

@@ -0,0 +1,42 @@
+{
+    "path": "/root/models/hf_convert_queue/Mistral-Small-24B-Instruct",
+    "type": null,
+    "tokenizer_path": "/root/models/hf_convert_queue/Mistral-Small-24B-Instruct",
+    "eagle_path": null,
+    "lora_path": null,
+    "gptq_path": null,
+    "dst_path": "/root/models/hf_uploads/Mistral-Small-24B-Instruct-MNN",
+    "verbose": false,
+    "test": null,
+    "export": "mnn",
+    "onnx_slim": false,
+    "quant_bit": 4,
+    "quant_block": 128,
+    "visual_quant_bit": null,
+    "visual_quant_block": null,
+    "lm_quant_bit": 4,
+    "lm_quant_block": 128,
+    "mnnconvert": "../../../build/MNNConvert",
+    "ppl": false,
+    "awq": false,
+    "hqq": false,
+    "omni": false,
+    "transformer_fuse": false,
+    "group_conv_native": false,
+    "smooth": false,
+    "sym": false,
+    "visual_sym": false,
+    "seperate_embed": true,
+    "lora_split": false,
+    "calib_data": null,
+    "act_bit": 16,
+    "embed_bit": 16,
+    "act_sym": false,
+    "quant_config": null,
+    "generate_for_npu": false,
+    "skip_weight": false,
+    "omni_epochs": 20,
+    "omni_lr": 0.005,
+    "omni_wd": 0.0001,
+    "tie_word_embeddings": false
+}

llm.mnn

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2c55cc27d7e48c7c81513d6ec72f41df4cca38fc73cb4d97705bcc808d370e30
+size 695576

llm.mnn.weight

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a0721394f97c7e276177e35b96182903cae4d237876600e84a1bd5cfd6643760
+size 12885080106

llm_config.json

@@ -0,0 +1,12 @@
+{
+    "model_type": "mistral",
+    "hidden_size": 5120,
+    "attention_mask": "float",
+    "attention_type": "full",
+    "is_mrope": false,
+    "jinja": {
+        "chat_template": "{%- set today = strftime_now(\"%Y-%m-%d\") %}\n{%- set default_system_message = \"You are Mistral Small 3, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\\nYour knowledge base was last updated on 2023-10-01. The current date is \" + today + \".\\n\\nWhen you're not sure about some information, you say that you don't have the information and don't make up anything.\\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \\\"What are some good restaurants around me?\\\" => \\\"Where are you?\\\" or \\\"When is the next flight to Tokyo\\\" => \\\"Where do you travel from?\\\")\" %}\n\n{{- bos_token }}\n\n{%- if messages[0]['role'] == 'system' %}\n    {%- set system_message = messages[0]['content'] %}\n    {%- set loop_messages = messages[1:] %}\n{%- else %}\n    {%- set system_message = default_system_message %}\n    {%- set loop_messages = messages %}\n{%- endif %}\n{{- '[SYSTEM_PROMPT]' + system_message + '[/SYSTEM_PROMPT]' }}\n\n{%- for message in loop_messages %}\n    {%- if message['role'] == 'user' %}\n        {{- '[INST]' + message['content'] + '[/INST]' }}\n    {%- elif message['role'] == 'system' %}\n        {{- '[SYSTEM_PROMPT]' + message['content'] + '[/SYSTEM_PROMPT]' }}\n    {%- elif message['role'] == 'assistant' %}\n        {{- message['content'] + eos_token }}\n    {%- else %}\n        {{- raise_exception('Only user, system and assistant roles are supported!') }}\n    {%- endif %}\n{%- endfor %}",
+        "bos": "<s>",
+        "eos": "</s>"
+    }
+}