DeepSeek R1 Distill Llama 8B (MNN)

DeepSeek's R1 Distill Llama 8B reasoning model as a 4-bit MNN bundle for chatting on your phone with TokForge.

Published by TokForge for on-device inference.

What it is Reasoning chat model
Format MNN bundle, 4-bit weights (block 128)
Runs on Android phones (CPU, OpenCL or Vulkan GPU)
Phone class No catalog class; needs more free RAM than its 5.27 GB of weights
Download 5.28 GB (10 files)
Base model deepseek-ai/DeepSeek-R1-Distill-Llama-8B
License MIT (DeepSeek); Llama base under the Llama 3.1 Community License

TokForge

Runs on-device in the TokForge app.

Use it in TokForge

  1. Install TokForge from Google Play, or join the iOS TestFlight.
  2. On Android, open Models > Browse, search for DeepSeek-R1-Distill-Llama-8B-MNN and download it.
  3. Load it and start a chat. The model runs on the phone; once it is downloaded, chatting needs no connection.

Collections: TokForge Chat Models (MNN), TokForge Llama, Mistral and Dolphin (MNN).

Files

File Size What it is
config.json 210 B MNN runtime settings
embeddings_bf16.bin 1.05 GB Token embeddings (bf16)
export_args.json 1.14 KB Export settings used for the conversion
llm.mnn 558 KB Model graph
llm.mnn.json 1.12 MB Model graph as JSON (reference copy)
llm.mnn.weight 4.22 GB Model weights
llm_config.json 2.54 KB Model settings and chat template
tokenizer.txt 1.41 MB Tokenizer
LICENSE 7.63 KB Llama 3.1 Community License (Meta's text, for the Llama base)
NOTICE 116 B Llama 3.1 notice (Meta)

About the model

DeepSeek's R1 reasoning capability distilled into a Llama 3.1 8B body: DeepSeek fine-tuned Llama 3.1 8B on reasoning samples curated with DeepSeek-R1. It writes out its thinking step by step before the answer, which suits math, logic puzzles, coding and multi-step analysis.

Architecture Llama 3.1 (standard attention, 32 layers, GQA, distilled from DeepSeek-R1)
Parameters 8B (4-bit quantized)
Quantization W4A16 (4-bit weights, block size 128)
Vocab 128,256 tokens

Recommended settings

Setting Value
Backend OpenCL (Qualcomm) / Vulkan (MediaTek) / CPU (fallback)
Precision Low
Threads 4
Thinking On (the chat template always opens a <think> block)

DeepSeek recommends a temperature between 0.5 and 0.7 (0.6 recommended) and no system prompt: put all instructions in your message.

How it was made

Converted using MNN's llmexport pipeline:

python llmexport.py --path deepseek-ai/DeepSeek-R1-Distill-Llama-8B --export mnn --quant_bit 4 --quant_block 128

Limitations

  • Intended for TokForge / MNN on-device inference on Android
  • This is a runtime bundle, not a standard Transformers training checkpoint
  • Quantization (Q4) may slightly reduce quality compared to the full-precision original
  • The reasoning comes before every answer, so replies are longer than from a plain chat model

License

DeepSeek-R1-Distill-Llama-8B is released by DeepSeek under the MIT License. It is derived from Meta's Llama 3.1 8B, which is licensed under the Llama 3.1 Community License. Meta's license text is in LICENSE and the required notice is in NOTICE. Built with Llama.

Credits

  • DeepSeek-R1-Distill-Llama-8B by DeepSeek (Hugging Face). Used under the MIT license.
  • Llama 3.1 8B by Meta (Hugging Face). Used under the Llama 3.1 Community License. The distilled weights derive from it.
  • Built with Llama. Llama 3.1 is licensed under the Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. The license text and this notice are in the LICENSE and NOTICE files in this repo.
  • Converted with llmexport from MNN by Alibaba (Apache-2.0).
Downloads last month
70
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for darkmaniac7/DeepSeek-R1-Distill-Llama-8B-MNN

Quantized
(214)
this model

Collections including darkmaniac7/DeepSeek-R1-Distill-Llama-8B-MNN