edgeimpulse/edgeimpulse-docs-qwen-0.5b

Edge Impulse Docs Assistant — Qwen 0.5B (GGUF)

A compact (0.5B parameter) chat model tuned to answer Edge Impulse questions — building, training, optimizing and deploying edge AI / TinyML models. It is distributed as a single quantized GGUF file so it runs locally with llama.cpp or Ollama on a laptop, mini-PC, or edge device — no GPU required.

• Base model: Qwen/Qwen2.5-Coder-0.5B-Instruct
• Format: GGUF, Q4_K_M quantization (~398 MB)
• Prompt format: ChatML (<|im_start|> / <|im_end|>)
• Context length: 32K tokens
• Good for: on-device documentation Q&A, quick "how do I…" answers, drafting Edge Impulse CLI / Studio API snippets.

This repository intentionally ships only what you need to run the model. Training data, retrieval indexes, and the training pipeline are not included.

Recommended sampling settings

This is a small 0.5B model. With greedy / low-temperature decoding and no repetition penalty it will loop and repeat sentences. Always run it with a repeat penalty:

Setting	Value
temperature	0.3
top_p	0.9
repeat_penalty (Ollama) / --repeat-penalty (llama.cpp)	1.2

The bundled Modelfile already applies these.

Run it with Ollama (easiest)

Ollama can pull the GGUF straight from this repo:

ollama run hf.co/edgeimpulse/edgeimpulse-docs-qwen-0.5b

Then ask it something:

>>> How do I export a model for a Linux target with the Edge Impulse CLI?

Ollama with the bundled Modelfile (recommended)

The Modelfile adds the tuned system prompt and the repeat penalty above:

# download the two small files
huggingface-cli download edgeimpulse/edgeimpulse-docs-qwen-0.5b \
  qwen-edgeai-q4_k_m.gguf Modelfile --local-dir edgeimpulse-docs-qwen

cd edgeimpulse-docs-qwen
ollama create edgeimpulse-docs-qwen -f Modelfile
ollama run edgeimpulse-docs-qwen

Run it with llama.cpp

# get the model
huggingface-cli download edgeimpulse/edgeimpulse-docs-qwen-0.5b \
  qwen-edgeai-q4_k_m.gguf --local-dir .

# one-shot prompt (note the repeat penalty)
llama-cli -m qwen-edgeai-q4_k_m.gguf --jinja \
  --temp 0.3 --repeat-penalty 1.2 --top-p 0.9 \
  -p "How do I create an Edge Impulse API key?" -st

# or an OpenAI-compatible local server
llama-server -m qwen-edgeai-q4_k_m.gguf -c 4096 --jinja
# -> http://127.0.0.1:8080  (POST /v1/chat/completions; send repeat_penalty: 1.2)

llama.cpp reads the ChatML chat template embedded in the GGUF (--jinja), so llama-cli chat mode and the llama-server chat endpoints work out of the box.

Prompt format

The model uses the standard Qwen/ChatML template:

<|im_start|>system
You are the Edge Impulse documentation assistant.<|im_end|>
<|im_start|>user
How do I deploy a model to an Arduino board?<|im_end|>
<|im_start|>assistant

Intended use & limitations

• Intended use: a lightweight, local assistant for Edge Impulse documentation and workflow questions, and as a small building block inside a larger retrieval-augmented (RAG) or agent setup.
• Size trade-off: at 0.5B parameters this model is fast and tiny, but it can be wrong, invent commands/APIs, or be out of date — for example it may fabricate CLI flags or code that does not exist. Verify anything important against the official docs: https://docs.edgeimpulse.com.
• Best paired with retrieval. For accurate, current answers, feed it relevant documentation snippets as context (RAG) rather than relying on its parametric memory alone.
• Requires a repeat penalty (see settings above) to avoid repetitive output.
• Not a safety-critical system. Do not rely on it for security, compliance, or production configuration without human review.

License

Released under the Apache-2.0 license, inheriting the license of the base Qwen/Qwen2.5-Coder-0.5B-Instruct model.