| --- |
| base_model: |
| - Qwen/Qwen3.5-2B |
| tags: |
| - llamafile |
| - qwen3_5 |
| - image-text-to-text |
| - conversational |
| - text-generation-inference |
| notes: |
| - llamafile-0.10.0 |
| - based on GGUF format quantizations by unsloth |
| --- |
| |
| ## About |
|
|
| * ***Update(1)*** `2026-04-24`: apparently, VirusTotal does not like the llamafiles and marks them "suspicious". There's not much I can do there, I guess. Let us wait untill the other scanners try it; until then -- feel free to e.g. ClamAV scan them after download, but also feel free to redo it yourself following the links provided: your result will be binary compatible with what is uploaded here. |
| * ***Update(2)*** `2026-04-24`: hmm, the "suspicious" flag seems to be cleared on some binaries. Good. May be just give it time to update the databases. |
| * ***Update(3)*** `2026-04-24`: testing virustotal reaction on other files, see [Qwen3.5-4B-llamafile/README.md](https://huggingface.co/thread13/Qwen3.5-4B-llamafile/blob/main/README.md) |
| * Also, apparently only <u>one</u> of all those 75 engines (named "SecureAge") marks this binary as "malicious", and its comment says _"Acronis (Static ML)"_ -- whatever that means. Well, the binary _does_ have an embedded LLM (which is like the whole point here), so may be this is what causes this misunderstanding. |
|
|
| A [llamafile](https://github.com/mozilla-ai/llamafile/blob/main/docs/quickstart.md) is a universal ([APE](https://justine.lol/ape.html)) single-file executable, |
| which is based on [llama.cpp](https://github.com/ggml-org/llama.cpp/releases)-built binaries. It contains an embedded LLM model and can provide a console and/or a web interface to chat with it. |
|
|
| * "and/or" here has a literal meaning -- one can _either_ run a console chat (`--chat`), or a web server (`--server`), or -- by default -- both. |
| * NB(1): by default the web server attempts to listen on _all_ available interfaces -- run it as `--host 127.0.0.1` to make it private to your system only! |
| * NB(2): there's also a cli completion interface (`--cli`), if you wish -- [check the docs!](https://mozilla-ai.github.io/llamafile/) |
|
|
| ## How to use: |
| * https://github.com/mozilla-ai/llamafile/blob/main/docs/running_llamafile.md |
| * https://github.com/mozilla-ai/llamafile/blob/main/docs/quickstart.md |
| |
| ## Building references: |
| * https://github.com/mozilla-ai/llamafile/blob/main/docs/creating_llamafiles.md |
| * https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/blob/main/mmproj-BF16.gguf |
| * https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/blob/main/Qwen3.5-2B-UD-Q8_K_XL.gguf |
| * https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/blob/main/Qwen3.5-2B-UD-Q4_K_XL.gguf |
| * https://huggingface.co/unsloth/Qwen3.5-2B-GGUF/blob/main/Qwen3.5-2B-IQ4_XS.gguf |
| |
| ## Default arguments: |
| * https://unsloth.ai/docs/models/qwen3.5 |
| ``` |
| temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0 |
|
|
| // recommended (unsloth) for coding in thinking mode: |
| --ctx-size 16384 --temp 0.6 --top-p 0.95 --top-k 20 --min-p 0.00 \ |
| --chat-template-kwargs '{"enable_thinking":true}' |
| |
| // recommended (unsloth) for reasoning tasks: |
| --ctx-size 16384 --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.00 \ |
| ``` |
| |
| ### To enable "thinking mode" (slower but better results): |
| * https://unsloth.ai/docs/models/qwen3.5#how-to-enable-or-disable-reasoning-and-thinking |
| ``` |
| --chat-template-kwargs '{"enable_thinking":true}' |
| ``` |
| |
| ### Extra recommendations -- try the following if generation falls into an endless loop: |
| * https://unsloth.ai/docs/models/qwen3.5 (same source) |
| |
| ``` |
| presence_penalty = 0.0 to 2.0 |
| ``` |
| |
| ## Quirks: |
| * https://github.com/mozilla-ai/llamafile/issues/373 |
| * https://github.com/mozilla-ai/llamafile/tree/0.9.0?tab=readme-ov-file#gotchas-and-troubleshooting |
| |
| For some quirks and workarounds (like how to make this run on WSL), see the second link above; |
| in particular, if you are seeing a message like this: |
| ``` |
| Cannot open assembly '...': File does not contain a valid CIL image. |
| ``` |
| |
| The reason might be that you have wine installed and it tries to run the executable (and fails). |
| There are workarounds available -- see above; if that does not work, please open an issue, |
| and when I see it (which might not happen immediately), we will think of something. |
| |
