Instructions to use leok7v/Qwen3.5-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use leok7v/Qwen3.5-4B with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf leok7v/Qwen3.5-4B
# Run inference directly in the terminal:
llama cli -hf leok7v/Qwen3.5-4B

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf leok7v/Qwen3.5-4B
# Run inference directly in the terminal:
llama cli -hf leok7v/Qwen3.5-4B

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf leok7v/Qwen3.5-4B
# Run inference directly in the terminal:
./llama-cli -hf leok7v/Qwen3.5-4B

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf leok7v/Qwen3.5-4B
# Run inference directly in the terminal:
./build/bin/llama-cli -hf leok7v/Qwen3.5-4B

Use Docker

docker model run hf.co/leok7v/Qwen3.5-4B

LM Studio
Jan

vLLM

How to use leok7v/Qwen3.5-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "leok7v/Qwen3.5-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leok7v/Qwen3.5-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/leok7v/Qwen3.5-4B

Ollama
How to use leok7v/Qwen3.5-4B with Ollama:
```
ollama run hf.co/leok7v/Qwen3.5-4B
```

Unsloth Studio

How to use leok7v/Qwen3.5-4B with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for leok7v/Qwen3.5-4B to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for leok7v/Qwen3.5-4B to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for leok7v/Qwen3.5-4B to start chatting

How to use leok7v/Qwen3.5-4B with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf leok7v/Qwen3.5-4B

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "leok7v/Qwen3.5-4B"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use leok7v/Qwen3.5-4B with Docker Model Runner:
```
docker model run hf.co/leok7v/Qwen3.5-4B
```

Lemonade

How to use leok7v/Qwen3.5-4B with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull leok7v/Qwen3.5-4B

Run and chat with the model

lemonade run user.Qwen3.5-4B-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use leok7v/Qwen3.5-4B with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf leok7v/Qwen3.5-4B

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default leok7v/Qwen3.5-4B

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use leok7v/Qwen3.5-4B with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf leok7v/Qwen3.5-4B

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "leok7v/Qwen3.5-4B" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Model Card for Qwen3.5-4B (2-bit GGUF)

A 2-bit GGUF build of Qwen3.5-4B, packaged to run entirely on device. The tokenizer, the merge table and the chat template are embedded in the weight file, so nothing calls out to a server at inference time.

This repository holds the quantized weights only, not training data or the original checkpoint.

This file uses a non-standard block type and stock llama.cpp will not read it. The weights are stored in a custom 2-bit format that upstream ggml does not implement, so the file is intended for a companion on-device runtime that does. If you are looking for a build that runs in llama.cpp, Ollama or LM Studio today, use one of the standard-quant GGUF releases instead.

Files

File	Purpose
`Qwen3.5-4B-Q2_E8.gguf`	the weights, tokenizer and chat template
`generation_config.json`	the sampling parameters, per mode

Both are needed. The sampling card is read from beside the weight file; a runtime that cannot find it should refuse to guess rather than substitute another model's settings.

Model Details

Model Description

Qwen3.5-4B is a dense hybrid. Its layers alternate three Gated DeltaNet linear-attention blocks to one full attention block. The linear blocks carry a small fixed-size recurrent state instead of a growing key/value cache, so memory stays flat as context grows, while the periodic attention blocks preserve exact long-range recall.

This build is text-only. The base model is multimodal, but no vision tower is included here, so image and video input are not available from this file.

A multi-token-prediction layer is present, which a runtime that supports self-speculative decoding can use as a drafter; one that does not will ignore it.

Developed by: Qwen, Alibaba (base model); quantization by leok7v
Model type: Hybrid Gated DeltaNet + attention causal language model
Language(s): English and the languages of the base model
License: Apache 2.0, inherited from the base model
Quantized from model: Qwen/Qwen3.5-4B

Model Sources

Repository (this build): https://huggingface.co/leok7v/Qwen3.5-4B
Base model: Qwen/Qwen3.5-4B

Uses

Direct Use

On-device chat, summarization, drafting and question answering over supplied text, in offline or privacy-sensitive settings where prompts must not leave the device.

Out-of-Scope Use

Anything the base model's licence and card exclude. This build adds no alignment, no safety tuning and no filtering of its own; it is the upstream model in a different numeric format. Not a basis for legal, medical, financial or safety decisions, and not intended for high-stakes automated decisions without human review.

Bias, Risks, and Limitations

The base model's biases and failure modes carry over unchanged.

Two bits is aggressive, and it shows. Outputs will not match the upstream model token for token, and the gap is widest where the model was already uncertain. In practice the structure of an answer survives better than its details: a chain of reasoning can be laid out correctly and still contain an arithmetic slip, and long multi-step derivations are where errors accumulate.

Anyone depending on this build should evaluate it on their own task rather than assume it behaves like the full-precision model or like a 4-bit one.

Recommendations

Treat generations as drafts to verify, not as fact. For anything where accuracy matters, and for arithmetic in particular, check the result.

How to Get Started with the Model

The tokenizer and the chat template are embedded, so a runtime reads both from the GGUF rather than from sibling files. The chat template documents the roles, the thinking block and the tool-call format.

The base model card recommends these sampling parameters, and generation_config.json carries them:

mode	temperature	top_p	top_k	presence_penalty
Thinking	1.0	0.95	20	1.5
Instruct (non-thinking)	0.7	0.80	20	1.5

reasoning_effort is supported by the template; low and medium cost fewer tokens per turn than the default.

Technical Specifications

Model Architecture and Objective

Property	Value
Objective	Autoregressive next-token prediction
Hidden size	2560
Layers	32 (Gated DeltaNet + attention, three to one) + 1 MTP
Attention heads	16 query / 4 key-value (grouped-query)
Head dimension	256 (partial rotary, 64)
Linear-attention state	128, 16 groups, inner size 4096
Feed-forward	9216
Vocabulary	248320
Context length	up to 262144 tokens

Weight format

1.47 GiB, 441 tensors, GGUF v3, 16384-byte tensor alignment.

Type	Tensors	Covers
2-bit	200	the transformer trunk and the embedding table
BF16	8	held wider on purpose
Q4_0	1	held wider on purpose
F32	232	norms, layer scalars, the recurrent-state parameters

The 2-bit type is a custom ggml block: each group of eight weights is snapped to the nearest point of an E8 lattice codebook and stored as one 16-bit index, which is exactly two bits per weight. That is why stock llama.cpp cannot read the file.

Citation

Please cite the upstream Qwen3.5-4B model. See the base model card at Qwen/Qwen3.5-4B for the canonical reference and license.

Model Card Contact

leok7v on Hugging Face.

Downloads last month: -

GGUF

Model size

4B params

Architecture

qwen35

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for leok7v/Qwen3.5-4B

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Quantized

(374)

this model