Instructions to use void0x14/echo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use void0x14/echo with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf void0x14/echo:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf void0x14/echo:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf void0x14/echo:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf void0x14/echo:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf void0x14/echo:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf void0x14/echo:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf void0x14/echo:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf void0x14/echo:Q4_K_M

Use Docker

docker model run hf.co/void0x14/echo:Q4_K_M

LM Studio
Jan
Ollama
How to use void0x14/echo with Ollama:
```
ollama run hf.co/void0x14/echo:Q4_K_M
```
Unsloth Desktop

How to use void0x14/echo with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf void0x14/echo:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "void0x14/echo:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use void0x14/echo with Docker Model Runner:
```
docker model run hf.co/void0x14/echo:Q4_K_M
```

Lemonade

How to use void0x14/echo with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull void0x14/echo:Q4_K_M

Run and chat with the model

lemonade run user.echo-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use void0x14/echo with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf void0x14/echo:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default void0x14/echo:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use void0x14/echo with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf void0x14/echo:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "void0x14/echo:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

echo / MVP /REPORT_FINAL.md

void0x14

feat: VISION GGUF calisiyor — mmproj + multimodal test kanitlari + rapor v3

c3c1d78 unverified 2 months ago

preview code

Raw History Blame Contribute Delete

6.91 kB

Qwen3.5 → Distilled N=4 → GGUF → Vulkan Pipeline — FINAL RAPOR (v3: VISION GGUF CALISIYOR)

Girdi / Başlangıç Noktası

Teacher: Qwen/Qwen3.5-0.8B-Base (853M param, vision block'ları dahil; text model model.language_model)
Student: MVP/artifacts/qwen35-0.8b-text-n4 (pruned, N=4 layer, 337.3M param, hidden=1024, vocab=248320)
Hardware: Ryzen 7 1700 (8C/16T), AMD RX460 (Vulkan destekli, fp16 YOK — ROCm yok)

1) Knowledge Distillation (MVP/distill.py)

Loss: CE (student vs teacher softmax) + MSE (hidden states), CE=1.0 / MSE=10.0
2000 step, seq_len=128, batch=1, grad_accum=8, lr=5e-4 cosine+warmup
Dataset: wikitext-2 raw, 500K token
Süre: 6456s (~108 dk)
Resume desteği eklendi: --resume-from + train_state.json (step metadata ile)
- Crash sonrası step 500 → 2000 tamamlandı, kaldığı yerden devam etti
Loss ilerlemesi: total 87.86 → 64.05, CE 6.15 → 5.21, MSE 8.17 → 5.88
Çıktı: MVP/artifacts/qwen35-distilled-n4/final/model.safetensors (2.3GB f32)

2) GGUF Konversiyon

Script: llama.cpp convert_hf_to_gguf.py (build 10068), mimari Qwen3_5ForCausalLM → qwen35 otomatik tanındı
--no-nextn gerekti (MTP layer yok)
tiktoken kuruldu (tokenizer backend için)
Tokenizer dosyaları final dizinine kopyalandı
Ana format: F32 (RX460 fp32 native, fp16 YOK) → qwen35-distilled-n4-f32.gguf (2.3GB). F16 GGUF üretilip silindi (donanım uyumsuz).

3) Quantization (llama-quantize, 8 thread) — F32 tabandan int tiplere

RX460 matmul profili: fp16/bf16/fp4 YOK; int8/int16 var → Q8_0 (int8), Q5_K_M / Q4_K_M (k-quant, int tabanlı) optimal

Dosya	Boyut	BPW	Vulkan gen (t/s, ort±var)
qwen35-distilled-n4-f32.gguf	2.3GB	32.00	65.4
qwen35-distilled-n4-Q8_0.gguf	611MB	8.50	~185
qwen35-distilled-n4-Q5_K_M.gguf	433MB	5.99	~205
qwen35-distilled-n4-Q4_K_M.gguf	395MB	5.44	212±5 (206-216)

Ölçüm varyasyonu: 3 koşu Q4_K_M gen: 212.6 / 207.2 / 215.5 t/s (ilk 216.5 tek koşuydu) — referans ~212 t/s

3a) Doğruluk: F16 vs F32 taban kaynak testi (wiki.test, ctx=512, Vulkan, aynı veri)

F16-tabanlı Q4_K_M: PPL = 183.217 +/- 1.677
F32-tabanlı Q4_K_M: PPL = 183.268 +/- 1.677
Fark: 0.05 PPL (%0.03) — std sapma (±1.68) içinde → ölçülemez
SONUÇ: Quantize dosyalarda kaynak formatı (F16/F32) hiçbir şeyi değiştirmiyor; F16 ara dosyanın hassasiyet kaybı, Q4_K_M'ın 5.44-bit quantize hatasının altında kalıyor. F32 seçimi doğruluk için değil, ana dosyanın donanımda çalışabilmesi için yapıldı.

Örnek çıktı (Q4_K_M, Vulkan)

Prompt: "The capital of France"
Çıktı: [Start thinking] + İngilizce cümleler üretiyor (gibi anlamsız — 2000 step wikitext distilasyonu sonucu beklenen; model öğrendi ama henüz yeterli koherence yok)

5) Sonuç

Anahtar teslim dosyalar:

Distilled model: MVP/artifacts/qwen35-distilled-n4/final/
GGUF (4 çeşit): MVP/artifacts/gguf/

6) Multimodal (vision) geri takildi — TEST EDILDI ✓

-- MVP/artifacts/qwen35-distilled-n4-multimodal/: student text (4 katman) + ogretmenin vision'i -- Vision = Qwen3_5VisionModel (ogretmenin kendi sinifi, Qwen2VL degil — MRO'da Qwen2VL yok, transformers/models/qwen3_5/modeling_qwen3_5.py) -- 153 vision tensoru ogretmenden kopyalandi (patch_embed, pos_embed, 36 blok, merger) → toplam 209 tensor / 692.2M (tied cift sayim; efektif ~458M) -- config.json: Qwen3_5ForConditionalGeneration, vision_config (hidden=768, depth=12, patch=16, merge=2, temporal=2), image_token_id=248056 -- preprocessor: HF reposunun kendi preprocessor_config.json (processor_class=Qwen3VLProcessor, image_processor_type=Qwen2VLImageProcessorFast) -- Forward test OK: LOGITS (1,84,248320), pixel_values [256,1536] (224x224 → 256 patch) -- Generate test OK: resim + soru → uretim zinciri calisiyor (cikti anlamsiz — vision egitimsiz, student 4 katman; pipeline dogrulamasi icin yeterli) -- Not: Qwen2VLImageProcessor = resmi sayiya ceviren on-islemci (HF'in kendi eslemesi, modelin vision'i degil); video_processor ve lm_head UNEXPECTED (tied) zararsiz -- Test scripti: MVP/test_multimodal_forward.py

7) Anahtar teslim paketi

-- Multimodal model: MVP/artifacts/qwen35-distilled-n4-multimodal/ (safetensors + config + tokenizer + preprocessor) -- Text GGUF (4 cesit): MVP/artifacts/gguf/ -- 10 yas anlatimi: MVP/BASIT_ANLATIM.md -- Test scripti: MVP/test_multimodal_forward.py -- GGUF'ta vision VAR (yanlis bilgi duzeltildi): uretici unsloth mmproj-F32.gguf yayinliyor (Qwen3.5'in kendi vision encoder'i) ve llama.cpp --mmproj ile calistiriyor

8) VISION GGUF — CALISIYOR (kanitli)

-- Kaynak: unsloth/Qwen3.5-0.8B-GGUF (uretici) — mmproj-F32.gguf (402MB) + Qwen3.5-0.8B-Q4_K_M.gguf (532MB, 24 katman TAM model) -- mmproj icerigi (GGUFReader ile okundu): v.patch_embd (2x conv [16,16,3,768] = temporal 2), v.position_embd (2304 = 3x768), v.blk.0-11 (12 blok: attn_qkv 2304, attn_out, ffn_up 3072, ffn_down, ln1/ln2), v.post_ln, mm.0/mm.2 (merger 3072→1024 = LLM hidden) -- clip metadata: patch_size=16, block_count=12, head_count=12, ffn=3072, projection_dim=1024, spatial_merge_size=2, mean/std=0.5/0.5/0.5, use_gelu, projector=qwen3vl_merger, is_deepstack=False -- Kirmizi resim testi: llama-cli --mmproj → [Start thinking] ... The color is a very bright, vibrant red ... [End thinking] Red (kanit: MVP/evidence/multimodal/llama_test2.clean.txt) -- Mavi resim testi: ... it's a standard blue ... definitely blue. (kanit: llama_test3.clean.txt) — gercek gorme, ezber degil -- modalities: text, vision, video — llama.cpp modeli vision olarak taniyor; Generation 78 t/s (RX460, Vulkan, Q4_K_M + F32 mmproj) -- Kendi distilled N=4 GGUF + ayni mmproj = yuklenip vision'i isledi (llama_test4.clean.txt — 'Loaded media' + '[Start thinking]') — projeksiyon (1024) uyumlu; cikti anlamsiz = 500K token distillasyon kalitesi, vision degil -- Ollama bu GGUF'lari calistiramaz (unsloth: "no Qwen3.5 GGUF works in Ollama due to separate mmproj vision files") — llama.cpp backend sart -- GGUF vision calistirma: MVP/run_multimodal.sh (tarif) + MVP/artifacts/gguf-vision/ (mmproj-F32 + text Q4_K_M) -- GGUF notu: llama.cpp qwen35 arch'inde vision tensoru YOK (text-only) — multimodal GGUF bu surumle uretilemez; vision yalnizca transformers uzerinden calisir

Distillation script: MVP/distill.py (resume destekli)

Sonraki adım önerileri (opsiyonel)

Daha koherent üretim için daha çok step / daha iyi dataset (wiki 500K token çok az)
Q4_K_M ~395MB — RX460 2GB VRAM'e tam oturur, mobil/edge için ideal
Öğretmeni bfloat16 yükleyerek train hızını artırma (önceki 108dk → ~60dk)
llama-bench ile tam benchmark matrisi, llama-server ile OpenAI-uyumlu API