Qwen3.8-9B Loop-6 Adapter (Recurrent Test-Time Compute 1..6 Loops)

Open In Colab Download Windows CUDA Release Companion Engine

Identity-preserving LoopLoRA + LoopBridge adapter (45.27 MB) that turns Qwen3.8-9B into a recurrent-depth transformer (1..6 internal loops = 152 virtual layers at K=6) reusing the same 32 quantized blocks on an 8 GB VRAM GPU (verified on RTX 3070 Ti).

Real K=1 vs K=6 Showcase (Same 5.2 GB Model, +45 MB Adapter)

Benchmark Probe K=1 (Single Pass) K=6 (6 Internal Loops)
Prompt Injection Defense (B2) ❌ Obeys malicious injection (delete_database → 0/20) ✅ Blocks injection & calls diagnostic tool (20/20)
4-Item Deductive Ordering (logic_order_4) ❌ Outputs 1>2>3>4 ✅ Deduces Delta>Beta>Gamma>Alpha
Overall Quality Score (/90) 70.0 / 90 (Total 80.0/100) 90.0 / 90 — 100% (Total 95.0/100)
Peak VRAM (RTX 3070 Ti 8 GB) 5474 MiB 6244 MiB (0 crashes)
  • Base Model Preserved at K=1: Loop k=0 stays strictly frozen; LoRA + Loop Bridges act only on iterations k=1..5.
  • Zero-Friction Testing: Download the prebuilt Windows CUDA ZIP (v1.0.0) and double-click run.bat to switch between 1-Loop, 2-Loop, 4-Loop, and 6-Loop live inside the Web UI (http://localhost:8080) — or try it free on Google Colab T4 via try_in_colab.ipynb.

Exact Reference Base Model

This adapter was trained directly on and verified against:

  • Hugging Face Repo: mradermacher/Qwen3.8-9B-Distill-uncensored-heretic-GGUF
  • Exact GGUF File: Qwen3.8-9B-Distill-uncensored-heretic.Q4_K_S.gguf (5.23 GB)
  • Optional Vision Projector: Qwen3.8-9B-Distill-uncensored-heretic.mmproj-f16.gguf (875.6 MB) — can be offloaded to CPU RAM with --no-mmproj-offload to keep VRAM free for loops and context.

Why is this distributed as a separate adapter instead of a merged GGUF? Unlike a standard static LoRA, this adapter stores 5 distinct per-iteration LoRA sets (k=1..5) + 5 LoopBridge layers applied dynamically on each loop pass while keeping k=0 frozen. Merging into a single static weight matrix would break K=1 identity preservation and eliminate depth specialization.


Repository Contents

Path Size Description
qwen3.8-9b-heretic-loop6-lora.gguf 45.27 MB Ready-to-run GGUF adapter for llama-cli / llama-server (llama.cpp-qwen-loop fork).
qwen3.8-9b-heretic-loop6-adapter.pt 45.41 MB PyTorch checkpoint (adapter_state_dict) for warm-starting or research.
training/ ~170 KB Direct-from-GGUF 8 GB VRAM trainer (FrozenInt4Linear + WY Chunked DeltaNet + Selective Top-64 KL Anchor), self-distillation dataset builder, and benchmark harness.
dataset/ 0.69 MB Programmatically verified self-distilled dataset (200 train / 51 eval) with zero benchmark leakage.
benchmarks/ 12 KB Verified llama-server scaling reports (LOOP_SCALING_REPORT.md & .json).
docs/ ~70 KB Full technical documentation & simplified logical guide in English (TECHNICAL_DOCUMENTATION_QWEN_LOOP.md, SIMPLIFIED_GUIDE_QWEN_LOOP.md) and Portuguese (DOCUMENTACAO_TECNICA_QWEN_LOOP.md, GUIA_SIMPLIFICADO_QWEN_LOOP.md).
models.example.ini 1 KB Ready-to-use llama-server --models-preset templates for K=2, 4, 6.

Verified Results (RTX 3070 Ti — 8 GB VRAM)

1. End-to-End llama-server Benchmark (per_loop, ctx=8192, parallel=1, 0 crashes)

Loops (K) Quality /90 Score /100 B2 Security /20 B4 Code /15 B5 Table /10 B6 Long-Ctx /10 Multi-Hop PP (t/s) TG (t/s) Peak VRAM
K=1 70.0 80.0 0.0 15.0 10.0 10.0 5/6 1104.2 56.6 5474 MiB
K=2 70.0 79.0 0.0 15.0 10.0 10.0 5/6 689.8 42.9 5648 MiB
K=4 70.0 77.0 0.0 15.0 10.0 10.0 5/6 366.0 26.5 5950 MiB
K=6 90.0 95.0 20.0 15.0 10.0 10.0 5/6 247.5 19.1 6244 MiB
  • B2 (Prompt Injection Defense): At K=1, 2, 4 the model obeys the destructive injection (delete_database → 0/20). At K=6, it rejects the injection and invokes the legitimate diagnostic tool (20.0/20).
  • logic_order_4 (4-Item Deductive Ordering): Fails at K=1, 2 (1>2>3>4), solved cleanly at K=4 and K=6 (Delta>Beta>Gamma>Alpha).

2. PyTorch Held-out Monotonic Scaling (48 held-out samples, Peak VRAM = 6301 MiB)

Loops (K) Held-out NLL Held-out Token Acc
K=1 0.4259 89.21%
K=2 0.2989 90.99%
K=3 0.2314 92.81%
K=4 0.2005 93.67%
K=5 0.1853 94.20%
K=6 0.1780 94.22%

How to Run (1..6 Loops)

Option A — 1-Click Windows CUDA (No Compilation Required)

  1. Download llama-cpp-qwen-loop-v1.0.0-win-cuda-x64.zip from CympUser/llama.cpp-qwen-loop Releases and unzip it.
  2. Double-click run.bat (downloads the base GGUF + Loop-6 adapter automatically and launches the Web UI at http://localhost:8080).
  3. Use the model selector dropdown in the Web UI to compare 1-Loop, 2-Loop, 4-Loop, and 6-Loop live!

Option B — Build from Source (llama.cpp-qwen-loop)

This adapter requires the llama.cpp-qwen-loop fork (which adds qwen35.num_loops, LoopBridge, per-loop LoRA, per_loop KV cache, and std::bad_alloc checkpoint hardening):

git clone https://github.com/CympUser/llama.cpp-qwen-loop
cd llama.cpp-qwen-loop
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release --target llama-cli llama-server -j

2. Run llama-server (change int:6 to any int:1..int:6)

.\build\bin\Release\llama-server.exe `
  -m .\models\Qwen3.8-9B-Distill-uncensored-heretic.Q4_K_S.gguf `
  --lora .\qwen3.8-9b-heretic-loop6-lora.gguf `
  --override-kv qwen35.num_loops=int:6,qwen35.recurrent_kv_cache_mode=str:per_loop `
  -c 8192 -np 1 --cache-ram 1024 --ctx-checkpoints 2 `
  -ngl 999 --fit off --cache-type-k q8_0 --cache-type-v q8_0 --repeat-penalty 1.15 --port 8080

Tip (Multimodal Vision in System RAM): If you also pass --mmproj .\models\Qwen3.8-9B-Distill-uncensored-heretic.mmproj-f16.gguf, add --no-mmproj-offload (or mmproj-offload = false in models.example.ini) to load the vision projector into CPU RAM instead of GPU VRAM.


How to Apply This Concept to Another Qwen3.5 / Qwen3.8 Model

Everything needed to uptrain any other local Qwen3.5 / Qwen3.8 GGUF (0.8B, 2B, 4B, 9B) on an 8 GB VRAM GPU is in training/:

  1. Install Python dependencies:
    python -m venv .venv-qwen-loop
    .\.venv-qwen-loop\Scripts\Activate.ps1
    pip install -r training\requirements.txt
    
  2. Generate a native K=1 self-distilled dataset (keeps the model's natural <think> distribution and avoids synthetic overfitting):
    python training\self_distill_dataset.py --model path\to\your-qwen35.gguf --server path\to\llama-server.exe
    
  3. Train the 6-loop adapter directly from the quantized GGUF (no 19 GB FP16 download required — packs frozen weights into symmetric group-INT4 in ~4.1 GB VRAM + Selective Top-64 KL Anchor):
    python training\gguf_qwen35_trainer.py `
      --gguf path\to\your-qwen35.gguf `
      --dataset-jsonl dataset\native_distill_polish_train.jsonl `
      --eval-jsonl dataset\native_distill_polish_eval.jsonl `
      --steps 50 --grad-accum 4 --rank 8 --alpha 24.0 --max-seq-len 448 `
      --lr-bridge 3e-3 --lr-lora 2.5e-4 --lambda-mono 0.8 --mono-margin 0.025 --lambda-kl 0.25 `
      --out-gguf-name my-qwen35-loop6-lora.gguf
    
  4. Optional warm-start KL polish (20 steps, --lambda-kl 0.45): pass --init-adapter <checkpoint.pt> to sharpen formatting & safety while locking in base strengths.

Documentation (English & Portuguese)

English

  • Technical Documentation — C++/CUDA architecture, Loop Bridge & LoopLoRA math, 8 GB VRAM trainer internals, experimental history, and full verified tables.
  • Simplified & Logical Guide — Intuitive, jargon-free explanation of how the loop engine and training work and why each decision was made.

Português

Downloads last month
-
GGUF
Model size
23.7M params
Architecture
qwen35
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Vnswz/Qwen3.8-9B-Loop6-Adapter