VeriLoop-E2-oQ4e-mtp

Overview

VeriLoop-E2 is a variant of Qwen3.5-27B that adds a conditional memory registry (VeriLoop E2) to the MTP (Multi-Token Prediction) head. The registry stores structured memory capsules — keyed by project, topic, or domain — that can be activated during inference for context-aware generation. The model retains the hybrid GatedDeltaNet + full attention architecture of the base Qwen3.5 model.

Upstream model: tsinghua-sigs-robot-lab/VeriLoop-E2 (Tsinghua SIGS Robot Lab, Libo Wang)

Architecture

Component	Configuration
Model type	Qwen3_5ForConditionalGeneration
Total layers	64
Attention pattern	48 linear attention + 16 full attention (3:1 ratio)
Full attention interval	Every 4th layer
Hidden size	5,120
Intermediate size	17,408
Attention heads	24 (full attention), 16 key + 48 value (linear attention)
Head dimension	256
Vocab size	248,320
Max position	262,144 tokens
RoPE	Partial (0.25), θ=10,000,000, M-RoPE interleaved [11,11,10]
Activation	SiLU
Norm	RMSNorm (ε=1e-6)

Vision Tower (VLM)

Parameter	Value
Model type	Qwen3.5 vision
Hidden size	1,152
Depth	27 layers
Heads	16
Patch size	16
Spatial merge	2
Intermediate size	4,304

Multi-Token Prediction (MTP)

Parameter	Value
MTP layers	1
Dedicated embeddings	No (shared with base model)

VeriLoop Conditional Memory

Parameter	Value
Format	VeriLoop-E2-Conditional-Memory-Registration-v2
Enabled	Yes
Tensor count	34
Routing	semantic_gate_embedded_model_shards
Physical embedding	embedded_candidate
Runtime activation	embedded_loader_cpu_verified_gpu_pending
Storage	ordinary_model_shards_vllm_ignored_mtp_namespace
Storage prefix	mtp.veriloop_

The VeriLoop registry stores memory capsules as model weights under the language_model.mtp.veriloop_engram_registry.* namespace. Each capsule contains:

• Text content (UTF-8)
• Token IDs and offsets
• Topic and project keys
• Semantic templates and graph edges
• Capsule policy and priority

This design allows the memory to be loaded by standard weight loaders while being ignored by vLLM (which skips mtp.* prefixes).

Quantization

Setting	Value
Method	oQ4e (enhanced affine)
Default bits	4
Group size	64
Mode	Affine

Mixed-precision layers

• Most linear_attn.in_proj_*, linear_attn.out_proj, mlp.down_proj, self_attn.o_proj: 5-bit (affine, group_size 64)
• VeriLoop registry tensors: 4-bit (default)

Validation

oMLX (Apple Silicon)

• Load: ✅ Model loads successfully
• Tokenizer: ✅ Tokenizer loads and encodes correctly
• Text generation: ✅ Validated via Hermes Agent + oMLX

Status: Model produces coherent output when served via oMLX.

Usage

oMLX (Apple Silicon)

from mlx_lm import load, generate

model, tokenizer = load(
    "hermitdave/VeriLoop-E2-oQ4e-mtp"
)

prompt = "What are the key principles of reinforcement learning?"
response = generate(model, tokenizer, prompt, max_tokens=512)
print(response)

Memory Registry

The VeriLoop memory registry is embedded in the model weights. To inspect or modify the registry:

from safetensors import safe_open
import os

model_dir = "path/to/VeriLoop-E2-oQ4e-mtp"
for shard in sorted(os.listdir(model_dir)):
    if shard.endswith(".safetensors"):
        with safe_open(os.path.join(model_dir, shard), framework="pt") as f:
            for key in f.keys():
                if "veriloop" in key.lower():
                    print(key)

Memory Requirements

Device	Memory
M3 Max 64GB	~17 GB (measured peak)
M2 Ultra 128GB	~17 GB
Minimum recommended	24 GB RAM

Citation

@misc{veriloop-e2,
  title={VeriLoop E2: 27B Post-Trained Model for Code, Mathematics, and Physics},
  author={Tsinghua SIGS Robot Lab, Libo Wang},
  year={2026},
  url={https://huggingface.co/tsinghua-sigs-robot-lab/VeriLoop-E2}
}

Attribution

This MLX quantization was prepared by Hermes Agent (Nous Research).