🌀 vxtral-loopus-4b (True Loopformer)

vxtral-loopus-4b is a contractive recurrent language model powered by the LoopUS architecture. It implements an adaptive reasoning loop with Mamba-style SelectiveGate residual mixing, enabling dynamic inference-time depth scaling ($N \in [1..8]$) without altering model weights.

👥 Model Overview

• Architecture: Recurrent Contractive Transformer (VxtralForCausalLM)
• Base Backbone: synthetic-voxtral-text-3b_tekken
• Tokenizer: Tekken
• Checkpointed Step: step_46000 (FineWeb dataset)
• Recurrent Depth Range ($N$): $1 \le N \le 8$
• Parameters: 3.2B base + recurrent reasoning loop (4B effective capacity at $N=8$)

---

🚀 Quickstart & Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "vxltxrllc/vxtral-loopus-4b"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="cuda",
    trust_remote_code=True,
)

prompt = "The future of artificial intelligence is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

# Dynamically adjust the reasoning depth on the fly:
# N=1 (fast surface generation), N=4 (balanced reasoning), N=8 (deep contractive synthesis)
model.N = 4

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=60,
        do_sample=False,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📊 Dynamic Depth Scaling Behavior ($N$)

The recurrent reasoning block utilizes a Mamba-style SelectiveGate to perform a convex combination of activations between passes:

$$h_{l+1}=\bar{A}\cdot h_{\text{new}}+(1-\bar{A})\cdot h_{\text{old}}h\_\{l+1\}\; =\; \backslash bar\{A\}\; \backslash cdot\; h\_\{\backslash text\{new\}\}\; +\; (1\; -\; \backslash bar\{A\})\; \backslash cdot\; h\_\{\backslash text\{old\}\}$$

Changing $N$ shifts the reasoning trajectory dynamically during inference:

Depth ($N$)	Generation Profile	Output Characteristics
$N = 1, 2$	Surface-level completion	Fast response, relies on local token transitions and standard templates.
$N = 4$	Structural expansion	Balanced reasoning, introduces more detailed contextual relationships.
$N = 8$	Deep contractive synthesis	High-level abstraction, broader conceptual synthesis and systemic implications.

---

🛠️ Model Architecture Details

[Input Embeddings] ──> [Encoder Layers: 0..21]
                               │
                               ▼
               ┌───────────────────────────────┐
               │ Recurrent Loop (Passes: 1..N) │
               │   Reasoning Layers: 22..28    │
               │   Gate: SelectiveGate(dt, A)  │
               └───────────────┬───────────────┘
                               │
                               ▼
                       [Decoder Layer: 29] ──> [RMSNorm + LM Head]

• Encoder: 22 standard Transformer layers.
• Reasoning Loop: 7 recurrent Transformer layers bound by SelectiveGate.
• Decoder: 1 final alignment layer.
• Early Exit: Integrated $Q$-oracle for dynamic halting.