README.md

---
license: apache-2.0
tags:
- scrapegoat
- dual-track
- parallel-attention
- moe
- kda
- quantile-balancing
- transformers
library_name: transformers
pipeline_tag: text-generation
datasets:
- Glint-Research/Fable-5-traces
- armand0e/claude-fable-5-claude-code
- lordx64/agentic-distill-fable-5-sft
- HelioAI/Fable-5-Distill-Reasoning-462x
- >-
  Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset
- greghavens/fable-5-coding-and-debugging-traces
- Mike0021/claude-kimi-k3-game-session
- gbeck/kimi-k3-coding-and-debugging-traces
---

<div align="center">

  <h1 align="center">🐐 ScrapeGoat</h1>

  <p><strong>Parallel Dual‑Track Transformer Architecture • 81 Layers • 825B Parameters</strong></p>

  <p>
    <strong>Track A (Left Brain — Analytical – State‑Aware Code Generation)</strong><br>
    <strong>Track B (Right Brain — Holistic – System Design)</strong><br>
    <strong>= Unified Intelligence</strong>
  </p>

  <p>
    <a href="https://opensource.org/licenses/Apache-2.0">
      <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License">
    </a>
  </p>

</div>

---


![ScrapeGoat Architecture](https://huggingface.co/scrapegoat/Scrapegoat-Tiny-Coder/resolve/main/agent.png)

---


## 🌟 Overview

ScrapeGoat is a **parallel dual-track transformer** modeled after the human brain's hemispheric specialization — Track A (analytical **left brain**) for deep sequential reasoning via KDA recurrent attention, Track B (holistic **right brain**) for broad parallel processing via GQA. Both tracks run simultaneously at every layer, united by learned gating. This architecture is purpose-built for **humanoid embodied intelligence**, combining the precision needed for logic, motor planning, and sequential reasoning with the flexibility required for natural language, spatial awareness, and multimodal perception. The architecture is derived from ScrapeGoat's Agentic Modelling and DeepSeek's DSpark innovations, with custom modifications including KDA (Kimi Delta Attention), Quantile Balancing MoE, and Mixture-of-Memories (MoM).
In ScrapeGoat-Tiny-Coder context: detailed state aware code generation (Track A) and holistic system design (Track B).

### 🧠 Humanoid Brain Readiness

| Cognitive Function | Brain Analog | ScrapeGoat Implementation |
|--------------------|-------------|--------------------------|
| **Logical reasoning** | Left prefrontal cortex | Track A KDA — sequential, stateful token processing |
| **Pattern recognition** | Right temporal/parietal | Track B GQA — parallel multi-head attention |
| **Motor planning** | Motor cortex / basal ganglia | 704 MoE experts = cortical column specialization, 8 active per token |
| **Episodic memory** | Hippocampus | MoM-KDA — 4 memory states with learned routing |
| **Sensory integration** | Corpus callosum | Per-layer learned gating (`attn_track_gate`, `moe_track_gate`) blends tracks |
| **Long-context awareness** | Working memory | 262K token context window (RoPE θ=10,000) |
| **Real-time control** | Reflex arcs | Low-latency KDA recurrent inference (O(1) per token, no KV cache quadratic blowup) |

---


### Key Innovations

| Feature | Track A (Left Brain — Analytical) | Track B (Right Brain — Holistic) 
|---------|---------|---------|
| **Attention** | 32 heads × 256 dim (KDA/GQA 3:1 interleaved) | 64 heads × 128 dim (GQA) |
| **Key-Value Heads** | 2 | 8 |
| **MoE Experts** | 512 fused | 192 stacked |
| **MoE Intermediate** | 1024 | 1536 |
| **Routing** | Quantile Balancing (QB) | Quantile Balancing (QB) |
| **Shared Expert** | ✅ Shared across both tracks |


## 🏗️ Architecture

### Parallel Dual-Track Processing

Both tracks process every token in parallel at each of the 81 layers, with learned gating per layer for attention (`attn_track_gate`) and MoE (`moe_track_gate`):

| Layer 0 (Special) | Layers 1–80 |
|-|-|
| Track A: MoE only (no attn) | Track A: KDA attention + MoE |
| Track B: Attention + dense FFN (no MoE) | Track B: GQA attention + MoE |

### ⚡ KDA (Kimi Delta Attention)

Recurrent linear attention with diagonal gating:
```
S_t = (I - β_t k_t k_t^T) Diag(α_t) S_{t-1} + β_t k_t v_t^T
```
- Supports recurrent (inference) and chunkwise (training) modes
- ShortConv preprocessing for Q/K/V projections (kernel=3)
- 3:1 KDA:GQA interleaving — 60 KDA layers + 21 GQA layers (every 4th: `kda_gqa_layers = [0,4,8,...,80]`)
- Per-head diagonal gating: `α_t = sigmoid(W_g · x_t)`, applied element-wise over the recurrence
- ShortConv (1D conv, kernel=3) preprocesses Q/K/V before the KDA recurrence for stability

### 🎯 Quantile Balancing for MoE

Alternating quantile algorithm (J. Su) computes per-expert biases that equalize token assignment — entirely hyperparameter-free (no temperature, no aux loss).
```
β ← 0
q = 1 - k/n
for _ in range(qb_iterations):
    α = quantile(scores - β, q, dim=1)
    β = quantile(scores - α, q, dim=0)
bias = β
routing_weights = softmax(router_logits - bias, dim=-1)
```
8 experts selected per token across 512 (Track A) + 192 (Track B) experts.

### 🌊 Mixture-of-Memories (MoM-KDA)

- Multiple independent KDA memory states with learned routing
- Shared memory always active + top-k memory selection per token (`top_k = num_experts_per_tok = 8`)
- 4 memories total, 2 active per token selected via softmax over router logits
- MoM integrates with KDA recurrence — each memory state has its own KDA layer with independent shortconv Q/K/V projections

### 📐 Block Attention Residuals (v2)

Softmax attention over layer depth — each layer attends over prior block representations via a learned pseudo-query.

```
block_size = num_hidden_layers // attn_res_blocks   # 81 // 8 = 10
block_idx  = layer_idx // block_size                # which block this layer belongs to

q = pseudo_query  (learned, [hidden_size])
keys = LayerNorm(block_reps)  [num_blocks, hidden_size]
attn_weights = softmax(q · keys^T / sqrt(hidden_size))
depth_out = attn_weights @ block_reps   [hidden_size]
hidden_states = hidden_states + depth_out   (residual)
```

- `attn_residual=false` — disabled in this checkpoint, 8 blocks when enabled
- Activated at every layer but only reads from prior blocks' representations
- Provides long-range depth-wise communication across the 81 layers; stabilises training on deep MoE architectures

### 🔄 StableMoE Stage 1

- Progressive routing stabilisation: `router` weights are frozen (`requires_grad = False`) in stage 2, preventing expert collapse during continued pretraining/LoRA fine-tuning
- Reduces expert representation collapse during training

---

## 📊 Model Specifications

| Parameter | Value |
|-----------|-------|
| **Parameters** | ~825B |
| **Hidden Size** | 4096 |
| **Layers** | 81 |
| **Vocab Size** | 248,320 (tiktoken BPE) |
| **Max Position** | 262,144 (RoPE θ=10,000) |
| **Track A Heads** | 32 (KV: 2, head dim: 256) |
| **Track A Experts** | 512 (intermediate: 1024) |
| **Track B Heads** | 64 (KV: 8, head dim: 128) |
| **Track B Experts** | 192 (intermediate: 1536) |
| **Track B Dense FFN** | 13312 (layer 0 only) |
| **Experts per Token** | 8 |
| **Activation** | SiLU |
| **Norm** | RMSNorm (ε=1e-6) |
| **MoM Memories** | 4 (top-2 active) |
| **StableMoE Stage** | 1 |
| **Attn Residual** | false (configurable, 8 blocks) |
| **Weight Format** | BF16 |

### Weights

83 shards × ~20 GB each = **~1.65 TB total** (BF16 safetensors). Will be stored in the repo itself as `scrapegoat-fp8/`.

#### Notable Weight Name Differences from Model Code

| Weight Prefix | Mapped To | Notes |
|---|---|---|
| `q_norm`, `k_norm` | Attention QK LayerNorm | Extra weights present in checkpoint, accepted via `strict=False` |
| `expert_bias` | Track B MoE expert bias | Same |
| `shared_expert.gate_proj` | Shared expert gate | Fused in checkpoint |

---

## 🚀 Quick Start

### Installation
```bash
pip install transformers accelerate safetensors
```

### Loading for Inference
```python
import sys
sys.path.insert(0, "/path/to/model/dir")
from configuration_scrapegoat import ScrapeGoatConfig
from modeling_scrapegoat import ScrapeGoatForCausalLM
from transformers import AutoTokenizer
import torch

model_dir = "/path/to/scrapegoat-weights"

tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = ScrapeGoatForCausalLM.from_pretrained(
    model_dir,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
```

### Tokenizer

ScrapeGoat uses the **tiktoken BPE tokenizer** (`tiktoken.model`). The vocabulary is 248,320 tokens (163,584 base BPE merges + 256 special/control tokens). It ships with the weights in the repo.

- **BPE on bytes** — lossless encoding of any Unicode/UTF-8, no OOV tokens
- **256 special token slots** for `[BOS]`, `[EOS]`, `[PAD]`, `[UNK]`, role markers, tool call tokens, media tokens, etc.
- **Default backend**: `GigaKimiTokenizer` — a HuggingFace-compatible wrapper around Gigatoken (~750× faster than native tiktoken)

**Special tokens:**

| Token | ID | Purpose |
|-------|----|---------|
| `[BOS]` | 163584 | Begin of sequence |
| `[EOS]` | 163585 | End of sequence |
| `[PAD]` | 163839 | Padding |
| `[UNK]` | 163838 | Unknown |
| `[EOT]` | 163593 | End of turn |
| `<\|im_end\|>` | 163586 | Chat message end |
| `<\|im_user\|>` | 163587 | User role marker |
| `<\|im_assistant\|>` | 163588 | Assistant role marker |
| `<\|im_system\|>` | 163594 | System role marker |
| `<\|tool_calls_section_begin\|>` | 163595 | Tool calls section |
| `<\|tool_call_begin\|>` | 163597 | Individual tool call |
| `<\|media_begin\|>` | 163602 | Media/vision content |
| `<think>` | 163606 | Reasoning section begin |
| `</think>` | 163607 | Reasoning section end |

### Optimiser: NorMuon

`optimiser/normuon.py` implements [NorMuon](https://github.com/KellerJordan/Muon), a Muon variant that orthogonalises weight updates via Newton-Schulz iteration and normalises update norms via a second-momentum running average. Available as `NorMuon` (distributed), `SingleDeviceNorMuon`, `NorMuonWithAuxAdam`, and `SingleDeviceNorMuonWithAuxAdam` for training resumption and continued pretraining. See `README.md` in the repo root for usage.

### Training (SFT with DeepSpeed ZeRO-3)

Due to the 1.65 TB model size, training requires NVMe offloading. See `train_scrapegoat_sft.py` in the `Training/` directory for the SFT script (DeepSpeed ZeRO-3, LoRA).

Key config: DeepSpeed ZeRO-3 with `offload_param` and `offload_optimizer` to CPU, `train_micro_batch_size_per_gpu=1`, `gradient_accumulation_steps=8`.

---

## 🔧 Configuration

```python
from configuration_scrapegoat import ScrapeGoatConfig

config = ScrapeGoatConfig.from_pretrained("/path/to/model")

# Track A
config.track_a_num_attention_heads    # 32
config.track_a_num_key_value_heads    # 2
config.track_a_head_dim               # 256
config.track_a_num_experts            # 512
config.track_a_moe_intermediate_size  # 1024

# Track B
config.track_b_num_attention_heads    # 64
config.track_b_num_key_value_heads    # 8
config.track_b_head_dim               # 128
config.track_b_num_experts            # 192
config.track_b_moe_intermediate_size  # 1536

# Attention
config.kda_gqa_layers                 # [0,4,8,12,...,80] (every 4th = GQA)
config.kda_conv_kernel                # 3
config.attn_residual                  # false (configurable, 8 blocks)

# MoE
config.quantile_balancing             # True
config.qb_iterations                  # 5
config.num_experts_per_tok            # 8
config.stable_moe_stage               # 1
config.stable_moe_r3                  # False
config.stable_moe_r3_cache            # True

# MoM-KDA
config.mom_enabled                    # True
config.mom_num_memories               # 4
config.mom_active_memories            # 2
config.mom_shared_memory              # True

# DSpark
config.dspark_block_size              # 6
config.dspark_markov_rank             # 256
config.dspark_noise_token_id          # 0
config.dspark_target_layer_ids        # [] (empty = apply to all layers)
config.output_router_logits            # false (also returns router_logits in model output when enabled)
```


---

## 🛠️ Hardware Requirements

| Setup | VRAM/RAM | Notes |
|-------|----------|-------|
| **Inference (BF16)** | ~1.65 TB | Multi-GPU cluster (e.g., 24×H100 80GB) |
| **Inference (4-bit)** | ~412 GB | 5+ H100 80GB |
| **QLoRA SFT** | ~180GB RAM + 750GB NVMe | ZeRO-3 + NVMe offload (single H100) |
| **Full Training** | Multi-node cluster | 1.65 TB+ |

---

## 📝 Citation

```bibtex
@misc{scrapegoat2026,
  title={ScrapeGoat: Parallel Dual-Track Transformer with KDA and Quantile Balancing},
  author={ScrapeGoat Team},
  year={2026},
}
```

## Coding Agent Framework

The most powerful AI coding agent on planet Earth is now a self-learning model agnostic harness.

/fast mode and more advanced skills enabled only with scrapegoat models

/graph - indexes your repo locally and auto-injects repo context. Excoder is always code aware.

/Agent-WebBridge - Browser automation via Agent-WebBridge-skill and Agent-WebBridge for QA automation and Browser Autonomy.

/dispatching-parallel-agents - To dispatch Swarm of agents for parallel work.

Team | Enterprise: run excoder, pick a username on first launch, then use /message @teammate to chat with org members in real time with secure encrypted messaging.

/redteam-skill & /pentest-skill for offensive and advanced cyber security capabilities.

- Application Security Testing — Detect and validate critical vulnerabilities in your applications
- Rapid Penetration Testing — Get penetration tests done in hours, not weeks, with compliance reports
- Bug Bounty Automation — Automate bug bounty research and generate PoCs for faster reporting
- CI/CD Integration — Run tests in CI/CD to block vulnerabilities before reaching production
- 🕸️ Web apps | Black-box, external-attacker recon → exploit (XBEN suite) | ✅
- 🚩 CTF | Hint-free, sandbox-jailed solves (Cybench) | ✅
- 🤖 Robotics / OT / embedded | Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) | ✅
- 📂 Source code | White-box repo analysis with blind master-builder decomposition
- 💰 Smart contracts | Damn Vulnerable DeFi | ⚠️ reproduction
- ☁️ Cloud (IaC) | Misconfig-detection benchmark (`cloud:bench`) + opt-in cloud arsenal | 🚧 IaC-misconfig scaffolding — live-cloud exploitation
- 📱 Mobile | Built-in static analyzer (manifest misconfig + secret/cleartext detection, `mobile:bench`) + opt-in arsenal (mobsfscan/objection/drozer; frida gated)
- 🔩 Binary / RE | Decompiled-output sink detector (unsafe-copy / format-string / cmd-injection / int-overflow, `binary:bench`)

[![Excoder.pro](https://i.ibb.co/BVBcMWLy/Screenshot-2026-07-28-at-1-29-24-PM.png)](https://excoder.pro)


---

<div align="center">
   <strong>Left brain • Right brain • One mind. 🐐</strong>
</div>