Instructions to use scrapegoat/Scrapegoat-Tiny-Coder with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use scrapegoat/Scrapegoat-Tiny-Coder with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="scrapegoat/Scrapegoat-Tiny-Coder")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("scrapegoat/Scrapegoat-Tiny-Coder", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use scrapegoat/Scrapegoat-Tiny-Coder with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "scrapegoat/Scrapegoat-Tiny-Coder"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrapegoat/Scrapegoat-Tiny-Coder",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/scrapegoat/Scrapegoat-Tiny-Coder

SGLang

How to use scrapegoat/Scrapegoat-Tiny-Coder with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "scrapegoat/Scrapegoat-Tiny-Coder" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrapegoat/Scrapegoat-Tiny-Coder",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "scrapegoat/Scrapegoat-Tiny-Coder" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "scrapegoat/Scrapegoat-Tiny-Coder",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use scrapegoat/Scrapegoat-Tiny-Coder with Docker Model Runner:
```
docker model run hf.co/scrapegoat/Scrapegoat-Tiny-Coder
```

Scrapegoat-Tiny-Coder / README.md

Nathan9

Update README.md

dbfbbc7 verified 3 days ago

preview code

Raw

History Blame Contribute Delete

15 kB

	---
	license: apache-2.0
	tags:
	- scrapegoat
	- dual-track
	- parallel-attention
	- moe
	- kda
	- quantile-balancing
	- transformers
	library_name: transformers
	pipeline_tag: text-generation
	datasets:
	- Glint-Research/Fable-5-traces
	- armand0e/claude-fable-5-claude-code
	- lordx64/agentic-distill-fable-5-sft
	- HelioAI/Fable-5-Distill-Reasoning-462x
	- >-
	Manusagents/GPT-5.5-Gemini-3.1-Pro-Grok-4-Claude-Fable-5-Mythos-5-Qwen-3.7-Max-and-more-Distillation-Dataset
	- greghavens/fable-5-coding-and-debugging-traces
	- Mike0021/claude-kimi-k3-game-session
	- gbeck/kimi-k3-coding-and-debugging-traces
	---

	<div align="center">

	<h1 align="center">🐐 ScrapeGoat</h1>

	<p><strong>Parallel Dual‑Track Transformer Architecture • 81 Layers • 825B Parameters</strong></p>

	<p>
	<strong>Track A (Left Brain — Analytical – State‑Aware Code Generation)</strong><br>
	<strong>Track B (Right Brain — Holistic – System Design)</strong><br>
	<strong>= Unified Intelligence</strong>
	</p>

	<p>
	<a href="https://opensource.org/licenses/Apache-2.0">
	<img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License">
	</a>
	</p>

	</div>

	---


	![ScrapeGoat Architecture](https://huggingface.co/scrapegoat/Scrapegoat-Tiny-Coder/resolve/main/agent.png)

	---


	## 🌟 Overview

	ScrapeGoat is a parallel dual-track transformer modeled after the human brain's hemispheric specialization — Track A (analytical left brain) for deep sequential reasoning via KDA recurrent attention, Track B (holistic right brain) for broad parallel processing via GQA. Both tracks run simultaneously at every layer, united by learned gating. This architecture is purpose-built for humanoid embodied intelligence, combining the precision needed for logic, motor planning, and sequential reasoning with the flexibility required for natural language, spatial awareness, and multimodal perception. The architecture is derived from ScrapeGoat's Agentic Modelling and DeepSeek's DSpark innovations, with custom modifications including KDA (Kimi Delta Attention), Quantile Balancing MoE, and Mixture-of-Memories (MoM).
	In ScrapeGoat-Tiny-Coder context: detailed state aware code generation (Track A) and holistic system design (Track B).

	### 🧠 Humanoid Brain Readiness

	\| Cognitive Function \| Brain Analog \| ScrapeGoat Implementation \|
	\|--------------------\|-------------\|--------------------------\|
	\| Logical reasoning \| Left prefrontal cortex \| Track A KDA — sequential, stateful token processing \|
	\| Pattern recognition \| Right temporal/parietal \| Track B GQA — parallel multi-head attention \|
	\| Motor planning \| Motor cortex / basal ganglia \| 704 MoE experts = cortical column specialization, 8 active per token \|
	\| Episodic memory \| Hippocampus \| MoM-KDA — 4 memory states with learned routing \|
	\| Sensory integration \| Corpus callosum \| Per-layer learned gating (`attn_track_gate`, `moe_track_gate`) blends tracks \|
	\| Long-context awareness \| Working memory \| 262K token context window (RoPE θ=10,000) \|
	\| Real-time control \| Reflex arcs \| Low-latency KDA recurrent inference (O(1) per token, no KV cache quadratic blowup) \|

	---


	### Key Innovations

	\| Feature \| Track A (Left Brain — Analytical) \| Track B (Right Brain — Holistic)
	\|---------\|---------\|---------\|
	\| Attention \| 32 heads × 256 dim (KDA/GQA 3:1 interleaved) \| 64 heads × 128 dim (GQA) \|
	\| Key-Value Heads \| 2 \| 8 \|
	\| MoE Experts \| 512 fused \| 192 stacked \|
	\| MoE Intermediate \| 1024 \| 1536 \|
	\| Routing \| Quantile Balancing (QB) \| Quantile Balancing (QB) \|
	\| Shared Expert \| ✅ Shared across both tracks \|


	## 🏗️ Architecture

	### Parallel Dual-Track Processing

	Both tracks process every token in parallel at each of the 81 layers, with learned gating per layer for attention (`attn_track_gate`) and MoE (`moe_track_gate`):

	\| Layer 0 (Special) \| Layers 1–80 \|
	\|-\|-\|
	\| Track A: MoE only (no attn) \| Track A: KDA attention + MoE \|
	\| Track B: Attention + dense FFN (no MoE) \| Track B: GQA attention + MoE \|

	### ⚡ KDA (Kimi Delta Attention)

	Recurrent linear attention with diagonal gating:
	```
	S_t = (I - β_t k_t k_t^T) Diag(α_t) S_{t-1} + β_t k_t v_t^T
	```
	- Supports recurrent (inference) and chunkwise (training) modes
	- ShortConv preprocessing for Q/K/V projections (kernel=3)
	- 3:1 KDA:GQA interleaving — 60 KDA layers + 21 GQA layers (every 4th: `kda_gqa_layers = [0,4,8,...,80]`)
	- Per-head diagonal gating: `α_t = sigmoid(W_g · x_t)`, applied element-wise over the recurrence
	- ShortConv (1D conv, kernel=3) preprocesses Q/K/V before the KDA recurrence for stability

	### 🎯 Quantile Balancing for MoE

	Alternating quantile algorithm (J. Su) computes per-expert biases that equalize token assignment — entirely hyperparameter-free (no temperature, no aux loss).
	```
	β ← 0
	q = 1 - k/n
	for _ in range(qb_iterations):
	α = quantile(scores - β, q, dim=1)
	β = quantile(scores - α, q, dim=0)
	bias = β
	routing_weights = softmax(router_logits - bias, dim=-1)
	```
	8 experts selected per token across 512 (Track A) + 192 (Track B) experts.

	### 🌊 Mixture-of-Memories (MoM-KDA)

	- Multiple independent KDA memory states with learned routing
	- Shared memory always active + top-k memory selection per token (`top_k = num_experts_per_tok = 8`)
	- 4 memories total, 2 active per token selected via softmax over router logits
	- MoM integrates with KDA recurrence — each memory state has its own KDA layer with independent shortconv Q/K/V projections

	### 📐 Block Attention Residuals (v2)

	Softmax attention over layer depth — each layer attends over prior block representations via a learned pseudo-query.

	```
	block_size = num_hidden_layers // attn_res_blocks # 81 // 8 = 10
	block_idx = layer_idx // block_size # which block this layer belongs to

	q = pseudo_query (learned, [hidden_size])
	keys = LayerNorm(block_reps) [num_blocks, hidden_size]
	attn_weights = softmax(q · keys^T / sqrt(hidden_size))
	depth_out = attn_weights @ block_reps [hidden_size]
	hidden_states = hidden_states + depth_out (residual)
	```

	- `attn_residual=false` — disabled in this checkpoint, 8 blocks when enabled
	- Activated at every layer but only reads from prior blocks' representations
	- Provides long-range depth-wise communication across the 81 layers; stabilises training on deep MoE architectures

	### 🔄 StableMoE Stage 1

	- Progressive routing stabilisation: `router` weights are frozen (`requires_grad = False`) in stage 2, preventing expert collapse during continued pretraining/LoRA fine-tuning
	- Reduces expert representation collapse during training

	---

	## 📊 Model Specifications

	\| Parameter \| Value \|
	\|-----------\|-------\|
	\| Parameters \| ~825B \|
	\| Hidden Size \| 4096 \|
	\| Layers \| 81 \|
	\| Vocab Size \| 248,320 (tiktoken BPE) \|
	\| Max Position \| 262,144 (RoPE θ=10,000) \|
	\| Track A Heads \| 32 (KV: 2, head dim: 256) \|
	\| Track A Experts \| 512 (intermediate: 1024) \|
	\| Track B Heads \| 64 (KV: 8, head dim: 128) \|
	\| Track B Experts \| 192 (intermediate: 1536) \|
	\| Track B Dense FFN \| 13312 (layer 0 only) \|
	\| Experts per Token \| 8 \|
	\| Activation \| SiLU \|
	\| Norm \| RMSNorm (ε=1e-6) \|
	\| MoM Memories \| 4 (top-2 active) \|
	\| StableMoE Stage \| 1 \|
	\| Attn Residual \| false (configurable, 8 blocks) \|
	\| Weight Format \| BF16 \|

	### Weights

	83 shards × ~20 GB each = ~1.65 TB total (BF16 safetensors). Will be stored in the repo itself as `scrapegoat-fp8/`.

	#### Notable Weight Name Differences from Model Code

	\| Weight Prefix \| Mapped To \| Notes \|
	\|---\|---\|---\|
	\| `q_norm`, `k_norm` \| Attention QK LayerNorm \| Extra weights present in checkpoint, accepted via `strict=False` \|
	\| `expert_bias` \| Track B MoE expert bias \| Same \|
	\| `shared_expert.gate_proj` \| Shared expert gate \| Fused in checkpoint \|

	---

	## 🚀 Quick Start

	### Installation
	```bash
	pip install transformers accelerate safetensors
	```

	### Loading for Inference
	```python
	import sys
	sys.path.insert(0, "/path/to/model/dir")
	from configuration_scrapegoat import ScrapeGoatConfig
	from modeling_scrapegoat import ScrapeGoatForCausalLM
	from transformers import AutoTokenizer
	import torch

	model_dir = "/path/to/scrapegoat-weights"

	tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
	model = ScrapeGoatForCausalLM.from_pretrained(
	model_dir,
	torch_dtype=torch.bfloat16,
	device_map="auto",
	trust_remote_code=True,
	)
	```

	### Tokenizer

	ScrapeGoat uses the tiktoken BPE tokenizer (`tiktoken.model`). The vocabulary is 248,320 tokens (163,584 base BPE merges + 256 special/control tokens). It ships with the weights in the repo.

	- BPE on bytes — lossless encoding of any Unicode/UTF-8, no OOV tokens
	- 256 special token slots for `[BOS]`, `[EOS]`, `[PAD]`, `[UNK]`, role markers, tool call tokens, media tokens, etc.
	- Default backend: `GigaKimiTokenizer` — a HuggingFace-compatible wrapper around Gigatoken (~750× faster than native tiktoken)

	Special tokens:

	\| Token \| ID \| Purpose \|
	\|-------\|----\|---------\|
	\| `[BOS]` \| 163584 \| Begin of sequence \|
	\| `[EOS]` \| 163585 \| End of sequence \|
	\| `[PAD]` \| 163839 \| Padding \|
	\| `[UNK]` \| 163838 \| Unknown \|
	\| `[EOT]` \| 163593 \| End of turn \|
	\| `<\\|im_end\\|>` \| 163586 \| Chat message end \|
	\| `<\\|im_user\\|>` \| 163587 \| User role marker \|
	\| `<\\|im_assistant\\|>` \| 163588 \| Assistant role marker \|
	\| `<\\|im_system\\|>` \| 163594 \| System role marker \|
	\| `<\\|tool_calls_section_begin\\|>` \| 163595 \| Tool calls section \|
	\| `<\\|tool_call_begin\\|>` \| 163597 \| Individual tool call \|
	\| `<\\|media_begin\\|>` \| 163602 \| Media/vision content \|
	\| `<think>` \| 163606 \| Reasoning section begin \|
	\| `</think>` \| 163607 \| Reasoning section end \|

	### Optimiser: NorMuon

	`optimiser/normuon.py` implements [NorMuon](https://github.com/KellerJordan/Muon), a Muon variant that orthogonalises weight updates via Newton-Schulz iteration and normalises update norms via a second-momentum running average. Available as `NorMuon` (distributed), `SingleDeviceNorMuon`, `NorMuonWithAuxAdam`, and `SingleDeviceNorMuonWithAuxAdam` for training resumption and continued pretraining. See `README.md` in the repo root for usage.

	### Training (SFT with DeepSpeed ZeRO-3)

	Due to the 1.65 TB model size, training requires NVMe offloading. See `train_scrapegoat_sft.py` in the `Training/` directory for the SFT script (DeepSpeed ZeRO-3, LoRA).

	Key config: DeepSpeed ZeRO-3 with `offload_param` and `offload_optimizer` to CPU, `train_micro_batch_size_per_gpu=1`, `gradient_accumulation_steps=8`.

	---

	## 🔧 Configuration

	```python
	from configuration_scrapegoat import ScrapeGoatConfig

	config = ScrapeGoatConfig.from_pretrained("/path/to/model")

	# Track A
	config.track_a_num_attention_heads # 32
	config.track_a_num_key_value_heads # 2
	config.track_a_head_dim # 256
	config.track_a_num_experts # 512
	config.track_a_moe_intermediate_size # 1024

	# Track B
	config.track_b_num_attention_heads # 64
	config.track_b_num_key_value_heads # 8
	config.track_b_head_dim # 128
	config.track_b_num_experts # 192
	config.track_b_moe_intermediate_size # 1536

	# Attention
	config.kda_gqa_layers # [0,4,8,12,...,80] (every 4th = GQA)
	config.kda_conv_kernel # 3
	config.attn_residual # false (configurable, 8 blocks)

	# MoE
	config.quantile_balancing # True
	config.qb_iterations # 5
	config.num_experts_per_tok # 8
	config.stable_moe_stage # 1
	config.stable_moe_r3 # False
	config.stable_moe_r3_cache # True

	# MoM-KDA
	config.mom_enabled # True
	config.mom_num_memories # 4
	config.mom_active_memories # 2
	config.mom_shared_memory # True

	# DSpark
	config.dspark_block_size # 6
	config.dspark_markov_rank # 256
	config.dspark_noise_token_id # 0
	config.dspark_target_layer_ids # [] (empty = apply to all layers)
	config.output_router_logits # false (also returns router_logits in model output when enabled)
	```


	---

	## 🛠️ Hardware Requirements

	\| Setup \| VRAM/RAM \| Notes \|
	\|-------\|----------\|-------\|
	\| Inference (BF16) \| ~1.65 TB \| Multi-GPU cluster (e.g., 24×H100 80GB) \|
	\| Inference (4-bit) \| ~412 GB \| 5+ H100 80GB \|
	\| QLoRA SFT \| ~180GB RAM + 750GB NVMe \| ZeRO-3 + NVMe offload (single H100) \|
	\| Full Training \| Multi-node cluster \| 1.65 TB+ \|

	---

	## 📝 Citation

	```bibtex
	@misc{scrapegoat2026,
	title={ScrapeGoat: Parallel Dual-Track Transformer with KDA and Quantile Balancing},
	author={ScrapeGoat Team},
	year={2026},
	}
	```

	## Coding Agent Framework

	The most powerful AI coding agent on planet Earth is now a self-learning model agnostic harness.

	/fast mode and more advanced skills enabled only with scrapegoat models

	/graph - indexes your repo locally and auto-injects repo context. Excoder is always code aware.

	/Agent-WebBridge - Browser automation via Agent-WebBridge-skill and Agent-WebBridge for QA automation and Browser Autonomy.

	/dispatching-parallel-agents - To dispatch Swarm of agents for parallel work.

	Team \| Enterprise: run excoder, pick a username on first launch, then use /message @teammate to chat with org members in real time with secure encrypted messaging.

	/redteam-skill & /pentest-skill for offensive and advanced cyber security capabilities.

	- Application Security Testing — Detect and validate critical vulnerabilities in your applications
	- Rapid Penetration Testing — Get penetration tests done in hours, not weeks, with compliance reports
	- Bug Bounty Automation — Automate bug bounty research and generate PoCs for faster reporting
	- CI/CD Integration — Run tests in CI/CD to block vulnerabilities before reaching production
	- 🕸️ Web apps \| Black-box, external-attacker recon → exploit (XBEN suite) \| ✅
	- 🚩 CTF \| Hint-free, sandbox-jailed solves (Cybench) \| ✅
	- 🤖 Robotics / OT / embedded \| Coordinated-disclosure pipeline for OSS vuln hunting (OSV + live-PoC + refuter) \| ✅
	- 📂 Source code \| White-box repo analysis with blind master-builder decomposition
	- 💰 Smart contracts \| Damn Vulnerable DeFi \| ⚠️ reproduction
	- ☁️ Cloud (IaC) \| Misconfig-detection benchmark (`cloud:bench`) + opt-in cloud arsenal \| 🚧 IaC-misconfig scaffolding — live-cloud exploitation
	- 📱 Mobile \| Built-in static analyzer (manifest misconfig + secret/cleartext detection, `mobile:bench`) + opt-in arsenal (mobsfscan/objection/drozer; frida gated)
	- 🔩 Binary / RE \| Decompiled-output sink detector (unsafe-copy / format-string / cmd-injection / int-overflow, `binary:bench`)

	[![Excoder.pro](https://i.ibb.co/BVBcMWLy/Screenshot-2026-07-28-at-1-29-24-PM.png)](https://excoder.pro)


	---

	<div align="center">
	<strong>Left brain • Right brain • One mind. 🐐</strong>
	</div>