design_document.md

Multi-Expert Smart Contract Vulnerability Detection — Design Document

Current Results Diagnosis

Critical Failures

• 25% parse failures (302/1206): Model output format is inconsistent
• Access Control F1: 0.0235 — Model essentially never detects this (recall 0.012)
• Reentrancy precision: 0.0663 — Model hallucinates Reentrancy everywhere (196 predictions vs 52 actual)
• Only Integer Overflow performs well (F1: 0.8207)

Root Causes

1. Severe class imbalance in training data: Integer Overflow 3412 vs tx.origin 11
2. Single model trying to learn 6 distinct vulnerability patterns simultaneously — patterns are very different (state updates for Reentrancy, auth checks for Access Control, arithmetic for Integer Overflow)
3. No reasoning chains — model memorizes shortcuts rather than reasoning about code semantics
4. Output format too rigid — model doesn't consistently follow the structured template

Research-Backed Solution: Multi-Expert Architecture

Key Papers Referenced

• VulnLLM-R (2512.07533): Reasoning models with distillation, constitution-based correction
• Smart-LLaMA-DPO (2506.18245): Balanced detection + explanation loss, DPO for explanation quality
• SmartLLM (2502.13167): Multi-role pipeline (Detector → Reasoner → Verificator)
• SmartVD (2409.10574): Composite function F(C) = (binary, type, severity)

Architecture: Router + Type-Specific Expert Adapters

┌─────────────────┐     ┌─────────────────────────────────────────┐
│   Input Code    │────▶│  Stage 1: Router (Binary + Type Hint) │
└─────────────────┘     └─────────────────────────────────────────┘
                                     │
                    ┌────────────────┼────────────────┐
                    ▼                ▼                ▼
          ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
          │Expert 1:    │  │Expert 2:    │  │Expert 3:    │
          │Reentrancy   │  │Access Ctrl  │  │Integer Ov/  │
          │LoRA (r=32) │  │LoRA (r=32)  │  │Under LoRA   │
          └─────────────┘  └─────────────┘  └─────────────┘
          ┌─────────────┐  ┌─────────────┐
          │Expert 4:    │  │Expert 5:    │
          │Timestamp    │  │Unchecked    │
          │Dependence   │  │Calls        │
          └─────────────┘  └─────────────┘
                    │                │
                    └────────────────┘
                                     │
                    ┌────────────────┼────────────────┐
                    ▼                ▼                ▼
            ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
            │  Aggregate  │──│   Stage 3:  │──│   Severity  │
            │   Results   │  │  Finalize   │  │  (if vuln)  │
            └─────────────┘  └─────────────┘  └─────────────┘

Why This Works

1. Each expert only solves a binary problem: "Is this Reentrancy?" vs "Is this NOT Reentrancy?"
2. No inter-type confusion: Reentrancy expert never confuses with Access Control
3. Router provides type hints: Reduces the search space
4. Small adapters (r=32): Total trainable params ~6 × 50M = 300M vs current single adapter ~616M
5. Can be trained in parallel: 6 independent training jobs

Training Strategy Per Expert

Each expert trains on:

• Positive: All contracts with that specific vulnerability type
• Negative: Safe contracts + contracts with OTHER vulnerability types (hard negatives)
• This teaches the expert: "What makes THIS type different from ALL others?"

Reasoning Chains (from VulnLLM-R)

Each expert outputs reasoning before classification:

<analysis>
1. Identify external calls: `msg.sender.call{value: bal}("")`
2. Check state update timing: `balances[msg.sender] = 0` is AFTER the call
3. Follows checks-effects-interactions? No — external call before state update
</analysis>
<answer>
Vulnerable: Yes
Type: Reentrancy
Severity: Critical
</answer>

Output Format Fix

Use XML-style tags instead of markdown — easier for the model to learn and parse deterministically.