design_document.md

# Multi-Expert Smart Contract Vulnerability Detection — Design Document

## Current Results Diagnosis

### Critical Failures
- **25% parse failures** (302/1206): Model output format is inconsistent
- **Access Control F1: 0.0235** — Model essentially never detects this (recall 0.012)
- **Reentrancy precision: 0.0663** — Model hallucinates Reentrancy everywhere (196 predictions vs 52 actual)
- **Only Integer Overflow performs well** (F1: 0.8207)

### Root Causes
1. **Severe class imbalance in training data**: Integer Overflow 3412 vs tx.origin 11
2. **Single model trying to learn 6 distinct vulnerability patterns simultaneously** — patterns are very different (state updates for Reentrancy, auth checks for Access Control, arithmetic for Integer Overflow)
3. **No reasoning chains** — model memorizes shortcuts rather than reasoning about code semantics
4. **Output format too rigid** — model doesn't consistently follow the structured template

## Research-Backed Solution: Multi-Expert Architecture

### Key Papers Referenced
- **VulnLLM-R** (2512.07533): Reasoning models with distillation, constitution-based correction
- **Smart-LLaMA-DPO** (2506.18245): Balanced detection + explanation loss, DPO for explanation quality
- **SmartLLM** (2502.13167): Multi-role pipeline (Detector → Reasoner → Verificator)
- **SmartVD** (2409.10574): Composite function F(C) = (binary, type, severity)

### Architecture: Router + Type-Specific Expert Adapters

```
┌─────────────────┐     ┌─────────────────────────────────────────┐
│   Input Code    │────▶│  Stage 1: Router (Binary + Type Hint) │
└─────────────────┘     └─────────────────────────────────────────┘
                                     │
                    ┌────────────────┼────────────────┐
                    ▼                ▼                ▼
          ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
          │Expert 1:    │  │Expert 2:    │  │Expert 3:    │
          │Reentrancy   │  │Access Ctrl  │  │Integer Ov/  │
          │LoRA (r=32) │  │LoRA (r=32)  │  │Under LoRA   │
          └─────────────┘  └─────────────┘  └─────────────┘
          ┌─────────────┐  ┌─────────────┐
          │Expert 4:    │  │Expert 5:    │
          │Timestamp    │  │Unchecked    │
          │Dependence   │  │Calls        │
          └─────────────┘  └─────────────┘
                    │                │
                    └────────────────┘
                                     │
                    ┌────────────────┼────────────────┐
                    ▼                ▼                ▼
            ┌─────────────┐  ┌─────────────┐  ┌─────────────┐
            │  Aggregate  │──│   Stage 3:  │──│   Severity  │
            │   Results   │  │  Finalize   │  │  (if vuln)  │
            └─────────────┘  └─────────────┘  └─────────────┘
```

### Why This Works
1. **Each expert only solves a binary problem**: "Is this Reentrancy?" vs "Is this NOT Reentrancy?"
2. **No inter-type confusion**: Reentrancy expert never confuses with Access Control
3. **Router provides type hints**: Reduces the search space
4. **Small adapters (r=32)**: Total trainable params ~6 × 50M = 300M vs current single adapter ~616M
5. **Can be trained in parallel**: 6 independent training jobs

### Training Strategy Per Expert
Each expert trains on:
- **Positive**: All contracts with that specific vulnerability type
- **Negative**: Safe contracts + contracts with OTHER vulnerability types (hard negatives)
- This teaches the expert: "What makes THIS type different from ALL others?"

### Reasoning Chains (from VulnLLM-R)
Each expert outputs reasoning before classification:
```
<analysis>
1. Identify external calls: `msg.sender.call{value: bal}("")`
2. Check state update timing: `balances[msg.sender] = 0` is AFTER the call
3. Follows checks-effects-interactions? No — external call before state update
</analysis>
<answer>
Vulnerable: Yes
Type: Reentrancy
Severity: Critical
</answer>
```

### Output Format Fix
Use XML-style tags instead of markdown — easier for the model to learn and parse deterministically.