docs/token-optimization-guide.md

# Token Optimization Guide

> Practical strategies for reducing token consumption while maintaining quality.

---

## Why Token Optimization Matters

| Issue | Impact |
|-------|--------|
| Excessive file loading | Higher costs, slower responses |
| Context accumulation | Quality degradation after 50% |
| Re-reading files | Wasted tokens on understood content |
| Full files when snippets suffice | 10x token usage |

**Goal:** Maximize output quality per token spent.

---

## The Token Efficiency Stack

```
┌─────────────────────────────────────┐
│ 1. Search-First (context-fetch)    │ ← Find before loading
├─────────────────────────────────────┤
│ 2. Budget Tracking (token-budget)  │ ← Know your limits
├─────────────────────────────────────┤
│ 3. Compression (context-compressor)│ ← Minimize footprint
├─────────────────────────────────────┤
│ 4. Health Monitoring               │ ← Prevent degradation
└─────────────────────────────────────┘
```

---

## Quick Reference: Token Costs

### By Content Type

| Type | Tokens/Line | 100 Lines = |
|------|-------------|-------------|
| Dense code | 6 | ~600 tokens |
| Standard code | 4 | ~400 tokens |
| Markdown | 3 | ~300 tokens |
| Sparse YAML | 2 | ~200 tokens |

### By File Size

| File Lines | Strategy |
|------------|----------|
| <50 | Load freely |
| 50-200 | Outline first |
| 200-500 | Search + snippets |
| 500+ | Never load fully |

---

## Optimization Patterns

### Pattern 1: Search → Outline → Target

```
Step 1: Search for "handlePayment"
  → Found in: payment.ts:45, checkout.ts:120

Step 2: Get outline of payment.ts
  → L45-80: handlePayment function

Step 3: Load only L45-80
  → 35 lines (~140 tokens) vs 400 lines (~1600 tokens)
  → Saved: ~90%
```

### Pattern 2: Summarize After Understanding

```
After reading auth.ts:

## Summary: auth.ts
- Exports: login, logout, validateToken
- Pattern: Express middleware
- DB: queries users table
- JWT: uses jose library

Next time: Reference summary, don't reload
```

### Pattern 3: Progressive Disclosure

```
Need: Understand login flow

Level 1: Outline
  → "login at L45, uses validateCredentials at L67"
  → Often sufficient

Level 2: Key function
  → Load L45-65 only
  → Understand core logic

Level 3: Dependencies
  → Load validateCredentials (L67-85)
  → Only if L2 insufficient

Level 4: Full file
  → Last resort, re-compress after
```

---

## Anti-Patterns to Avoid

### ❌ The "Context Dump"

```
BAD:
"Let me read all the files in src/ to understand the project"
→ 50 files × 200 lines × 4 tokens = 40,000 tokens

GOOD:
"Let me search for 'main entry' and 'router'"
→ 2 targeted searches, ~500 tokens
```

### ❌ The "Just In Case" Load

```
BAD:
"Loading utils.ts in case I need it later"
→ Probably won't need it, wasted tokens

GOOD:
"Noting utils.ts exists, will load if needed"
→ Zero tokens until actually needed
```

### ❌ The Re-Read

```
BAD:
"Reading config.ts again to check the port"
→ Already read it twice = 1200 tokens

GOOD:
"From my earlier analysis, port is on L15"
→ Zero additional tokens
```

---

## Budget Checkpoints

### Before Starting Work

```markdown
□ Do I know my current budget usage?
□ Have I tried searching before loading?
□ Am I loading files I've already understood?
```

### During Execution

```markdown
□ Am I at >50%? Time to compress.
□ Am I re-reading files? Use summaries.
□ Can I use outline instead of full file?
```

### After Each Wave

```markdown
□ Have I compressed context for next wave?
□ Are summaries documented in STATE.md?
□ Would a fresh session be more efficient?
```

---

## Integration with GSD

| GSD Workflow | Token Optimization |
|--------------|-------------------|
| `/map` | Generate outline, not full read |
| `/plan` | Budget estimate per task |
| `/execute` | Load minimal per task |
| `/verify` | Targeted evidence only |
| `/pause` | Compress and dump state |

---

## Metrics

Track these for improvement:

| Metric | Good | Poor |
|--------|------|------|
| Files fully loaded | <3 per wave | 10+ |
| Search:Load ratio | 3:1 | 1:3 |
| Re-reads | 0 | 3+ |
| Budget at wave end | <50% | >70% |

---

*See also:*
- *[.agents/skills/token-budget/SKILL.md](.agents/skills/token-budget/SKILL.md)*
- *[.agents/skills/context-compressor/SKILL.md](.agents/skills/context-compressor/SKILL.md)*
- *[PROJECT_RULES.md](PROJECT_RULES.md) — Token Efficiency Rules*