OPTIMIZATION_PLAN.md

# Skill Extractor Optimization Plan

## Current Performance
- **Inference time**: 40-60 seconds per request
- **Hardware**: 2 vCPUs, 18GB RAM
- **Model**: Qwen2.5-1.5B-Instruct (Q4_K_M quantization)

## Optimization Strategies (Ranked by Impact)

### 1. Switch to Smaller/Faster Model ⚡⚡⚡
**Impact**: 3-5x speedup | **Effort**: Medium

**Options:**
- **Qwen2.5-0.5B-Instruct-Q4_K_M** (~1.5GB vs 1GB)
  - Pros: Much faster inference, lower memory usage
  - Cons: Slightly less accurate on complex skills
  - Expected speedup: 3-4x

- **Phi-3-mini-4k-instruct-Q4_K_M** (~2GB)
  - Pros: Optimized for speed, good reasoning
  - Cons: Different model, may need prompt tuning
  - Expected speedup: 2-3x

- **Gemma-2-2b-it-Q4_K_M** (~1.6GB)
  - Pros: Fast, good instruction following
  - Cons: Slightly larger than 0.5B options
  - Expected speedup: 2-3x

**Implementation:**
- Update Dockerfile to download new model
- Change default model path in `get_instance()`
- Test accuracy on sample job descriptions

---

### 2. Remove Grammar Constraint ⚡⚡
**Impact**: 20-30% speedup | **Effort**: Low

**Current bottleneck:** JSON grammar validation adds significant overhead

**Approach:**
- Remove `grammar=self.grammar` from `create_chat_completion()`
- Use simpler regex-based JSON parsing
- Add fallback for malformed output

**Trade-off:** Slightly less structured output, but parsing is robust

---

### 3. Aggressive Prompt Compression ⚡⚡
**Impact**: 15-25% speedup | **Effort**: Low

**Current prompt:** ~200 tokens
**Optimized prompt:** ~80 tokens

**Changes:**
```python
# Current (verbose)
"Extract skills from the job posting as short 2-4 word phrases. "
"Ignore company boilerplate. Never output fragments. "
"required_skills = explicitly mandatory. "
"nice_to_have_skills = preferred/optional. "
"Include mandatory degrees/certifications in required_skills."

# Optimized (concise)
"Extract skills as 2-4 word phrases. "
"required=mandatory, nice_to_have=optional. "
"Include degrees/certs in required."
```

---

### 4. Reduce Output Tokens ⚡
**Impact**: 10-15% speedup | **Effort**: Low

**Current:** `_MAX_TOKENS = 64`
**Optimized:** `_MAX_TOKENS = 32`

Most skill extractions need <20 tokens. 32 provides safety margin.

---

### 5. Add Response Caching ⚡⚡
**Impact**: Near-instant for duplicates | **Effort**: Medium

**Strategy:**
- Hash job_description + job_title
- Cache results in memory (dict) or Redis
- Set TTL of 24-48 hours
- Cache hit rate typically 20-30% for job boards

**Implementation:**
```python
_cache = {}
def extract(self, job_description: str, job_title: str = ""):
    key = hashlib.md5(f"{job_title}:{job_description}".encode()).hexdigest()
    if key in _cache:
        return _cache[key]
    result = # ... extraction logic
    _cache[key] = result
    return result
```

---

### 6. Hybrid Approach: LLM + Rule-Based ⚡⚡⚡
**Impact**: 5-10x speedup for common patterns | **Effort**: High

**Strategy:**
- Build regex/keyword-based extractor for common skills
- Use LLM only for complex/ambiguous cases
- 60-80% of requests can be handled by rules

**Rule-based examples:**
- "Python", "Java", "JavaScript" → programming languages
- "AWS", "Azure", "GCP" → cloud platforms
- "Docker", "Kubernetes" → devops tools
- "Bachelor's", "Master's" → degrees

**Fallback:** If rule extraction fails or confidence low, use LLM

---

### 7. Optimize Context Window ⚡
**Impact**: 5-10% speedup | **Effort**: Low

**Current:** `n_ctx = 1024`
**Optimized:** Dynamic based on input length

**Strategy:**
- Calculate input tokens
- Set `n_ctx = input_tokens + _MAX_TOKENS + 100`
- Minimum 512, maximum 2048

---

### 8. Batch Processing (For Bulk Operations) ⚡⚡
**Impact**: 2-3x throughput | **Effort**: Medium

**Use case:** Processing multiple job descriptions

**Strategy:**
- Add `/extract_batch` endpoint
- Process 5-10 descriptions in parallel
- Use threading with model instance lock

---

### 9. Use GPU Acceleration ⚡⚡⚡
**Impact**: 5-10x speedup | **Effort**: Low (if GPU available)

**Current:** CPU-only inference
**Optimized:** GPU offloading

**Implementation:**
- Already have `n_gpu_layers=-1` in code
- Ensure CUDA-compatible llama-cpp-python is installed
- If no GPU available, this has no effect

**Docker changes:**
```dockerfile
# Install CUDA toolkit if GPU available
RUN pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
```

---

### 10. Alternative: Use vLLM or TensorRT-LLM ⚡⚡⚡
**Impact**: 2-5x speedup | **Effort**: High

**Strategy:**
- Replace llama-cpp-python with vLLM
- Better batching, optimized inference
- Requires more setup but significant speed gains

**Trade-off:** More complex deployment, larger memory footprint

---

## Recommended Implementation Order

### Phase 1: Quick Wins (Total: 2-3x speedup)
1. Remove grammar constraint (20-30%)
2. Compress prompt (15-25%)
3. Reduce max tokens to 32 (10-15%)
4. Optimize context window (5-10%)

**Expected result:** 15-25 seconds per request

### Phase 2: Model Change (Total: 3-5x from baseline)
5. Switch to Qwen2.5-0.5B or Phi-3-mini

**Expected result:** 8-15 seconds per request

### Phase 3: Advanced Optimizations
6. Add response caching
7. Implement hybrid rule-based approach
8. Enable GPU acceleration (if available)

**Expected result:** 3-8 seconds per request (with cache hits near-instant)

---

## Alternative: Non-LLM Approach

### Pure Rule-Based Extraction
**Speed:** <100ms | **Accuracy:** 70-80%

**Strategy:**
- Build comprehensive skill taxonomy (500-1000 known skills)
- Use fuzzy matching (Levenshtein, TF-IDF)
- Add NLP for context (spaCy, NLTK)
- No LLM required

**Pros:**
- Extremely fast
- Predictable costs
- No model management

**Cons:**
- Lower accuracy on novel skills
- Requires manual taxonomy maintenance
- Misses contextual nuances

---

## Performance Targets

| Approach | Target Time | Accuracy | Effort |
|----------|-------------|----------|--------|
| Current | 40-60s | 85-90% | - |
| Phase 1 | 15-25s | 85-90% | Low |
| Phase 1+2 | 8-15s | 80-85% | Medium |
| Phase 1+2+3 | 3-8s | 80-85% | High |
| Hybrid | 1-5s | 75-85% | High |
| Rule-based | <0.1s | 70-80% | High |

---

## Next Steps

1. **Implement Phase 1** (grammar removal, prompt compression)
2. **Test accuracy** on sample job descriptions
3. **If acceptable**, deploy and monitor
4. **If too slow**, proceed to Phase 2 (model change)
5. **Consider hybrid approach** for production scale