OPTIMIZATION_PLAN.md

Skill Extractor Optimization Plan

Current Performance

• Inference time: 40-60 seconds per request
• Hardware: 2 vCPUs, 18GB RAM
• Model: Qwen2.5-1.5B-Instruct (Q4_K_M quantization)

Optimization Strategies (Ranked by Impact)

1. Switch to Smaller/Faster Model ⚡⚡⚡

Impact: 3-5x speedup | Effort: Medium

Options:

• Qwen2.5-0.5B-Instruct-Q4_K_M (~1.5GB vs 1GB)

  • Pros: Much faster inference, lower memory usage
  • Cons: Slightly less accurate on complex skills
  • Expected speedup: 3-4x

• Phi-3-mini-4k-instruct-Q4_K_M (~2GB)

  • Pros: Optimized for speed, good reasoning
  • Cons: Different model, may need prompt tuning
  • Expected speedup: 2-3x

• Gemma-2-2b-it-Q4_K_M (~1.6GB)

  • Pros: Fast, good instruction following
  • Cons: Slightly larger than 0.5B options
  • Expected speedup: 2-3x

Implementation:

• Update Dockerfile to download new model
• Change default model path in get_instance()
• Test accuracy on sample job descriptions

---

2. Remove Grammar Constraint ⚡⚡

Impact: 20-30% speedup | Effort: Low

Current bottleneck: JSON grammar validation adds significant overhead

Approach:

• Remove grammar=self.grammar from create_chat_completion()
• Use simpler regex-based JSON parsing
• Add fallback for malformed output

Trade-off: Slightly less structured output, but parsing is robust

---

3. Aggressive Prompt Compression ⚡⚡

Impact: 15-25% speedup | Effort: Low

Current prompt: ~200 tokens Optimized prompt: ~80 tokens

Changes:

# Current (verbose)
"Extract skills from the job posting as short 2-4 word phrases. "
"Ignore company boilerplate. Never output fragments. "
"required_skills = explicitly mandatory. "
"nice_to_have_skills = preferred/optional. "
"Include mandatory degrees/certifications in required_skills."

# Optimized (concise)
"Extract skills as 2-4 word phrases. "
"required=mandatory, nice_to_have=optional. "
"Include degrees/certs in required."

---

4. Reduce Output Tokens ⚡

Impact: 10-15% speedup | Effort: Low

Current: _MAX_TOKENS = 64 Optimized: _MAX_TOKENS = 32

Most skill extractions need <20 tokens. 32 provides safety margin.

---

5. Add Response Caching ⚡⚡

Impact: Near-instant for duplicates | Effort: Medium

Strategy:

• Hash job_description + job_title
• Cache results in memory (dict) or Redis
• Set TTL of 24-48 hours
• Cache hit rate typically 20-30% for job boards

Implementation:

_cache = {}
def extract(self, job_description: str, job_title: str = ""):
    key = hashlib.md5(f"{job_title}:{job_description}".encode()).hexdigest()
    if key in _cache:
        return _cache[key]
    result = # ... extraction logic
    _cache[key] = result
    return result

---

6. Hybrid Approach: LLM + Rule-Based ⚡⚡⚡

Impact: 5-10x speedup for common patterns | Effort: High

Strategy:

• Build regex/keyword-based extractor for common skills
• Use LLM only for complex/ambiguous cases
• 60-80% of requests can be handled by rules

Rule-based examples:

• "Python", "Java", "JavaScript" → programming languages
• "AWS", "Azure", "GCP" → cloud platforms
• "Docker", "Kubernetes" → devops tools
• "Bachelor's", "Master's" → degrees

Fallback: If rule extraction fails or confidence low, use LLM

---

7. Optimize Context Window ⚡

Impact: 5-10% speedup | Effort: Low

Current: n_ctx = 1024 Optimized: Dynamic based on input length

Strategy:

• Calculate input tokens
• Set n_ctx = input_tokens + _MAX_TOKENS + 100
• Minimum 512, maximum 2048

---

8. Batch Processing (For Bulk Operations) ⚡⚡

Impact: 2-3x throughput | Effort: Medium

Use case: Processing multiple job descriptions

Strategy:

• Add /extract_batch endpoint
• Process 5-10 descriptions in parallel
• Use threading with model instance lock

---

9. Use GPU Acceleration ⚡⚡⚡

Impact: 5-10x speedup | Effort: Low (if GPU available)

Current: CPU-only inference Optimized: GPU offloading

Implementation:

• Already have n_gpu_layers=-1 in code
• Ensure CUDA-compatible llama-cpp-python is installed
• If no GPU available, this has no effect

Docker changes:

# Install CUDA toolkit if GPU available
RUN pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

---

10. Alternative: Use vLLM or TensorRT-LLM ⚡⚡⚡

Impact: 2-5x speedup | Effort: High

Strategy:

• Replace llama-cpp-python with vLLM
• Better batching, optimized inference
• Requires more setup but significant speed gains

Trade-off: More complex deployment, larger memory footprint

---

Recommended Implementation Order

Phase 1: Quick Wins (Total: 2-3x speedup)

1. Remove grammar constraint (20-30%)
2. Compress prompt (15-25%)
3. Reduce max tokens to 32 (10-15%)
4. Optimize context window (5-10%)

Expected result: 15-25 seconds per request

Phase 2: Model Change (Total: 3-5x from baseline)

5. Switch to Qwen2.5-0.5B or Phi-3-mini

Expected result: 8-15 seconds per request

Phase 3: Advanced Optimizations

6. Add response caching
7. Implement hybrid rule-based approach
8. Enable GPU acceleration (if available)

Expected result: 3-8 seconds per request (with cache hits near-instant)

---

Alternative: Non-LLM Approach

Pure Rule-Based Extraction

Speed: <100ms | Accuracy: 70-80%

Strategy:

• Build comprehensive skill taxonomy (500-1000 known skills)
• Use fuzzy matching (Levenshtein, TF-IDF)
• Add NLP for context (spaCy, NLTK)
• No LLM required

Pros:

• Extremely fast
• Predictable costs
• No model management

Cons:

• Lower accuracy on novel skills
• Requires manual taxonomy maintenance
• Misses contextual nuances

---

Performance Targets

Approach	Target Time	Accuracy	Effort
Current	40-60s	85-90%	-
Phase 1	15-25s	85-90%	Low
Phase 1+2	8-15s	80-85%	Medium
Phase 1+2+3	3-8s	80-85%	High
Hybrid	1-5s	75-85%	High
Rule-based	<0.1s	70-80%	High

---

Next Steps

1. Implement Phase 1 (grammar removal, prompt compression)
2. Test accuracy on sample job descriptions
3. If acceptable, deploy and monitor
4. If too slow, proceed to Phase 2 (model change)
5. Consider hybrid approach for production scale