TROUBLESHOOTING_DYNAMIC_CACHE.md

# Troubleshooting: DynamicCache 'seen_tokens' Error

## Error Message
```
ERROR: Local model error: 'DynamicCache' object has no attribute 'seen_tokens'
```

## What This Means

This error occurs when using local model inference (Phi-3, Llama, Mistral, etc.) with the `transformers` library. It's caused by a version incompatibility in the internal caching mechanism used during text generation.

**Impact**:
- Transcripts process but get Quality Score 0.00
- LLM analysis fails for all chunks
- No insights extracted from transcripts
- System still generates outputs but they're empty/error messages

---

## Root Cause

The `transformers` library changed its internal `Cache` implementation between versions:
- **Older versions (< 4.36)**: Used simpler cache without `seen_tokens` attribute
- **Newer versions (>= 4.36)**: Introduced `DynamicCache` with `seen_tokens` attribute
- **Version mismatch**: Code expects one format but library provides another

The error specifically occurs during the `model.generate()` call when the library tries to manage the key-value cache for efficient generation.

---

## Quick Fix (Applied)

**File**: `llm.py` (lines 460-480)

The code has been updated with:

```python
# Fix for DynamicCache 'seen_tokens' error
outputs = query_llm_local.model.generate(
    **inputs,
    max_new_tokens=max_tokens,
    temperature=temperature,
    do_sample=temperature > 0,
    pad_token_id=query_llm_local.tokenizer.eos_token_id,
    use_cache=False  # ← Disable caching to avoid DynamicCache errors
)
```

**What this does**: Disables the key-value caching mechanism entirely, forcing the model to recompute at each step.

**Trade-off**: Slightly slower generation (~10-20%) but avoids the error completely.

---

## Solutions (In Order of Preference)

### Solution 1: Upgrade Transformers Library ✅ **RECOMMENDED**

```bash
pip install --upgrade transformers
```

**Expected version**: 4.36.0 or higher

**Verify installation**:
```bash
python -c "import transformers; print(transformers.__version__)"
```

**Expected output**: `4.36.0` or higher

**Why this works**: Newer versions have the `seen_tokens` attribute properly implemented.

---

### Solution 2: Use HuggingFace API Instead 🚀 **EASIEST**

Instead of running models locally, use HuggingFace's cloud API.

**Advantages**:
- No local model loading (saves RAM)
- Faster processing
- No compatibility issues
- Access to larger, better models

**Setup**:

1. Get a HuggingFace token: https://huggingface.co/settings/tokens
2. Create token with "Read" access
3. Set environment variables:

```bash
export HUGGINGFACE_TOKEN='hf_your_token_here'
export USE_HF_API=True
```

Or in `.env` file:
```
HUGGINGFACE_TOKEN=hf_your_token_here
USE_HF_API=True
```

**Verify**:
```bash
python -c "import os; print('HF Token:', os.getenv('HUGGINGFACE_TOKEN')[:20])"
```

---

### Solution 3: Use LMStudio 🖥️ **BEST FOR OFFLINE**

LMStudio provides a GUI for running local models with better compatibility.

**Advantages**:
- Better compatibility than raw transformers
- Easy model management with GUI
- Local/offline processing
- No API costs

**Setup**:

1. Download LMStudio: https://lmstudio.ai/
2. Install and open LMStudio
3. Download a model (recommended: Phi-3-mini or Mistral-7B)
4. Start the local server:
   - Open LMStudio
   - Go to "Server" tab
   - Click "Start Server"
   - Default: http://localhost:1234

5. Set environment variables:

```bash
export USE_LMSTUDIO=True
export LMSTUDIO_URL=http://localhost:1234
```

Or in `.env` file:
```
USE_LMSTUDIO=True
LMSTUDIO_URL=http://localhost:1234
```

**Verify**:
```bash
curl http://localhost:1234/v1/models
```

Should return JSON with available models.

---

### Solution 4: Use Diagnostic Script

Run the diagnostic script to automatically detect and fix issues:

```bash
python fix_local_model.py
```

This script will:
1. Check your transformers version
2. Test local model functionality
3. Provide specific recommendations
4. Guide you through setup alternatives

**Example output**:
```
==================================================================
Local Model DynamicCache Error Fix
==================================================================

[Step 1] Diagnosing current environment...
✓ Transformers version: 4.35.0
⚠️  Transformers 4.35.0 is outdated
   Recommended: >= 4.36.0

[Step 2] Attempting to fix...
Upgrade transformers library? (y/n): y
✓ Transformers upgraded successfully
✓ Please restart your application
```

---

## Verification Steps

After applying any fix, verify it works:

### Test 1: Check Versions
```bash
python -c "import transformers, torch; print(f'Transformers: {transformers.__version__}'); print(f'PyTorch: {torch.__version__}')"
```

**Expected**:
```
Transformers: 4.36.0 or higher
PyTorch: 2.1.0 or higher
```

### Test 2: Quick LLM Test
```bash
python -c "from llm import query_llm_local; print(query_llm_local('Test', max_tokens=10))"
```

**Expected**: Some text output (not an error message)

### Test 3: Full Integration Test
Process a single transcript through the app and check:
- Quality Score > 0.00 ✓
- Structured data extracted ✓
- No DynamicCache errors in logs ✓

---

## Understanding Quality Score 0.00

If you see `Quality Score: 0.00` for all transcripts, it means:

**Cause**: LLM analysis is failing (likely due to this error)

**How Quality Score is calculated** (validation.py):
```python
def validate_transcript_quality(full_text, structured_data, interviewee_type):
    score = 0.0

    # Text length check (0.3 points)
    if len(full_text) > 100: score += 0.3

    # Structured data check (0.4 points)
    if has_structured_data: score += 0.4

    # Specificity check (0.3 points)
    if has_specific_terms: score += 0.3

    return score, issues
```

**If LLM fails**:
- `full_text` = "[Error] Local model failed: ..."
- `structured_data` = {} (empty)
- **Result**: Score = 0.00

**Fix**: Resolve the DynamicCache error → LLM works → Quality Score improves to 0.7-1.0

---

## Prevention & Best Practices

### 1. Pin Dependency Versions
In `requirements.txt`:
```
transformers>=4.36.0,<5.0.0
torch>=2.1.0,<2.3.0
```

**Why**: Ensures compatible versions are installed together

### 2. Use Virtual Environments
```bash
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate  # Windows
pip install -r requirements.txt
```

**Why**: Isolates dependencies, prevents conflicts with other projects

### 3. Regular Updates
```bash
pip install --upgrade transformers torch accelerate
```

**When**:
- After any error
- Monthly maintenance
- Before deploying to production

### 4. Prefer Cloud APIs for Production

For production deployments:
- **Use HuggingFace API** for reliability
- **Use LMStudio** for on-premise/offline requirements
- **Avoid local transformers** unless you control the environment

---

## Environment-Specific Notes

### Docker / HuggingFace Spaces
```dockerfile
# In Dockerfile or requirements
RUN pip install transformers>=4.36.0 torch>=2.1.0 accelerate
```

### Windows
```powershell
# Install in PowerShell with admin rights
pip install --upgrade transformers torch accelerate
```

### Linux / WSL
```bash
pip3 install --upgrade transformers torch accelerate
```

### macOS
```bash
pip3 install --upgrade transformers torch accelerate
```

---

## Still Having Issues?

### Debug Mode
Enable detailed logging:
```python
import os
os.environ["DEBUG_MODE"] = "True"
```

Then check logs for detailed error messages.

### Check Full Error Stack
Look for the full traceback in console output:
```
ERROR: Local model error: 'DynamicCache' object has no attribute 'seen_tokens'
Traceback (most recent call last):
  File "llm.py", line 459, in query_llm_local
    outputs = query_llm_local.model.generate(...)
  ...
```

### Contact Support
If the issue persists:
1. Run diagnostic script: `python fix_local_model.py`
2. Capture full logs
3. Note your environment:
   - OS (Windows/Linux/Mac)
   - Python version
   - Transformers version
   - PyTorch version
4. Report issue with logs

---

## Summary Checklist

- [ ] Updated transformers: `pip install --upgrade transformers`
- [ ] Verified version: `python -c "import transformers; print(transformers.__version__)"`
- [ ] Applied code fix (use_cache=False) - already done in llm.py
- [ ] Tested with sample transcript
- [ ] Quality Score > 0.00 ✓
- [ ] OR: Switched to HF API / LMStudio instead

**If all checked**: ✓ Problem solved!

**If still failing**: Use HF API or LMStudio (Solutions 2-3 above)

---

## Related Files

- `llm.py` - Contains the fix (lines 460-480)
- `fix_local_model.py` - Diagnostic script
- `requirements.txt` - Dependency versions
- `ENHANCEMENTS.md` - Recent improvements documentation

---

## Technical Details (For Developers)

### Why `use_cache=False` Works

**Normal generation with caching**:
```python
# Step 1: Generate token 1
cache = DynamicCache()  # Create cache
cache.seen_tokens = 1   # Track position

# Step 2: Generate token 2
cache.seen_tokens = 2   # Update position
# ... uses previous key/values from cache

# Faster but requires cache.seen_tokens attribute
```

**Generation without caching**:
```python
# Step 1: Generate token 1
# No cache used

# Step 2: Generate token 2
# Recompute everything from scratch

# Slower (~10-20%) but no cache dependencies
```

### Future Improvements

We're monitoring:
- Transformers library updates
- Alternative caching implementations
- Model-specific optimizations

Stay updated: Check `ENHANCEMENTS.md` for latest improvements.