docs/LOCAL_MODEL_SETUP.md

Local Model Setup - Solution Summary

🎯 Problem Resolved

Issue: Training failed with OSError: [Errno 116] Stale file handle when trying to download/use models from HuggingFace cache.

Root Cause: Corrupted NFS file handle in HuggingFace cache directory preventing model access.

Solution: Downloaded Mistral-7B-v0.1 model directly to workspace, bypassing the corrupted cache.

---

📦 Model Location

/workspace/ftt/base_models/Mistral-7B-v0.1

Size: 28 GB (includes both PyTorch and SafeTensors formats)

Contents:

• ✓ Model weights (model-00001-of-00002.safetensors, model-00002-of-00002.safetensors)
• ✓ Tokenizer (tokenizer.model, tokenizer.json)
• ✓ Configuration files (config.json, generation_config.json)

---

🔧 Changes Made

1. Downloaded Model Locally

Used huggingface-cli to download model directly to workspace:

huggingface-cli download mistralai/Mistral-7B-v0.1 \
  --local-dir /workspace/ftt/base_models/Mistral-7B-v0.1 \
  --local-dir-use-symlinks False

2. Updated Gradio Interface

File: /workspace/ftt/semicon-finetuning-scripts/interface_app.py

Change: Updated default base model path from HuggingFace ID to local path:

# Before:
value="mistralai/Mistral-7B-v0.1"

# After:
value="/workspace/ftt/base_models/Mistral-7B-v0.1"

3. Restarted Interface

Killed old Gradio process and started fresh instance with updated configuration.

---

🚀 How to Use

Starting Training

1. Access Gradio Interface:

   • The interface is running on port 7860
   • Access via the public link displayed in the terminal

2. Fine-tuning Tab:

   • Base Model field now defaults to: /workspace/ftt/base_models/Mistral-7B-v0.1
   • You can still use HuggingFace model IDs if needed
   • Upload your dataset or use HuggingFace datasets
   • Configure training parameters
   • Click "Start Fine-tuning"

3. Monitor Training:

   • Status updates in real-time
   • Progress bar shows epoch and loss
   • Logs are scrollable with copy functionality

Using Other Models

If you want to use a different base model:

Option 1: Download Another Model Locally

cd /workspace/ftt
source /venv/main/bin/activate

# Download model
huggingface-cli download <model-id> \
  --local-dir /workspace/ftt/base_models/<model-name> \
  --local-dir-use-symlinks False

# Use the path in Gradio:
# /workspace/ftt/base_models/<model-name>

Option 2: Use HuggingFace ID Directly

• Simply enter the model ID in the Base Model field (e.g., mistralai/Mistral-7B-Instruct-v0.2)
• The script will download it if not cached (may hit cache issues if they persist)

---

🔍 Verification

Check Model is Accessible

python3 << 'EOF'
from transformers import AutoTokenizer, AutoConfig

model_path = "/workspace/ftt/base_models/Mistral-7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
config = AutoConfig.from_pretrained(model_path, local_files_only=True)

print(f"✓ Tokenizer: {len(tokenizer)} tokens")
print(f"✓ Model: {config.model_type}")
EOF

Check Gradio Status

# Check process
ps aux | grep interface_app.py

# Check port
lsof -i :7860

# View logs (if started with nohup)
tail -f /tmp/gradio_interface.log

---

📊 Interface Features

Fine-tuning Section

• ✓ File upload support (JSON/JSONL)
• ✓ HuggingFace dataset integration
• ✓ Automatic train/validation/test split
• ✓ Max sequence length up to 6000
• ✓ GPU-based parameter recommendations
• ✓ Detailed tooltips for all parameters
• ✓ Real-time progress tracking
• ✓ Checkpoint/resume functionality

API Hosting Section

• ✓ Host fine-tuned models from local paths
• ✓ Host models from HuggingFace repositories
• ✓ FastAPI with automatic documentation
• ✓ Health checks and status monitoring

Test Inference Section

• ✓ Test local fine-tuned models
• ✓ Test HuggingFace models
• ✓ Adjustable max-length (up to 6000)
• ✓ Temperature control with tooltips
• ✓ Uses API if running, otherwise direct loading

UI Controls

• ✓ Stop Training button
• ✓ Refresh Status button
• ✓ Scrollable logs with copy functionality
• ✓ Progress bars for training
• ✓ 🛑 Shutdown Gradio Server button (System Controls)

---

🐛 Troubleshooting

Issue: Cache errors persist

Solution: Always use local model paths from /workspace/ftt/base_models/

Issue: Training logs not updating

Solution:

1. Click "Refresh Status" button
2. Check that training process is running: ps aux | grep finetune_mistral

Issue: Interface not accessible

Solution:

# Check if running
lsof -i :7860

# Restart if needed
pkill -f interface_app.py
cd /workspace/ftt/semicon-finetuning-scripts
python3 interface_app.py

Issue: Out of memory during training

Solution:

1. Reduce batch size
2. Reduce max sequence length
3. Enable gradient checkpointing (already enabled in script)
4. Use LoRA with lower rank (r=8 instead of r=16)

---

📝 Technical Details

Training Script

Location: /workspace/ftt/semicon-finetuning-scripts/models/msp/ft/finetune_mistral7b.py

Key Features:

• LoRA fine-tuning for memory efficiency
• Gradient checkpointing enabled
• Automatic device detection (CUDA/MPS/CPU)
• Resume from checkpoint support
• JSON configuration export

Fine-tuning Command (Generated by Interface)

python3 -u /workspace/ftt/semicon-finetuning-scripts/models/msp/ft/finetune_mistral7b.py \
  --base-model /workspace/ftt/base_models/Mistral-7B-v0.1 \
  --dataset /path/to/your/dataset.jsonl \
  --output-dir ./your-finetuned-model \
  --max-length 2048 \
  --num-epochs 3 \
  --batch-size 4 \
  --learning-rate 2e-4 \
  --lora-r 16 \
  --lora-alpha 32

---

🎉 Success Criteria

You'll know everything is working when:

1. ✅ Gradio interface loads without errors
2. ✅ Base model field shows local path
3. ✅ Training starts without cache errors
4. ✅ Progress updates appear in UI
5. ✅ Model weights are saved to output directory

---

📚 Related Files

• Interface: /workspace/ftt/semicon-finetuning-scripts/interface_app.py
• Training Script: /workspace/ftt/semicon-finetuning-scripts/models/msp/ft/finetune_mistral7b.py
• Base Model: /workspace/ftt/base_models/Mistral-7B-v0.1/
• Startup Script: /workspace/ftt/semicon-finetuning-scripts/start_interface.sh
• Requirements: /workspace/ftt/semicon-finetuning-scripts/requirements_interface.txt

---

🆘 Support

If you encounter any issues:

1. Check this document's troubleshooting section
2. Review the training logs in the UI
3. Check process status: ps aux | grep -E "interface_app|finetune_mistral"
4. Review cache directories are clear: ls -lh /workspace/.hf_home/hub/

---

Last Updated: 2025-11-24 Solution: Local model download to bypass corrupted HuggingFace cache