Hugging Face's logo

Elinnos
/
codellama-fine-tuning

Model card Files
xet
Community
codellama-fine-tuning / FILE_INVENTORY.md
Prithvik-1's picture
Prithvik-1
Upload FILE_INVENTORY.md with huggingface_hub
51c7198 verified
preview code
|
raw
history blame contribute delete
7.73 kB

📁 Complete File Inventory for CodeLlama Model Migration

📊 Overview

This document lists all files created/modified for the CodeLlama model fine-tuning project.

📝 Documentation Files (.md)

Migration & Progress Tracking

  1. MIGRATION_PROGRESS.md - Main migration tracking document
  2. TRAINING_STARTED_SUMMARY.md - Initial training summary
  3. TRAINING_COMPLETE.md - Training completion report (chat format model)
  4. FINAL_ANSWER.md - Final answer about format issues and solutions

Analysis & Guides

  1. HYPERPARAMETER_ANALYSIS.md - Optimal hyperparameters for CodeLlama
  2. HYPERPARAMETER_TUNING_GUIDE.md - Guide for tuning inference parameters
  3. DATASET_SPLIT_VALIDATION_GUIDE.md - Dataset splitting guidelines
  4. FORMAT_ISSUE_ANALYSIS.md - Analysis of format mismatch issues
  5. SOLUTION_DATASET_REFORMAT.md - Solution for dataset reformatting

Training Guides

  1. TRAINING_GUIDE.md - General training guide
  2. RETRAIN_WITH_CHAT_FORMAT.md - Instructions for retraining with chat format

Testing & Evaluation

  1. TEST_COMMANDS.md - Various testing commands
  2. QUICK_TEST_COMMAND.md - Quick reference for testing
  3. TEST_RESULTS_NEW_MODEL.md - Test results for new chat format model
  4. EVALUATION_REPORT.md - Detailed evaluation report
  5. EVALUATION_SUMMARY.md - Summary of evaluation results
  6. COMPARISON_REPORT.md - Detailed comparison: Expected vs Generated
  7. QUICK_COMPARISON_SUMMARY.md - Quick comparison summary

References

  1. INFERENCE_GUIDE.md - Inference usage guide
  2. QUICK_REFERENCE.md - Quick reference guide
  3. SUMMARY_FIX.md - Summary of fixes applied

Current Document

  1. FILE_INVENTORY.md - This file (complete file listing)

🐍 Python Scripts (.py)

Dataset Processing

  1. reformat_dataset_for_codellama.py - Reformat dataset to CodeLlama chat format
  2. scripts/dataset_split.py - Split dataset into train/val/test
  3. scripts/validate_dataset.py - Validate dataset format and quality

Training Scripts

  1. scripts/training/finetune_codellama.py - Main fine-tuning script for CodeLlama

Inference Scripts

  1. scripts/inference/inference_codellama.py - Inference script (adapted for CodeLlama)

Testing Scripts

  1. test_samples.py - Test model on multiple samples from dataset
  2. test_single_sample.py - Test on a single training sample
  3. test_single_training_sample.py - Test with exact training format
  4. test_exact_training_format.py - Test with exact format matching
  5. test_new_model.py - Test the new fine-tuned model

🔧 Shell Scripts (.sh)

  1. start_training.sh - Start training with original format
  2. start_training_chat_format.sh - Start training with chat format dataset
  3. test_inference.sh - Quick inference test script

📊 Dataset Files

Raw Datasets

  1. datasets/raw/elinnos_fifo_mistral_100samples_converted.jsonl - Original converted dataset

Processed Datasets

  1. datasets/processed/elinnos_fifo_codellama_v1.jsonl - Initial CodeLlama formatted dataset (94 samples)
  2. datasets/processed/elinnos_fifo_codellama_chat_format.jsonl - Chat template format dataset (94 samples)

Split Datasets (Original Format)

  1. datasets/processed/split/train.jsonl - Training split (71 samples)
  2. datasets/processed/split/val.jsonl - Validation split (9 samples)
  3. datasets/processed/split/test.jsonl - Test split (14 samples)

Split Datasets (Chat Format)

  1. datasets/processed/split_chat_format/train.jsonl - Chat format training split (70 samples)
  2. datasets/processed/split_chat_format/val.jsonl - Chat format validation split (9 samples)
  3. datasets/processed/split_chat_format/test.jsonl - Chat format test split (15 samples)

🤖 Model Files

Base Model

  • models/base-models/CodeLlama-7B-Instruct/ - Base CodeLlama model directory
    • Contains all base model files (config.json, tokenizer files, model weights, etc.)

Fine-Tuned Models

Model v1 (Original Format - Has Issues)

  • training-outputs/codellama-fifo-v1/ - First fine-tuned model
    • adapter_model.safetensors - LoRA adapter weights
    • adapter_config.json - LoRA configuration
    • training_config.json - Training configuration
    • checkpoint-25/ - Checkpoint at step 25
    • checkpoint-45/ - Checkpoint at step 45

Model v2 (Chat Format - Working!)

  • training-outputs/codellama-fifo-v2-chat/ - Fine-tuned model with chat format ✅
    • adapter_model.safetensors - LoRA adapter weights (458M)
    • adapter_config.json - LoRA configuration
    • training_config.json - Training configuration
    • chat_template.jinja - Chat template file
    • checkpoint-25/ - Final checkpoint (completed training)

📋 Configuration & Log Files

Logs

  1. training_fresh_start.log - Log from initial training run
  2. training_chat_format.log - Log from chat format training run
  3. evaluation_output.log - Evaluation output log

JSON Files

  1. evaluation_results.json - Evaluation results in JSON format

Download Files

  1. download_log.txt - Model download log
  2. download_pid.txt - Download process ID

📂 Directory Structure 

codellama-migration/
├── 📄 Documentation (22 .md files)
├── 🐍 Scripts/
│   ├── dataset_split.py
│   ├── validate_dataset.py
│   ├── training/
│   │   └── finetune_codellama.py
│   └── inference/
│       └── inference_codellama.py
├── 📊 Datasets/
│   ├── raw/
│   │   └── elinnos_fifo_mistral_100samples_converted.jsonl
│   └── processed/
│       ├── elinnos_fifo_codellama_v1.jsonl
│       ├── elinnos_fifo_codellama_chat_format.jsonl
│       ├── split/ (train/val/test - original format)
│       └── split_chat_format/ (train/val/test - chat format)
├── 🤖 Models/
│   ├── base-models/
│   │   └── CodeLlama-7B-Instruct/ (Base model)
│   └── training-outputs/
│       ├── codellama-fifo-v1/ (Old model - has issues)
│       └── codellama-fifo-v2-chat/ (New model - working! ✅)
└── 🔧 Scripts & Tools/
    ├── reformat_dataset_for_codellama.py
    ├── start_training.sh
    ├── start_training_chat_format.sh
    ├── test_*.py (Multiple test scripts)
    └── *.log files

✅ Key Files Summary

Most Important Files:

  1. Training Script: scripts/training/finetune_codellama.py
  2. Inference Script: scripts/inference/inference_codellama.py
  3. Working Model: training-outputs/codellama-fifo-v2-chat/
  4. Chat Format Dataset: datasets/processed/split_chat_format/
  5. Training Script: start_training_chat_format.sh

Key Documentation:

  1. MIGRATION_PROGRESS.md - Overall progress tracking
  2. TRAINING_COMPLETE.md - Training completion details
  3. COMPARISON_REPORT.md - Expected vs Generated comparison
  4. FINAL_ANSWER.md - Summary of issues and solutions

📊 File Statistics

  • Total Documentation Files: 22
  • Total Python Scripts: 10
  • Total Shell Scripts: 3
  • Total Dataset Files: 9
  • Fine-Tuned Models: 2 (v1 has issues, v2 working ✅)
  • Total Files: ~100+ (including model checkpoints and configs)

🎯 Current Status

Working Model: training-outputs/codellama-fifo-v2-chat/ ✅
Dataset Used: datasets/processed/split_chat_format/ ✅
Status: Model is working correctly, generates valid Verilog code

Last Updated: After successful training with chat format dataset