IMPLEMENTATION.md

Implementation Guide

Goal

Build and run a local fine-tuning pipeline for a coding assistant model with:

• dataset generation
• LoRA fine-tuning
• local inference
• optional Hugging Face upload

Project Modules

• generate_dataset.py
  • Generates training samples into JSON.
  • Supports dataset size range: 5000 to 10000.
• finetune_coding_llm_colab.py
  • Main training module (local usage).
  • Supports dataset generation, training, and optional HF upload.
• run_pipeline.py
  • Orchestrates generate -> train -> upload in one command.
  • Reads defaults from training_config.json.
• infer_local.py
  • Runs inference from local trained output.
  • Handles both LoRA adapter output and full model output.
  • Returns structured JSON fields including code, explanation, confidence, relevancy, hallucination check, and latency.
• infer_cloud.py
  • Runs inference through the Hugging Face API using an HF token.
  • Reuses the local structured-output parser and repair checks so API output matches the local JSON contract.
  • Falls back to the local model/ folder when Hugging Face does not serve the custom repo through an inference provider.
• handler.py
  • Custom Hugging Face Dedicated Inference Endpoint handler.
  • Loads the LoRA adapter/full model and returns the same structured JSON contract directly from the hosted endpoint.
• evaluate_model.py
  • Runs a multi-prompt evaluation and reports pass rate (accuracy) for schema + quality checks.
• upload_to_hf.py
  • Uploads local model folder to Hugging Face model repo.

Environment Setup

1. Use Python 3.10+ (recommended).
2. Install dependencies:
   • pip install -r requirements.txt
3. (Optional) Login to Hugging Face before upload:
   • huggingface-cli login

Standard Execution Flow

1. Generate dataset:
   • python generate_dataset.py --size 8000 --out train.json
2. Train model:
   • python finetune_coding_llm_colab.py --dataset-size 8000 --train-file train.json --output-dir model --skip-dataset-gen
3. Test inference:
   • python infer_local.py --model-path model --prompt "Fix this code: def add(a,b) return a+b"
   • Add --allow-downloads on a fresh machine if the base model is not cached locally.
4. Evaluate quality:
   • python evaluate_model.py --model-path model
5. Upload (optional):
   • python upload_to_hf.py --model-dir model --repo-id your-username/your-model-name
6. Test cloud inference (optional):
   • PowerShell: $env:HF_TOKEN="your_huggingface_token"
   • python infer_cloud.py --repo-id your-username/your-model-name --prompt "Fix this code: def add(a,b) return a+b"
   • If you already logged in with hf auth login, the saved token can be used without setting HF_TOKEN.
   • Add --no-local-fallback if you want the command to fail when HF cloud serving is unavailable.
   • Add --allow-downloads if local fallback needs to download missing base-model files.
   • For true cloud execution, deploy a Hugging Face Dedicated Inference Endpoint and call:
     • python infer_cloud.py --endpoint-url "https://your-endpoint-url.endpoints.huggingface.cloud" --prompt "Fix this code: def add(a,b) return a+b" --no-local-fallback
   • Users should set their own token with $env:HF_TOKEN="their_huggingface_token" before calling the endpoint.

One-Command Execution

• Run full pipeline without upload:

  • python run_pipeline.py --dataset-size 8000 --skip-upload

• Run with upload:

  • python run_pipeline.py --dataset-size 8000 --hf-repo your-username/your-model-name

Performance Recommendations

• CPU quick validation:
  • python run_pipeline.py --dataset-size 5000 --max-train-samples 20 --epochs 0.1 --skip-upload
• Full quality run:
  • python run_pipeline.py --dataset-size 8000 --epochs 3 --batch-size 2 --learning-rate 1e-4 --max-length 512 --use-4bit --skip-upload

Error Handling Rules

• If dataset file is missing, run generate_dataset.py.
• If model folder is missing, run training first.
• If HF upload fails, verify:
  • huggingface-cli whoami
  • repo permission and repo id format (username/repo)

Integration Notes

• run_pipeline.py is the recommended entrypoint for regular usage.
• training_config.json provides default values and can be overridden by CLI flags.
• Inference works with LoRA adapters and full models automatically.

Hugging Face Existing Model Update

To update an already published Hugging Face model with current project behavior:

1. Retrain with latest code:
   • python run_pipeline.py --dataset-size 8000 --skip-upload
2. Validate local inference + evaluation:
   • python infer_local.py --model-path model --prompt "Fix this code: def add(a,b) return a+b"
   • python evaluate_model.py --model-path model
3. Upload to same repo id:
   • python upload_to_hf.py --model-dir model --repo-id your-username/your-existing-model-name

Optional safer rollout:

• Upload to a revision branch first and test before merging to main.

Current Output Contract

infer_local.py returns JSON with:

• code
• explanation
• confidence
• important_tokens
• relevancy_score
• hallucination
• hallucination_check_reason
• latency_ms

infer_cloud.py returns the same JSON keys through the Hugging Face API, or through local fallback if HF cannot serve the custom repo. Cloud responses may not include token-level probabilities, so important_tokens can be empty and confidence can be 0.0 unless the serving endpoint exposes token details.

For users calling the hosted model with their own token/API key, deploy the repository as a Hugging Face Dedicated Inference Endpoint. The included handler.py makes endpoint responses use the same JSON pattern:

• code
• explanation
• confidence
• important_tokens
• relevancy_score
• hallucination
• hallucination_check_reason
• latency_ms

Direct Hugging Face serverless calls to the model repo are not guaranteed to run custom LoRA repos. Dedicated endpoints or a cloud VM are required for true cloud execution.