Raghul09
/

llama-code-gen-lora

@@ -1,199 +1,132 @@
 ---
-library_name: transformers
-tags: []
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 ---
+base_model: meta-llama/Llama-3.2-3B
+library_name: peft
+tags: [lora, qlora, code-generation, python]
+datasets: [sahil2801/CodeAlpaca-20k]
+license: llama3.2
 ---
+# Python code generation — LLaMA 3.2 3B + QLoRA
+**HumanEval pass@1: 40.5% → 54.1%** on the uncontaminated subset — a 13.6-point gain
+from a 9.2M-parameter adapter (0.285% of the model), trained in under two hours on a
+single free-tier T4.
+The base model already writes correct code. It just writes it in JavaScript 42% of the
+time, wrapped in markdown fences. This adapter makes Python the default and the output
+directly executable.
+## Results
+**HumanEval, first 50 problems** (greedy decoding, deterministic):
+| | Base | Fine-tuned | Δ |
+|---|---|---|---|
+| **Clean subset (37 problems)** | **40.5%** | **54.1%** | **+13.6 pts** |
+| All 50 | 46.0% | 58.0% | +12.0 pts |
+The clean subset excludes 13 problems whose function names appear in the training data
+(see Contamination). It is the number to cite.
+**Custom eval set** — 40 hand-written Python problems with executable unit tests, all 40
+reference solutions verified to pass before use:
+| | Base | Fine-tuned |
+|---|---|---|
+| Valid Python (free-form instruction) | 57.5% | **100%** |
+| Language correct | 62.5% | **100%** |
+| pass@1 | 82.5% | **90%** |
+Without a specified function signature, the base model emitted non-Python for 15 of 40
+problems and markdown-fenced (non-executable) output for 2 more. The adapter reaches
+100% executable Python with no prompt engineering.
+## Usage
+The prompt format is load-bearing.
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
+from peft import PeftModel
+import torch
+bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=torch.float16, bnb_4bit_use_double_quant=True)
+base = AutoModelForCausalLM.from_pretrained(
+    "meta-llama/Llama-3.2-3B", quantization_config=bnb, device_map="auto")
+model = PeftModel.from_pretrained(base, "Raghul09/llama-code-gen-lora")
+tok = AutoTokenizer.from_pretrained("Raghul09/llama-code-gen-lora")
+prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
+```
+## Training
+| | |
+|---|---|
+| Base | meta-llama/Llama-3.2-3B |
+| Quantization | 4-bit NF4 + double quant — 6.43 GB → 2.20 GB (66%) |
+| LoRA | r=16, α=32, q/k/v/o across 28 layers |
+| Trainable | 9,175,040 / 3,221,924,864 (0.285%) |
+| Data | CodeAlpaca-20K, AST-filtered to Python — 6,418 / 802 / 803 |
+| max_length | 256, set from measured distribution (p50=89, p95=211) |
+| Hardware | single T4, 1.88 h, **4.17 GB peak VRAM** |
+| Checkpoint | epoch 2 of 3 — val loss 0.475; epoch 3 rose to 0.485 |
+Full fine-tuning of this model requires roughly 50 GB. QLoRA brought it to 4.17 GB,
+a 12x reduction, which is what made it feasible on free-tier hardware.
+## Rank sweep
+| Rank | Trainable | Val loss | Valid Python | pass@1 |
+|---|---|---|---|---|
+| 8 | 4,587,520 | 0.4795 | 100% | 82.5% |
+| **16** | **9,175,040** | **0.4753** | **100%** | **90.0%** |
+| 32 | 18,350,080 | 0.4739 | 100% | 82.5% |
+Quadrupling rank bought 1.2% lower validation loss and no consistent gain in pass@1.
+The target behaviour is genuinely low-rank — consistent with the LoRA paper's central
+hypothesis, tested here rather than assumed.
+## Contamination
+13 of the 50 HumanEval problems have their function names defined in CodeAlpaca. The
+fine-tuned model scores 69.2% on those versus ~52% on clean problems, under both prompt
+formats tested. Headline numbers use the clean subset only.
+Function-name matching catches exact reuse but misses paraphrased problems, so 26% is a
+lower bound on overlap, not an estimate.
+## Limitations
+- **Synthetic training data.** CodeAlpaca-20K is GPT-generated via self-instruct,
+  unverified, stylistically homogeneous. Quality is bounded by the teacher model.
+- **Conformance over capability.** With a function signature specified in the prompt,
+  the base model already reaches 97.5% valid Python and 82.5% pass@1. This adapter's
+  main contribution is removing the need for that prompt engineering.
+- **Python only.** Training data was AST-filtered to Python.
+- **Short outputs.** Median training example was 89 tokens; long generations degrade.
+- **Known failure modes:** repetition loops causing mid-generation truncation; calling
+  helper functions it never defines.
+## Methodology notes
+Three measurement errors found and corrected during evaluation:
+1. **Naming confound.** Initial pass@1 read 35% base / 50% fine-tuned. 17 of 20 failures
+   were `NameError` — correct code under a different function name than the test called.
+   Specifying signatures corrected the baseline by 47 points, to 82.5%.
+2. **Indentation destruction.** Raw-format HumanEval returned 0% for both models. The
+   fence-stripper called `.strip()`, removing leading indentation from function bodies.
+   All 50 failures were `IndentationError`.
+3. **Training-data contamination.** A keyword-based Python filter kept 58.7% of
+   CodeAlpaca; AST-parsing a 300-example sample showed 38% weren't valid Python — mostly
+   Java and JavaScript matching on shared keywords like `for` and `class`. Replaced with
+   `ast.parse` plus a syntax-tree check: 0% contamination on re-check.
+Sequence packing was disabled after batch inspection revealed it silently disabled
+completion-only loss masking. Packing requires Flash Attention for block-diagonal
+masking, which requires Ampere; the T4 is Turing.