UAB-NLP
/

ProjGen_Finetuned_llama

@@ -2,199 +2,232 @@
 library_name: transformers
 tags:
 - unsloth
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
 ## Model Details
 ### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 ### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
 ## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
 ### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
 ## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 library_name: transformers
 tags:
 - unsloth
+- qlora
+- lora
+- llama-3.2
+- instruction-tuned
+- bf16
+- 4bit
 ---
+# Model Card: UAB-NLP/ProjGen_Finetuned_llama
+A **LLaMA-3.2** based instruction-tuned model fine-tuned with **Unsloth + QLoRA** using 🤗 **Transformers**.
+This model is part of the **ProjGen project**, aimed at enhancing developer productivity through automated project generation and structured code scaffolding.
+---
 ## Model Details
 ### Model Description
+- **Base model:** `meta-llama/Llama-3.2-<SIZE>-Instruct` <!-- replace SIZE with e.g. 8B/70B -->
+- **Finetuning method:** Unsloth + QLoRA (LoRA adapters)
+- **Precision (train):** 4-bit NF4 quantization (bitsandbytes) + bf16 compute
+- **Context length:** 4096
+- **Task(s):** Instruction following & project/code generation
+- **License:** Inherits from Meta’s LLaMA-3.2 license
+- **Developed by:** UAB-NLP Group (Sai Praneeth Kumar, University of Alabama at Birmingham)
+- **Finetuned from:** `meta-llama/Llama-3.2-<SIZE>-Instruct`
+- **Shared by:** [UAB-NLP](https://huggingface.co/UAB-NLP)
+### Model Sources
+- **Repository:** [UAB-NLP/ProjGen_Finetuned_llama](https://huggingface.co/UAB-NLP/ProjGen_Finetuned_llama)
+- **Project Paper:** ProjGen – Enhanced Developer Productivity for Flask Project Generation with a RAG-Enhanced Fine-Tuned Local LLM
+---
+## Intended Uses & Limitations
 ### Direct Use
+- Generating Flask/Django/Streamlit project structures automatically.
+- Instruction-following tasks related to software engineering and code generation.
+### Downstream Use
+- Further fine-tuning on domain-specific datasets (e.g., medical imaging, finance, etc.).
+- Integration into developer assistants and productivity tools.
+### Out-of-Scope / Limitations
+- Not suitable for medical, legal, or financial decision-making without human review.
+- May hallucinate or produce insecure/inefficient code if not monitored.
+---
 ## Bias, Risks, and Limitations
+The model inherits risks from the base **LLaMA-3.2** model:
+- Possible hallucinations and factual inaccuracies.
+- Dataset/domain biases reflected in responses.
+- Outputs should be validated before production deployment.
+**Recommendation:** Always pair outputs with testing, validation, and human oversight.
+---
+## Getting Started
+### Inference (PEFT adapter form)
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
+model_id = "UAB-NLP/ProjGen_Finetuned_llama"
+tok = AutoTokenizer.from_pretrained(model_id)
+bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    quantization_config=bnb,
+    device_map="auto",
+    torch_dtype="auto"
+)
+prompt = "Generate a Flask project with login, dashboard, and reports."
+inputs = tok(prompt, return_tensors="pt").to(model.device)
+outputs = model.generate(**inputs, max_new_tokens=512)
+print(tok.decode(outputs[0], skip_special_tokens=True))
+```
+---
+## Training Details
+### Data
+- **Dataset:** Custom **ProjGen dataset** built from structured Flask/Django/Streamlit projects and instructions.
+- **Size:** [Fill in #samples / tokens]
+- **Preprocessing:** Chat-style instruction formatting (system/user/assistant), deduplication, truncation at 4096 tokens.
 ### Training Procedure
+- **Quantization:** 4-bit NF4 + double quantization (bitsandbytes)
+- **LoRA Config:**
+  - `r`: 16
+  - `alpha`: 32
+  - `dropout`: 0.05
+  - Target modules: q_proj, k_proj, v_proj, o_proj, gate_up_proj, down_proj
+- **Optimizer:** Paged AdamW (32-bit)
+- **LR / Schedule:** 2e-4 with cosine decay + warmup
+- **Batch size:** [fill in effective batch size]
+- **Epochs/Steps:** 60 steps
+- **Precision:** bf16 mixed precision
+- **Grad checkpointing:** Enabled
+- **Flash attention:** Enabled (Unsloth optimization)
+### Training Hardware
+- **GPU:** RTX 4070 (12GB VRAM) [replace with actual if different]
+- **Training runtime:** 331.47 seconds
+- **Steps per second:** 0.181
+- **Samples per second:** 1.448
+- **Final training loss:** 0.4899
+- **Total FLOPs:** 3.8674e15 FLOPs
+- **Checkpoint size:** ~ (adapter size: ~200MB; merged model size depends on base LLaMA size)
+---
 ## Evaluation
+### Training Loss Curve
+![Training Loss Curve](sandbox:/mnt/data/training_loss_curve.png)
+### Training Loss Table (per step)
+| Step | Training Loss |
+|------|---------------|
+| 1 | 0.9276 |
+| 2 | 1.0030 |
+| 3 | 1.0463 |
+| 4 | 0.9592 |
+| 5 | 0.9903 |
+| 6 | 0.9239 |
+| 7 | 0.7762 |
+| 8 | 0.6905 |
+| 9 | 0.6130 |
+| 10 | 0.5687 |
+| 11 | 0.6292 |
+| 12 | 0.5927 |
+| 13 | 0.5880 |
+| 14 | 0.5021 |
+| 15 | 0.5303 |
+| 16 | 0.4216 |
+| 17 | 0.4692 |
+| 18 | 0.5330 |
+| 19 | 0.4350 |
+| 20 | 0.4003 |
+| 21 | 0.3515 |
+| 22 | 0.4201 |
+| 23 | 0.4200 |
+| 24 | 0.3666 |
+| 25 | 0.4260 |
+| 26 | 0.4261 |
+| 27 | 0.3206 |
+| 28 | 0.4385 |
+| 29 | 0.3475 |
+| 30 | 0.4438 |
+| 31 | 0.4648 |
+| 32 | 0.4088 |
+| 33 | 0.4422 |
+| 34 | 0.4209 |
+| 35 | 0.3593 |
+| 36 | 0.3433 |
+| 37 | 0.3874 |
+| 38 | 0.3604 |
+| 39 | 0.4374 |
+| 40 | 0.4048 |
+| 41 | 0.3604 |
+| 42 | 0.4087 |
+| 43 | 0.3240 |
+| 44 | 0.4375 |
+| 45 | 0.4195 |
+| 46 | 0.3881 |
+| 47 | 0.4383 |
+| 48 | 0.3506 |
+| 49 | 0.4687 |
+| 50 | 0.3709 |
+| 51 | 0.3951 |
+| 52 | 0.4012 |
+| 53 | 0.4020 |
+| 54 | 0.3977 |
+| 55 | 0.2816 |
+| 56 | 0.4136 |
+| 57 | 0.4400 |
+| 58 | 0.3268 |
+| 59 | 0.4218 |
+| 60 | 0.3629 |
+**Final averaged train loss:** 0.4899
+---
+## Environmental Impact (estimate)
+- **Hardware:** RTX 4070 (12GB VRAM) [replace with actual]
+- **Hours:** ~0.09 h (331 seconds)
+- **Region/Provider:** [cloud/on-prem]
+- **Estimated CO₂e:** Use [ML CO₂ Impact](https://mlco2.github.io/impact#compute)
+---
+## Citation
+If you use this model, please cite the base model and this project:
+**BibTeX (base, example):**
+```bibtex
+@article{touvron2023llama,
+  title={LLaMA: Open and Efficient Foundation Language Models},
+  author={Touvron, Hugo and others},
+  journal={arXiv preprint arXiv:XXXX.XXXXX},
+  year={2023}
+}
+```
+**Your work (fill in):**
+```bibtex
+@misc{projgen2025,
+  title = {ProjGen: Enhanced Developer Productivity for Flask Project Generation with a RAG-Enhanced Fine-Tuned Local LLM},
+  author = {Sai Praneeth, Renduchinthala and UAB-NLP Group},
+  year = {2025},
+  howpublished = {\url{https://huggingface.co/UAB-NLP/ProjGen_Finetuned_llama}}
+}
+```
+---
+## Contact
+- **Author:** Sai Praneeth Kumar (UAB, UAB-NLP Group)
+- **HF Profile:** [UAB-NLP](https://huggingface.co/UAB-NLP)