| # chipforge-llm | |
| LoRA adapters for Verilog HDL generation, fine-tuned on Nemotron-30B. | |
| ## phase_1/v0.1/ | |
| | Field | Value | | |
| |-------|-------| | |
| | Base model | NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 | | |
| | Training data | 36,321 Verilog modules (filtered ≤10k tokens) | | |
| | Method | LoRA CPT (Continued Pre-Training) | | |
| | LoRA rank | 8 | | |
| | Trainable params | ~4.5M (0.015% of 30B) | | |
| | Max seq len | 1024 | | |
| | Epochs | 2 | | |
| | Final train loss | 0.42 | | |
| | Final eval loss | 0.43 | | |
| | Token accuracy | ~89% | | |
| | Hardware | 2x A100-80GB, FSDP | | |
| ### Usage | |
| ``\python | |
| from peft import PeftModel | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| model = AutoModelForCausalLM.from_pretrained( | |
| "NVIDIA/Nemotron-3-Nano-30B-A3B-BF16", | |
| torch_dtype="bfloat16", | |
| device_map="auto", | |
| trust_remote_code=True, | |
| ) | |
| model = PeftModel.from_pretrained(model, "Ashx098/chipforge-llm/phase_1/v0.1") | |
| tokenizer = AutoTokenizer.from_pretrained( | |
| "NVIDIA/Nemotron-3-Nano-30B-A3B-BF16", | |
| trust_remote_code=True, | |
| ) | |
| ``` | |
| ### wandb | |
| Run: https://wandb.ai/avinash-mynampati-juspay/vaschpforge-llm/runs/c5gid36w | |