| --- |
| license: apache-2.0 |
| base_model: google/gemma-4-12b |
| language: |
| - en |
| tags: |
| - code |
| - coding |
| - agentic |
| - gemma-4 |
| - code-generation |
| - perciqa |
| - aurora |
| - mini |
| - lora |
| - sft |
| - qlora |
| datasets: |
| - perciqa/aurora-code-sft-v1 |
| pipeline_tag: text-generation |
| library_name: peft |
| --- |
| |
| # Aurora-Code-Mini-1 |
|
|
| > Compact. Capable. Completely open. |
|
|
| **Aurora-Code-Mini-1** is a 12B dense coding model built by [Perciqa](https://perciqa.com), a Canadian AI company. Fine-tuned from Gemma 4 12B on curated agentic coding instruction pairs using QLoRA. |
|
|
| ## Training Details |
|
|
| | Field | Value | |
| |---|---| |
| | Base Model | google/gemma-4-12b | |
| | Method | QLoRA (4-bit NF4) | |
| | LoRA Rank | 16 | |
| | LoRA Alpha | 32 | |
| | LoRA Dropout | 0.0 | |
| | Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj | |
| | Precision | BF16 compute, NF4 storage | |
| | Context Length | 2048 tokens | |
| | Epochs | 3 | |
| | Batch Size | 2 (effective 16 with grad accum 8) | |
| | Learning Rate | 2e-4 | |
| | Warmup Steps | 10 | |
| | Optimizer | AdamW | |
| | Gradient Checkpointing | True | |
| |
| ## Training Data |
| |
| - **Source:** perciqa/aurora-code-sft-v1 |
| - **Train:** 1,992 records |
| - **Validation:** 225 records |
| - **Categories:** Code generation, debugging, code review, agentic coding, refactoring, test writing |
| |
| ## Hardware |
| |
| - **GPU:** AMD Radeon W7900 (48 GB VRAM) |
| - **Framework:** ROCm 7.2.4, PyTorch 2.10.0, Transformers 5.14.1, TRL 0.24.0, PEFT 0.19.1 |
| - **Training time:** ~10 hours |
| |
| ## Quickstart |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_name = "Perciqa/Aurora-Code-Mini-1" |
|
|
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model = AutoModelForCausalLM.from_pretrained( |
| model_name, |
| torch_dtype="auto", |
| device_map="auto", |
| ) |
| |
| system_prompt = ( |
| "You are Aurora, an AI code assistant built by Perciqa. " |
| "You help developers write, review, and understand code. " |
| "You provide clear, correct, and complete solutions. " |
| "When you're unsure, you say so." |
| ) |
| |
| messages = [ |
| {"role": "system", "content": system_prompt}, |
| {"role": "user", "content": "Write a Python function to merge two sorted lists."}, |
| ] |
| |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer([text], return_tensors="pt").to(model.device) |
| |
| outputs = model.generate( |
| **inputs, |
| max_new_tokens=1024, |
| temperature=0.7, |
| do_sample=True, |
| ) |
| response = tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True) |
| print(response) |
| ``` |
| |
| ## License |
| |
| Apache 2.0 |
| |
| ## About Perciqa |
| |
| Perciqa is a Canadian AI company building enterprise models and tools that organisations can deploy, audit, and fully control — on their own infrastructure, on their own terms. |
| |
| [perciqa.com](https://perciqa.com) |
| |
