README.md

---
license: apache-2.0
base_model: google/gemma-4-12b
language:
  - en
tags:
  - code
  - coding
  - agentic
  - gemma-4
  - code-generation
  - perciqa
  - aurora
  - mini
  - lora
  - sft
  - qlora
datasets:
  - perciqa/aurora-code-sft-v1
pipeline_tag: text-generation
library_name: peft
---

# Aurora-Code-Mini-1

> Compact. Capable. Completely open.

**Aurora-Code-Mini-1** is a 12B dense coding model built by [Perciqa](https://perciqa.com), a Canadian AI company. Fine-tuned from Gemma 4 12B on curated agentic coding instruction pairs using QLoRA.

## Training Details

| Field | Value |
|---|---|
| Base Model | google/gemma-4-12b |
| Method | QLoRA (4-bit NF4) |
| LoRA Rank | 16 |
| LoRA Alpha | 32 |
| LoRA Dropout | 0.0 |
| Target Modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Precision | BF16 compute, NF4 storage |
| Context Length | 2048 tokens |
| Epochs | 3 |
| Batch Size | 2 (effective 16 with grad accum 8) |
| Learning Rate | 2e-4 |
| Warmup Steps | 10 |
| Optimizer | AdamW |
| Gradient Checkpointing | True |

## Training Data

- **Source:** perciqa/aurora-code-sft-v1
- **Train:** 1,992 records
- **Validation:** 225 records
- **Categories:** Code generation, debugging, code review, agentic coding, refactoring, test writing

## Hardware

- **GPU:** AMD Radeon W7900 (48 GB VRAM)
- **Framework:** ROCm 7.2.4, PyTorch 2.10.0, Transformers 5.14.1, TRL 0.24.0, PEFT 0.19.1
- **Training time:** ~10 hours

## Quickstart

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Perciqa/Aurora-Code-Mini-1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)

system_prompt = (
    "You are Aurora, an AI code assistant built by Perciqa. "
    "You help developers write, review, and understand code. "
    "You provide clear, correct, and complete solutions. "
    "When you're unsure, you say so."
)

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": "Write a Python function to merge two sorted lists."},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=1024,
    temperature=0.7,
    do_sample=True,
)
response = tokenizer.decode(outputs[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
print(response)
```

## License

Apache 2.0

## About Perciqa

Perciqa is a Canadian AI company building enterprise models and tools that organisations can deploy, audit, and fully control — on their own infrastructure, on their own terms.

[perciqa.com](https://perciqa.com)