README.md

---
license: apache-2.0
language:
- en
base_model:
- google/gemma-4-26B-A4B-it
base_model_relation: finetune
pipeline_tag: text-generation
library_name: transformers
tags:
- sft
- trl
- transformers
- gemma-4
- full-fine-tuning
- moe
---

# Lux-V1

**Lux-V1** is a **fully fine-tuned LLM** built on top of [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) by **PoSTMEDIA AI Lab**.

It is trained with PoSTMEDIA's in-house **Capability-Preserving Full Fine-Tuning** recipe — a full-parameter SFT pipeline designed so that customization does **not** erode the reasoning, instruction-following, and multilingual abilities of the Gemma-4 base model.

---

## Highlights

- **Full fine-tuning of Gemma-4-26B-A4B (MoE, ~4B active)** — not LoRA, not an adapter
- **Base capability preserved** — pretraining knowledge and reasoning skills remain intact after SFT
- **Dataset-flexible** — any combination of curated instruction / domain / persona datasets can be composed into a single full-FT run
- **Efficient inference** — MoE architecture keeps active parameters low at serving time

---

## Model Overview

| Specification | Details |
|---------------|---------|
| Base Model | [`google/gemma-4-26B-A4B-it`](https://huggingface.co/google/gemma-4-26B-A4B-it) |
| Parameters | 26B total / ~4B active |
| Architecture | Decoder-only Transformer (MoE) |
| Training Precision | BF16 |
| Inference Precision | BF16 |
| Context Length | Inherits from Gemma-4 base |
| Fine-Tuning Method | Full-parameter SFT (Capability-Preserving recipe) |

---

## Capability-Preserving Full Fine-Tuning

Naive full fine-tuning of large pretrained LLMs often damages the base model's general abilities — a well-known trade-off when SFT is pushed too far. PoSTMEDIA's recipe is built specifically to avoid this.

For Lux-V1, three design choices keep the Gemma-4 base intact while still allowing deep adaptation:

1. **Selective trainable modules for MoE.** Only the attention projections (`q_proj`, `k_proj`, `v_proj`, `o_proj`) are updated. Expert FFN layers are left frozen, which prevents the sparse routing structure of Gemma-4-26B-A4B from collapsing during SFT.
2. **Architecture-tuned learning rate.** The LR is calibrated for MoE training dynamics to keep optimization in the stable regime where the base distribution is preserved.
3. **Continuous base-capability evaluation.** Evaluation runs at the start of training and at every epoch, so any regression in base-model quality is caught early rather than discovered post-hoc.

This means Lux-V1 can be re-trained from the same base with **arbitrary mixtures of datasets** — identity, domain knowledge, instruction-style, reasoning — without losing what Gemma-4 already knows.

---

## Training Configuration

| Parameter | Value |
|-----------|-------|
| Fine-Tuning Method | Full-parameter SFT (attention-only, MoE-safe) |
| Precision | BF16 |
| Distributed Strategy | DeepSpeed ZeRO-3 + CPU offload |
| Training Infrastructure | NVIDIA H200 × 8 |

---

## Quick Start

```bash
pip install transformers accelerate
```

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "PoSTMEDIA/Lux-V1"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "Explain why preserving base-model capability matters during fine-tuning."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Use Cases

- Enterprise assistants requiring domain adaptation without losing general reasoning
- Persona / identity-aligned chat applications
- Instruction-tuned downstream tasks built on a Gemma-4 foundation
- Efficient on-premise serving where MoE active-parameter cost matters

---

## Safety & Limitations

- Inherits the safety characteristics of the Gemma-4 base; output guardrails are recommended for production.
- Not intended for medical, legal, or financial decision-making.
- May occasionally hallucinate — human review is recommended for critical outputs.

---

## Citation

```bibtex
@misc{lux_v1_2026,
  title  = {Lux-V1: Capability-Preserving Full Fine-Tuning of Gemma-4-26B-A4B},
  author = {PoSTMEDIA AI Lab},
  year   = {2026},
  publisher = {Hugging Face}
}
```

---

## Contact

**PoSTMEDIA AI Lab**
- Email: [dev.postmedia@gmail.com](mailto:dev.postmedia@gmail.com)
- Web: [https://postmedia.ai](https://postmedia.ai)
- Web: [https://postmedia.co.kr](https://postmedia.co.kr)