README.md

---
base_model: google/gemma-4-E2B-it-qat-mobile-transformers
library_name: peft
license: gemma
pipeline_tag: text-generation
language: [en]
tags:
- lora
- peft
- strands-agents
- code
- gemma-4
- domain-adaptation
---

# strands-gemma4-e2b — Strands Agents expert (Gemma 4 E2B LoRA)

LoRA adapter that post-tunes **google/gemma-4-E2B-it-qat-mobile-transformers** on the
full **Strands Agents / Strands Robots** codebase + docs corpus, turning the mobile-class
Gemma 4 E2B into a Strands-Agents domain expert.

Sister models: [cagataydev/strands-qwen3-vl-2b](https://huggingface.co/cagataydev/strands-qwen3-vl-2b)
(same corpus, Qwen3-VL-2B backbone). Part of the self-learning-model research project
([github.com/cagataycali/slm](https://github.com/cagataycali/slm)).

## Results

| Metric | Before | After | Δ |
|---|---|---|---|
| Holdout NLL (strands corpus) | **2.689** | **1.260** | **−1.43 (−53%)** |
| Train loss (600 steps) | 3.07 | 0.96 | |

## Training

- **Corpus**: 2,288 documents (~5M tokens) — strands-agents SDK source, strands-tools,
  strands-robots, docs, examples, and Golden-200 Q&A pairs
- **Method**: QAT checkpoint dequantized to bf16, then LoRA on all language-model
  attention + MLP projections (`q,k,v,o,gate,up,down_proj`), vision tower untouched
- **Config**: r=32, alpha=64, dropout=0.05, 600 steps, bs 2 × accum 4, lr 1e-4 cosine,
  block 1024, AdamW, gradient checkpointing
- **Trainable**: 48.3M params (adapter only; base frozen)
- **Hardware**: 1× NVIDIA L40S, ~2,570 tok/s

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-4-E2B-it-qat-mobile-transformers"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "cagataydev/strands-gemma4-e2b")

msgs = [{"role": "user", "content": "How do I create a custom tool in Strands Agents?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
```

> Note: the base is a QAT (quantization-aware-training) checkpoint. Training was done on
> the dequantized bf16 weights; for training-compatible loading, dequantize QAT wrappers
> or load in bf16 as above.

## Intended use & limitations

- Domain expert for the Strands Agents ecosystem (SDK APIs, tools, patterns, robots).
- Inherits Gemma 4 license/usage terms. Not evaluated for general-purpose safety beyond base.
- Trained on a code corpus snapshot (July 2026); APIs may drift.

## Reproducibility

Training script: `strands_tune/train_lora_any.py` in the research repo
(`--dequant-qat --targets attn_mlp --steps 600 --bs 2 --accum 4 --lr 1e-4 --r 32`).
Full step log in `train_log.json` in this repo.