File size: 2,924 Bytes
9426377 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
base_model: google/gemma-4-E2B-it-qat-mobile-transformers
library_name: peft
license: gemma
pipeline_tag: text-generation
language: [en]
tags:
- lora
- peft
- strands-agents
- code
- gemma-4
- domain-adaptation
---
# strands-gemma4-e2b — Strands Agents expert (Gemma 4 E2B LoRA)
LoRA adapter that post-tunes **google/gemma-4-E2B-it-qat-mobile-transformers** on the
full **Strands Agents / Strands Robots** codebase + docs corpus, turning the mobile-class
Gemma 4 E2B into a Strands-Agents domain expert.
Sister models: [cagataydev/strands-qwen3-vl-2b](https://huggingface.co/cagataydev/strands-qwen3-vl-2b)
(same corpus, Qwen3-VL-2B backbone). Part of the self-learning-model research project
([github.com/cagataycali/slm](https://github.com/cagataycali/slm)).
## Results
| Metric | Before | After | Δ |
|---|---|---|---|
| Holdout NLL (strands corpus) | **2.689** | **1.260** | **−1.43 (−53%)** |
| Train loss (600 steps) | 3.07 | 0.96 | |
## Training
- **Corpus**: 2,288 documents (~5M tokens) — strands-agents SDK source, strands-tools,
strands-robots, docs, examples, and Golden-200 Q&A pairs
- **Method**: QAT checkpoint dequantized to bf16, then LoRA on all language-model
attention + MLP projections (`q,k,v,o,gate,up,down_proj`), vision tower untouched
- **Config**: r=32, alpha=64, dropout=0.05, 600 steps, bs 2 × accum 4, lr 1e-4 cosine,
block 1024, AdamW, gradient checkpointing
- **Trainable**: 48.3M params (adapter only; base frozen)
- **Hardware**: 1× NVIDIA L40S, ~2,570 tok/s
## Usage
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE = "google/gemma-4-E2B-it-qat-mobile-transformers"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "cagataydev/strands-gemma4-e2b")
msgs = [{"role": "user", "content": "How do I create a custom tool in Strands Agents?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))
```
> Note: the base is a QAT (quantization-aware-training) checkpoint. Training was done on
> the dequantized bf16 weights; for training-compatible loading, dequantize QAT wrappers
> or load in bf16 as above.
## Intended use & limitations
- Domain expert for the Strands Agents ecosystem (SDK APIs, tools, patterns, robots).
- Inherits Gemma 4 license/usage terms. Not evaluated for general-purpose safety beyond base.
- Trained on a code corpus snapshot (July 2026); APIs may drift.
## Reproducibility
Training script: `strands_tune/train_lora_any.py` in the research repo
(`--dequant-qat --targets attn_mlp --steps 600 --bs 2 --accum 4 --lr 1e-4 --r 32`).
Full step log in `train_log.json` in this repo.
|