README.md

metadata

base_model: google/gemma-4-E2B-it-qat-mobile-transformers
library_name: peft
license: gemma
pipeline_tag: text-generation
language:
  - en
tags:
  - lora
  - peft
  - strands-agents
  - code
  - gemma-4
  - domain-adaptation

strands-gemma4-e2b — Strands Agents expert (Gemma 4 E2B LoRA)

LoRA adapter that post-tunes google/gemma-4-E2B-it-qat-mobile-transformers on the full Strands Agents / Strands Robots codebase + docs corpus, turning the mobile-class Gemma 4 E2B into a Strands-Agents domain expert.

Sister models: cagataydev/strands-qwen3-vl-2b (same corpus, Qwen3-VL-2B backbone). Part of the self-learning-model research project (github.com/cagataycali/slm).

Results

Metric	Before	After	Δ
Holdout NLL (strands corpus)	2.689	1.260	−1.43 (−53%)
Train loss (600 steps)	3.07	0.96

Training

• Corpus: 2,288 documents (~5M tokens) — strands-agents SDK source, strands-tools, strands-robots, docs, examples, and Golden-200 Q&A pairs
• Method: QAT checkpoint dequantized to bf16, then LoRA on all language-model attention + MLP projections (q,k,v,o,gate,up,down_proj), vision tower untouched
• Config: r=32, alpha=64, dropout=0.05, 600 steps, bs 2 × accum 4, lr 1e-4 cosine, block 1024, AdamW, gradient checkpointing
• Trainable: 48.3M params (adapter only; base frozen)
• Hardware: 1× NVIDIA L40S, ~2,570 tok/s

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "google/gemma-4-E2B-it-qat-mobile-transformers"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "cagataydev/strands-gemma4-e2b")

msgs = [{"role": "user", "content": "How do I create a custom tool in Strands Agents?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Note: the base is a QAT (quantization-aware-training) checkpoint. Training was done on the dequantized bf16 weights; for training-compatible loading, dequantize QAT wrappers or load in bf16 as above.

Intended use & limitations

• Domain expert for the Strands Agents ecosystem (SDK APIs, tools, patterns, robots).
• Inherits Gemma 4 license/usage terms. Not evaluated for general-purpose safety beyond base.
• Trained on a code corpus snapshot (July 2026); APIs may drift.

Reproducibility

Training script: strands_tune/train_lora_any.py in the research repo (--dequant-qat --targets attn_mlp --steps 600 --bs 2 --accum 4 --lr 1e-4 --r 32). Full step log in train_log.json in this repo.