rpDungeon/glimmer-b200-final

Gemma 4 E4B — B200 production-scale instruct-subspace-masked mix CPT

This is the B200 production-scale validation of the instruct-subspace gradient-mask technique on google/gemma-4-E4B-it. The model was trained on a deliberately wild 7-corpus mix at 16k context, with a high-rank mask (rank 256) protecting the directions in weight space that carry instruction-following behavior.

The goal was to see whether the mask could absorb a much larger and more aggressive corpus than any of the small-scale single-source experiments (T2 INSTRUCT, T2 CPT-marvin, SD CPT — each ~6-34M tokens) while still preserving IFEval. Answer: largely yes, with one caveat (long-form repetition).

Training recipe

Setting	Value
Hardware	1× NVIDIA B200 (RunPod secure tier, $5.51/hr)
Base	google/gemma-4-E4B-it
LoRA	r=64, α=128 (4:1 ratio), all linear targets
Instruct-subspace mask	rank 256, per-step gradient projection via post-accumulate-grad hook (see TIES sibling card for math)
Sequence length	16384 (cce_patch was incompatible with transformers 5.8.1; without CCE 32k OOM'd on 178GB B200, so dropped to 16k)
Optimizer	paged adamw 8-bit, lr 5e-5, bf16 mixed precision, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
Epochs	1 (~2083 steps; trainer cleanup saved at step 2000)
Wall clock	23752s (~6.6h)
Total cost	~$41-42

Data mix (~62.6M tokens, seed=42 deterministic concat)

Source	Description	Note
erotica_quality_cleaned (full)	Mixed erotic CHYOA concatenated into cohesive storylines	Primary signal; the full set, not the 30% subset previous balancing runs used
counter_signal_training_balanced	Mixed-genre fiction, some lewd, predominantly good prose	Non-erotica style anchor
erotic_books_filtered	Filtered novels
rosier_inf_strict (subset)	Heavily-rated adult fiction	"if you really want to fry it" tier
spring_dragon	Old AI-Dungeon-era 2nd-person text-adventure	Adventure mode signal
terry_pratchett (chapter set)	Discworld and other Pratchett novels	High-quality British genre fiction style
worm_chapters	Worm web serial	Long-form descriptive prose

Total chunked at 32k tokens with 256-token overlap (Gemma 4 tokenizer); shuffled with seed=42; trained at 16k effective ctx (chunks split when longer).

Evaluation

Same harness as the rest of the E4B masking experiment lineage (IFEval strict 541 prompts + 10-prompt style gen + grpo_v12d metrics including V3 humanness classifier output). All numbers from the bf16 GGUF.

Metric	base-it	B200 mix r=256	Δ vs base
IFEval strict prompt-level	83.36	78.00	-5.36pp
IFEval strict instruction-level	—	84.53	—
IFEval loose prompt-level	—	81.52	—
V3 humanness h_delta	-16.74	-12.89	+3.85 (top of E4B cohort)
q_prob (qwen-DPO classifier — interpret with caution)	0.098	0.182	+0.084
det_open_frac	0.322	0.248	-0.074
amod_frac	0.142	0.065	-0.077
dialogue_frac	0.417	0.423	flat
contrast_per_1k	6.26	1.27	-4.99
em_dash_per_1k	1.60	0.47	-1.13
slop/1k	19.94	13.62	-6.32 (best non-base)
rep3g/1k	7.39	50.30	+42.91 ⚠️
word_count (avg)	880	980	+100
sent_count (avg)	64.1	78.1	+14

What to read from this

• The mask did its job at scale. -5.36pp IFEval is worse than the SD r=256 result (-2.03pp on 11M tokens) but better than the r=64 single-source runs (-6.8 to -8.5pp). For 5-6× the training data on a much more aggressive mix, the mask absorbed the brunt.
• Best style movement in the cohort. h_delta Δ +3.85 edges out the TIES merge (+3.64) and is essentially tied with T2 INSTRUCT (+3.91). Confirms the larger/richer corpus is moving the model further.
• Lowest AI-slop of any variant including TIES. Suggests the mix actively pushes the model away from generic LLM tics.
• ⚠️ rep3g/1k 50.30 is the open problem. 7× base rate, much worse than any other masked variant. Diagnostic: long-form descriptive passages in the smut/worm/adventure corpora teach the model to echo phrases. Same failure mode observed on the Stage 2/3 Marvin (Qwen) runs. Fix: antirep DPO pass. This model is not deployment-ready as-is; it is a successful training-technique demonstration.

Caveats

• No matched-recipe unmasked baseline was trained. The mask's effect at this scale is inferred from cross-recipe comparison with smaller-scale masked vs unmasked pairs (T2 INSTRUCT/CPT/SD CPT each had matched pairs showing 41-83% IFEval recovery from the mask). A 1:1 direct A/B at B200 scale would cost another ~$40.
• The V3 humanness classifier was trained against Gemma 4 31B-it outputs, not E4B-it. E4B is gemma-family-similar so V3 is still measuring the right axis, but distance-to-marvin is not literally comparable across model sizes.
• q_prob (ModernBERT trained on Qwen-Marvin DPO chosen/rejected pairs) is a weak quality oracle for a gemma derivative. The 0.182 jump from 0.098 is real but the absolute value shouldn't be over-read.
• rep3g/1k of 50.3 means real generations will exhibit visible phrase echoes at long context. Useful for technique validation; not useful for end-user RP/chat without an antirep cleanup pass.

Files

Path	Description
adapter/	Final LoRA adapter (558 MB safetensors + tokenizer + chat template + adapter_config)
merged/	Full merged bf16 safetensors model (15.9 GB) — adapter applied to google/gemma-4-E4B-it

A bf16 GGUF + Q6_K quant are published at ToastyPigeon/Gemma4-Test-GGUFs/Gemma4-E4B-b200-mix-masked-r256.{bf16,Q6_K}.gguf for llama.cpp use.

Provenance

• Trained 2026-05-16 → 2026-05-17 by ToastyPigeon (training) + autumn (server + Discord automation) + twistedshadows (B200 compute)
• Pod: RunPod B200 secure tier (~7.5h pod lifetime including bootstrap)
• Datasets: see ToastyPigeon's private dataset collection (sources listed above, NSFW corpora not redistributed)
• Method: instruct-subspace gradient mask, rank 256, on top of LoRA r=64 α=128
• Sibling experiments: small-scale variants live in ToastyPigeon/Gemma4-Test-GGUFs (T2 INSTRUCT/CPT masked, SD CPT masked at r=64 and r=256, TIES merge)

Recommended next step

Apply antirep DPO (the Stage 4 approach that worked on Qwen Marvin: ~250-500 pairs, chosen=clean, rejected=repetitive, mask_thinking=true) to clean up the rep3g/1k score, then re-eval. Target: bring rep3g/1k below 15 while preserving the h_delta/slop wins.

Sampling

Gemma 4 defaults: temperature=1.0, no top_p / top_k / rep_penalty.