ananddey/gemma-4-E2B-asm-init

Gemma 4 E2B — Assamese Pre-Initialized

Pre-initialized Gemma 4 E2B with a custom 32K Assamese SentencePiece tokenizer and FOCUS-aligned embeddings. Ready for continued pretraining (CPT) on Assamese text.

This is NOT a trained model — it is a CPT-ready initialization checkpoint. The base model (google/gemma-4-E2B) has been prepared for Assamese pretraining by replacing the tokenizer and warm-initializing the embedding table via FOCUS. Train it on Assamese corpus (e.g. AsmCorpus) to produce a full Assamese language model.

What Changed from Base Gemma 4 E2B

Component	Original	This Model
Tokenizer	Gemma-native (256K)	SentencePiece BPE (32K, Assamese-optimized)
Vocab size	256,000	32,000
Embedding table	Original	FOCUS-initialized (semantically aligned)
Transformer weights	Original	Unchanged
Architecture	Gemma4ForConditionalGeneration	Unchanged (35 layers, 1536 hidden, 1.6B params)

Tokenizer

A SentencePiece unigram tokenizer trained on the AsmCorpus dataset.

• Vocab size: 32,000
• Special tokens: <unk> (0), <s> (1), </s> (2)
• Space marker: ▁ (U+2581)
• Training data: Assamese monolingual text (~5.8B Gemma-equivalent tokens)
• File: tokenizer.model , tokenizer.json (HF fast tokenizer)

FOCUS Embedding Initialization

The embedding table was initialized using FOCUS (Fast Overlapping token Combinations Using Surface-form), which maps each new token's surface form through the original Gemma tokenizer and averages the corresponding base embeddings. This gives the model semantically meaningful embeddings from step 0, bypassing the need for a long embedding warmup phase.

• Special tokens (<unk>, <s>, </s>, <pad>) are copied exactly from the base model
• All 32K new tokens are initialized via FOCUS (surface-form mapping) with mean-embedding fallback
• LM head is tied to input embeddings (same as base Gemma 4)

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ananddey/gemma-4-E2B-asm-init",
    torch_dtype=torch.bfloat16,
    attn_implementation="sdpa",  # required: Gemma 4 global layers use head_dim=512 > FlashAttention limit
)
tokenizer = AutoTokenizer.from_pretrained("ananddey/gemma-4-E2B-asm-init")

To continue pretraining, use the training script:

python scripts/train_gemma_as.py --preset auto \
    --train-data data/train/mixed_train.jsonl.gz \
    --val-data data/train/mixed_val.jsonl.gz \
    --output-dir ./gemma-4-as

Intended Use

This checkpoint is a starting point for continued pretraining (CPT) on Assamese text. It is not instruction-tuned, not chat-aligned, and should not be used as an end-user assistant without further training.

How It Was Created

1. Trained a SentencePiece BPE tokenizer on AsmCorpus
2. Loaded google/gemma-4-E2B weights
3. Ran FOCUS embedding initialization (surface-form mapping through the base tokenizer)
4. Resized embedding table and lm_head to 32,000
5. Saved weights — transformer layers are untouched

Citation

@misc{gemma-4-asm-init,
  author = {Anand Dey},
  title = {Gemma 4 E2B — Assamese Pre-Initialized Checkpoint},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {https://huggingface.co/ananddey/gemma-4-E2B-asm-init},
}