MISE-FT: fine-tuned Gemma 4 E2B kitchen planner

Status: public release, part of MISE (project page; an English paper is in preparation). Part of an MSc thesis (University of Tehran): an LLM + RAG embodied agent that plans L1 atomic / L2 sequential / L3 abstract kitchen commands in AI2-THOR, with an honest perception stack (no simulator ground truth at test time).

MISE-FT is the small planner of MISE: Gemma 4 E2B (instruction-tuned) with a QLoRA adapter, trained to turn an English or Persian kitchen command into a plan for MISE's skill set. It is meant to be used together with Scene Memory, the retrieved text knowledge (what the robot has seen, stored procedures, an English rephrasing) that MISE puts in the planner prompt. Two training runs are released; they differ only in the training seed.

Files

File Role
adapters/run1/ PEFT LoRA adapter, training run 1 (adapter_config.json, adapter_model.safetensors)
adapters/run2/ PEFT LoRA adapter, training run 2 (same setup, different seed)
gguf/mise-ft-run1-q4_K_M.gguf run 1 adapter merged into the base model, quantized to Q4_K_M (the model used in the MISE experiments)
gguf/mise-ft-run2-q4_K_M.gguf the same for run 2
gguf/Modelfile Ollama Modelfile with the runtime settings used in the experiments

SHA256SUMS lists the exact bytes. The GGUF files are the Ollama models that produced the results below, exported unchanged. The adapters are the training checkpoints with the lowest validation loss; only their base_model_name_or_path was changed from a local path to google/gemma-4-E2B-it.

How to use

The model expects MISE's planner prompts: a system prompt that describes the skills, and a user message with the robot's observations, the Scene Memory block and the command. It answers with JSON, for example {"say": "...", "plan": [{"action": "open", "object": "Fridge", "to": null, "arg": null}, ...]}. The prompts are built by the code in the MISE repository; without them the output is not meaningful.

Ollama (download the gguf/ folder first):

cd gguf
ollama create mise-ft-run1 -f Modelfile
ollama run mise-ft-run1

PEFT + transformers:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "google/gemma-4-E2B-it"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "ArashNasr/mise-ft-gemma4-e2b", subfolder="adapters/run1")

msgs = [{"role": "system", "content": SYSTEM_PROMPT},   # MISE planner prompt
        {"role": "user", "content": USER_MESSAGE}]      # observations + Scene Memory + command
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=512)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

Training

Item Setting
base model Gemma 4 E2B, instruction-tuned (google/gemma-4-E2B-it)
method QLoRA: LoRA on the 4-bit NF4 base, rank 16, alpha 32, dropout 0.05, on the attention and feed-forward projections of the language model
loss / checkpoint loss on the plan tokens only; checkpoint with the lowest validation loss
optimizer AdamW, learning rate 1e-4, cosine decay, 13 warmup steps
schedule 2 epochs, gradient accumulation 8, 260 optimizer steps, max length 4,096 tokens
hardware one RTX 4060 Laptop GPU (8 GB), about 3 h per run
data 714 unique examples from the development kitchen (161 atomic, 193 sequential, 360 abstract), with Scene Memory in the prompts; 88 validation examples from other instruction groups
deployment adapter merged into the base model, quantized to Q4_K_M for Ollama

Training data. Instructions in English and Persian from the development kitchen (AI2-THOR FloorPlan1) only. Each target plan is either the reference plan of the instruction or a plan written by Opus 5.5 that succeeded in the simulator. No example comes from the held-out kitchens or from a test instruction; an automatic overlap check was run before training. The training data itself is not released here.

Results

Success rate on the held-out kitchens (mean of kitchens A and B, equal weight), with Scene Memory at test time:

Planner Success (%)
MISE (Gemma 4 E2B, not fine-tuned) 77.2
MISE-FT, run 1 80.8
MISE-FT, run 2 81.8

Over the two runs, MISE-FT is +4.0 points above MISE (paired comparison, 95% CI 1.3 to 6.8). Most of the gain is on abstract (L3) instructions.

Limitations

  • Simulation only: trained and tested in AI2-THOR kitchens, never on a real robot.
  • Trained on one development kitchen; tested on two held-out kitchens.
  • The outputs are plans for MISE's skill set, not general chat; the model is not intended as a general assistant.
  • It is meant to be used with Scene Memory in the prompt. A separate run trained without Scene Memory exists but is not part of MISE-FT and is not released here.

Licence and credits

Apache-2.0, following the base model: Gemma 4 is released by Google under the Gemma 4 licence (Apache-2.0). This repository holds a derivative of Gemma 4 E2B (google/gemma-4-E2B-it) by Google DeepMind. Simulator: AI2-THOR (Apache-2.0).

Downloads last month
99
GGUF
Model size
5B params
Architecture
gemma4
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for ArashNasr/mise-ft-gemma4-e2b

Adapter
(231)
this model