holodeck-parser-llama32-ft

Fine-tuned LoRA adapter for the Holodeck VR voice command parser.

Built on top of Llama 3.2-3B-Instruct, this model takes a voice transcript and a structured scene context and outputs a single JSON command for a 3D virtual environment engine.

Usage via Ollama (recommended)

ollama pull eyyrone/holodeck-parser-llama32-ft

The system prompt is baked into the Ollama model. Send user messages in this format:

Transcript: "move the red chair a bit to the left"
User context: {"id": "user_1", "position": {"x": 0, "y": 1.7, "z": 3}, "look_direction": {"x": 0, "y": 0, "z": -1}}
Voice context: {"lockedObjects": [], "fovObjects": [{"nodeId": "server_abc123", "meshName": "Red Chair", "type": "chair", "position": {"x": 2, "y": 0, "z": 1}, "rotation": {"x": 0, "y": 0, "z": 0, "w": 1}, "scale": {"x": 1, "y": 1, "z": 1}}], "raycastHit": null}

Expected output:

{"command": "edit", "id": "server_abc123", "changes": {"position_relative": {"direction": "left", "units": 1}}}

Note: Strip trailing $ and take text before the first \n\n to get clean JSON from the raw response.

Usage via PEFT (this repo)

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
model = PeftModel.from_pretrained(base, "trans-realities-lab/holodeck-parser-llama32-ft")

Output schema

Command	Description
spawn	Create a new object in the scene
edit	Move, rotate, scale, rename, or toggle visibility
delete	Remove an object
none	No actionable command detected

Edit supports both absolute (position) and relative (position_relative) moves. Relative directions (left, right, forward, back) are relative to the user's look direction; up/down are world-space.

Training details

Parameter	Value
Base model	meta-llama/Llama-3.2-3B-Instruct
Method	QLoRA (4-bit) via Unsloth
LoRA rank	16
LoRA alpha	16
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Dataset	270 synthetic examples (Holodeck parser training set)
Epochs	3
Learning rate	2e-4
Batch size	8 (4 per device × 2 grad accum)
Training loss	0.179
Training time	~8 minutes on RTX 4080 SUPER
Hardware	NVIDIA RTX 4080 SUPER (16 GB)