# MuJoCo → Unity → WebGL Pipeline
Export your MuJoCo scene to Unity, visualize the Φ+G representation, and deploy as WebGL for the browser.
---
## Overview
```
MuJoCo MJCF (.xml)
↓ Unity MuJoCo Plugin (import)
Unity Scene (3D objects + physics)
↓ Custom scripts (add Φ+G visualization)
Unity Scene with field/graph overlays
↓ Unity WebGL Build
Browser-ready .html + .wasm + .data
↓ Host on HuggingFace Spaces (static)
Interactive 3D playbook
```
---
## Step 1: Install MuJoCo Unity Plugin
### Prerequisites
- Unity 2021.3+ (LTS recommended)
- Git
- Windows/Mac/Linux
### Setup
```bash
# Clone the MuJoCo repo (or just the unity folder)
git clone https://github.com/google-deepmind/mujoco.git
cd mujoco
git checkout 3.2.4 # match your MuJoCo version
```
In Unity:
1. Open **Window → Package Manager**
2. Click **+** → **Add package from disk**
3. Navigate to `mujoco/unity/package.json` and select it
4. Unity will import the C# scripts
Then copy the MuJoCo native library:
- **Windows**: Copy `mujoco.dll` from your MuJoCo install to `Assets/Plugins/`
- **Mac**: Copy `libmujoco.dylib`
- **Linux**: Copy `libmujoco.so`
### Verify
After import, you should see **Assets → Import MuJoCo Scene** in the menu bar.
---
## Step 2: Import Your MJCF Scene
The scene XML files are in the `render_scripts/` folder. Here's the main tabletop scene:
```xml
```
### In Unity:
1. **Assets → Import MuJoCo Scene** → select `tabletop_scene.xml`
2. The plugin creates GameObjects for each body/geom with MuJoCo components
3. Each geom gets a MeshRenderer with procedural mesh
4. The scene runs MuJoCo physics in Play mode
---
## Step 3: Add Φ+G Visualization Overlays
Create these C# scripts to visualize the field and graph:
### `FieldVisualizer.cs` — Ownership heatmap + SDF wireframes
```csharp
using UnityEngine;
using System.Collections.Generic;
public class FieldVisualizer : MonoBehaviour
{
[Header("Objects")]
public Transform[] trackedObjects; // Drag MuJoCo bodies here
public Color[] objectColors; // One color per object
[Header("Ownership Grid")]
public bool showOwnership = false;
public float gridResolution = 0.01f;
public float gridExtent = 0.3f;
public float gridHeight = 0.455f;
[Header("SDF Wireframe")]
public bool showSDF = true;
private GameObject[,] ownershipTiles;
private List edgeRenderers = new List();
void Start()
{
if (objectColors.Length == 0)
objectColors = new Color[] {
new Color(1f, 0.45f, 0.09f), // orange
new Color(0.23f, 0.51f, 0.96f), // blue
new Color(0.13f, 0.77f, 0.37f), // green
new Color(0.55f, 0.36f, 0.96f), // purple
new Color(0.92f, 0.70f, 0.03f), // yellow
};
CreateOwnershipGrid();
}
void CreateOwnershipGrid()
{
int size = Mathf.CeilToInt(gridExtent * 2 / gridResolution);
ownershipTiles = new GameObject[size, size];
for (int y = 0; y < size; y++)
{
for (int x = 0; x < size; x++)
{
float wx = -gridExtent + x * gridResolution;
float wy = -gridExtent + y * gridResolution;
var tile = GameObject.CreatePrimitive(PrimitiveType.Quad);
tile.transform.SetParent(transform);
tile.transform.position = new Vector3(wx, wy, gridHeight + 0.001f);
tile.transform.localScale = Vector3.one * gridResolution;
tile.transform.rotation = Quaternion.identity;
var mat = new Material(Shader.Find("Unlit/Color"));
mat.color = Color.clear;
tile.GetComponent().material = mat;
tile.SetActive(false);
ownershipTiles[y, x] = tile;
}
}
}
void Update()
{
if (showOwnership)
UpdateOwnership();
// Toggle visibility
foreach (var tile in ownershipTiles)
if (tile != null) tile.SetActive(showOwnership);
}
void UpdateOwnership()
{
int size = ownershipTiles.GetLength(0);
for (int y = 0; y < size; y++)
{
for (int x = 0; x < size; x++)
{
var tile = ownershipTiles[y, x];
if (tile == null) continue;
Vector3 pos = tile.transform.position;
// Find nearest object (simplified gating)
float minDist = float.MaxValue;
int owner = -1;
for (int i = 0; i < trackedObjects.Length; i++)
{
float d = Vector3.Distance(pos, trackedObjects[i].position);
if (d < minDist) { minDist = d; owner = i; }
}
var mat = tile.GetComponent().material;
if (minDist < 0.05f && owner >= 0)
{
Color c = objectColors[owner % objectColors.Length];
c.a = Mathf.Clamp01(1f - minDist / 0.05f) * 0.4f;
mat.color = c;
}
else
{
mat.color = Color.clear;
}
}
}
}
}
```
### `GraphVisualizer.cs` — Contact edges + probabilities
```csharp
using UnityEngine;
using System.Collections.Generic;
public class GraphVisualizer : MonoBehaviour
{
public Transform[] objects;
public float contactThreshold = 0.08f; // meters
public Color edgeColor = new Color(1f, 0.45f, 0.09f, 0.8f);
public bool showEdges = true;
private List edges = new List();
void Update()
{
// Clear old edges
foreach (var lr in edges) if (lr != null) Destroy(lr.gameObject);
edges.Clear();
if (!showEdges) return;
// Create edges between close objects
for (int i = 0; i < objects.Length; i++)
{
for (int j = i + 1; j < objects.Length; j++)
{
float dist = Vector3.Distance(objects[i].position, objects[j].position);
float contactProb = 1f / (1f + Mathf.Exp((dist - contactThreshold) / 0.02f));
if (contactProb > 0.01f)
{
var go = new GameObject($"Edge_{i}_{j}");
go.transform.SetParent(transform);
var lr = go.AddComponent();
lr.positionCount = 2;
lr.SetPositions(new Vector3[] { objects[i].position, objects[j].position });
lr.startWidth = 0.002f + contactProb * 0.005f;
lr.endWidth = lr.startWidth;
lr.material = new Material(Shader.Find("Unlit/Color"));
Color c = edgeColor;
c.a = contactProb;
lr.startColor = c;
lr.endColor = c;
edges.Add(lr);
}
}
}
}
}
```
### `FieldQueryUI.cs` — Click-to-query the field
```csharp
using UnityEngine;
using UnityEngine.UI;
public class FieldQueryUI : MonoBehaviour
{
public Text positionText;
public Text sdfText;
public Text ownerText;
public Text frictionText;
public Transform[] objects;
public string[] objectNames;
public float[] objectFriction;
// Material data per object (set in inspector)
[System.Serializable]
public struct MaterialData {
public float friction;
public float density;
public float stiffness;
}
public MaterialData[] materials;
void Update()
{
if (Input.GetMouseButtonDown(0))
{
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
Plane plane = new Plane(Vector3.up, new Vector3(0, 0.455f, 0));
if (plane.Raycast(ray, out float enter))
{
Vector3 hitPoint = ray.GetPoint(enter);
QueryField(hitPoint);
}
}
}
void QueryField(Vector3 point)
{
positionText.text = $"({point.x:F3}, {point.y:F3}, {point.z:F3})";
float minDist = float.MaxValue;
int nearest = -1;
for (int i = 0; i < objects.Length; i++)
{
float d = Vector3.Distance(point, objects[i].position);
if (d < minDist) { minDist = d; nearest = i; }
}
sdfText.text = $"{minDist:F4}m";
ownerText.text = minDist < 0.03f ? objectNames[nearest] : "Background";
frictionText.text = minDist < 0.03f ? $"{materials[nearest].friction:F2}" : "—";
}
}
```
---
## Step 4: Build for WebGL
1. **File → Build Settings** → Select **WebGL**
2. Click **Switch Platform**
3. **Player Settings:**
- Set **Compression Format** to **Brotli** (smallest)
- Disable **Auto Graphics API** and add **WebGL 2.0**
- Set **Memory Size** to 256MB
- Under **Publishing Settings**, enable **Decompression Fallback**
4. Click **Build**
Unity produces:
```
Build/
index.html
Build/
Build.data.br (scene data)
Build.framework.js (runtime)
Build.loader.js (loader)
Build.wasm.br (compiled code)
```
### IMPORTANT: MuJoCo WASM Limitation
The MuJoCo Unity plugin uses native C libraries, which **do not compile to WebAssembly**. For WebGL builds, you have two options:
**Option A: Static scene (no live physics)**
- Import the scene in Unity, position everything, bake the visual state
- Remove all MuJoCo components before building
- The WebGL build shows the scene visually with your Φ+G overlays but without live MuJoCo physics
- This is sufficient for the playbook — you want to show the representation, not run live sim
**Option B: Pre-recorded simulation**
- Run the MuJoCo sim in the editor, record object trajectories as AnimationClips
- Replace MuJoCo physics with Unity's Animation system for WebGL playback
- Gives you "live" looking simulation without native MuJoCo at runtime
I recommend **Option A** for the playbook. The Three.js version (phi_g_interactive_scene.html, already built) gives you the interactive field query without Unity overhead.
---
## Step 5: Host on HuggingFace Spaces
### For the Unity WebGL build:
```bash
# Create HF Space
huggingface-cli repo create your-username/hybrid-world-model --type space --space_sdk static
# Copy build files
cp -r Build/* hybrid-world-model/
cd hybrid-world-model
git add .
git commit -m "Add Unity WebGL scene"
git push
```
### For the Three.js version (simpler, recommended):
```bash
# Just upload the HTML file as index.html
huggingface-cli repo create your-username/phi-g-scene --type space --space_sdk static
cp phi_g_interactive_scene.html index.html
git add index.html
git commit -m "Interactive Phi+G scene"
git push
```
The Three.js version is a single HTML file with no dependencies — it just works.
---
## Step 6: Recommended Approach
For the playbook, use **both**:
1. **Three.js scene** (phi_g_interactive_scene.html) — embed directly in the playbook HTML. Lightweight, no build step, works everywhere. Shows the Φ+G representation with clickable objects, field queries, graph edges, and ownership maps.
2. **Unity rendered videos** (from render scripts) — pre-rendered MP4s showing MuJoCo physics in action. Embed as `