Hugging Face's logo

Spaces:
Nirav-Madhani
/
WorldRepresentation
Running

App Files Community
WorldRepresentation / render_scripts /UNITY_PIPELINE.md
Nirav Madhani
Import World Representation Space
5af572e
preview code
|
Raw
History Blame Contribute Delete
13.6 kB
# MuJoCo β†’ Unity β†’ WebGL Pipeline
Export your MuJoCo scene to Unity, visualize the Ξ¦+G representation, and deploy as WebGL for the browser.
---
## Overview
```
MuJoCo MJCF (.xml)
 ↓ Unity MuJoCo Plugin (import)
Unity Scene (3D objects + physics)
 ↓ Custom scripts (add Ξ¦+G visualization)
Unity Scene with field/graph overlays
 ↓ Unity WebGL Build
Browser-ready .html + .wasm + .data
 ↓ Host on HuggingFace Spaces (static)
Interactive 3D playbook
```
---
## Step 1: Install MuJoCo Unity Plugin
### Prerequisites
- Unity 2021.3+ (LTS recommended)
- Git
- Windows/Mac/Linux
### Setup
```bash
# Clone the MuJoCo repo (or just the unity folder)
git clone https://github.com/google-deepmind/mujoco.git
cd mujoco
git checkout 3.2.4 # match your MuJoCo version
```
In Unity:
1. Open **Window β†’ Package Manager**
2. Click **+** β†’ **Add package from disk**
3. Navigate to `mujoco/unity/package.json` and select it
4. Unity will import the C# scripts
Then copy the MuJoCo native library:
- **Windows**: Copy `mujoco.dll` from your MuJoCo install to `Assets/Plugins/`
- **Mac**: Copy `libmujoco.dylib`
- **Linux**: Copy `libmujoco.so`
### Verify
After import, you should see **Assets β†’ Import MuJoCo Scene** in the menu bar.
---
## Step 2: Import Your MJCF Scene
The scene XML files are in the `render_scripts/` folder. Here's the main tabletop scene:
```xml
<!-- Save as tabletop_scene.xml -->
<mujoco model="tabletop_scene">
 <option timestep="0.002" gravity="0 0 -9.81"/>
 <worldbody>
 <light pos="0 0 2" dir="0 0 -1"/>
 <geom name="floor" type="plane" size="2 2 0.1" rgba="0.7 0.7 0.7 1"/>
 <body name="table" pos="0 0 0.4">
 <geom type="box" size="0.4 0.3 0.02" rgba="0.45 0.35 0.25 1"/>
 </body>
 <body name="wood_block" pos="-0.15 0.06 0.455">
 <joint type="free"/>
 <geom type="box" size="0.035 0.035 0.035" mass="0.6"
rgba="0.65 0.45 0.25 1" friction="0.4 0.005 0.0001"/>
 </body>
 <body name="rubber_ball" pos="0.05 -0.06 0.455">
 <joint type="free"/>
 <geom type="sphere" size="0.03" mass="1.1"
 rgba="0.18 0.18 0.20 1" friction="0.8 0.005 0.0001"/>
 </body>
 <body name="metal_cylinder" pos="0.18 0.07 0.455">
 <joint type="free"/>
 <geom type="cylinder" size="0.025 0.03" mass="3.0"
 rgba="0.7 0.72 0.75 1" friction="0.2 0.005 0.0001"/>
 </body>
 <body name="plastic_cup" pos="-0.08 -0.1 0.46">
 <joint type="free"/>
 <geom type="cylinder" size="0.022 0.035" mass="0.12"
 rgba="0.2 0.55 0.85 1" friction="0.35 0.005 0.0001"/>
 </body>
 <body name="ceramic_plate" pos="0.1 0.0 0.44">
 <joint type="free"/>
 <geom type="cylinder" size="0.05 0.008" mass="0.35"
 rgba="0.92 0.90 0.85 1" friction="0.25 0.005 0.0001"/>
 </body>
 </worldbody>
</mujoco>
```
### In Unity:
1. **Assets β†’ Import MuJoCo Scene** β†’ select `tabletop_scene.xml`
2. The plugin creates GameObjects for each body/geom with MuJoCo components
3. Each geom gets a MeshRenderer with procedural mesh
4. The scene runs MuJoCo physics in Play mode
---
## Step 3: Add Ξ¦+G Visualization Overlays
Create these C# scripts to visualize the field and graph:
### `FieldVisualizer.cs` β€” Ownership heatmap + SDF wireframes
```csharp
using UnityEngine;
using System.Collections.Generic;
public class FieldVisualizer : MonoBehaviour
{
 [Header("Objects")]
 public Transform[] trackedObjects; // Drag MuJoCo bodies here
 public Color[] objectColors; // One color per object
[Header("Ownership Grid")]
 public bool showOwnership = false;
 public float gridResolution = 0.01f;
 public float gridExtent = 0.3f;
 public float gridHeight = 0.455f;
[Header("SDF Wireframe")]
 public bool showSDF = true;
private GameObject[,] ownershipTiles;
 private List<LineRenderer> edgeRenderers = new List<LineRenderer>();
void Start()
 {
 if (objectColors.Length == 0)
 objectColors = new Color[] {
 new Color(1f, 0.45f, 0.09f), // orange
 new Color(0.23f, 0.51f, 0.96f), // blue
 new Color(0.13f, 0.77f, 0.37f), // green
 new Color(0.55f, 0.36f, 0.96f), // purple
 new Color(0.92f, 0.70f, 0.03f), // yellow
 };
CreateOwnershipGrid();
 }
void CreateOwnershipGrid()
 {
 int size = Mathf.CeilToInt(gridExtent * 2 / gridResolution);
 ownershipTiles = new GameObject[size, size];
for (int y = 0; y < size; y++)
 {
 for (int x = 0; x < size; x++)
 {
 float wx = -gridExtent + x * gridResolution;
 float wy = -gridExtent + y * gridResolution;
var tile = GameObject.CreatePrimitive(PrimitiveType.Quad);
 tile.transform.SetParent(transform);
 tile.transform.position = new Vector3(wx, wy, gridHeight + 0.001f);
 tile.transform.localScale = Vector3.one * gridResolution;
 tile.transform.rotation = Quaternion.identity;
var mat = new Material(Shader.Find("Unlit/Color"));
 mat.color = Color.clear;
 tile.GetComponent<Renderer>().material = mat;
 tile.SetActive(false);
ownershipTiles[y, x] = tile;
 }
 }
 }
void Update()
 {
 if (showOwnership)
 UpdateOwnership();
// Toggle visibility
 foreach (var tile in ownershipTiles)
 if (tile != null) tile.SetActive(showOwnership);
 }
void UpdateOwnership()
 {
 int size = ownershipTiles.GetLength(0);
 for (int y = 0; y < size; y++)
 {
 for (int x = 0; x < size; x++)
 {
 var tile = ownershipTiles[y, x];
 if (tile == null) continue;
Vector3 pos = tile.transform.position;
// Find nearest object (simplified gating)
 float minDist = float.MaxValue;
 int owner = -1;
for (int i = 0; i < trackedObjects.Length; i++)
 {
 float d = Vector3.Distance(pos, trackedObjects[i].position);
 if (d < minDist) { minDist = d; owner = i; }
 }
var mat = tile.GetComponent<Renderer>().material;
 if (minDist < 0.05f && owner >= 0)
 {
 Color c = objectColors[owner % objectColors.Length];
 c.a = Mathf.Clamp01(1f - minDist / 0.05f) * 0.4f;
 mat.color = c;
 }
 else
 {
 mat.color = Color.clear;
 }
 }
 }
 }
}
```
### `GraphVisualizer.cs` β€” Contact edges + probabilities
```csharp
using UnityEngine;
using System.Collections.Generic;
public class GraphVisualizer : MonoBehaviour
{
 public Transform[] objects;
 public float contactThreshold = 0.08f; // meters
 public Color edgeColor = new Color(1f, 0.45f, 0.09f, 0.8f);
 public bool showEdges = true;
private List<LineRenderer> edges = new List<LineRenderer>();
void Update()
 {
 // Clear old edges
 foreach (var lr in edges) if (lr != null) Destroy(lr.gameObject);
 edges.Clear();
if (!showEdges) return;
// Create edges between close objects
 for (int i = 0; i < objects.Length; i++)
 {
 for (int j = i + 1; j < objects.Length; j++)
 {
 float dist = Vector3.Distance(objects[i].position, objects[j].position);
 float contactProb = 1f / (1f + Mathf.Exp((dist - contactThreshold) / 0.02f));
if (contactProb > 0.01f)
 {
 var go = new GameObject($"Edge_{i}_{j}");
 go.transform.SetParent(transform);
 var lr = go.AddComponent<LineRenderer>();
 lr.positionCount = 2;
 lr.SetPositions(new Vector3[] { objects[i].position, objects[j].position });
 lr.startWidth = 0.002f + contactProb * 0.005f;
 lr.endWidth = lr.startWidth;
 lr.material = new Material(Shader.Find("Unlit/Color"));
Color c = edgeColor;
 c.a = contactProb;
 lr.startColor = c;
 lr.endColor = c;
edges.Add(lr);
 }
 }
 }
 }
}
```
### `FieldQueryUI.cs` β€” Click-to-query the field
```csharp
using UnityEngine;
using UnityEngine.UI;
public class FieldQueryUI : MonoBehaviour
{
 public Text positionText;
 public Text sdfText;
 public Text ownerText;
 public Text frictionText;
 public Transform[] objects;
 public string[] objectNames;
 public float[] objectFriction;
// Material data per object (set in inspector)
 [System.Serializable]
 public struct MaterialData {
 public float friction;
 public float density;
 public float stiffness;
 }
 public MaterialData[] materials;
void Update()
 {
 if (Input.GetMouseButtonDown(0))
 {
 Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
 Plane plane = new Plane(Vector3.up, new Vector3(0, 0.455f, 0));
if (plane.Raycast(ray, out float enter))
 {
 Vector3 hitPoint = ray.GetPoint(enter);
 QueryField(hitPoint);
 }
 }
 }
void QueryField(Vector3 point)
 {
 positionText.text = $"({point.x:F3}, {point.y:F3}, {point.z:F3})";
float minDist = float.MaxValue;
 int nearest = -1;
 for (int i = 0; i < objects.Length; i++)
 {
 float d = Vector3.Distance(point, objects[i].position);
 if (d < minDist) { minDist = d; nearest = i; }
 }
sdfText.text = $"{minDist:F4}m";
 ownerText.text = minDist < 0.03f ? objectNames[nearest] : "Background";
 frictionText.text = minDist < 0.03f ? $"{materials[nearest].friction:F2}" : "β€”";
 }
}
```
---
## Step 4: Build for WebGL
1. **File β†’ Build Settings** β†’ Select **WebGL**
2. Click **Switch Platform**
3. **Player Settings:**
 - Set **Compression Format** to **Brotli** (smallest)
 - Disable **Auto Graphics API** and add **WebGL 2.0**
 - Set **Memory Size** to 256MB
 - Under **Publishing Settings**, enable **Decompression Fallback**
4. Click **Build**
Unity produces:
```
Build/
 index.html
 Build/
 Build.data.br (scene data)
 Build.framework.js (runtime)
 Build.loader.js (loader)
 Build.wasm.br (compiled code)
```
### IMPORTANT: MuJoCo WASM Limitation
The MuJoCo Unity plugin uses native C libraries, which **do not compile to WebAssembly**. For WebGL builds, you have two options:
**Option A: Static scene (no live physics)**
- Import the scene in Unity, position everything, bake the visual state
- Remove all MuJoCo components before building
- The WebGL build shows the scene visually with your Ξ¦+G overlays but without live MuJoCo physics
- This is sufficient for the playbook β€” you want to show the representation, not run live sim
**Option B: Pre-recorded simulation**
- Run the MuJoCo sim in the editor, record object trajectories as AnimationClips
- Replace MuJoCo physics with Unity's Animation system for WebGL playback
- Gives you "live" looking simulation without native MuJoCo at runtime
I recommend **Option A** for the playbook. The Three.js version (phi_g_interactive_scene.html, already built) gives you the interactive field query without Unity overhead.
---
## Step 5: Host on HuggingFace Spaces
### For the Unity WebGL build:
```bash
# Create HF Space
huggingface-cli repo create your-username/hybrid-world-model --type space --space_sdk static
# Copy build files
cp -r Build/* hybrid-world-model/
cd hybrid-world-model
git add .
git commit -m "Add Unity WebGL scene"
git push
```
### For the Three.js version (simpler, recommended):
```bash
# Just upload the HTML file as index.html
huggingface-cli repo create your-username/phi-g-scene --type space --space_sdk static
cp phi_g_interactive_scene.html index.html
git add index.html
git commit -m "Interactive Phi+G scene"
git push
```
The Three.js version is a single HTML file with no dependencies β€” it just works.
---
## Step 6: Recommended Approach
For the playbook, use **both**:
1. **Three.js scene** (phi_g_interactive_scene.html) β€” embed directly in the playbook HTML. Lightweight, no build step, works everywhere. Shows the Ξ¦+G representation with clickable objects, field queries, graph edges, and ownership maps.
2. **Unity rendered videos** (from render scripts) β€” pre-rendered MP4s showing MuJoCo physics in action. Embed as `<video>` tags or link to them.
3. **Unity WebGL** (optional) β€” only if you want high-quality rendering with shadows, PBR materials, post-processing. Use Option A (static scene, no live MuJoCo). Worth it for a polished final version but not needed for v1.
The Three.js version already gives you the core demo: "click a point in 3D space, the field returns SDF + material properties + ownership." That's the pitch.