Spaces:

Nirav-Madhani
/

WorldRepresentation

Running

App Files Files Community

WorldRepresentation / render_scripts /UNITY_PIPELINE.md

Nirav Madhani

Import World Representation Space

5af572e 4 months ago

preview code

Raw

History Blame Contribute Delete

13.6 kB

MuJoCo → Unity → WebGL Pipeline

Export your MuJoCo scene to Unity, visualize the Φ+G representation, and deploy as WebGL for the browser.

Overview

MuJoCo MJCF (.xml)
    ↓  Unity MuJoCo Plugin (import)
Unity Scene (3D objects + physics)
    ↓  Custom scripts (add Φ+G visualization)
Unity Scene with field/graph overlays
    ↓  Unity WebGL Build
Browser-ready .html + .wasm + .data
    ↓  Host on HuggingFace Spaces (static)
Interactive 3D playbook

Step 1: Install MuJoCo Unity Plugin

Prerequisites

Unity 2021.3+ (LTS recommended)
Git
Windows/Mac/Linux

Setup

# Clone the MuJoCo repo (or just the unity folder)
git clone https://github.com/google-deepmind/mujoco.git
cd mujoco
git checkout 3.2.4  # match your MuJoCo version

In Unity:

Open Window → Package Manager
Click + → Add package from disk
Navigate to mujoco/unity/package.json and select it
Unity will import the C# scripts

Then copy the MuJoCo native library:

Windows: Copy mujoco.dll from your MuJoCo install to Assets/Plugins/
Mac: Copy libmujoco.dylib
Linux: Copy libmujoco.so

Verify

After import, you should see Assets → Import MuJoCo Scene in the menu bar.

Step 2: Import Your MJCF Scene

The scene XML files are in the render_scripts/ folder. Here's the main tabletop scene:

<!-- Save as tabletop_scene.xml -->
<mujoco model="tabletop_scene">
  <option timestep="0.002" gravity="0 0 -9.81"/>
  <worldbody>
    <light pos="0 0 2" dir="0 0 -1"/>
    <geom name="floor" type="plane" size="2 2 0.1" rgba="0.7 0.7 0.7 1"/>
    <body name="table" pos="0 0 0.4">
      <geom type="box" size="0.4 0.3 0.02" rgba="0.45 0.35 0.25 1"/>
    </body>
    <body name="wood_block" pos="-0.15 0.06 0.455">
      <joint type="free"/>
      <geom type="box" size="0.035 0.035 0.035" mass="0.6" 
            rgba="0.65 0.45 0.25 1" friction="0.4 0.005 0.0001"/>
    </body>
    <body name="rubber_ball" pos="0.05 -0.06 0.455">
      <joint type="free"/>
      <geom type="sphere" size="0.03" mass="1.1"
            rgba="0.18 0.18 0.20 1" friction="0.8 0.005 0.0001"/>
    </body>
    <body name="metal_cylinder" pos="0.18 0.07 0.455">
      <joint type="free"/>
      <geom type="cylinder" size="0.025 0.03" mass="3.0"
            rgba="0.7 0.72 0.75 1" friction="0.2 0.005 0.0001"/>
    </body>
    <body name="plastic_cup" pos="-0.08 -0.1 0.46">
      <joint type="free"/>
      <geom type="cylinder" size="0.022 0.035" mass="0.12"
            rgba="0.2 0.55 0.85 1" friction="0.35 0.005 0.0001"/>
    </body>
    <body name="ceramic_plate" pos="0.1 0.0 0.44">
      <joint type="free"/>
      <geom type="cylinder" size="0.05 0.008" mass="0.35"
            rgba="0.92 0.90 0.85 1" friction="0.25 0.005 0.0001"/>
    </body>
  </worldbody>
</mujoco>

In Unity:

Assets → Import MuJoCo Scene → select tabletop_scene.xml
The plugin creates GameObjects for each body/geom with MuJoCo components
Each geom gets a MeshRenderer with procedural mesh
The scene runs MuJoCo physics in Play mode

Step 3: Add Φ+G Visualization Overlays

Create these C# scripts to visualize the field and graph:

`FieldVisualizer.cs` — Ownership heatmap + SDF wireframes

using UnityEngine;
using System.Collections.Generic;

public class FieldVisualizer : MonoBehaviour
{
    [Header("Objects")]
    public Transform[] trackedObjects;  // Drag MuJoCo bodies here
    public Color[] objectColors;        // One color per object
    
    [Header("Ownership Grid")]
    public bool showOwnership = false;
    public float gridResolution = 0.01f;
    public float gridExtent = 0.3f;
    public float gridHeight = 0.455f;
    
    [Header("SDF Wireframe")]
    public bool showSDF = true;
    
    private GameObject[,] ownershipTiles;
    private List<LineRenderer> edgeRenderers = new List<LineRenderer>();
    
    void Start()
    {
        if (objectColors.Length == 0)
            objectColors = new Color[] {
                new Color(1f, 0.45f, 0.09f),  // orange
                new Color(0.23f, 0.51f, 0.96f), // blue
                new Color(0.13f, 0.77f, 0.37f), // green
                new Color(0.55f, 0.36f, 0.96f), // purple
                new Color(0.92f, 0.70f, 0.03f), // yellow
            };
        
        CreateOwnershipGrid();
    }
    
    void CreateOwnershipGrid()
    {
        int size = Mathf.CeilToInt(gridExtent * 2 / gridResolution);
        ownershipTiles = new GameObject[size, size];
        
        for (int y = 0; y < size; y++)
        {
            for (int x = 0; x < size; x++)
            {
                float wx = -gridExtent + x * gridResolution;
                float wy = -gridExtent + y * gridResolution;
                
                var tile = GameObject.CreatePrimitive(PrimitiveType.Quad);
                tile.transform.SetParent(transform);
                tile.transform.position = new Vector3(wx, wy, gridHeight + 0.001f);
                tile.transform.localScale = Vector3.one * gridResolution;
                tile.transform.rotation = Quaternion.identity;
                
                var mat = new Material(Shader.Find("Unlit/Color"));
                mat.color = Color.clear;
                tile.GetComponent<Renderer>().material = mat;
                tile.SetActive(false);
                
                ownershipTiles[y, x] = tile;
            }
        }
    }
    
    void Update()
    {
        if (showOwnership)
            UpdateOwnership();
        
        // Toggle visibility
        foreach (var tile in ownershipTiles)
            if (tile != null) tile.SetActive(showOwnership);
    }
    
    void UpdateOwnership()
    {
        int size = ownershipTiles.GetLength(0);
        for (int y = 0; y < size; y++)
        {
            for (int x = 0; x < size; x++)
            {
                var tile = ownershipTiles[y, x];
                if (tile == null) continue;
                
                Vector3 pos = tile.transform.position;
                
                // Find nearest object (simplified gating)
                float minDist = float.MaxValue;
                int owner = -1;
                
                for (int i = 0; i < trackedObjects.Length; i++)
                {
                    float d = Vector3.Distance(pos, trackedObjects[i].position);
                    if (d < minDist) { minDist = d; owner = i; }
                }
                
                var mat = tile.GetComponent<Renderer>().material;
                if (minDist < 0.05f && owner >= 0)
                {
                    Color c = objectColors[owner % objectColors.Length];
                    c.a = Mathf.Clamp01(1f - minDist / 0.05f) * 0.4f;
                    mat.color = c;
                }
                else
                {
                    mat.color = Color.clear;
                }
            }
        }
    }
}

`GraphVisualizer.cs` — Contact edges + probabilities

using UnityEngine;
using System.Collections.Generic;

public class GraphVisualizer : MonoBehaviour
{
    public Transform[] objects;
    public float contactThreshold = 0.08f; // meters
    public Color edgeColor = new Color(1f, 0.45f, 0.09f, 0.8f);
    public bool showEdges = true;
    
    private List<LineRenderer> edges = new List<LineRenderer>();
    
    void Update()
    {
        // Clear old edges
        foreach (var lr in edges) if (lr != null) Destroy(lr.gameObject);
        edges.Clear();
        
        if (!showEdges) return;
        
        // Create edges between close objects
        for (int i = 0; i < objects.Length; i++)
        {
            for (int j = i + 1; j < objects.Length; j++)
            {
                float dist = Vector3.Distance(objects[i].position, objects[j].position);
                float contactProb = 1f / (1f + Mathf.Exp((dist - contactThreshold) / 0.02f));
                
                if (contactProb > 0.01f)
                {
                    var go = new GameObject($"Edge_{i}_{j}");
                    go.transform.SetParent(transform);
                    var lr = go.AddComponent<LineRenderer>();
                    lr.positionCount = 2;
                    lr.SetPositions(new Vector3[] { objects[i].position, objects[j].position });
                    lr.startWidth = 0.002f + contactProb * 0.005f;
                    lr.endWidth = lr.startWidth;
                    lr.material = new Material(Shader.Find("Unlit/Color"));
                    
                    Color c = edgeColor;
                    c.a = contactProb;
                    lr.startColor = c;
                    lr.endColor = c;
                    
                    edges.Add(lr);
                }
            }
        }
    }
}

`FieldQueryUI.cs` — Click-to-query the field

using UnityEngine;
using UnityEngine.UI;

public class FieldQueryUI : MonoBehaviour
{
    public Text positionText;
    public Text sdfText;
    public Text ownerText;
    public Text frictionText;
    public Transform[] objects;
    public string[] objectNames;
    public float[] objectFriction;
    
    // Material data per object (set in inspector)
    [System.Serializable]
    public struct MaterialData {
        public float friction;
        public float density;
        public float stiffness;
    }
    public MaterialData[] materials;
    
    void Update()
    {
        if (Input.GetMouseButtonDown(0))
        {
            Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
            Plane plane = new Plane(Vector3.up, new Vector3(0, 0.455f, 0));
            
            if (plane.Raycast(ray, out float enter))
            {
                Vector3 hitPoint = ray.GetPoint(enter);
                QueryField(hitPoint);
            }
        }
    }
    
    void QueryField(Vector3 point)
    {
        positionText.text = $"({point.x:F3}, {point.y:F3}, {point.z:F3})";
        
        float minDist = float.MaxValue;
        int nearest = -1;
        for (int i = 0; i < objects.Length; i++)
        {
            float d = Vector3.Distance(point, objects[i].position);
            if (d < minDist) { minDist = d; nearest = i; }
        }
        
        sdfText.text = $"{minDist:F4}m";
        ownerText.text = minDist < 0.03f ? objectNames[nearest] : "Background";
        frictionText.text = minDist < 0.03f ? $"{materials[nearest].friction:F2}" : "—";
    }
}

Step 4: Build for WebGL

File → Build Settings → Select WebGL
Click Switch Platform
Player Settings:
- Set Compression Format to Brotli (smallest)
- Disable Auto Graphics API and add WebGL 2.0
- Set Memory Size to 256MB
- Under Publishing Settings, enable Decompression Fallback
Click Build

Unity produces:

Build/
  index.html
  Build/
    Build.data.br       (scene data)
    Build.framework.js  (runtime)
    Build.loader.js     (loader)
    Build.wasm.br       (compiled code)

IMPORTANT: MuJoCo WASM Limitation

The MuJoCo Unity plugin uses native C libraries, which do not compile to WebAssembly. For WebGL builds, you have two options:

Option A: Static scene (no live physics)

Import the scene in Unity, position everything, bake the visual state
Remove all MuJoCo components before building
The WebGL build shows the scene visually with your Φ+G overlays but without live MuJoCo physics
This is sufficient for the playbook — you want to show the representation, not run live sim

Option B: Pre-recorded simulation

Run the MuJoCo sim in the editor, record object trajectories as AnimationClips
Replace MuJoCo physics with Unity's Animation system for WebGL playback
Gives you "live" looking simulation without native MuJoCo at runtime

I recommend Option A for the playbook. The Three.js version (phi_g_interactive_scene.html, already built) gives you the interactive field query without Unity overhead.

Step 5: Host on HuggingFace Spaces

For the Unity WebGL build:

# Create HF Space
huggingface-cli repo create your-username/hybrid-world-model --type space --space_sdk static

# Copy build files
cp -r Build/* hybrid-world-model/
cd hybrid-world-model
git add .
git commit -m "Add Unity WebGL scene"
git push

For the Three.js version (simpler, recommended):

# Just upload the HTML file as index.html
huggingface-cli repo create your-username/phi-g-scene --type space --space_sdk static
cp phi_g_interactive_scene.html index.html
git add index.html
git commit -m "Interactive Phi+G scene"
git push

The Three.js version is a single HTML file with no dependencies — it just works.

Step 6: Recommended Approach

For the playbook, use both:

Three.js scene (phi_g_interactive_scene.html) — embed directly in the playbook HTML. Lightweight, no build step, works everywhere. Shows the Φ+G representation with clickable objects, field queries, graph edges, and ownership maps.
Unity rendered videos (from render scripts) — pre-rendered MP4s showing MuJoCo physics in action. Embed as <video> tags or link to them.
Unity WebGL (optional) — only if you want high-quality rendering with shadows, PBR materials, post-processing. Use Option A (static scene, no live MuJoCo). Worth it for a polished final version but not needed for v1.

The Three.js version already gives you the core demo: "click a point in 3D space, the field returns SDF + material properties + ownership." That's the pitch.

MuJoCo → Unity → WebGL Pipeline

Overview

Step 1: Install MuJoCo Unity Plugin

Prerequisites

Setup

Verify

Step 2: Import Your MJCF Scene

In Unity:

Step 3: Add Φ+G Visualization Overlays

FieldVisualizer.cs — Ownership heatmap + SDF wireframes

GraphVisualizer.cs — Contact edges + probabilities

FieldQueryUI.cs — Click-to-query the field

Step 4: Build for WebGL

IMPORTANT: MuJoCo WASM Limitation

Step 5: Host on HuggingFace Spaces

For the Unity WebGL build:

For the Three.js version (simpler, recommended):

Step 6: Recommended Approach

`FieldVisualizer.cs` — Ownership heatmap + SDF wireframes

`GraphVisualizer.cs` — Contact edges + probabilities

`FieldQueryUI.cs` — Click-to-query the field