Update README.md

README.md

@@ -1,3 +1,151 @@
 ---
-license: cc-by-nc-4.0
+language:
+- en
+tags:
+- robotics
+- vision-language-model
+- robot-learning
+- embodied-ai
 ---
+
+<!-- Add the final license and pipeline_tag before publication. -->
+
+# Isaac 0.5 by Perceptron
+
+Introducing Isaac 0.5, our open foundation model for robot learning.
+
+Isaac 0.5 brings multimodal video understanding, embodied reasoning, spatial grounding, task-progress estimation, and robot control into one 36-billion-parameter sparse model. It reads images, video, language instructions, robot state, and previous actions, then produces text, normalized coordinates, task-state outputs, or robot actions.
+
+To our knowledge, Isaac 0.5 is the first open model operating at the frontier of multimodal video understanding, embodied reasoning, and robot control.
+
+**[Read the paper](REPORT LINK) · [Download the weights](HUGGING FACE LINK) · [View the code](GITHUB LINK)**
+
+## Extending the frontier of open robot learning
+
+Isaac 0.5 is trained on more than 35 robot systems, 100,000 hours of robot experience, one million hours of general video, and three trillion multimodal tokens. Video understanding, spatial grounding, task progress, future-percept prediction, and robot action are co-trained from the beginning on one shared backbone.
+
+Teams can fine-tune Isaac as a robot policy or use its visual outputs inside a planner, controller, or data engine. The release includes base and action checkpoints, action-training and inference code, LeRobot integration, a reference policy server, evaluation tools, and the manifests needed to reproduce the model's data and checkpoint interfaces.
+
+## What's new in Isaac 0.5
+
+- **A scaling law for video and robot data:** Isaac measures how general video, egocentric video, UMI data, and teleoperation data trade off as training scales.
+- **Semantic world modeling:** The model predicts compact future percepts that capture task-relevant changes such as object motion, contact, grasp state, spatial relationships, and task progress.
+- **Unified perception, reasoning, and control:** One shared sparse backbone supports video understanding, pointing, tracking, task-state estimation, and robot action generation.
+- **Continuous and discrete action interfaces:** Isaac supports continuous control through a dedicated Flow expert and diffusion transformer, plus discrete control through a 2,048-token FAST action vocabulary.
+- **Real-time closed-loop control:** Isaac predicts the next action chunk while the current chunk is still executing, using the latest observation and previously issued commands.
+- **Open training and deployment stack:** The release includes model weights, training code, inference code, LeRobot integration, a reference policy server, evaluation code, and reproduction manifests.
+
+## A scaling law for video and robot experience
+
+Isaac 0.5 establishes a scaling law for choosing the mix of broad video and expensive robot demonstrations.
+
+We trained Isaac across a grid of general-video, egocentric-video, UMI, and teleoperation mixtures, then measured action-prediction loss on robot trajectories held out from training. The resulting contours map data mixtures that reach the same held-out action loss.
+
+At a target action loss of 2.50:
+
+- a model trained on 1,000 hours of general video requires about 5,900 hours of teleoperation;
+- a model trained on one million hours of general video requires about 28 hours of teleoperation.
+
+This is a 210× reduction in required teleoperation. The result gives teams a practical way to decide when another order of magnitude of video is valuable and when collecting more robot experience matters more.
+
+General video contributes broad coverage of objects, activities, environments, and tasks. Egocentric video adds close-up, task-relevant views of hands, motion, and contact. UMI data connects visual knowledge to manipulation. Teleoperation grounds that knowledge in a specific embodiment and action space. Isaac trains on these sources together so each shapes the representation used for control.
+
+> **[FIGURE: Scaling-law contours with teleoperation on the horizontal axis and general video on the vertical axis]**
+
+## Semantic world modeling
+
+Isaac learns from video by predicting future percepts: compact representations of changes that matter for a task. These include which object moved, how objects are related, whether contact occurred, whether a grasp held, and which stage of a task has been reached.
+
+Targets are derived from later frames, including general video without robot action labels. Given observations through time *t*, Isaac predicts the percept at a later time *t + Δ*:
+
+$$
+L_{\text{percept}} = \mathbb{E}\left[\ell\left(g_\theta(o_{\leq t}), z_{t+\Delta}\right)\right]
+$$
+
+Here, $o_{\leq t}$ is the observation history and $z_{t+\Delta}$ is the future percept. This objective trains the same backbone Isaac uses to produce actions.
+
+## Model architecture
+
+Isaac uses a 36B Qwen-family vision-language backbone with sparse experts. Text, vision, time, robot state, and action history enter one shared sequence.
+
+The autoregressive interface produces text, normalized coordinates, task-state outputs, and discrete actions. Discrete control uses a separate vocabulary of 2,048 FAST action tokens. For continuous control, backbone states condition a dedicated Flow expert and 36-block diffusion transformer that generates an action chunk.
+
+Each mixture-of-experts layer gives every token access to 256 learned experts and a null route. A token may use zero to eight routed experts, while a shared expert and residual path remain active. This lets Isaac vary routed compute across visual, language, state, and action tokens. The checkpoint contains 36B parameters in total.
+
+> **[FIGURE: Isaac 0.5 architecture and action interfaces]**
+
+## Training system
+
+mHarmony, our typed input compiler, converts every data source into one stream of text, vision, time, state, and action events while preserving provenance. Isaac uses the same format in training and serving.
+
+Training examples are packed into 16,384-token sequences with 2% padding across the full mixture. Video is decoded outside the training process so corrupted files can restart without interrupting a run. Visual blocks are distributed across workers before encoding to prevent one long video from slowing an entire step. The released hyper-sparse configuration reaches 24% model FLOP utilization.
+
+## Performance benchmarks
+
+We evaluate the same Isaac checkpoints across multimodal video understanding, spatial grounding, held-out action prediction, and closed-loop robot control.
+
+- **Video understanding:** [TK-VIDEO RESULT] on [TK-VIDEO EVALUATION]
+- **Pointing:** [TK-POINTING RESULT] on [TK-POINTING EVALUATION]
+- **Tracking:** [TK-TRACKING RESULT] on [TK-TRACKING EVALUATION]
+- **Closed-loop control:** [TK-HEADLINE-RESULT] across [TK-TASKS] standard manipulation tasks under the released protocol
+
+> **[FIGURE: Video understanding, pointing, tracking, task-progress, and future-percept results]**
+
+> **[FIGURE: Closed-loop task success and progress by robot and task family, with trial counts and confidence intervals]**
+
+## Robot-control evaluation
+
+Our YAM-native simulation and data engine is built around MolmoSpaces. It spans five hardware revisions and four grippers, producing 20 arm-and-gripper configurations with one common observation and action interface. Reach, joint limits, contact geometry, force limits, calibration, and collision envelopes remain specific to each configuration.
+
+The current zero-shot suite contains 42 tasks across nine manipulation families, including object interaction, spatial rearrangement, articulation, insertion, tool use, and multistage tasks. Compound tasks run continuously across subgoals, and grasp outcomes require valid contact with a stable object.
+
+We vary placement, lighting, instruction wording, and distractors while keeping each task and scorer fixed. Alongside task success, we report completion time, path length, jerk, collisions, slips, controller saturation, and safety interventions. Timeouts, invalid actions, dropped observations, and controller faults count as failures.
+
+Simulation provides repeatable coverage. Physical trials measure deployment directly. Where possible, we pair both and hold out complete robot families to measure transfer before hardware-specific adaptation. LIBERO and SIMPLER provide additional standardized comparisons under matched observations, instructions, action horizons, controllers, and serving paths.
+
+> **[MEDIA: Evaluation gallery filtered by task, robot, condition, and outcome, including failures, perturbations, and recoveries]**
+
+## Where Isaac sits among open models
+
+| Model | Robot training scope | Obs. steps | RTC-trained | Prev. actions | Mistake modeling | Non-robot video | Flow expert | Open source |
+| :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- | :---- |
+| **Isaac 0.5** | 35 embodiments | 1–3 | Yes | Yes | Yes | Yes | Yes | Yes |
+| π0.7 | Not reported | ≤6 / camera | Yes | No | Yes | Yes | Yes | No |
+| π0.5 | ~7 robots | 1 | No | No | No | Yes | Yes | Yes |
+| Qwen-VLA | ~10 robots | 1+ | No | No | No | Yes | Yes | No |
+| LingBot-VLA | 9 robots | 1 | No | No | No | No | Yes | Yes |
+| MolmoAct2 | ~5 embodiments | 1 | No | No | No | Yes | Yes | Yes |
+| SmolVLA | 1 embodiment | 1 | No | No | No | No | Yes | Yes |
+| Octo | 25 datasets | 2 | No | No | No | No | No | Yes |
+| OpenVLA | Not reported | 1 | No | No | No | No | No | Yes |
+
+## Building with Isaac in the open
+
+Isaac 0.5 can be fine-tuned on new demonstrations, used as a visual backbone inside a larger embodied system, or deployed as an action policy through LeRobot or our reference server.
+
+The release includes:
+
+- base and action-capable weights;
+- continuous Flow and discrete FAST action configurations;
+- action training and fine-tuning code;
+- text, pointing, tracking, and task-state output schemas;
+- LeRobot integration and a reference policy server;
+- evaluation code, task definitions, and rollout manifests;
+- checkpoint, data, and model-I/O manifests;
+- the technical report, model card, and reproduction guide.
+
+### Usage
+
+> **[TODO: Add final installation command and checkpoint-specific inference example.]**
+
+## Resources
+
+- **Weights:** [Hugging Face](HUGGING FACE LINK)
+- **Code:** [GitHub](GITHUB LINK)
+- **Technical report:** [Read the paper](REPORT LINK)
+- **Results and rollouts:** [View results](RESULTS LINK)
+- **Reproduction guide:** [View guide](REPRODUCTION GUIDE LINK)
+
+Open models are essential to robotics progress. We are releasing Isaac 0.5 with its weights, code, interfaces, benchmarks, and manifests so others can inspect, reproduce, and extend the work.
+
+For help deploying Isaac on your infrastructure, contact [sales@perceptron.inc](mailto:sales@perceptron.inc).