Do World Action Models Generalize Better than VLAs? A Robustness Study
Paper ā¢ 2603.22078 ā¢ Published ā¢ 8
Sajjad Pakdamansavoji
savoji
AI & ML interests
VLM, VLA, LLM, Robotics, Computer Vision
Recent Activity
posted an update about 13 hours ago
GPT-6.1 Terra scores 6 points higher than GPT-5.6 Terra on the Artificial Analysis Intelligence Index, at $0.40 per task instead of $1.40.
Per-token prices fell to $1/$5 per million input/output tokens, from $2/$12. For agent and computer-use workloads, cost stops being the blocker. The question becomes whether the model is reliable enough to act on its own.
The biggest reliability change: on AA-Omniscience it guesses 38% of the time when it does not know the answer, down from 88%, and accuracy rose 2 points.
š The numbers (Artificial Analysis)
ā¢ Low and medium effort are Pareto-optimal for cost; at lower effort the index gain is 7 to 13 points
ā¢ Terminal-Bench 4.0: about +20 points, rivaling GPT-6.1 Sol at max effort
ā¢ Coding Agent Index v2.0: 48, vs 54 for GPT-6.1 Sol and 31 for GPT-6 Luna
ā¢ Cached input $0.05 vs $0.20, cache write $1.25 per million tokens
ā ļø Caveats
ā¢ OpenAI says it matches GPT-6 Astra on Agents' Last Exam at one tenth the price per token and finishes tasks over 3x faster, but gives no absolute scores and there is no independent check yet
ā¢ Most other evals gain little, for example 5 points on GDP.pdf
ā¢ 38% guessing is still high for an agent that takes actions
ā¢ $0.40 per task is measured on AA's index, and your token usage will differ
š ļø Try it
ā¢ API model name: gpt-6.1-terra
ā¢ Rerun your own agent evals at low and medium effort and compare task success and error rates, not price
https://openai.com/index/introducing-gpt-6-1-terra/
https://x.com/ArtificialAnlys/status/2114781865811063259
If you run computer-use agents on GPT-5.6 Terra, what error rate would you need to see before switching?
ā¢ Medium: https://medium.com/p/2178290c2b22
ā¢ Substack: https://sajjadpakdamansavoji.substack.com/p/gpt-61-terra-settles-the-cost-question
ā¢ X: https://x.com/saavoji/status/2115105385388683774
ā¢ Website: https://sajjadpsavoji.github.io/blog#openais-gpt-6 posted an update 1 day ago
NVIDIA's GEAR lab says its open Flora models give Astra-level robotics data labeling at 125x lower cost. The models already label 200K hours of video a day on 500 GPUs.
Flora is not a robot policy. It is a suite of open perception models that watch raw human and robot video. They label what happened (timestamped instructions) and how well it went (rewards and failure reasons). Yunze Man says the team uses Flora to train NVIDIA GR00T, so this labeling layer helps build the policy.
The weights are public, so the data engine is no longer a private advantage. Anyone with footage can build the same curation, reward and evaluation loop on it.
š What Flora outputs
ā¢ Timestamped captions to curate and filter human egocentric and robot data
ā¢ Dense progress rewards for quality control and RL post-training
ā¢ Rubric scores for real-world benchmarking
ā¢ Failure reasons, so failed attempts become usable training data
š The numbers
ā¢ 125x lower cost at "Astra-level performance" (claimed)
ā¢ 200K hours of video labeled per day on 500 GPUs, in production at NVIDIA
ā¢ Works on human footage, which is cheaper to collect than robot data
ā ļø Caveats
ā¢ No benchmark, metric or cost breakdown supports the 125x figure, and Astra is never described
ā¢ The announcement gives no GPU type, model sizes, license or input specs
ā¢ Learned rewards could add bias or reward hacking to RL, and no accuracy numbers are published
ā¢ Every claim comes from the team that built it, so test it on your own data first
https://x.com/DrJimFan/status/2112597587987689912
If you train VLAs on egocentric video, which signal would you plug in first: captions for filtering, or progress rewards for RL?
ā¢ Medium: https://medium.com/p/2fd29045dffb
ā¢ Substack: https://sajjadpakdamansavoji.substack.com/p/nvidias-flora-puts-the-robot-data
ā¢ X: https://x.com/saavoji/status/2114758981474509137
ā¢ Website: https://sajjadpsavoji.github.io/blog#nvidia-just-opened posted an update 3 days ago
Openly available models now rank near the frontier on Artificial Analysis (AA). This week's most discussed agent work treats trust as something to engineer and verify, not assume.
- Kimi K3.1 Pro: multimodal MoE, 2.8T total / 104B active params, 1M-token context. It is 4th on the AA Intelligence Index (55, 11 points over K3) and scores 64% on the AA Cyber Index, #1 among openly available models. Its weights had not been published when it was scored.
- DeepSeek-V4.1-Flash-1017: an MIT-licensed minor update that cuts output length and task execution time. AA had named it #1 openly available on the Cyber Index before K3.1 Pro arrived. 85,275 downloads last month. https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
- OpenAI removed live internet access for all frontier models during training and gave them an offline, cached copy of the web instead.
- NVIDIA OpenShell (Apache-2.0, Rust, 0.1.x): one sandbox per agent, with kernel-enforced policy on files, system calls and network. Agents never see real credentials. Formal verification holds risky policy changes for human review. Anthropic argues LLMs make this kind of verification feasible at much larger scale.
- LGWM (173M params at inference) predicts the next screen's representation and compares it with the one observed. It reaches 0.987 AUC on RSWT-Bench at 17 ms per decision on one A100, against 0.995 for Gemini 3.7 Flash and 0.988 for GPT-5.6 Luna. The benchmark is the authors' own. https://huggingface.co/papers/2610.11942
Shortlist 1017 for agent work, or K3.1 Pro once its weights ship. Treat containment as a separate decision: sandbox the agent, keep real credentials out of its reach, and verify each transition, not just each screen.
ā¢ Medium: https://medium.com/p/4ce5f13afba9
ā¢ Substack: https://sajjadpakdamansavoji.substack.com/p/kimi-k31-pro-and-deepseek-v41-flash
ā¢ X: https://x.com/saavoji/status/2114480628121100394
ā¢ Website: https://sajjadpsavoji.github.io/blog#kimi-k3-1-proOrganizations
None yet
upvoted 2
papers about 2 months ago