simready-validator / tools /validation /playbooks /foundations-deviations.md
loginowskid's picture
Sync from simready-oem-library-pm@c858e9dd
cd53438 verified

SimReady Foundations β€” process deviations

Issues encountered while running the /simready-report skill against yaskawa_local that require changes in simready_foundations (or a clearly documented client-side workaround). These are not playbook bugs β€” the playbook is working around upstream gaps.

Source data for each item: paths/codes observed during the validation run on packages/yaskawa_local/ against Robot-Body-Isaac v1.0.0.


1. Profile.capabilities returns an empty list for every profile

Where: omni.asset_validator profile schema (Python registry).

Observed:

profile = oav.ProfileRegistry().find('Robot-Body-Isaac', '1.0.0')
profile.capabilities  # β†’ []
profile.features      # β†’ 2 features, 8 requirements

Every profile in the registry returns [] for .capabilities. Profiles now declare scope through .features β€” each feature owns its requirements directly. The capabilities attribute is a leftover from an older schema.

Impact: validate_yaskawa.py and earlier versions of this playbook's validate.py filtered the engine's enabled capabilities against profile.capabilities. Because the latter is always empty, the filter disabled every capability and reported enabled capabilities: 0 while producing 6,551 issues from rules that ignored the disable calls (see #2).

Required: remove Profile.capabilities, or populate it from the union of the features' capabilities, and update all reference scripts/docs that still use it.


2. engine.enable_* / disable_* do not constrain rule execution

Where: omni.asset_validator.ValidationEngine.

Observed: engine.enable_feature(f) and engine.enable_requirement(r) update engine.enabled_features / engine.enabled_requirements counters, but engine.validate(stage) still runs every rule registered via @register_rule. engine.enabled_rules stays at 0 even after enabling features and requirements; rules execute regardless.

Impact: profile-scoping has to happen post-hoc, by filtering issues against the profile's requirement codes. The current playbook does this in validate_one() (filters failures by code in profile_codes).

Required: either

  1. make enable_* / disable_* actually scope rule execution, or
  2. document explicitly that they are display-only and that callers must filter results themselves. Today the API name strongly implies (1).

3. Robot-Body-Isaac v1.0.0 silently drops 4 of 6 declared features

(TOML ↔ Python registry mismatch)

Where: simready_foundations/_build/target-deps/pip_prebundle/simready/foundation/core/profiles/profiles.toml vs the Features enum loaded from simready.foundation.core.features.

Observed: profiles.toml declares Robot-Body-Isaac v1.0.0 with six features:

TOML entry Loads? Notes
FET001_BASE_NEUTRAL v0.1.0 βœ“ 8 requirements
FET004_ROBOT_PHYSX v0.2.0 βœ— No such name in Features enum (only FET004_BASE_NEUTRAL)
FET021_ROBOT_CORE_ISAAC v0.2.0 βœ— Closest is FET_021_ROBOT_CORE v0.2.0 (note underscore + missing _ISAAC)
FET022_DRIVEN_JOINTS_ISAAC v0.1.0 βœ— Closest is FET_022_DRIVEN_JOINTS v0.1.0
FET024_BASE_ARTICULATION_PHYSX v0.1.0 βœ— Closest is FET_024_BASE_ARTICULATION v0.1.0
FET100_BASE_ISAACSIM v0.1.0 βœ“ 0 requirements

pr.find('Robot-Body-Isaac', '1.0.0').features therefore returns only the two features that successfully resolve. Four features (Robot Core, Driven Joints, Base Articulation, Robot Physx) are silently dropped β€” no warning, no error.

The rules associated with the dropped features (RC., JT., RB., AT.) still execute (per #2 the engine ignores feature gating), so they appear in the raw issue list, but they cannot be attributed to a profile feature for roll-up reporting.

Required: reconcile the names. Either rename the TOML entries to match the Features enum (FET_021_ROBOT_CORE etc.), or add the _ISAAC / _PHYSX variants to the foundation as real feature implementations. Also make profile-feature lookup fail loudly when a declared feature can't be resolved β€” silent drop is the worst outcome.

Validation also surfaced real failures with codes from features that aren't even declared in the TOML profile:

Code(s) Source Notes
RC.002, RC.004 (thumbnail-exist), RC.008, RC.009 capabilities/isaac_sim/robot_core Robot-core rules registered & raising issues, but no profile feature includes them
NP.001, NP.005, NP.006, NP.008 capabilities/visualization/nonvisual_materials
HI.001, HI.002 capabilities/hierarchy (HI.004 is in profile, others aren't)
VG.007, VG.009, VG.016, VG.019, VG.MESH.001 capabilities/visualization/geometry (VG.001 is in profile, the rest aren't)
EX.01, EX.03, GSP.001, ISA.001, VM.MDL.001 various

A "Robot-Body-Isaac" profile that does not require the asset to have a thumbnail, robot type, or pinned root joint, but the foundations repo ships those as registered rules, is a process gap.

Required: extend the profile's features to include the robot-core, nonvisual-materials, hierarchy-completeness, and geometry-completeness features that are already authored in the foundation; align the JSON capability configs (config/robot-core.json etc.) with the Python Profiles enum.


4. JSON capability config and Python profile registry are not linked

Where: simready_foundations/nv_core/sr_specs/config/robot-core.json vs simready.foundation.core Python plugin.

Observed: robot-core.json declares RobotCore capability v1.0.0 with the thumbnail-exist requirement (and others). The Python registry exposes that requirement to the rule machinery (issues with RC.004 are emitted), but no Profile exposes RobotCore as a feature, so client code that introspects profile.features to compute pass/fail per feature can't see it.

Impact: validation reports list RC.* codes in raw issues but cannot map them to a feature, so they appear as "uncategorized" failures.

Required: wire the JSON capability declarations into the Features / Profiles enums (or into a new profile.capabilities accessor that returns real entries β€” see #1).


5. ~87% of issues report code: UNKNOWN

Where: rules registered in omni.asset_validator and SimReady plugins.

Observed: of 6,551 issues across 7 robots, 5,695 have iss.requirement is None, so the issue cannot be attributed to any requirement code:

  5695  UNKNOWN
   129  NP.001
   116  HI.002
   116  VG.009
    96  VG.007
    ...

These come from rules that don't decorate with @register_requirements, so they run but have no requirement linkage.

Impact: profile-scoped filtering (#2's workaround) drops 87% of the signal; those failures appear in the raw issue list but never roll up to a feature pass/fail.

Required: every rule registered in foundations should declare the requirement(s) it enforces via @register_requirements, so issues are attributable.


6. init_rules defaults are inconsistent across consumers

Where: omni.asset_validator.ValidationEngine constructor and the SimReady SDK's engine wrapper.

Observed: validate_yaskawa.py (top-level reference) explicitly passes init_rules=True and notes that simready_sdk.core.engine passes init_rules=False, which leaves the engine empty:

The SDK passes init_rules=False which leaves the engine with no checks,
producing 'No rules or requirements have been enabled.' for every asset.

With init_rules=True the engine has 124 rules / 118 requirements. With False it has 0 / 0.

Required: make init_rules=True the default in the engine, or remove the parameter; update simready_sdk.core.engine to call it correctly. Today choosing the wrong path silently produces a "passes everything" report.


7. Foundation has the wrapper but ships no runnable thumbnail generator

Where: simready_foundations/nv_core/testing_tools/testing_framework/source/job_runner/core/engines/usdrecord/ plus lighting rigs at simready_foundations/nv_core/common_tools/thumbnails/auto_thumbnail_indoor_lighting_rig_{small,medium,large}.usd.

Observed: the foundation defines UsdRecordCommandBuilder which builds the command <usdrecord_exe> <asset.usd> <output.png> and references three preset lighting USDs intended to be composed with the asset before rendering. But:

  • The <usdrecord_exe> is taken from job.runner_config.executable. The foundation does not bundle the binary, the venv does not have it, and pxr.UsdAppUtils (the Python library backing usdrecord) is absent.
  • The auto_thumbnail_indoor_lighting_rig_*.usd rigs have no documented composition recipe β€” there is no published "open the rig, payload the asset, render to 256Γ—256" wrapper script.

usdrecord is part of OpenUSD; on this machine it must be installed externally. Three working install paths (commands documented for the record; none are run automatically by the validator):

  1. Kit Kernel from NGC (the user already has ngc on PATH):
    $env:KIT_VERSION = "106.5.0"
    ngc registry resource download-version "nvidia/omniverse/kit:$env:KIT_VERSION" `
      --dest "$env:USERPROFILE\.kit"
    # `usdrecord.bat` lives at <kit_root>\dev\tools\packman\repo\bld\target-deps\usd\release\bin\usdrecord.bat
    $env:USDRECORD_BIN = "$env:USERPROFILE\.kit\kit-$env:KIT_VERSION\...\usdrecord.bat"
    
  2. OpenUSD prebuilt (NVIDIA developer release):
    # Download from https://developer.nvidia.com/usd (signed-in NGC link),
    # extract, and point to the binary:
    $env:USDRECORD_BIN = "C:\openusd\bin\usdrecord.cmd"
    
  3. Build OpenUSD from source (heaviest, only if the prebuilt isn't acceptable):
    git clone https://github.com/PixarAnimationStudios/OpenUSD.git C:\src\OpenUSD
    python C:\src\OpenUSD\build_scripts\build_usd.py --tools C:\openusd
    $env:USDRECORD_BIN = "C:\openusd\bin\usdrecord.cmd"
    

Once USDRECORD_BIN is set (or the binary is on PATH), validate.py's _find_usdrecord() will pick it up and the validator will invoke it for each asset missing a thumbnail at the spec path. Generated PNGs go to report/<set>/_generated_thumbs/<filename>.png β€” never into the packaged asset tree. Every command run is logged in results.json under thumbnail_generation.attempts and surfaced in the dashboard's "External tooling used" panel.

For yaskawa_local, six of seven thumbnails were instead pulled from Assets/Isaac/6.0/Isaac/Robots/Yaskawa/Motoman Next/.../.thumbs/256x256/ on the Omniverse content S3 bucket (free; no install required). One robot (NHC10DE-A00) has no S3 thumbnail; without usdrecord installed, its tile renders the inline SVG "no thumbnail" placeholder.

Required: either ship usdrecord in the foundation's bundled toolchain, or distribute a foundation-side wrapper (e.g. a Kit ext that does UsdAppUtils.framerec.FrameRecorder programmatically) so simready ingest usd can produce the spec thumbnail without external installs.


8. Spec docs disagree on the thumbnail filename

Where:

  • Spec doc: nv_core/sr_specs/docs/capabilities/isaac_sim/robot_core/requirements/thumbnail-exist.md
  • Validation rule: nv_core/sr_specs/docs/capabilities/isaac_sim/robot_core/validation.py:213
  • OEM publishing guide: simready-oem-sdk-poc/docs/oem-publishing-guide-hf.md

Observed:

  • The spec doc's example uses …/Robot_Name/.thumbs/256x256/Robot.png (filename = USD stem only, no .usd suffix).
  • The rule code computes …/<filename>.png where <filename> is the full USD filename including the .usd extension, e.g. nex4_c00.usd.png.
  • The OEM publishing guide also uses robot-a.usd.png (with .usd.).
  • The Isaac S3 bucket uses NEX4_C00.usd.png (with .usd.).

The rule and most published assets agree (<filename>.png with .usd); the spec doc's example is wrong.

Required: fix the spec example in thumbnail-exist.md to match the rule.


9. simready ingest usd does not lift companion thumbnails

Where: simready-oem-sdk-poc/src/simready_sdk/packager/organizer.py.

Observed: simready ingest usd ./robot/robot.usd produces the spec layout (<name>/<name>.usd, configuration/, materials/, .thumbs/256x256/ folder) but does not copy/rename a thumbnail from any companion location into .thumbs/256x256/<file>.usd.png. The packager docs (SKILL.md) call it "thumbnail placeholder" β€” the directory is created empty.

For yaskawa_local/images/<stem>.usd.usd.png, no automated path exists for the packager to know that file should land at the spec's thumbnail path.

Required: the packager should accept a companion-images directory hint (e.g. --thumbs-from <dir> or a sibling images/ convention) and lift matching files into .thumbs/256x256/<file>.usd.png during organize.


10. A foundation rule crashes with 'NoneType' object has no attribute 'JointStateAPI'

Where: somewhere in the simready.foundation.core plugin's joint-state checker (rule not yet pinned down β€” issue is reported as code: UNKNOWN, rule: None).

Observed: the rule fires on every prim it visits and produces an Uncaught error issue, accounting for between 566 and 698 issues per asset in our run β€” i.e., most of what makes the report look noisy:

UNKNOWN  Γ—668  Uncaught error: 'NoneType' object has no attribute 'JointStateAPI'
UNKNOWN  Γ—566  Uncaught error: 'NoneType' object has no attribute 'JointStateAPI'
UNKNOWN  Γ—698  Uncaught error: 'NoneType' object has no attribute 'JointStateAPI'

The remaining ~150 issues per asset (HI., NP., RC.*, etc.) are the real findings.

The crash strongly suggests the rule reads stage.GetPrimAtPath(...).GetAPISchemas()-style and assumes the result is non-None. A defensive if foo is None: continue would silence the noise; the underlying joint-state coverage is presumably what the rule was meant to be checking, but currently it produces nothing useful.

Required: pin down the rule (grep JointStateAPI across simready.foundation.core), add the missing None-guard, and either register the rule's requirement properly or remove it from the engine until it works.


11. NP.005 β€” layout mismatch between the rule's spec and simready ingest usd, plus a path-printing bug

Where:

  • Rule code: simready_foundations/nv_core/sr_specs/docs/capabilities/core/naming_paths/validation.py:271-320 (AssetFolderStructureChecker).
  • Rule spec doc: simready_foundations/nv_core/sr_specs/docs/capabilities/core/naming_paths/requirements/asset-folder-structure.md.
  • Packager output: simready-oem-sdk-poc/src/simready_sdk/packager/organizer.py (invoked via simready ingest usd).

11a. Layout mismatch

The NP.005 spec doc requires this structure:

<asset>/                  ← asset folder
  β”œβ”€β”€ <intermediate>/     ← exactly ONE intermediate folder (any name)
  β”‚   └── <main>.usd      ← main USD; NO other .usd files here or below
  └── materials/          ← supporting folders are SIBLINGS of <intermediate>/

The reference dashboard's FANUC content follows this exactly:

robots/CR/cr_50f_16b/isaac_ready_usd/cr_50f_16b.usd.usd
robots/CR/cr_50f_16b/configuration/...

But simready ingest usd produces a flat layout β€” main USD at the asset root, sublayers nested inside the same dir:

<asset>/
  β”œβ”€β”€ <asset>.usd
  └── configuration/
      β”œβ”€β”€ <asset>_base.usd
      β”œβ”€β”€ <asset>_physics.usd
      └── <asset>_sensor.usd

The rule walks os.path.dirname(stage_path) (which is <asset>/ in the flat layout) and flags every sublayer as "stray". The sublayers aren't stray β€” they're the spec-required configuration files β€” but the rule's walk-set includes them because the packager produced an NP.005-non-compliant layout.

So: two foundation artifacts disagree on the canonical asset layout. The packaging spec (simready-packaging's SKILL.md) and simready ingest usd produce one layout; NP.005's spec doc and the FANUC reference content use a different layout.

11b. Path-printing bug (independent)

# capabilities/core/naming_paths/validation.py:308-313
for root, dirs, files in os.walk(current_dir):
    for file in files:
        found_file = os.path.join(root, file).replace("\\", "/")
        if file.endswith((".usd", ".usda")) and found_file != stage_path:
            relative_path = os.path.relpath(file, current_dir)   # ← bug
            other_usd_files.append(relative_path)

file here is just the basename returned by os.walk (e.g. nex10_c00_base.usd). os.path.relpath resolves it against os.getcwd(), not against root. With CWD = <workspace>/.claude/skills/simready-report/, the rule prints:

..\..\..\.claude\skills\simready-report\nex10_c00_base.usd

β€” a path back to the validator's CWD, where the file emphatically does not exist. Should be os.path.relpath(found_file, current_dir) (found_file is the correct joined path, already computed on the line above and otherwise unused). With that one-character fix the cited path becomes configuration/nex10_c00_base.usd β€” pointing at the real file.

Required

Pick one canonical layout and reconcile both sides. Either:

  • Update the packager (simready ingest usd) to emit the NP.005-compliant <asset>/<intermediate>/<main>.usd structure with configuration/ and materials/ as siblings of <intermediate>/. This is what the FANUC reference content already uses. The packaging skill doc and any consumers (manifest tooling, dashboard, etc.) need updating in lockstep.

  • Or update NP.005's spec doc and rule to bless the flat layout that simready ingest usd actually produces β€” i.e., explicitly whitelist the spec-mandated subdirectories (configuration/, materials/, .thumbs/, .simready/) when scanning for stray USDs.

Whichever is chosen, fix the path-printing bug at the same time (relpath(file, …) β†’ relpath(found_file, …)) so the message names a real on-disk path.


12. The "example" capability is active in production validation

Where: simready_foundations/nv_core/sr_specs/docs/capabilities/example/example.py, codes EX.01, EX.02, EX.03.

Observed: every asset fails:

  • EX.01 β€” "Stage has no mesh prims."
  • EX.03 β€” "Test prim 'TestBlob' not found under default prim."

The source comment on this capability is literally "example" and the rule docs (example.md) describe it as a tutorial β€” EX.03 requires a prim named TestBlob to exist as a child of the default prim. That's clearly not an asset requirement; it's pedagogical scaffolding to teach plugin authors how to register a rule.

Required: unregister the Example capability from production rule loading (move it to test fixtures, gate it behind an env var, or delete it from the shipped plugin). Today every real asset takes 2–3 spurious EX failures.


13. EX.01 and VG.MESH.001 are duplicate checks under different codes

Where:

  • capabilities/example/example.py:52 β€” EX.01: "Stage has no mesh prims."
  • capabilities/visualization/geometry/validation.py:836 β€” VG.MESH.001: "Stage does not contain any meshes."

Observed: both fire on every asset, with effectively identical semantics. EX.01 is the example/tutorial version; VG.MESH.001 is the real version. Reporting both is noise.

Required: when #12 is fixed, this disappears for free. Otherwise, EX.01 should be retired and any consumer that still references it should be redirected to VG.MESH.001.


14. HI.002 rejects valid USD xform-op decompositions

Where: capabilities/hierarchy/validation.py:62-104 (ExclusiveXFormParentChecker).

Observed: ~14 issues per asset, message "Prim Parent has no xformOp:rotate." The rule requires every Gprim parent to author both xformOp:translate and (a substring-matching) xformOp:rotate*:

if not any("xformOp:rotate" in op.GetAttr().GetName() for op in xform_ops):
    self._AddFailedCheck("Prim Parent has no xformOp:rotate.", …)

That's an over-strict reading of USD. Valid alternatives the rule rejects:

  • xformOp:transform (single 4Γ—4 matrix β€” encodes translate + rotate + scale together).
  • No xform ops at all (identity transform β€” also a valid stateless prim).
  • Any combination authored on an inherited parent and overridden by the Gprim's own ops.

Yaskawa's robots author matrix transforms in many places, which is what trips this. The check should accept any xformable spec that resolves to a defined transform β€” likely via Xformable.GetLocalTransformation() returning a non-error result β€” rather than pattern-matching specific op names.

Required: rewrite the check to validate that the prim resolves to a defined transform, not that it authored a particular op vocabulary.


15. NP.001 flags the conventional /Looks scope as a naming violation

Where: capabilities/core/naming_paths/validation.py:125-160 (PrimNamingConventionChecker).

Observed: every asset fails NP.001 Γ—8 with "Prim 'Looks' does not follow consistent naming convention (camelCase or snake_case)."

Looks is the long-established USD convention for the materials scope under an asset's default prim β€” it's used by Pixar's published USD samples, NVIDIA's Isaac/Omniverse content, and most authoring tools (Houdini, Maya USD, Kit). The rule's CAMEL_CASE_PATTERN and SNAKE_CASE_PATTERN regexes don't accept the Looks PascalCase single-word, so it reports a violation on a name the broader USD ecosystem treats as canonical.

Required: either whitelist the well-known USD scope names (Looks, Geometry, Cameras, Lights, Animation, Render, etc.) or accept PascalCase as a valid convention alongside camelCase / snake_case.


16. VM.MDL.001 and NP.008 double-report the same MDL resolution failure

Where:

  • capabilities/visualization/materials/... (VM.MDL.001 β€” "The path OmniPBR.mdl does not exist.")
  • capabilities/core/naming_paths/... (NP.008 β€” "Asset path 'OmniPBR.mdl' on attribute 'info:mdl:sourceAsset' at prim '/.../Looks/.../...' does not resolve to an existing file.")

Observed: every asset emits the same number of VM.MDL.001 and NP.008 issues (9 each on nex10_c00, 6 each on nex20_c00, etc.) β€” both rules fire on the same info:mdl:sourceAsset = @OmniPBR.mdl@ attribute. NP.008 gives a useful prim path; VM.MDL.001 just says "doesn't exist".

Separately: OmniPBR.mdl is a standard NVIDIA MDL that ships with Kit and Isaac. The validator's asset resolver doesn't know where to find it (likely missing an MDL search path in the SimReady plugin's resolver config), so a perfectly portable, NVIDIA-blessed material reference is flagged as broken on every asset.

Required:

  1. Configure the SimReady validator plugin to register Kit's standard MDL search paths (mdl/, omni/mdl/, etc.) so OmniPBR.mdl and other shipped MDLs resolve without the customer having to vendor them.
  2. Pick one of VM.MDL.001 / NP.008 as the canonical "MDL doesn't resolve" code. Have the other defer when the first has fired against the same attribute.

17. RC.002 flags spec-mandated robot metadata as "overrides"

Where: capabilities/isaac_sim/robot_core/..., RC.002.

Observed: every asset fails RC.002 with messages like:

Prim is overridden: /nex10_c00, ['isaac:description', 'isaac:license',
'isaac:namespace', 'isaac:robotType']

isaac:description, isaac:license, isaac:namespace, isaac:robotType look like exactly the kind of metadata the SimReady robot spec wants you to author on the robot's default prim β€” but RC.002 is treating any authored override of the default prim's properties as a violation.

If those four isaac:* attributes are spec-mandated, the rule should require them, not flag their presence. If they're optional metadata, the rule should ignore them β€” overriding metadata fields on the default prim is a normal authoring pattern, not a layout error.

Required: clarify the spec around the isaac:* namespace on the robot default prim and update RC.002 to either require those fields, or exclude them from the "this prim shouldn't be overridden" check, or split into two rules (one for "structural" overrides, one for the spec metadata).


Summary

# Severity Area One-line
1 high profile schema Profile.capabilities is dead; should be removed or populated
2 high engine API enable_*/disable_* don't actually scope rule execution
3 high profile content Robot-Body-Isaac v1.0.0 silently drops 4 of 6 declared features
4 high profile schema JSON capability configs aren't linked to the Python profile registry
5 high rule registration Most rules don't declare @register_requirements β†’ 87% UNKNOWN
6 medium engine API init_rules defaults inconsistent; SDK silently turns off all checks
7 medium tooling No headless thumbnail generator ships with foundations
8 low spec docs thumbnail-exist.md example contradicts the actual rule
9 medium packaging simready ingest usd doesn't lift companion thumbnails into spec layout
10 high rule reliability Rule crashes on JointStateAPI access β†’ ~600 UNKNOWN issues per asset
11 high spec/packager mismatch NP.005 expects <asset>/<intermediate>/<main>.usd; packager produces flat layout. Plus relpath(file,…) should be relpath(found_file,…)
12 high rule scoping "Example" tutorial capability (EX.*) is active in production validation
13 medium rule duplication EX.01 and VG.MESH.001 are the same check
14 high rule correctness HI.002 rejects matrix transforms (xformOp:transform) and identity transforms
15 medium rule correctness NP.001 rejects the conventional /Looks scope name
16 high rule correctness OmniPBR.mdl (standard NVIDIA MDL) doesn't resolve; both VM.MDL.001 and NP.008 fire on the same attribute
17 medium spec ambiguity RC.002 flags spec-looking isaac:* metadata as illegal "overrides"