arozantsev commited on
Commit
8bc8124
·
verified ·
1 Parent(s): 923558a

Deploy simready-assets run 20261020-a3cb7f2-captions (GPU variant)

Browse files
Dockerfile CHANGED
@@ -1,4 +1,4 @@
1
- # deploy: 20261020-a3cb7f2 (simready-assets)
2
  # GPU variant of the single-container MVP image for Hugging Face Spaces
3
  # (Docker SDK): everything docker/Dockerfile.hf-space has (OpenSearch + Neo4j
4
  # + Redis + deepsearch-api + asset-graph-service + nginx under supervisord),
@@ -185,12 +185,15 @@ RUN python3.12 -m venv /app/.venv \
185
 
186
  # The vision checks' prepared asset views on disk (docker/hf-space/check_image_cache.py), on while the
187
  # CHECK_IMAGE_CACHE_DIR Space Variable is set (pipeline/config.py). The .pth loads it in every venv python. The
188
- # build fails unless it wraps the vendored llm_client and cached views come back byte-identical (--self-test).
 
189
  COPY docker/hf-space/check_image_cache.py docker/hf-space/check_image_cache.pth /tmp/check-image-hook/
190
  RUN SP=$(env -u PYTHONPATH /app/.venv/bin/python -c 'import sysconfig; print(sysconfig.get_path("purelib"))') \
191
  && install -m 644 /tmp/check-image-hook/check_image_cache.py /tmp/check-image-hook/check_image_cache.pth "$SP/" \
192
  && rm -rf /tmp/check-image-hook \
193
  && env -u PYTHONPATH CHECK_IMAGE_CACHE_DIR=/tmp/check-image-selftest /app/.venv/bin/python -m check_image_cache --self-test \
 
 
194
  && rm -rf /tmp/check-image-selftest
195
 
196
  # hf-storage-api — see docker/Dockerfile.hf-space's own copy of this block
 
1
+ # deploy: 20261020-a3cb7f2-captions (simready-assets)
2
  # GPU variant of the single-container MVP image for Hugging Face Spaces
3
  # (Docker SDK): everything docker/Dockerfile.hf-space has (OpenSearch + Neo4j
4
  # + Redis + deepsearch-api + asset-graph-service + nginx under supervisord),
 
185
 
186
  # The vision checks' prepared asset views on disk (docker/hf-space/check_image_cache.py), on while the
187
  # CHECK_IMAGE_CACHE_DIR Space Variable is set (pipeline/config.py). The .pth loads it in every venv python. The
188
+ # build fails unless it wraps the vendored llm_client and cached views come back byte-identical (--self-test), as PNG
189
+ # (the default) and as JPEG (USDSEARCH_VISION_VALIDATION_IMAGE_FORMAT=jpeg, the Space Variable prod runs with).
190
  COPY docker/hf-space/check_image_cache.py docker/hf-space/check_image_cache.pth /tmp/check-image-hook/
191
  RUN SP=$(env -u PYTHONPATH /app/.venv/bin/python -c 'import sysconfig; print(sysconfig.get_path("purelib"))') \
192
  && install -m 644 /tmp/check-image-hook/check_image_cache.py /tmp/check-image-hook/check_image_cache.pth "$SP/" \
193
  && rm -rf /tmp/check-image-hook \
194
  && env -u PYTHONPATH CHECK_IMAGE_CACHE_DIR=/tmp/check-image-selftest /app/.venv/bin/python -m check_image_cache --self-test \
195
+ && env -u PYTHONPATH CHECK_IMAGE_CACHE_DIR=/tmp/check-image-selftest USDSEARCH_VISION_VALIDATION_IMAGE_FORMAT=jpeg \
196
+ /app/.venv/bin/python -m check_image_cache --self-test \
197
  && rm -rf /tmp/check-image-selftest
198
 
199
  # hf-storage-api — see docker/Dockerfile.hf-space's own copy of this block
docker/hf-space/check_image_cache.py CHANGED
@@ -1,8 +1,9 @@
1
  """The vision checks' prepared asset views, kept on disk for every deepsearch-api process of the Space.
2
 
3
  Before each check, usdsearch's llm_client prepares the asset's stored view: it decodes the stored thumbnail, fits it
4
- to USDSEARCH_VISION_VALIDATION_IMAGE_MAX_PX and re-encodes it as base64 PNG (``Validation._encode_reference_images``
5
- -> ``_encode_image``). That is most of a check's API CPU, and a stored view never changes. With
 
6
  ``CHECK_IMAGE_CACHE_DIR`` set (a Space Variable, pipeline/config.py), each view ``_encode_reference_images`` prepares
7
  is kept in ``<dir>/<settings>/<sha256 of the stored base64>.b64``: a hit reads the file; a miss prepares the view as
8
  before and writes it (a temp file, then ``os.replace``), so the API processes share one copy. ``<settings>`` is a
@@ -39,7 +40,7 @@ import zlib
39
 
40
  ENV = "CHECK_IMAGE_CACHE_DIR"
41
  TARGET = "llm_client.validation.validation"
42
- MAX_BYTES = 2_000_000_000 # ~15k views at ~130 KB (448-px PNG); the Space's ~2,600 stored images take ~340 MB
43
  MIN_FREE = 12_000_000_000 # above the 8 GB the media loader keeps free
44
  RECHECK_BYTES, RECHECK_S = 64_000_000, 60 # each process re-reads the size and the free space this often
45
  _DIGEST = re.compile(r"^[0-9a-f]{24}$")
@@ -382,8 +383,9 @@ def sweep(directory, older_than=600):
382
 
383
  def self_test():
384
  """The image build's check (Dockerfile.hf-space-gpu): the .pth loaded this module and it wrapped the vendored
385
- llm_client, views read back from the cache are byte-identical to llm_client's own, and the warm-up's imports
386
- exist. Raises on any difference. Writes only in CHECK_IMAGE_CACHE_DIR, which the build then removes."""
 
387
  import base64
388
  import io
389
 
@@ -409,6 +411,10 @@ def self_test():
409
  images = [b64(rgb, "JPEG", quality=90), b64(rgba, "PNG"), b64(rgb.resize((300, 200)), "JPEG", quality=90)]
410
  config = ValidationSettings(image_max_px=448).to_validation_config()
411
  plain = [Validation(config=config)._encode_image(image) for image in images]
 
 
 
 
412
  miss = Validation(config=config)._encode_reference_images(images)
413
  reader = Validation(config=config)
414
  hit = reader._encode_reference_images(images)
 
1
  """The vision checks' prepared asset views, kept on disk for every deepsearch-api process of the Space.
2
 
3
  Before each check, usdsearch's llm_client prepares the asset's stored view: it decodes the stored thumbnail, fits it
4
+ to USDSEARCH_VISION_VALIDATION_IMAGE_MAX_PX and re-encodes it as base64 PNG, or JPEG with
5
+ USDSEARCH_VISION_VALIDATION_IMAGE_FORMAT=jpeg as on the Space (``Validation._encode_reference_images`` ->
6
+ ``_encode_image``). That is most of a check's API CPU, and a stored view never changes. With
7
  ``CHECK_IMAGE_CACHE_DIR`` set (a Space Variable, pipeline/config.py), each view ``_encode_reference_images`` prepares
8
  is kept in ``<dir>/<settings>/<sha256 of the stored base64>.b64``: a hit reads the file; a miss prepares the view as
9
  before and writes it (a temp file, then ``os.replace``), so the API processes share one copy. ``<settings>`` is a
 
40
 
41
  ENV = "CHECK_IMAGE_CACHE_DIR"
42
  TARGET = "llm_client.validation.validation"
43
+ MAX_BYTES = 2_000_000_000 # 448-px views: ~28 KB as JPEG, the Space's ~2,600 images ~75 MB (PNG ~121 KB, ~340 MB)
44
  MIN_FREE = 12_000_000_000 # above the 8 GB the media loader keeps free
45
  RECHECK_BYTES, RECHECK_S = 64_000_000, 60 # each process re-reads the size and the free space this often
46
  _DIGEST = re.compile(r"^[0-9a-f]{24}$")
 
383
 
384
  def self_test():
385
  """The image build's check (Dockerfile.hf-space-gpu): the .pth loaded this module and it wrapped the vendored
386
+ llm_client, views read back from the cache are byte-identical to llm_client's own and in the format the
387
+ environment names (USDSEARCH_VISION_VALIDATION_IMAGE_FORMAT, unset: PNG), and the warm-up's imports exist.
388
+ Raises on any difference. Writes only in CHECK_IMAGE_CACHE_DIR, which the build then removes."""
389
  import base64
390
  import io
391
 
 
411
  images = [b64(rgb, "JPEG", quality=90), b64(rgba, "PNG"), b64(rgb.resize((300, 200)), "JPEG", quality=90)]
412
  config = ValidationSettings(image_max_px=448).to_validation_config()
413
  plain = [Validation(config=config)._encode_image(image) for image in images]
414
+ # Wheels that dropped the format setting would pass the build's JPEG run with PNG views.
415
+ wanted = os.environ.get("USDSEARCH_VISION_VALIDATION_IMAGE_FORMAT") or "png"
416
+ magic = {"png": "iVBORw0KGgo", "jpeg": "/9j/"}[wanted]
417
+ assert all(view.startswith(magic) for view in plain), f"the views are not {wanted}"
418
  miss = Validation(config=config)._encode_reference_images(images)
419
  reader = Validation(config=config)
420
  hit = reader._encode_reference_images(images)
docker/hf-space/search_fields.yaml CHANGED
@@ -15,18 +15,26 @@
15
  # parser-only (no `label:`) — neither is in nginx.conf.template's
16
  # railFilters allow-list, so they're invisible to the rail regardless of
17
  # being registered here; only the LLM query parser picks them up. Verified
18
- # live via /search/stats/usd_properties: mass is numeric (kg), continuous
19
- # (1407 distinct values across 1484 of 1488 assets, range 0.001-217652.74,
20
- # median ~1.77); rigid_body_count is numeric (integer), 37 distinct values
21
- # across 1484 of 1488 assets, range 1-106, most common value 1 (904
22
- # assets).
 
 
 
 
 
 
 
 
23
  #
24
  # Sizes (2026-10-14): `dimension` (Central's catalog field, every side) and
25
  # min_height / max_height / max_size_x / max_size_y write the bounding-box
26
  # params min_bbox_* / max_bbox_* in metres, so the Explorer shows them as its
27
  # Dimensions (m) X/Y/Z chips and sliders (the rail's "dimension" control,
28
- # railFilters in nginx.conf.template). Every asset of the 1,401 carries
29
- # usd_dimensions, Z up (scaled sides: median 0.25 x 0.29 x 0.18 m, max 12 m).
30
  # The props face either axis (desk 0.80 x 1.80, counter 0.68 x 3.02, bulk rack
31
  # 2.45 x 1.23 m), so X and Y are bounded together, once, by the LARGEST
32
  # horizontal size the query gives (the length, or the width when it is the
@@ -50,10 +58,10 @@ fields:
50
  operators:
51
  - eq
52
  - contains
53
- description: SimReady validation profile. The only values are 'Robotics-Prop', 'Robot-Body' and
54
- 'Robot-Gripper'. Set it ONLY when the query says "profile" or names one of these values. Words for what
55
  the asset is (robot, gripper, arm, prop, simready) and maker or product names (NVIDIA, LG, Robotiq,
56
- Standard Bots, 2F-85, "made by X", "from X") are never a profile; they stay in semantic_query. Another
57
  profile name, a profile version ("3.2.0") or a negated profile ("not X", "other than X", "outside X") is
58
  an unmapped_constraint and no filter.
59
  examples:
@@ -73,12 +81,16 @@ fields:
73
  operators:
74
  - eq
75
  - contains
76
- description: Asset license. The only value is 'CC-BY-4.0' (Creative Commons Attribution 4.0); "CC BY",
77
- "CC-BY 4.0" and "creative commons attribution" mean it. Set it ONLY when the query names a license; it is
78
- never a stand-in for another constraint (a date, a file type, a format). Another license (MIT, Apache,
79
- GPL, proprietary) or a negated license ("not CC-BY") is an unmapped_constraint and no filter.
 
 
80
  examples:
81
- - '"CC BY licensed jugs" -> simready_metadata_asset_license eq "CC-BY-4.0", semantic_query "jugs"'
 
 
82
  - '"GPL licensed lamps" -> no filter, semantic_query "lamps", unmapped "GPL licensed"'
83
  - '"glb files of lamps" -> no filter, semantic_query "lamps", unmapped "glb files"'
84
  label: License
@@ -93,7 +105,7 @@ fields:
93
  - lt
94
  - lte
95
  - eq
96
- description: Mass in kilograms (median 1.5 kg, from 0.001 to 217,000). Convert g / 1000, lb x 0.4536, t or
97
  ton x 1000. "heavier than / over N" -> gt N; "under / lighter than N" -> lt N; "no more than / not
98
  heavier than N" -> lte N; "between A and B" -> gte A and lte B, exactly A and B; only "about / around
99
  N" -> gte N x 0.9 and lte N x 1.1, never eq. Only a number with a weight unit is a mass; a length, a
@@ -115,8 +127,8 @@ fields:
115
  - lt
116
  - lte
117
  - eq
118
- description: Number of rigid bodies (parts that move separately) in the physics setup, an integer from 1 to
119
- 106; most assets have 1. "single-body" or "one rigid body" -> eq 1; "multi-body" or "several rigid
120
  bodies" -> gt 1. Only a count of rigid bodies is this field; a size, a mass, a polygon or triangle count
121
  never is.
122
  examples:
 
15
  # parser-only (no `label:`) — neither is in nginx.conf.template's
16
  # railFilters allow-list, so they're invisible to the rail regardless of
17
  # being registered here; only the LLM query parser picks them up. Verified
18
+ # live via /search/stats/usd_properties: mass is numeric (kg), continuous;
19
+ # rigid_body_count is numeric (integer), most common value 1.
20
+ #
21
+ # Values and ranges in the descriptions below are the index's (run
22
+ # 20261020-a3cb7f2-captions, 3,599 assets): license 'CC-BY 4.0' 2,513,
23
+ # 'CC-BY-4.0' 1,049, 'cc-by-4.0' 22, 'BSD-3-Clause' 4, 'Apache-2.0' 2,
24
+ # 'LicenseRef-UR-Graphical-Documentation AND BSD-3-Clause' 1, none 8;
25
+ # profiles Robotics-Prop 3,559, Robot-Body 18, AIF-Entity 15, Robot-Gripper 7;
26
+ # mass 0.00013-501,226 kg, median 0.68; rigid bodies 0-126. A property_match
27
+ # eq is an exact, case-sensitive match, so the license (three spellings of
28
+ # CC-BY) is matched with contains (=~*stem*, case-insensitive). They go stale
29
+ # with every re-index: re-check them (docs/development.md, "search_fields.yaml
30
+ # is hand-curated").
31
  #
32
  # Sizes (2026-10-14): `dimension` (Central's catalog field, every side) and
33
  # min_height / max_height / max_size_x / max_size_y write the bounding-box
34
  # params min_bbox_* / max_bbox_* in metres, so the Explorer shows them as its
35
  # Dimensions (m) X/Y/Z chips and sliders (the rail's "dimension" control,
36
+ # railFilters in nginx.conf.template). Every asset carries usd_dimensions, Z up
37
+ # (2026-10-20, 3,599 assets: scaled sides median 0.19 x 0.17 x 0.11 m, max 33 m).
38
  # The props face either axis (desk 0.80 x 1.80, counter 0.68 x 3.02, bulk rack
39
  # 2.45 x 1.23 m), so X and Y are bounded together, once, by the LARGEST
40
  # horizontal size the query gives (the length, or the width when it is the
 
58
  operators:
59
  - eq
60
  - contains
61
+ description: SimReady validation profile. The only values are 'Robotics-Prop', 'Robot-Body', 'Robot-Gripper'
62
+ and 'AIF-Entity'. Set it ONLY when the query says "profile" or names one of these values. Words for what
63
  the asset is (robot, gripper, arm, prop, simready) and maker or product names (NVIDIA, LG, Robotiq,
64
+ Standard Bots, Vertiv, 2F-85, "made by X", "from X") are never a profile; they stay in semantic_query. Another
65
  profile name, a profile version ("3.2.0") or a negated profile ("not X", "other than X", "outside X") is
66
  an unmapped_constraint and no filter.
67
  examples:
 
81
  operators:
82
  - eq
83
  - contains
84
+ description: Asset license. The index spells one license several ways, so ALWAYS use contains with the
85
+ short name below, never eq. Creative Commons Attribution 4.0 ("CC BY", "CC-BY 4.0", "creative commons
86
+ attribution"; almost every asset) -> contains "CC-BY". Apache 2.0 ("Apache", "Apache-2.0") -> contains
87
+ "Apache". BSD 3-Clause ("BSD", "BSD-3-Clause") -> contains "BSD". Set it ONLY when the query names a
88
+ license; it is never a stand-in for another constraint (a date, a file type, a format). Any other license
89
+ (MIT, GPL, proprietary) or a negated license ("not CC-BY") is an unmapped_constraint and no filter.
90
  examples:
91
+ - '"CC BY licensed jugs" -> simready_metadata_asset_license contains "CC-BY", semantic_query "jugs"'
92
+ - '"grippers released under the Apache license" -> simready_metadata_asset_license contains "Apache",
93
+ semantic_query "grippers"'
94
  - '"GPL licensed lamps" -> no filter, semantic_query "lamps", unmapped "GPL licensed"'
95
  - '"glb files of lamps" -> no filter, semantic_query "lamps", unmapped "glb files"'
96
  label: License
 
105
  - lt
106
  - lte
107
  - eq
108
+ description: Mass in kilograms (median 0.7 kg, from 0 to 501,300). Convert g / 1000, lb x 0.4536, t or
109
  ton x 1000. "heavier than / over N" -> gt N; "under / lighter than N" -> lt N; "no more than / not
110
  heavier than N" -> lte N; "between A and B" -> gte A and lte B, exactly A and B; only "about / around
111
  N" -> gte N x 0.9 and lte N x 1.1, never eq. Only a number with a weight unit is a mass; a length, a
 
127
  - lt
128
  - lte
129
  - eq
130
+ description: Number of rigid bodies (parts that move separately) in the physics setup, an integer from 0 to
131
+ 126; most assets have 1. "single-body" or "one rigid body" -> eq 1; "multi-body" or "several rigid
132
  bodies" -> gt 1. Only a count of rigid bodies is this field; a size, a mass, a polygon or triangle count
133
  never is.
134
  examples:
docker/hf-space/wheels/SOURCE.txt CHANGED
@@ -1,3 +1,3 @@
1
  # Generated by scripts/extract_usdsearch_wheels.sh — do not hand-edit.
2
- source_image=gitlab-master.nvidia.com:5005/omniverse/deeptag/usdsearch/usdsearch:fd61a332
3
- extracted_at=2026-10-09T14:24:46Z
 
1
  # Generated by scripts/extract_usdsearch_wheels.sh — do not hand-edit.
2
+ source_image=gitlab-master.nvidia.com:5005/omniverse/deeptag/usdsearch/usdsearch:d427fefd
3
+ extracted_at=2026-10-20T20:21:26Z
docker/hf-space/wheels/asset_graph_client-0.1.0-py3-none-any.whl CHANGED
Binary files a/docker/hf-space/wheels/asset_graph_client-0.1.0-py3-none-any.whl and b/docker/hf-space/wheels/asset_graph_client-0.1.0-py3-none-any.whl differ
 
docker/hf-space/wheels/asset_graph_service-0.1.0-py3-none-any.whl CHANGED
Binary files a/docker/hf-space/wheels/asset_graph_service-0.1.0-py3-none-any.whl and b/docker/hf-space/wheels/asset_graph_service-0.1.0-py3-none-any.whl differ
 
docker/hf-space/wheels/base_logging-0.1.0-py3-none-any.whl CHANGED
Binary files a/docker/hf-space/wheels/base_logging-0.1.0-py3-none-any.whl and b/docker/hf-space/wheels/base_logging-0.1.0-py3-none-any.whl differ
 
docker/hf-space/wheels/deepsearch_api-0.1.0-py3-none-any.whl CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:380e2f6e9f261b3aedc41c6f3647017002cf68209fed3577eef9e3157209a272
3
- size 221113
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3ab3d0469632a4e8f31ae2cd19c46f23bb662afabe286266e42520a1aa4c8a32
3
+ size 221753
docker/hf-space/wheels/llm_client-0.1.0-py3-none-any.whl CHANGED
Binary files a/docker/hf-space/wheels/llm_client-0.1.0-py3-none-any.whl and b/docker/hf-space/wheels/llm_client-0.1.0-py3-none-any.whl differ
 
docker/hf-space/wheels/search_utils-0.1.0-py3-none-any.whl CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:35f1cc7cf94f7bf845036cd7332ce2888944eb1c1309532b8863f9e7097c3092
3
- size 286722
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e498c1662a82bf7c712e50ed86bb9e6b423dff1f160b113b32b201a2a2b6e4b
3
+ size 287870
docker/hf-space/wheels/siglip2_triton_client-0.6.1-py3-none-any.whl CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:daba44a4ac31b836f233e1e7945ab5aaa8b8a2ebcf2702375166f87248f49fc0
3
  size 5482694
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:117363f7769e221649ae6e758ba82826ac9e8e00ce227ebfceebf2950d3e8d1e
3
  size 5482694
docker/hf-space/wheels/storage_api-0.1.0-py3-none-any.whl CHANGED
Binary files a/docker/hf-space/wheels/storage_api-0.1.0-py3-none-any.whl and b/docker/hf-space/wheels/storage_api-0.1.0-py3-none-any.whl differ