AbstractPhil commited on
Commit
22c5eca
·
verified ·
1 Parent(s): 3106dc1

anima_closeout v10: latent caching died on its FIRST image read — the fork's ParquetImageReader whole-file column read cannot deliver >2GiB nested (struct) columns (ArrowNotImplementedError), and the 8192-row split monolith crosses that line. Fixed in the fork (8ed4b6b: per-row-group reads assembled into a ChunkedArray; proven against a 1200x2MB struct parquet reproducing the exact error). CELL 0 asserts the fixed reader pre-spend. No CELL 1 re-run needed: the monolith is readable as-is.

Browse files
Files changed (1) hide show
  1. colab/anima_closeout.ipynb +5 -0
colab/anima_closeout.ipynb CHANGED
@@ -123,6 +123,11 @@
123
  "# crash, ModuleNotFoundError: hyvideo \u2014 the recorded exp004 + smoke killer)\n",
124
  "assert \"_HAS_HYVIDEO\" in (FORK / \"utils\" / \"patches.py\").read_text(), \\\n",
125
  " \"fork main lacks the guarded hyvideo import (utils/patches.py)\"\n",
 
 
 
 
 
126
  "\n",
127
  "if not ANIMA.get(\"DEPS_DONE\"):\n",
128
  " sh(f\"pip install -q -r {FORK}/requirements.txt\")\n",
 
123
  "# crash, ModuleNotFoundError: hyvideo \u2014 the recorded exp004 + smoke killer)\n",
124
  "assert \"_HAS_HYVIDEO\" in (FORK / \"utils\" / \"patches.py\").read_text(), \\\n",
125
  " \"fork main lacks the guarded hyvideo import (utils/patches.py)\"\n",
126
+ "# the 8192-row split monolith needs the per-row-group image reader (fork\n",
127
+ "# 8ed4b6b) \u2014 the whole-file read dies on >2GiB nested columns\n",
128
+ "assert \"pa.chunked_array(chunks)\" in \\\n",
129
+ " (FORK / \"utils\" / \"parquet_source.py\").read_text(), \\\n",
130
+ " \"fork main lacks the per-row-group ParquetImageReader (parquet_source.py)\"\n",
131
  "\n",
132
  "if not ANIMA.get(\"DEPS_DONE\"):\n",
133
  " sh(f\"pip install -q -r {FORK}/requirements.txt\")\n",