docs: verify setup recovery

6e38481 3 days ago

6.32 kB

# Repository and preservation setup

Implemented 2026-10-21. This document supersedes the separate-dataset proposal in the original planning pass. The user chose one HF repository with normal Git code/tests and Xet-backed reference payloads, matching the inspected `urban-terror-rs` configuration. `glade` currently has a different, still-unbound separate-dataset design.

## Repository identity and storage

- Local checkout name: `nanobrush`; remote `hf`: `git@hf.co:nmarshall/nanobrush`.
- The HF initial commit is preserved as the parent of the setup commit.
- Root `.gitattributes` selects `nanobrush-code` by default. It selects LFS with binary-safe attributes for `assets/**`, `references/**`, and `**/references/**`.
- `scripts/setup-repository.py` installs code filters, Git LFS, Git Xet registration, and hooks locally. It refuses to replace an unrelated hook path. It does not modify global Git configuration.
- Staged-object and history checks reject ordinary payload blobs in reference paths, binaries outside reference paths, unexpected filters, invalid asset attributes, symlinks/gitlinks, and unresolved index entries. LFS's specified empty-file representation is allowed with LFS attributes.
- The pre-push hook preserves update stdin for both the history guard and `git lfs pre-push`, which invokes Xet through the configured transfer agent.
- Source and tests are normal Git files. All original ZIPs, extracted IXPL/shaders/config/text/binaries, provenance files, and generated inventories are reference payloads.

## Environment

The flake uses flake-parts, pinned `my-nix` essentials/Rust modules, and Linux/Darwin system declarations. A watched optional local override follows the recent nix-rust `.envrc` pattern. The shell provides Rust/C build tools, Git/LFS/Xet, Python/HF tooling, archives/checksums, formatting, shell checks, and Linux graphics dependencies. Shared editor/global mutation hooks are replaced with the repository-local installer.

Nix checks use an explicit code-only fileset excluding assets; the normal flake source import can still see hydrated Git-tracked assets. This is a consequence of the requested single-repository layout. Run code-only CI with `GIT_LFS_SKIP_SMUDGE=1` and hydrate assets only for reference checks. The application/Bevy workspace and real graphics probes remain future work.

## Preservation and extraction

All 14 original collection files were copied to the canonical reference layout and byte-compared before removing the old root locations. Original manifest/checksum/report bytes remain under provenance. `config/references.json` is the small authored verification manifest. Relocated archive checksums are in `assets/references/SHA256SUMS`.

`scripts/references.py` offers `list`, `verify`, `unpack`, and `index`; build selection, dry-run, and extracted-output verification are available. It verifies all selected archives before output changes, rejects unsafe paths/types/collisions/expansion, validates member CRCs, preserves original bytes, records timestamps/attributes in inventories, uses a writer lock, atomically publishes each completed build, and refuses changed output. On a hydrated clone it first validates every file, then recreates the original empty directories omitted by Git. Read-only commands do not create output directories.

The verified corpus is five archives, 560 regular file occurrences, and 132 distinct paths. Extraction intentionally does not restore executable permissions; original ZIP attributes are retained in the inventory. No original executable, BAT file, compiler, or plugin has been run.

## Validation record

- Network-free storage and adversarial archive tests: 28 passing on the host Python and in the pinned Nix shell.
- All five original archive SHA-256 values and internal CRCs verified.
- All 560 extracted files verified against archive-member hashes; repeat unpack reports every build unchanged.
- Nix shell enters successfully on the current Apple Silicon macOS host; environment tool versions are verified.
- All four declared system outputs evaluate; the actual Apple Silicon `checks.aarch64-darwin.setup` derivation builds successfully with an asset-free source fileset.
- Setup hooks, staged pointer inspection, direnv reload, and post-rename path checks pass. The initial upload succeeded, and HF tree metadata independently reports all 580 asset files as Xet-backed with LFS metadata.

Linux, Intel macOS, runtime graphics, full resource dependency indexing, and asset conversion/recode tools remain future tasks. The original software license/permission investigation remains a separate open research item; this setup does not mark it resolved.

## Fresh-clone extraction correction

The first hydrated-clone test found that Git omits empty original ZIP directories (for example `data/models`). The unpacker now verifies all existing file bytes before restoring missing original directories, and still refuses changed files without writing into them. A dedicated regression covers both restoration and refusal after a file edit.

## Completed remote recovery check

Implementation commits: `29ed697` (setup) and `81d1607` (empty-directory recovery). The user-created HF initial commit is retained.

A fresh SSH clone with skip-smudge first built the code-only Nix check while reference files were still pointers. With explicitly empty LFS and Xet caches, `assets.py fetch` recovered all 580 payloads and matched every staged pointer hash. `references.py unpack` restored the original empty directories; `verify --check-extracted` then checked all 560 extracted files across all five builds. The temporary clone remained clean and was removed afterward.

The installed Git Xet agent handles Xet upload integration. This client's download trace selects the standard LFS `basic` adapter/compatibility bridge; HF metadata confirms the backend objects are Xet-backed for every file, including small TXT/CFG/IXPL resources. Download transport does not turn those payloads into ordinary Git blobs.

All four declared Nix system outputs evaluate. Actual shell, direnv/local overrides, setup tests (28 passing), code-only Nix derivation, and reference round trip were exercised on Apple Silicon macOS. Linux and Intel macOS execution remain unverified. No original program or plugin was executed.

	# Repository and preservation setup

	Implemented 2026-10-21. This document supersedes the separate-dataset proposal in the original planning pass. The user chose one HF repository with normal Git code/tests and Xet-backed reference payloads, matching the inspected `urban-terror-rs` configuration. `glade` currently has a different, still-unbound separate-dataset design.

	## Repository identity and storage

	- Local checkout name: `nanobrush`; remote `hf`: `git@hf.co:nmarshall/nanobrush`.
	- The HF initial commit is preserved as the parent of the setup commit.
	- Root `.gitattributes` selects `nanobrush-code` by default. It selects LFS with binary-safe attributes for `assets/`, `references/`, and `/references/`.
	- `scripts/setup-repository.py` installs code filters, Git LFS, Git Xet registration, and hooks locally. It refuses to replace an unrelated hook path. It does not modify global Git configuration.
	- Staged-object and history checks reject ordinary payload blobs in reference paths, binaries outside reference paths, unexpected filters, invalid asset attributes, symlinks/gitlinks, and unresolved index entries. LFS's specified empty-file representation is allowed with LFS attributes.
	- The pre-push hook preserves update stdin for both the history guard and `git lfs pre-push`, which invokes Xet through the configured transfer agent.
	- Source and tests are normal Git files. All original ZIPs, extracted IXPL/shaders/config/text/binaries, provenance files, and generated inventories are reference payloads.

	## Environment

	The flake uses flake-parts, pinned `my-nix` essentials/Rust modules, and Linux/Darwin system declarations. A watched optional local override follows the recent nix-rust `.envrc` pattern. The shell provides Rust/C build tools, Git/LFS/Xet, Python/HF tooling, archives/checksums, formatting, shell checks, and Linux graphics dependencies. Shared editor/global mutation hooks are replaced with the repository-local installer.

	Nix checks use an explicit code-only fileset excluding assets; the normal flake source import can still see hydrated Git-tracked assets. This is a consequence of the requested single-repository layout. Run code-only CI with `GIT_LFS_SKIP_SMUDGE=1` and hydrate assets only for reference checks. The application/Bevy workspace and real graphics probes remain future work.

	## Preservation and extraction

	All 14 original collection files were copied to the canonical reference layout and byte-compared before removing the old root locations. Original manifest/checksum/report bytes remain under provenance. `config/references.json` is the small authored verification manifest. Relocated archive checksums are in `assets/references/SHA256SUMS`.

	`scripts/references.py` offers `list`, `verify`, `unpack`, and `index`; build selection, dry-run, and extracted-output verification are available. It verifies all selected archives before output changes, rejects unsafe paths/types/collisions/expansion, validates member CRCs, preserves original bytes, records timestamps/attributes in inventories, uses a writer lock, atomically publishes each completed build, and refuses changed output. On a hydrated clone it first validates every file, then recreates the original empty directories omitted by Git. Read-only commands do not create output directories.

	The verified corpus is five archives, 560 regular file occurrences, and 132 distinct paths. Extraction intentionally does not restore executable permissions; original ZIP attributes are retained in the inventory. No original executable, BAT file, compiler, or plugin has been run.

	## Validation record

	- Network-free storage and adversarial archive tests: 28 passing on the host Python and in the pinned Nix shell.
	- All five original archive SHA-256 values and internal CRCs verified.
	- All 560 extracted files verified against archive-member hashes; repeat unpack reports every build unchanged.
	- Nix shell enters successfully on the current Apple Silicon macOS host; environment tool versions are verified.
	- All four declared system outputs evaluate; the actual Apple Silicon `checks.aarch64-darwin.setup` derivation builds successfully with an asset-free source fileset.
	- Setup hooks, staged pointer inspection, direnv reload, and post-rename path checks pass. The initial upload succeeded, and HF tree metadata independently reports all 580 asset files as Xet-backed with LFS metadata.

	Linux, Intel macOS, runtime graphics, full resource dependency indexing, and asset conversion/recode tools remain future tasks. The original software license/permission investigation remains a separate open research item; this setup does not mark it resolved.

	## Fresh-clone extraction correction

	The first hydrated-clone test found that Git omits empty original ZIP directories (for example `data/models`). The unpacker now verifies all existing file bytes before restoring missing original directories, and still refuses changed files without writing into them. A dedicated regression covers both restoration and refusal after a file edit.

	## Completed remote recovery check

	Implementation commits: `29ed697` (setup) and `81d1607` (empty-directory recovery). The user-created HF initial commit is retained.

	A fresh SSH clone with skip-smudge first built the code-only Nix check while reference files were still pointers. With explicitly empty LFS and Xet caches, `assets.py fetch` recovered all 580 payloads and matched every staged pointer hash. `references.py unpack` restored the original empty directories; `verify --check-extracted` then checked all 560 extracted files across all five builds. The temporary clone remained clean and was removed afterward.

	The installed Git Xet agent handles Xet upload integration. This client's download trace selects the standard LFS `basic` adapter/compatibility bridge; HF metadata confirms the backend objects are Xet-backed for every file, including small TXT/CFG/IXPL resources. Download transport does not turn those payloads into ordinary Git blobs.

	All four declared Nix system outputs evaluate. Actual shell, direnv/local overrides, setup tests (28 passing), code-only Nix derivation, and reference round trip were exercised on Apple Silicon macOS. Linux and Intel macOS execution remain unverified. No original program or plugin was executed.