docs/TECHNICAL_DIRECTION.md

Proposed technical direction

Storage, environment, hydration, and extraction are implemented; see SETUP.md. The mb3-dat and mb3-txt crates plus nanobrush-cli now implement native TXT/DAT format handling (compatibility profile); the broader Rust/rendering/conversion boundaries below remain proposed and must be validated through the TODO phases.

Storage and repository boundaries

Follow the active urban-terror-rs pattern selected by the user: one Hugging Face Git repository, normal Git source/tests, and Xet-backed assets and references. The initial separate-dataset proposal is superseded.

nanobrush/                         # remote hf: git@hf.co:nmarshall/nanobrush
  .envrc, flake.nix, flake.lock     # pinned my-nix environment
  .gitattributes                   # code filter; assets/references use LFS + Xet
  .githooks/                       # staged-object/history guards plus LFS pre-push
  config/references.json           # authored preservation metadata; ordinary Git
  scripts/                        # setup, storage, hydration, extraction, checks
  tests/                          # ordinary Git test source
  docs/                           # ordinary Git authored findings/specifications
  crates/                         # native DAT/TXT libraries and nanobrush CLI
  assets/references/              # every file is Xet-backed, including small text
    archives/                     # unchanged ZIPs
    provenance/collection/        # original reports/manifest/checksum files
    provenance/historical/        # historical pages/indexes
    extracted/<build>/            # immutable raw members, including binaries
    inventories/<build>.json      # generated metadata payloads
    SHA256SUMS                    # relocated archive paths

Any additional references/ directory, including root references/ or nested docs/references/, is also LFS-filtered. Keep authored source/tests/docs outside these payload directories. All pre-existing collection downloads have been relocated beneath assets/references/ and byte-verified.

There is no nested .git, dataset, separate asset branch, or asset lockfile. Each normal Git commit pins asset content through its LFS pointer SHA-256 values. assets-fetch hydrates the checked-out revision; assets-check verifies working bytes against staged pointers.

Use filter=lfs attributes plus Git Xet's custom transfer registration. All reference resources intentionally follow this policy regardless of size. Commit/push guards check staged and historical objects so raw payloads cannot slip into ordinary Git, even when a clean filter is bypassed. The LFS-specified empty-file representation carries no payload bytes.

Build outputs and disposable runtime copies remain ignored. Keep immutable references unchanged; transformations write separate derived assets and provenance. Nix derivation source filesets exclude payloads, while Git-backed flake source imports can still contain hydrated assets; use a skip-smudge clone for code-only CI.

Environment shape

Use flake-parts and github:nmrshll/my-nix, following the current local exports flakeModuleGroups.essentials and flakeModules.rust. Keep nixpkgs and flake-parts follows consistent. Rust checks and the CLI package use the same pinned toolchain and explicit asset-free source fileset.

Pin remote inputs in flake.lock. An optional developer .envrc helper may watch local my-nix / tools checkouts and pass overrides without rewriting the shared lock; the committed flake remains usable without those checkouts. Separate platform-specific graphics libraries from portable archive/source tooling.

Provide a lightweight preservation/research shell first. Add the Rust workspace, native graphics dependencies, codec tools, and analysis tools as needed. Keep Windows/Wine/.NET oracle requirements in a separate documented environment; native Nanobrush must not inherit them.

Planned commands and responsibilities

Setup/storage/hydration/reference commands and a converter-only Mono oracle are implemented. The native CLI supports inspect/validate/convert for TXT and both DAT versions, with a Nix package; other formats and editor/replay interfaces remain future work. The remaining asset-conversion, general oracle/replay and GUI-packaging interfaces below remain proposed.

Command	Inputs	Output and acceptance contract
setup-repository	Current checkout and committed policy	Repository-local filters/hooks; detect conflicting setup; no global config edits
check-storage	Index/history or source export	Reject code payloads, invalid pointers, unsafe entry types, or missing policy
assets-check / assets-fetch	Same-repo staged pointer identities / checked-out revision	Verified LFS/Xet setup and checksummed hydration; no separate asset checkout
verify-references	Pinned archives and original manifest	Original-byte hash/CRC results with missing/pointer-stub failures
unpack-references	Manifest, selected versions, explicit destination	Traversal-safe deterministic extraction into assets/references/extracted/; no execution
index-references	Archives/extractions	Full inventory, hash/version matrix, include/resource graph, missing dependency report
diff-references	Two or more build inventories	Added/removed/renamed/changed candidates with separate semantic review status
recode-reference-text	Raw file plus declared encoding recipe	UTF-8 derivative and source/output hash mapping; raw bytes retained
convert-assets	Named recipes and pinned inputs	Native model/image/font/config derivatives with provenance and validation
oracle-run	Pinned isolated oracle and experiment recipe	Captured commands, logs, exit status, state/file diffs, permitted fixture/capture payloads
nanobrush inspect/validate/convert	Legacy/native document and explicit format flags	Rust format handling with structured diagnostics/loss report and bounded processing
nanobrush replay	Native document and command/input trace	Deterministic semantic result plus state/geometry/render summaries
check-environment / probe-renderer	Host/toolchain and synthetic scene	Tool/SDK/native-library and adapter/backend/feature/limit report, no false support claims
package / verify-package	Git revision, target, release profile	Native package, checksums/notices, clean-machine launch and asset-lookup checks

Use Python for archive handling, inventory/provenance, and process orchestration; standard-library ZIP/hash/JSON support is sufficient for initial extraction. Put canonical geometry and document parsing/conversion in Rust and call its CLI from scripts to avoid two competing implementations. Pin any third-party image/font/recode tooling actually selected.

Every conversion recipe identifies its input hashes/version, destination, tool/code revision, options, source/destination encoding or color/coordinate conventions, output hashes, license/disposition, and validation rule. Hash inputs and recipes for incremental rebuilds; generate atomically and report stale/unexpected outputs. A force operation must have a bounded owned destination.

Rust module boundaries

Suggested crate names are provisional; combine them if separate crates would add overhead without isolation value.

Boundary	Owns	Must not depend on
nanobrush-geometry	Planes, brush construction, CSG operations actually specified, Patch9, triangulation, bounds/intersections	UI, GPU, filesystem state
nanobrush-document	Stable world/object IDs, properties, visibility/selection, native schema and migrations	Bevy transient entity identity
nanobrush-commands	Transactions, tool-independent edits, undo/redo, dirty state, deterministic replay	Widgets or render passes
nanobrush-formats	MB3 TXT/DAT, MAP/VMF, OBJ export, IXModel conversion, diagnostics/loss reporting	Original converter executable
nanobrush-assets	Asset manifests, recipes, validation, runtime lookup	Mutable reference originals
nanobrush-editor	Input/tool state machines, views/cameras/grids, UI, settings, native file workflows	Undocumented host-global behavior
nanobrush-render	Bevy meshes/materials/WGSL, diagnostic modes, overlays, Solari integration, capability policy	Authoritative editable document storage
nanobrush-cli	Inspection, conversion, validation, replay and headless integration	A GPU for ordinary document work
nanobrush-app	Native startup, plugin composition, lifecycle, packaging integration	Windows, .NET, IXPL runtime

The authoritative document lives outside render entities. Commands create transactional changes; background geometry jobs use document revision IDs and discard stale results. Stable object/face identity maps render meshes and pick results back to the document. UI preview is cancellable and creates a coherent undo step only on commit.

Separate document data from user preferences, per-view state, global UI layout, and ephemeral tool state. Determine actual reference ownership before selecting which state is saved in which file. A future plugin interface should expose these deliberate boundaries rather than all host filesystem/process privileges automatically.

Rendering and platform strategy

Keep editing diagnostics in a portable Bevy path. Required modes include grid, cursor, face/border/wireframe/normal/see-through displays and selection overlays, with consistent picking regardless of illumination. Solari supplies an additional lighting path over editable brush/patch meshes, rather than becoming the document or tool engine.

The early spike must probe required features and limits, create a real scene, perform mesh edits, and exercise multiple views on macOS Metal and Linux Vulkan. Bevy's tagged Solari source exposes a required-feature function; using it alone still needs runtime testing for limits and backend-specific failures. The M2 upstream report makes macOS adapter qualification a concrete task.

Maintain per-view temporal state. Geometry/material/visibility changes, undo, camera cuts, projection changes, resize, and viewport destruction must invalidate or update relevant lighting resources correctly. Measure BLAS/TLAS rebuild/update costs and memory under interactive edits. Editor grids/icons/gizmos must remain readable and must not accidentally cast lighting into the scene.

Use the upstream example as a spike reference for mesh/material/camera integration, but pin the selected release's API rather than copying moving main. DLSS is optional; document quality/performance on other hardware. Shader/material compatibility, orthographic rendering, denoising, and multiview behavior are experiments, not assumptions.

Validation layers

1. Preservation: exact hashes, CRCs, original-byte retention, safe extraction, and remote payload recovery.
2. Behavior: source/API coverage plus oracle experiments and semantic differential tests.
3. Core correctness: geometry invariants, parser fuzzing, round trips, undo/replay, numerical boundary tests.
4. Interaction: real input/focus/dialog/multiview workflows on both target operating systems.
5. Rendering: analytic visual cases, measured tolerances, named hardware/backend results, dynamic-edit stress.
6. Distribution: clean-machine install/launch, native runtime dependency audit, required assets/notices, offline operation, migration/recovery.

Keep synthetic independently authored tests available in code-only CI; run reference-corpus jobs separately when permitted and accessible. Never substitute software-rendered/headless CI for a claim that Solari works on actual target GPUs.