docs/TECHNICAL_DIRECTION.md

# Proposed technical direction

Storage, environment, hydration, and extraction are implemented; see [SETUP.md](SETUP.md). The `mb3-dat` and `mb3-txt` crates plus `nanobrush-cli` now implement native TXT/DAT format handling ([compatibility profile](RUST_DAT.md)); the broader Rust/rendering/conversion boundaries below remain proposed and must be validated through the [TODO phases](TODO.md).

## Storage and repository boundaries

Follow the active `urban-terror-rs` pattern selected by the user: one Hugging Face Git repository, normal Git source/tests, and Xet-backed assets and references. The initial separate-dataset proposal is superseded.

```text
nanobrush/                         # remote hf: git@hf.co:nmarshall/nanobrush
  .envrc, flake.nix, flake.lock     # pinned my-nix environment
  .gitattributes                   # code filter; assets/references use LFS + Xet
  .githooks/                       # staged-object/history guards plus LFS pre-push
  config/references.json           # authored preservation metadata; ordinary Git
  scripts/                        # setup, storage, hydration, extraction, checks
  tests/                          # ordinary Git test source
  docs/                           # ordinary Git authored findings/specifications
  crates/                         # native DAT/TXT libraries and nanobrush CLI
  assets/references/              # every file is Xet-backed, including small text
    archives/                     # unchanged ZIPs
    provenance/collection/        # original reports/manifest/checksum files
    provenance/historical/        # historical pages/indexes
    extracted/<build>/            # immutable raw members, including binaries
    inventories/<build>.json      # generated metadata payloads
    SHA256SUMS                    # relocated archive paths
```

Any additional `references/` directory, including root `references/` or nested `docs/references/`, is also LFS-filtered. Keep authored source/tests/docs outside these payload directories. All pre-existing collection downloads have been relocated beneath `assets/references/` and byte-verified.

There is no nested `.git`, dataset, separate asset branch, or asset lockfile. Each normal Git commit pins asset content through its LFS pointer SHA-256 values. `assets-fetch` hydrates the checked-out revision; `assets-check` verifies working bytes against staged pointers.

Use `filter=lfs` attributes plus Git Xet's custom transfer registration. All reference resources intentionally follow this policy regardless of size. Commit/push guards check staged and historical objects so raw payloads cannot slip into ordinary Git, even when a clean filter is bypassed. The LFS-specified empty-file representation carries no payload bytes.

Build outputs and disposable runtime copies remain ignored. Keep immutable references unchanged; transformations write separate derived assets and provenance. Nix derivation source filesets exclude payloads, while Git-backed flake source imports can still contain hydrated assets; use a skip-smudge clone for code-only CI.

## Environment shape

Use `flake-parts` and `github:nmrshll/my-nix`, following the current local exports `flakeModuleGroups.essentials` and `flakeModules.rust`. Keep nixpkgs and flake-parts follows consistent. Rust checks and the CLI package use the same pinned toolchain and explicit asset-free source fileset.

Pin remote inputs in `flake.lock`. An optional developer `.envrc` helper may watch local `my-nix` / tools checkouts and pass overrides without rewriting the shared lock; the committed flake remains usable without those checkouts. Separate platform-specific graphics libraries from portable archive/source tooling.

Provide a lightweight preservation/research shell first. Add the Rust workspace, native graphics dependencies, codec tools, and analysis tools as needed. Keep Windows/Wine/.NET oracle requirements in a separate documented environment; native Nanobrush must not inherit them.

## Planned commands and responsibilities

Setup/storage/hydration/reference commands and a converter-only Mono oracle are implemented. The native CLI supports inspect/validate/convert for TXT and both DAT versions, with a Nix package; other formats and editor/replay interfaces remain future work. The remaining asset-conversion, general oracle/replay and GUI-packaging interfaces below remain proposed.

| Command | Inputs | Output and acceptance contract |
| --- | --- | --- |
| `setup-repository` | Current checkout and committed policy | Repository-local filters/hooks; detect conflicting setup; no global config edits |
| `check-storage` | Index/history or source export | Reject code payloads, invalid pointers, unsafe entry types, or missing policy |
| `assets-check` / `assets-fetch` | Same-repo staged pointer identities / checked-out revision | Verified LFS/Xet setup and checksummed hydration; no separate asset checkout |
| `verify-references` | Pinned archives and original manifest | Original-byte hash/CRC results with missing/pointer-stub failures |
| `unpack-references` | Manifest, selected versions, explicit destination | Traversal-safe deterministic extraction into `assets/references/extracted/`; no execution |
| `index-references` | Archives/extractions | Full inventory, hash/version matrix, include/resource graph, missing dependency report |
| `diff-references` | Two or more build inventories | Added/removed/renamed/changed candidates with separate semantic review status |
| `recode-reference-text` | Raw file plus declared encoding recipe | UTF-8 derivative and source/output hash mapping; raw bytes retained |
| `convert-assets` | Named recipes and pinned inputs | Native model/image/font/config derivatives with provenance and validation |
| `oracle-run` | Pinned isolated oracle and experiment recipe | Captured commands, logs, exit status, state/file diffs, permitted fixture/capture payloads |
| `nanobrush inspect/validate/convert` | Legacy/native document and explicit format flags | Rust format handling with structured diagnostics/loss report and bounded processing |
| `nanobrush replay` | Native document and command/input trace | Deterministic semantic result plus state/geometry/render summaries |
| `check-environment` / `probe-renderer` | Host/toolchain and synthetic scene | Tool/SDK/native-library and adapter/backend/feature/limit report, no false support claims |
| `package` / `verify-package` | Git revision, target, release profile | Native package, checksums/notices, clean-machine launch and asset-lookup checks |

Use Python for archive handling, inventory/provenance, and process orchestration; standard-library ZIP/hash/JSON support is sufficient for initial extraction. Put canonical geometry and document parsing/conversion in Rust and call its CLI from scripts to avoid two competing implementations. Pin any third-party image/font/recode tooling actually selected.

Every conversion recipe identifies its input hashes/version, destination, tool/code revision, options, source/destination encoding or color/coordinate conventions, output hashes, license/disposition, and validation rule. Hash inputs and recipes for incremental rebuilds; generate atomically and report stale/unexpected outputs. A `force` operation must have a bounded owned destination.

## Rust module boundaries

Suggested crate names are provisional; combine them if separate crates would add overhead without isolation value.

| Boundary | Owns | Must not depend on |
| --- | --- | --- |
| `nanobrush-geometry` | Planes, brush construction, CSG operations actually specified, Patch9, triangulation, bounds/intersections | UI, GPU, filesystem state |
| `nanobrush-document` | Stable world/object IDs, properties, visibility/selection, native schema and migrations | Bevy transient entity identity |
| `nanobrush-commands` | Transactions, tool-independent edits, undo/redo, dirty state, deterministic replay | Widgets or render passes |
| `nanobrush-formats` | MB3 TXT/DAT, MAP/VMF, OBJ export, IXModel conversion, diagnostics/loss reporting | Original converter executable |
| `nanobrush-assets` | Asset manifests, recipes, validation, runtime lookup | Mutable reference originals |
| `nanobrush-editor` | Input/tool state machines, views/cameras/grids, UI, settings, native file workflows | Undocumented host-global behavior |
| `nanobrush-render` | Bevy meshes/materials/WGSL, diagnostic modes, overlays, Solari integration, capability policy | Authoritative editable document storage |
| `nanobrush-cli` | Inspection, conversion, validation, replay and headless integration | A GPU for ordinary document work |
| `nanobrush-app` | Native startup, plugin composition, lifecycle, packaging integration | Windows, .NET, IXPL runtime |

The authoritative document lives outside render entities. Commands create transactional changes; background geometry jobs use document revision IDs and discard stale results. Stable object/face identity maps render meshes and pick results back to the document. UI preview is cancellable and creates a coherent undo step only on commit.

Separate document data from user preferences, per-view state, global UI layout, and ephemeral tool state. Determine actual reference ownership before selecting which state is saved in which file. A future plugin interface should expose these deliberate boundaries rather than all host filesystem/process privileges automatically.

## Rendering and platform strategy

Keep editing diagnostics in a portable Bevy path. Required modes include grid, cursor, face/border/wireframe/normal/see-through displays and selection overlays, with consistent picking regardless of illumination. Solari supplies an additional lighting path over editable brush/patch meshes, rather than becoming the document or tool engine.

The early spike must probe required features and limits, create a real scene, perform mesh edits, and exercise multiple views on macOS Metal and Linux Vulkan. Bevy's [tagged Solari source](https://raw.githubusercontent.com/bevyengine/bevy/v0.20.0/crates/bevy_solari/src/lib.rs) exposes a required-feature function; using it alone still needs runtime testing for limits and backend-specific failures. The [M2 upstream report](https://github.com/bevyengine/bevy/issues/26131) makes macOS adapter qualification a concrete task.

Maintain per-view temporal state. Geometry/material/visibility changes, undo, camera cuts, projection changes, resize, and viewport destruction must invalidate or update relevant lighting resources correctly. Measure BLAS/TLAS rebuild/update costs and memory under interactive edits. Editor grids/icons/gizmos must remain readable and must not accidentally cast lighting into the scene.

Use the upstream example as a spike reference for mesh/material/camera integration, but pin the selected release's API rather than copying moving `main`. DLSS is optional; document quality/performance on other hardware. Shader/material compatibility, orthographic rendering, denoising, and multiview behavior are experiments, not assumptions.

## Validation layers

1. **Preservation:** exact hashes, CRCs, original-byte retention, safe extraction, and remote payload recovery.
2. **Behavior:** source/API coverage plus oracle experiments and semantic differential tests.
3. **Core correctness:** geometry invariants, parser fuzzing, round trips, undo/replay, numerical boundary tests.
4. **Interaction:** real input/focus/dialog/multiview workflows on both target operating systems.
5. **Rendering:** analytic visual cases, measured tolerances, named hardware/backend results, dynamic-edit stress.
6. **Distribution:** clean-machine install/launch, native runtime dependency audit, required assets/notices, offline operation, migration/recovery.

Keep synthetic independently authored tests available in code-only CI; run reference-corpus jobs separately when permitted and accessible. Never substitute software-rendered/headless CI for a claim that Solari works on actual target GPUs.