Rust DAT codec and compatibility profile — F29
Implemented in crates/mb3-dat, a dependency-free Rust library in the root Cargo workspace. The binary contract remains the format specification. This implementation handles versions 201501280 and 201512200; it does not require .NET, IXPL, Bevy, a GPU or the original converter at runtime. The package is unpublished pending the project's source-license decision.
API and data model
use mb3_dat::{read, write, Document, Limits, Version};
fn migrate_property_free_file(input: impl std::io::Read,
output: impl std::io::Write)
-> Result<(), mb3_dat::Error>
{
let limits = Limits::default();
let mut document: Document = read(input, &limits)?;
document.version = Version::V201512200;
write(output, &document, &limits)
}
decode(&[u8], &Limits) detects the header version; encode(&Document, &Limits) writes the selected version. read buffers a bounded input and handles short/interrupted reads; write completes encoding/validation before touching the caller's writer, handles short/interrupted writes, and reports I/O failures with a completed-prefix byte offset. write does not flush the caller's writer. It cannot make filesystem writes atomic: callers must implement temporary-file/rename transactions. The current library does not open or overwrite files itself.
Document contains Version and World. World retains the computed flag, separate ordered string/UV name tables, brushes and patches. Plane triples are P0/U/V, not three absolute points. Uv names both axes, scales and shifts; serialization uses the recovered ten-float wire order. Property<T> stores an index and value in a sparse vector, retaining absent versus present-empty values. Decode sorts entries by index; encode requires unique, increasing indices. Table order, unused names, duplicate table names, object order and finite f32 bits (including negative zero and subnormals) are preserved. Property tables are not optimized or sorted by usage.
Each plane has polygon: Option<Vec<Vec3>>; each patch has computed: Option<PatchComputed>. These must be present exactly when the world flag is true, even for an empty polygon. Patch anchor lengths must equal anchors_u * anchors_v. Computed points and normals must each match the implicit grid-size formula in the specification. Encode rejects inconsistent models rather than dropping arrays or producing an incoherent file. This is structural serialization validation, not convexity, brush validity, normal-length or patch evaluation. Zero/even anchor dimensions and zero tessellation remain representable when their array lengths agree, including the unusual width-isolation converter fixtures.
Errors expose ErrorKind and offset; I/O causes remain available through std::error::Error::source. There is no permissive recovery mode. A version change on a decoded property-free document performs actual hierarchy migration. Selecting the old version with property names or plane property values returns LegacyProperties; callers must make and report any loss decision themselves.
Intentional differences from the converters
| Area | Legacy behavior (E02) | Rust behavior |
|---|---|---|
| Version dispatch | Each executable accepts only its own DAT version | Reader automatically handles either; writer uses explicit Document.version |
| Short numeric reads | Reuses stale buffer bytes and can succeed with corrupt values | Every field must be complete; every tested proper prefix is rejected |
| Unterminated strings | EOF can create an unbounded byte list | Terminator, per-string limit, input limit and allocation budget checked |
| ASCII | Non-ASCII writes become ?; high-bit bytes decode lossily |
Rejects non-ASCII on read/write; rejects NUL in in-memory strings |
| Float payloads | No explicit finiteness checks | NaN/infinity rejected; finite binary32 values retained bit-for-bit |
| Duplicate slots | Binary last-wins | Rejected; duplicate table names remain legal and index-distinct |
| Slot ordering | Accepts arbitrary order | Accepts unique arbitrary wire order, returns sorted sparse entries; requires sorted entries when writing |
| Reserved flags/selector 3 | Ignored or treated as a four-byte selector | Rejected, including an old boolean byte other than 0/1 |
| Valid nonminimal widths | Accepted and minimized on save | Accepted and minimized on save; original width bytes are not retained |
| Trailing bytes | Ignored/dropped | Rejected; reader expects one complete document |
| Property normalization | Latest CLI removes unused names/reorders used ones | Preserves names/order/indices; no hidden normalization |
| 2015 counts 32,768–65,535 | Writer chooses signed i16 and produces negative reader counts | Writer chooses i32; reader rejects the malformed signed-i16 representation |
| Computed data | TXT can emit missing/wrong-sized arrays; no evaluation | Writer requires consistent arrays; no synthesis or geometry evaluation |
| Count arithmetic/allocation | Unchecked products/counts and allocations | Checked products/sums, wire-length checks, aggregate budgets, fallible reservation |
| Save behavior | File.Create may truncate before a later failure |
No I/O until validation/encoding succeeds; later I/O failure can still leave a prefix |
The legacy count repair applies consistently to plane counts, shared contour widths and paired patch dimensions. It intentionally changes bytes where the old writer was broken, while using the four-byte branch already present in the recovered old reader. Rust synthetic boundary tests cover this; direct execution of the legacy converter on the repaired large-count output remains an interoperability follow-up. Existing E02 outputs within the old valid count range are byte-identical after Rust round trips.
Resource policy
Defaults are application policy, not file-format maxima:
| Limit | Default | Scope |
|---|---|---|
max_bytes |
64 MiB | Complete input and encoded output; streaming input observes at most one excess byte |
max_string_bytes |
1 MiB | Each ASCII name/value, excluding terminator |
max_items |
1,000,000 | Aggregate names, brushes, planes, sparse entries, patches and vector-array elements |
max_owned_bytes |
128 MiB | Decoder's requested owned record/array/sparse-capacity/string storage |
Decoded storage budget excludes caller-owned input, allocator overhead and transient reallocations. Streaming read also owns its input buffer, bounded separately by max_bytes; encoding owns an output buffer bounded by max_bytes. max_owned_bytes does not limit an already caller-owned document or the encoded buffer. These are explicit accounting limits, not an exact process-RSS guarantee. All heap reservations in codec parsing/serialization are fallible; unsafe is forbidden by workspace lint. Integer fields must fit their version's nonnegative count representation. Sparse vector growth is geometric, capped by the table size, with capacity charged to the decoder budget; duplicate checking sorts once instead of repeatedly scanning prior entries. There is no recursive input traversal.
Reproduction and evidence
Run inside the pinned Nix shell:
cargo fmt --all --check
cargo clippy --workspace --all-targets --locked --offline -- -D warnings
analyze-converter check
cargo test --workspace --locked --offline
The converter integration test requires hydrated E02 assets under assets/analysis/converter/probes. Missing files, pointer stubs or changed corpus counts fail; no fixture test silently skips. An explicit alternate corpus path can be supplied with NANOBRUSH_CONVERTER_FIXTURES. analyze-converter check verifies the original SHA-256-indexed captures before Rust tests; the fixtures remain Xet payloads and the test source remains ordinary Git.
Validation:
- All 58 E02 DAT output files (25 old, 33 latest) decode and re-encode byte-for-byte, including computed brush/patch arrays, sparse UV/string data, width transitions and normalized outputs from malformed legacy inputs. A valid normalized output cannot reveal or repair corruption already introduced by the converter; the captured truncated source inputs are separately rejected.
- Five integration tests cover the entire output corpus, asymmetric vectors/property/UV/patch semantics, 11 captured malformed/ambiguous inputs, automatic cross-version detection, explicit lossy-downgrade refusal and computed property-free migration matching both converters' bytes, and mixed nonempty brush/patch worlds assembled independently from captured record bytes.
- 24 synthetic unit tests cover headers, every proper prefix of rich documents, binary ordering/duplicates, limits, string policies, computed consistency, all compact count boundaries through 65,536 (planes, names/UV slots, contours and patch dimensions), arithmetic overflow, finite extremes/non-finites, short/interrupted I/O, partial I/O errors, validation before writing, reserved bits and normalization of valid nonminimal widths. A deterministic 5,000-mutation test checks rejection or stable canonical round trips under tight limits; this is a smoke test, not coverage-guided fuzzing.
- One compiled API doctest exercises an empty latest world.
The DAT crate’s 24 unit tests and one doctest run inside the asset-free checks.<system>.rust workspace check, which now includes TXT/CLI tests (44 unit tests and two doctests in E04). Formatting and Clippy cover all targets. Full integration tests run separately against hydrated, verified captures. The setup derivation still runs the 52 Python regressions and code-tree storage checks. Only macOS ARM64 execution is validated in this stage; Linux/Intel execution, native editor interoperability, real user documents, long-running fuzzing, hostile-stream timeouts, full crash-durability workflows and the remaining MAP/VMF/OBJ codecs stay open. TXT serialization and the headless CLI are now implemented in F30/F31. The declared Rust minimum is 1.85; the measured pinned toolchain is 1.99.0, not a separate minimum-version validation.