| # Rust MB3 TXT codec — F30 |
|
|
| `crates/mb3-txt` implements the supplied latest TXT grammar and the older property-free subset, using the existing `mb3_dat::World` model. TXT has no format-version header; the reader does not guess a historical release. The [CLI](CLI.md) converts TXT to either explicit DAT version. The only Cargo dependency is the local DAT model/codec crate; no third-party runtime packages, original converter, graphics stack or .NET runtime are required. |
|
|
| ## Public API |
|
|
| ```rust |
| use mb3_dat::{Limits, World}; |
| let world = World::default(); |
| let bytes = mb3_txt::encode(&world, &Limits::default())?; |
| assert_eq!(mb3_txt::decode(&bytes, &Limits::default())?, world); |
| # Ok::<(), mb3_txt::Error>(()) |
| ``` |
|
|
| `decode` / `encode` work on bytes; `read` / `write` support standard Rust I/O, short operations and `Interrupted`. Input must be UTF-8, with an optional leading BOM. Output is canonical **UTF-8 with a BOM**, LF line endings and shortest round-tripping finite f32 decimals. Invariant number formatting preserves finite f32 values, signed zero and subnormals. Comments, whitespace, original decimal spelling and brush/patch interleaving are not retained: the model stores brushes and patches separately, and the writer emits brushes first. Names, their original table order, unused names, property indices, empty versus absent values, and all explicit geometry arrays are retained. |
|
|
| The parser accepts line/block comments, exact case-sensitive keywords, signed decimal integers, decimal-point/exponent floats, ordinary strings and the six supplied escape forms (`\r`, `\n`, `\t`, `\'`, `\"`, `\\`). Single quotes inside strings must be escaped, as in the supplied grammar. Strings may contain Unicode directly; general `\u`, hex and unknown escapes are rejected. Non-nested block comments terminate at the first closing delimiter. Line comments may end at EOF. Unknown fields, trailing objects/text, malformed punctuation and duplicate property names/slots are errors. Sparse slots can arrive out of index order and are sorted after parsing; writer input requires unique increasing indices. |
|
|
| Plane tuples are P0/U/V, and UV tuples are axis/scale/shift pairs; the shared model names these fields to avoid confusing them with DAT's ten-float wire order. Dimension/slot integers must be nonnegative i32 values. Numeric tokens are capped at 128 bytes; integer tokens with leading zeroes are rejected rather than imitating possible token splitting by the historical lexer. Finite floats can have leading zeroes before a decimal point. No expression language, recursive nesting of brushes/worlds, general parser AST, geometry calculation or property optimization is introduced. |
|
|
| Computed features are a consistency contract: each plane must have a polygon iff the world flag is true, and each patch must have both points/normals iff true. Anchor lengths must equal the dimensions' product; point and normal lengths must equal the recovered grid formula. Missing arrays and optional computed arrays under a false flag are rejected, instead of being synthesized, ignored or serialized incoherently. This validates structure, not convexity, geometric degeneracy, normal lengths, patch smoothness or editor usability. Zero/even dimensions remain representable with matching arrays. |
|
|
| ## Resource and error policy |
|
|
| The API shares `mb3_dat::Limits`: 64 MiB input/output, 1 MiB per decoded UTF-8 string, 1,000,000 aggregate logical elements and 128 MiB requested decoder-owned storage by default. Limits are caller policy. Element counts cover names, parent records, sparse entries and vector-array elements. Geometric vector/string capacity growth and string payloads are charged before fallible reservations. Name uniqueness uses a fallibly allocated, budgeted index scratch array and sorting; duplicate slots sort once rather than repeatedly scanning earlier entries. |
|
|
| Storage accounting includes decoder capacity and name-validation scratch requests, excluding the caller's input, allocator overhead and transient reallocations. `read` owns a separate bounded input buffer. Writer output has its own byte limit; writer-owned name-validation scratch uses the owned-storage limit, while the caller already owns the model. These are explicit accounting limits, not a precise RSS limit. There is no recursive input traversal. All numeric sums/products needed for array sizes are checked. String scanning never loops at EOF. |
|
|
| `Error` carries a byte offset and typed `ErrorKind`, including syntax, UTF-8, escape, number, limits, allocation, property, computed-data and I/O errors. Byte positions include an input BOM. `write` validates and buffers before writing; it does not flush or make file writes atomic. Later I/O failure can leave a prefix; use the CLI's publication workflow or implement the equivalent at the caller. |
|
|
| ## Compatibility differences |
|
|
| | Area | Recovered converter | Native TXT codec | |
| | --- | --- | --- | |
| | Encoding | Guesses Latin-1 without an initial UTF-8 marker; its BOM-less UTF-8 output can drift on reread | Strict UTF-8 input; BOM output protects Unicode when handed to the latest converter's documented detection path | |
| | ASCII DAT conversion | Replaces non-ASCII with `?` | TXT retains Unicode; conversion to DAT fails explicitly, preserving source/destination | |
| | Property tables | Latest CLI drops unused names and reorders by usage | Retains table order/unused names and index meaning | |
| | Duplicate names | DAT can contain duplicate names, TXT loader rejects them | TXT reader/writer rejects them; DAT→TXT can therefore fail on an otherwise valid DAT document | |
| | Computed flag | False can discard optional arrays; true can crash if arrays omitted | Requires arrays to agree with flag; never silently drops supplied data | |
| | Patch lengths | TXT constructors can retain inconsistent arrays and emit invalid DAT | Checks exact array lengths with checked products | |
| | Numbers | Invariant f32 parsing; runtime-dependent default printing, no explicit finite policy | Finite f32 only, shortest round-tripping output, 128-byte token limit, explicit integer bounds | |
| | Historical syntax | 2015 grammar lacks names/strings/comments | Reader is the latest superset; property-free output omits tables and comments so it fits the old syntax | |
| | Formatting | Historical layout/newlines/decimal output | Stable canonical layout and BOM; semantic TXT comparison, not byte-identical text | |
|
|
| Native Unicode output follows the recovered grammar/encoding contract; a broad new Mono/Windows output-consumption campaign was not performed in this stage. The E02 Unicode inputs demonstrate the legacy encoding difference. There is no lossy conversion switch: callers must explicitly transform unsupported data before a DAT downgrade or Unicode-to-ASCII export. |
|
|
| ## Fixture validation (E04) |
|
|
| - All **58 DAT output fixtures** decode, serialize to TXT, reparse and encode back to the same DAT version with **exact original bytes**. This includes UV/property ordering, computed brush/patch data, old/new hierarchy differences and compact widths. |
| - All **seven converter TXT outputs** parse and stabilize under native canonicalization; text bytes intentionally differ. |
| - **16 TXT-input/DAT-output pairs** match exact converter DAT bytes where the input did not undergo property optimization or character loss. |
| - **Nine** invalid or silently lossy TXT source fixtures are rejected, including duplicate names/slots, bad indices, missing computed arrays, syntax errors and ignored polygons. |
| - The property-optimization fixture retains `[unused, rare, common]`, while the original CLI emitted `[common, rare]`; Unicode source fixtures retain `é`, while DAT conversion refuses replacement and the historical BOM-less path drifted. |
|
|
| Twelve unit tests cover lexer errors, comments, exponent syntax, escapes/UTF-8, finite extremes, signed zero, proper-prefix truncations, all resource policies, array consistency, streaming I/O, 3,000 bounded deterministic mutations and rejection of recursive-looking syntax. Five integration tests require hydrated E02 fixtures. Coverage-guided fuzzing, large real TXT files, explicit legacy Latin-1 recoding, native-editor interoperability and cross-platform execution remain open. See [E04](experiments/txt-cli-2026-10-21.md) for reproduction. |
|
|
