nanobrush / docs /MB3_DAT.md
nmarshall's picture
formats: add safe dat codecs
918c06e
|
Raw History Blame Contribute Delete
10.1 kB
# MB3 DAT binary contract — F24/F25
Status: **observed managed implementation plus 101 controlled Mono probes across two distinct binaries**, E02, 2026-10-21. This specifies the converter's format. The native editor's reader/writer has not been traced or tested. See [converter behavior](CONVERTER.md), [experiment and reproduction](experiments/converter-2026-10-21.md), and [raw identities/method index](../assets/analysis/converter/static.json).
## F24 — Binary identity and anchors
All paths below are within `assets/references/extracted/<build>/tools/Mb3 file formats.exe`.
| Build(s) | SHA-256 | Size | Embedded tool version | Required DAT version |
| --- | --- | ---: | --- | ---: |
| All four 2015 archives | `66f1e3d9d53acfbe12a2be0e2a2bde64509d61d451ba7773bda68cff9d54abe5` | 78,336 | 1.4.0 @ 2015-05-02 | 201501280 |
| 2016-03-10-c | `0789c98b4150f281f92181a3368c86f5a81e033b7ecf58b8d320c5365546b5a6` | 91,136 | 1.5.0 @ 2015-12-20 | 201512200 |
Both are PE32/I386 (`machine=0x14c`), CLR header RVA `0x2008`, file offset `0x208`, CorFlags `0x3` (IL-only plus 32-bit-required), entrypoint token `0x06000001`. The flag names/values follow [Microsoft’s CorFlags definition](https://learn.microsoft.com/en-us/dotnet/api/system.reflection.portableexecutable.corflags). Assembly `Project` version `1.0.0.0` targets .NET Framework 4.0. Assembly references are `mscorlib`, `System`, `System.Core`, all version 4.0.0.0. The sole PE import is the managed bootstrap `mscoree.dll!_CorExeMain`. ManifestResource and ImplMap tables are empty: no embedded managed resources or P/Invoke declarations were found. This does not inventory native editor dependencies.
RVA means the method start, not an offset within decompiled C#. All IL instruction offsets and 183/193 method entries are retained in the Xet evidence.
| Method in `Shrinker.Mb3.World` unless qualified | Latest RVA | 2015 RVA |
| --- | --- | --- |
| `Shrinker.Main_.Main` | `0x2050` | `0x2050` |
| `Optimize` | `0x365c` | absent |
| `LoadFromDATFile` | `0x3a80` | `0x32cc` |
| `LoadFromTXTFile` | `0x40b0` | `0x3698` |
| `SaveToDATFile` | `0x4274` | `0x37c4` |
| `SaveToTXTFile` | `0x4ad0` | `0x3c94` |
| TXT `processBrush` | `0x61e4` | `0x4ffc` |
| TXT `processPatch9` | `0x6594` | `0x516c` |
| `read` | `0x6f14` | `0x5b0c` |
| `readVec3` | `0x6f42` | `0x5b3a` |
| `Shrinker.Common.Common.readBytesTo0Terminator` | `0x2740` | absent |
## Shared primitives
On the recovered Windows targets and measured ARM64 Mono host, integer and IEEE-754 binary32 float bytes are **little endian**. The implementation delegates to `BitConverter`, with no explicit endian conversion; a hypothetical big-endian runtime is not specified. A vector is three f32 values, X then Y then Z (12 bytes). No alignment padding is inserted.
A two-bit width selector `W` means:
| W | Latest reader | 2015 reader | Writer selection |
| --- | --- | --- | --- |
| 0 | u8 | u8 | 0–255 |
| 1 | u16 | **i16** | 256–65,535 |
| 2 | i32 | i32 | larger |
| 3 | i32 | i32 | never emitted |
The 2015 writer chooses the two-byte branch up to 65,535 but casts to signed `short`; its own reader sees 32,768–65,535 as negative. The `signed-16-count` probe confirms rejection at 32,768. The new tool fixes these two-byte operations to unsigned. Selectors and width thresholds were measured at 255/256 planes, latest 255/256 property names, contour counts, and patch tessellation dimensions 256/65,536; large plane/property 65,535/65,536 thresholds are established statically, not allocation-scale runtime tests.
`C(W)` below is an integer using that selector. Counts should be nonnegative; the original does not consistently validate them. Reserved selector 3 is accepted as four bytes; unassigned flag bits are ignored. The writer emits minimal width and clears unused bits.
## Latest layout: version 201512200
Fields occur sequentially with no footer, checksum, offsets, IDs, compression or chunk tags.
| Order / initial offset | Field |
| --- | --- |
| 0–7 | ASCII bytes `Mb3World`, no terminator |
| 8–11 | i32 version, exactly `201512200` (`08 d5 02 0c`) |
| 12 | World flags: bit 0 computed features; bits 1–2 string width `S`; bits 3–4 UV width `T`; bits 5–7 unused |
| 13 onward | `C(S)` string-property count; that many ASCII zero-terminated names |
| following | `C(T)` UV-property count; that many ASCII zero-terminated names |
| following | i32 brush count, then brush records |
| following brushes | i32 patch count, then Patch9 records |
ASCII encoding is lossy: non-ASCII characters become `?`. Names and string values are terminated by a single zero byte, with no length field. An empty value is distinct from an absent slot. Names are indexed by table position. DAT loading does not check name uniqueness; TXT loading does. If a table is empty, its per-plane list is entirely absent, including its terminator.
An empty, uncomputed latest world is 23 bytes:
```text
4d 62 33 57 6f 72 6c 64 08 d5 02 0c 00 00 00 00 00 00 00 00 00 00 00
```
### Brush and plane records
Each brush begins with one flag byte: bits 0–1 select plane-count width `P`; bits 2–3 select contour-count width `V`. Then `C(P)` plane count and plane records. `V` is shared by **every plane in that brush**, chosen from its largest contour count if computed features are present.
Each plane contains, in this order:
1. If the string table is nonempty: repeated `C(S)` index + zero-terminated ASCII value, ended by all-ones index (`ff`, `ffff`, or `ffffffff`). No separate entry count.
2. If the UV table is nonempty: repeated `C(T)` index + ten f32 values, ended by the same width-dependent all-ones sentinel.
3. Three vectors: `P0`, `U`, `V` (36 bytes). These are an **origin and two direction vectors**, not three absolute plane points. TXT transfers these directly. MAP/VMF point conversion is specified separately.
4. Only when world computed bit is set: `C(V)` contour-vertex count and that many vectors. No contour normal, triangle indices or extra geometry metadata follows.
UV's ten floats are **Ux Uy Uz Vx Vy Vz UScale UShift VScale VShift**. This differs from TXT's two five-number tuples, which interleave each axis with its scale and shift. Probe `properties-to-dat` uses ten distinct numbers to distinguish the order.
Valid property indices range from zero to table count minus one. The maximum index of each compact width is reserved for the terminator, which is compatible with count 255 using indices 0–254 and count 65,535 using indices 0–65,534. Binary duplicate slots overwrite earlier values; TXT duplicate slots produce an error. DAT out-of-range indices reach an array access rather than a format-specific diagnostic.
### Patch9 records
Each patch begins with a flag byte: bits 0–1 select anchor-dimension width `A`; bits 2–3 select tessellation-dimension width `Q`. Then:
```text
C(A) anchorsU
C(A) anchorsV
C(Q) quadsPerTileU
C(Q) quadsPerTileV
vec3 anchors[anchorsU * anchorsV]
if computed:
vec3 points[N]
vec3 normals[N]
```
`N = (floor(anchorsU / 2) * quadsPerTileU + 1) * (floor(anchorsV / 2) * quadsPerTileV + 1)` for nonnegative dimensions. Anchors use `u + anchorsU * v` (U varies fastest); points use U-fastest rows in OBJ export. Points and normals are separate arrays, not interleaved. There are no explicit lengths for any patch vector array, material fields or UV fields.
The DAT reader derives lengths from dimensions. The TXT loader and constructors do not establish those consistency constraints: the DAT writer emits the actual anchor array and emits points/normals whenever `Points != null`, without checking the world flag or expected size. Controlled normal inputs agree; malformed TXT can therefore yield incoherent DAT. The 3×3, 2×2 computed fixture has nine anchors, nine points, nine normals. This converter does **not** evaluate a patch or generate a brush contour; it copies supplied computed geometry.
## Older layout: version 201501280
Header remains eight-byte `Mb3World` + i32 exact version (`60 aa 02 0c`). Offset 12 is a boolean byte: **any nonzero byte** means computed. There are no property tables or per-plane property lists.
At offset 13 come **both** i32 brush count and i32 patch count, followed by all brush records and then all patch records. Brush flags, plane origin/directions, computed contours and patch shapes otherwise use the same structure, with the signed two-byte difference above. Empty world size is 21 bytes; a single uncomputed one-plane brush is 59 bytes (latest: 61).
Both converters reject the other DAT version at the header. Renaming the extension or merely changing the version number is not migration. A future migrator must decode the correct hierarchy and re-encode the target structure, with an explicit loss policy for properties going to the old format.
## F25 — Validation boundaries and rewrite requirements
Observed probes establish empty/computed worlds, asymmetric plane vectors, sparse properties/UV order, property optimization, computed contours, Patch9 arrays, compact-width transitions, accepted extra bytes, cross-version rejection, duplicate slots, and short-read corruption. See [F28](CONVERTER.md#f28--failure-and-filesystem-contract) for failure details.
The research [inspector](../scripts/inspect_mb3dat.py) independently reads both layouts and reports offsets and trailing bytes. It deliberately checks every read and bounds allocation: 64 MiB input and 100,000 elements per count/vector array are **research-tool limits**, not recovered format maxima. It neither validates convex geometry nor writes files nor claims Rust compatibility.
The implemented [Rust codec](RUST_DAT.md) uses explicit LE primitives, checked arithmetic, finite-number policy, bounded allocations, checked string termination/indexing and complete reads. It rejects unknown bits and trailing data and does not reproduce stale-buffer behavior. Both DAT versions pass all 58 converter-output byte round trips. Native editor compatibility, real user documents, non-finite floats, enormous counts, embedded NUL handling, malformed patch lengths and broader fuzzing remain P6.19/P6.20 work.