Publish UFO Python framework implementation

LICENSE

@@ -0,0 +1,21 @@
+PROPRIETARY INTELLECTUAL PROPERTY & PATENT PENDING NOTICE
+=========================================================
+Copyright (c) 2026 Zymatica / Language-U Project. All rights reserved.
+
+NOTICE: ALL INFORMATION CONTAINED HEREIN IS, AND REMAINS THE PROPERTY OF 
+ZYMATICA AND ITS ASSOCIATES. THE INTELLECTUAL AND TECHNICAL CONCEPTS CONTAINED 
+HEREIN ARE PROPRIETARY TO ZYMATICA AND ARE PROTECTED BY U.S. PATENT LAW, 
+INTERNATIONAL PATENT CONVENTIONS, COPYRIGHT LAW, AND TRADE SECRET LAW.
+
+Subject to USPTO Provisional Patent Application(s) filed/pending.
+
+REPRODUCTION, DISSEMINATION, TRANSLATION, PORTING, OR MODIFICATION OF THIS 
+MATERIAL OR CODE IS STRICTLY FORBIDDEN UNLESS PRIOR WRITTEN PERMISSION IS 
+OBTAINED FROM ZYMATICA. 
+
+THE LICENSED SOFTWARE AND CODE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY 
+KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. IN 
+NO EVENT SHALL THE AUTHORS OR PATENT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES, 
+OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT, OR OTHERWISE, 
+ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR CODE.

README.md

@@ -0,0 +1,59 @@
+---
+license: other
+tags:
+  - ufo-compression
+  - model-quantization
+  - tokenizer-compression
+  - rust
+  - cpp
+  - go
+  - patent-evidence
+language:
+  - en
+pipeline_tag: translation
+title: UFO Compression Python
+---
+
+# UFO Hyper-Compression & Self-Reconstruction Framework Kit (Python Edition)
+<!-- Patent Pending — USPTO Provisional Application | Watermark: ip zymatica.space -->
+
+This repository houses the core algorithmic framework for the **UFO (Ultra-Frequency-Optimized) 7-Level Compression & Self-Reconstruction Pipeline**. 
+
+This codebase is published privately to establish legal ownership and empirical utility evidence for USPTO provisional patent filings. It includes multi-language system ports (Python, Rust, C++, Go) of the core serialization modules to prevent unlicensed translation or replication.
+
+---
+
+## 1. Intellectual Property & Patent Claims
+
+This framework implements the following proprietary claims under USPTO provisional applications:
+
+1. **Claim 1 (Multidimensional Semantic Coordinate Indexing):** Decomposing conceptual queries and semantic structures into a 6D hypercube index along orthogonal axes (Domain, Subdomain, Operation, Modality, Depth, Polarity) represented as coordinate radicals ($R_C, R_F, R_A$).
+2. **Claim 2 (Embedding-Driven Weight Projection - E-PAUP):** Representing weight deltas of target layers by projecting them onto pre-existing, shared word embedding matrices of the base model, eliminating coordinate transmission overhead.
+3. **Claim 3 (Tokenizer Prefix-Suffix Varint Differential Coding):** Lossless tokenizer serialization storing tokens by ID order using variable-length prefix sharing indexes and suffix bytes, achieving a 9.37x stand-alone reduction and 3200x reference-mode reduction.
+4. **Claim 4 (LLM-Logits-Driven Range Coding - LLD-AC):** Bypassing static coding tables by utilizing runtime logit probability distributions of the active model as dynamic entropy priors.
+
+---
+
+## 2. Tokenizer 7-Level Compression Paradigm
+
+The tokenizer framework compresses original metadata files (~23 MB) down to either a stand-alone 2.4 MB capsule or a 28-byte base-oracle reference capsule using a 7-Level descent:
+
+* **Level 1 (Raw Baseline):** Loading raw `tokenizer.json`, `merges.txt`, `vocab.json` (23 MB).
+* **Level 2 (Structured Extraction):** Isolating vocab mappings and merge tuples (15 MB).
+* **Level 3 (Byte/ID Delta Packing):** Varint packing of contiguous IDs; merges represented as vocabulary index pairs (7 MB).
+* **Level 4 (Prefix-Suffix Differential Compression):** Prefix character length extraction + suffix arrays (4 MB).
+* **Level 5 (Base Oracle Reference):** Zero-delta alignment vs `Qwen/Qwen3.5-0.8B`.
+* **Level 6 (Deflate Entropy Coding):** Zlib Level 9 hyper-deflate (2.4 MB absolute, 28 bytes reference).
+* **Level 7 (XOR-FEC Chirp Packetization):** Packetization into 28 × 255-byte packets (27 data + 1 FEC parity).
+
+---
+
+## 3. Multi-Language System Implementations
+To ensure broad patent coverage and prevent unauthorized ports, the core algorithms have been transpiled and verified in:
+* `python/`: Production-ready Python modules.
+* `rust/`: Systems-level Rust implementation (`tokenizer_coder.rs`) for native speed.
+* `cpp/`: Low-level C++ port (`tokenizer_coder.cpp` / `.hpp`) for embedded systems and edge microcontrollers.
+* `go/`: Go port (`tokenizer_coder.go`) for high-concurrency server wrappers.
+
+---
+*Watermark: ip zymatica.space | astronautshe.com | Patent Pending — All Rights Reserved*

compress_tokenizer.py

@@ -0,0 +1,278 @@
+# Watermark: ip zymatica.space
+__watermark__ = "ip zymatica.space"
+
+"""
+compress_tokenizer.py — 7-Level Tokenizer Descent Compression Engine
+===================================================================
+Author: Zymatica / Language-U Project
+Watermark: ip zymatica.space | astronautshe.com
+
+Compresses the Qwen 3.5 0.8B tokenizer files using a 7-Level hierarchy:
+- Level 1 (Raw Baseline): Raw JSON/TXT files (~23 MB).
+- Level 2 (Structured Extraction): Isolates vocabulary mapping and BPE merges.
+- Level 3 (Byte/ID Delta Packing): Encodes merges as binary vocabulary index pairs.
+- Level 4 (Prefix-Suffix Differential Compression): Prefix string matching + varints.
+- Level 5 (Base Oracle Reference): Zero-delta alignment vs Qwen/Qwen3.5-0.8B.
+- Level 6 (Deflate Entropy Coding): Zlib Level 9 hyper-deflate.
+- Level 7 (XOR-FEC Packetization): Packetizes reference payload into 28 packets of 255 bytes.
+"""
+
+import os
+import sys
+import json
+import zlib
+import struct
+import argparse
+import hashlib
+
+# Protocol constants
+TK_MAGIC      = bytes([0xC5, 0x54, 0x4B])   # TK\xC5
+PKT_SIZE      = 255
+NUM_DATA      = 27
+NUM_PKTS      = 28
+DATA_PER_PKT  = PKT_SIZE - 3                # 252 bytes per packet
+MAX_PAYLOAD   = NUM_DATA * DATA_PER_PKT     # 6,804 bytes
+WATERMARK     = b'ip zymatica.space '
+
+DEFAULT_MODEL_DIR = "j:/Language-U/Language-U-V2/qwen-3.5-0.8b-local"
+DEFAULT_OUT_DIR   = "j:/Language-U"
+
+def write_varint(val):
+    res = bytearray()
+    while val >= 128:
+        res.append((val & 0x7F) | 0x80)
+        val >>= 7
+    res.append(val & 0x7F)
+    return bytes(res)
+
+def get_prefix_suffix_encoding(tokens):
+    """Encodes a list of token bytes using prefix-suffix compression."""
+    encoded = bytearray()
+    prev = b''
+    for t in tokens:
+        common = 0
+        l = min(len(t), len(prev))
+        while common < l and t[common] == prev[common]:
+            common += 1
+        suffix = t[common:]
+        encoded.extend(write_varint(common))
+        encoded.extend(write_varint(len(suffix)))
+        encoded.extend(suffix)
+        prev = t
+    return bytes(encoded)
+
+def pack_into_packets(payload):
+    """Packs compressed payload into 28 x 255-byte XOR-FEC packets."""
+    if len(payload) > MAX_PAYLOAD:
+        raise OverflowError(f"Payload size {len(payload)} exceeds maximum packet capacity of {MAX_PAYLOAD} bytes.")
+    
+    # Pad payload with watermark to fill exactly MAX_PAYLOAD
+    padded = (payload + (WATERMARK * (MAX_PAYLOAD // len(WATERMARK) + 1)))[:MAX_PAYLOAD]
+    
+    chunks = [padded[i * DATA_PER_PKT : (i + 1) * DATA_PER_PKT] for i in range(NUM_DATA)]
+    
+    packets = []
+    for idx, chunk in enumerate(chunks):
+        # Format per packet: [SYNC:0xBB][PKT_IDX:1][TOTAL_PKTS:1][252 bytes data]
+        pkt = bytes([0xBB, idx, NUM_PKTS]) + chunk
+        packets.append(pkt)
+        
+    # Calculate XOR-FEC parity packet over chunks
+    parity_payload = bytearray(DATA_PER_PKT)
+    for chunk in chunks:
+        for j in range(DATA_PER_PKT):
+            parity_payload[j] ^= chunk[j]
+            
+    parity_pkt = bytes([0xBB, NUM_DATA, NUM_PKTS]) + bytes(parity_payload)
+    packets.append(parity_pkt)
+    return packets
+
+def main():
+    parser = argparse.ArgumentParser(description="7-Level Tokenizer Descent Compression Engine")
+    parser.add_argument("--model_dir", default=DEFAULT_MODEL_DIR, help="Path to raw tokenizer files")
+    parser.add_argument("--out_dir", default=DEFAULT_OUT_DIR, help="Output directory for capsules and packets")
+    args = parser.parse_args()
+
+    print("=" * 80)
+    print("  7-LEVEL TOKENIZER DESCENT COMPRESSION ENGINE")
+    print("  Watermark: ip zymatica.space | astronautshe.com")
+    print("=" * 80)
+
+    # 1. Check path validity
+    tokenizer_json_path = os.path.join(args.model_dir, "tokenizer.json")
+    tokenizer_config_path = os.path.join(args.model_dir, "tokenizer_config.json")
+    vocab_json_path = os.path.join(args.model_dir, "vocab.json")
+    merges_txt_path = os.path.join(args.model_dir, "merges.txt")
+
+    for p in [tokenizer_json_path, tokenizer_config_path, vocab_json_path, merges_txt_path]:
+        if not os.path.exists(p):
+            print(f"[-] Error: Missing required file: {p}")
+            sys.exit(1)
+
+    # Load raw inputs
+    print(f"\n[L1] Loading baseline files from {args.model_dir} ...")
+    with open(tokenizer_json_path, "r", encoding="utf-8") as f:
+        t_json = json.load(f)
+    with open(tokenizer_config_path, "r", encoding="utf-8") as f:
+        t_config = json.load(f)
+    with open(vocab_json_path, "r", encoding="utf-8") as f:
+        vocab_dict = json.load(f)
+    with open(merges_txt_path, "r", encoding="utf-8") as f:
+        merges_lines = [line.strip() for line in f if line.strip() and not line.startswith('#')]
+
+    print(f"  - tokenizer.json:        {os.path.getsize(tokenizer_json_path):,} bytes")
+    print(f"  - merges.txt:            {os.path.getsize(merges_txt_path):,} bytes")
+    print(f"  - vocab.json:            {os.path.getsize(vocab_json_path):,} bytes")
+    print(f"  - tokenizer_config.json: {os.path.getsize(tokenizer_config_path):,} bytes")
+
+    # ── Phase 2: Extract structured configurations (L2) ────────────────────────
+    print("\n[L2] Extracting structural configurations...")
+    config_meta = {
+        "version": t_json.get("version"),
+        "truncation": t_json.get("truncation"),
+        "padding": t_json.get("padding"),
+        "added_tokens": t_json.get("added_tokens"),
+        "normalizer": t_json.get("normalizer"),
+        "pre_tokenizer": t_json.get("pre_tokenizer"),
+        "post_processor": t_json.get("post_processor"),
+        "decoder": t_json.get("decoder"),
+        
+        "model_type": t_json["model"]["type"],
+        "model_dropout": t_json["model"].get("dropout"),
+        "model_unk_token": t_json["model"].get("unk_token"),
+        "model_continuing_subword_prefix": t_json["model"].get("continuing_subword_prefix"),
+        "model_end_of_word_suffix": t_json["model"].get("end_of_word_suffix"),
+        "model_fuse_unk": t_json["model"].get("fuse_unk"),
+        "model_byte_fallback": t_json["model"].get("byte_fallback"),
+        "model_ignore_merges": t_json["model"].get("ignore_merges"),
+        
+        # Include original tokenizer_config.json metadata
+        "tokenizer_config": t_config
+    }
+    config_str = json.dumps(config_meta, ensure_ascii=False)
+    config_bytes = config_str.encode("utf-8")
+    comp_config = zlib.compress(config_bytes, 9)
+    print(f"  Config metadata: {len(config_bytes):,} bytes -> compressed: {len(comp_config):,} bytes")
+
+    # ── Phase 3: Binary Merges delta mapping (L3) ──────────────────────────────
+    print("\n[L3] Mapping BPE merges to binary index pairs...")
+    # Normal vocab tokens (ignoring added tokens mapped >= 248044)
+    normal_vocab = sorted([(k, v) for k, v in vocab_dict.items() if v < 248044], key=lambda x: x[1])
+    vocab_list = [t[0].encode("utf-8", errors="replace") for t in normal_vocab]
+    
+    # Construct binary merges
+    merges_data = bytearray()
+    for line in merges_lines:
+        parts = line.split()
+        if len(parts) != 2:
+            continue
+        idx0 = vocab_dict.get(parts[0], -1)
+        idx1 = vocab_dict.get(parts[1], -1)
+        if idx0 == -1 or idx1 == -1:
+            print(f"  [-] Warning: Merge token not found in vocab: {parts}")
+            continue
+        merges_data.extend(struct.pack('>I', idx0)[1:])
+        merges_data.extend(struct.pack('>I', idx1)[1:])
+    print(f"  BPE Merges: {len(merges_lines):,} items -> encoded: {len(merges_data):,} bytes")
+
+    # ── Phase 4: Prefix-Suffix Vocab Compression (L4) ──────────────────────────
+    print("\n[L4] Performing prefix-suffix spectral vocabulary compression...")
+    vocab_data = get_prefix_suffix_encoding(vocab_list)
+    print(f"  Vocabulary: {len(vocab_list):,} strings -> encoded: {len(vocab_data):,} bytes")
+
+    # ── Phase 5/6: Absolute Mode Capsule Assembly (L6) ──────────────────────────
+    print("\n[L6] Assembling Absolute Mode capsule...")
+    # Absolute Capsule Layout:
+    # [Magic: 3B][Mode: 1B = 0x01][comp_config_len: 4B][comp_config: var][vocab_num: 4B][vocab_len: 4B][vocab_data: var][merges_num: 4B][merges_data: var]
+    raw_absolute_payload = bytearray()
+    raw_absolute_payload.extend(TK_MAGIC)
+    raw_absolute_payload.append(0x01) # mode = Absolute
+    raw_absolute_payload.extend(struct.pack('>I', len(comp_config)))
+    raw_absolute_payload.extend(comp_config)
+    raw_absolute_payload.extend(struct.pack('>I', len(vocab_list)))
+    raw_absolute_payload.extend(struct.pack('>I', len(vocab_data)))
+    raw_absolute_payload.extend(vocab_data)
+    raw_absolute_payload.extend(struct.pack('>I', len(merges_lines)))
+    raw_absolute_payload.extend(merges_data)
+
+    print(f"  Raw Absolute capsule: {len(raw_absolute_payload):,} bytes")
+    absolute_capsule = zlib.compress(raw_absolute_payload, 9)
+    print(f"  Compressed Absolute capsule (Zlib): {len(absolute_capsule):,} bytes")
+
+    # Save Absolute Capsule
+    abs_path = os.path.join(args.out_dir, "qwen-3.5-0.8b-28chirps-tokenizer.capsule")
+    with open(abs_path, "wb") as f:
+        f.write(absolute_capsule)
+    print(f"  [+] Wrote Absolute capsule to: {abs_path}")
+
+    # ── Phase 5/6: Reference Mode Capsule Assembly (L5/L6) ──────────────────────
+    print("\n[L5/L6] Assembling Reference Mode capsule (Zero-delta vs Base Oracle)...")
+    # Reference Capsule Layout:
+    # [Magic: 3B][Mode: 1B = 0x02][base_repo_len: 2B][base_repo: var]
+    base_repo = "Qwen/Qwen3.5-0.8B"
+    base_repo_bytes = base_repo.encode("utf-8")
+    
+    raw_ref_payload = bytearray()
+    raw_ref_payload.extend(TK_MAGIC)
+    raw_ref_payload.append(0x02) # mode = Reference
+    raw_ref_payload.extend(struct.pack('>H', len(base_repo_bytes)))
+    raw_ref_payload.extend(base_repo_bytes)
+
+    print(f"  Raw Reference capsule: {len(raw_ref_payload)} bytes")
+    ref_capsule = zlib.compress(raw_ref_payload, 9)
+    print(f"  Compressed Reference capsule (Zlib): {len(ref_capsule)} bytes")
+
+    # Save Reference Capsule
+    ref_path = os.path.join(args.out_dir, "qwen-3.5-0.8b-28chirps-tokenizer-ref.capsule")
+    with open(ref_path, "wb") as f:
+        f.write(ref_capsule)
+    print(f"  [+] Wrote Reference capsule to: {ref_path}")
+
+    # ── Phase 7: XOR-FEC Chirp Packetization (L7) ───────────────────────────────
+    print(f"\n[L7] Packetizing Reference Mode capsule into {NUM_PKTS} × {PKT_SIZE}-byte LoRa chirps...")
+    packets = pack_into_packets(ref_capsule)
+    
+    pkt_dir = os.path.join(args.out_dir, "packets_tokenizer")
+    os.makedirs(pkt_dir, exist_ok=True)
+    
+    sha_all = hashlib.sha256()
+    for idx, pkt in enumerate(packets):
+        p_path = os.path.join(pkt_dir, f"packet_tokenizer_{idx}.bin")
+        with open(p_path, "wb") as f:
+            f.write(pkt)
+        sha_all.update(pkt)
+        ptype = "DATA" if idx < NUM_DATA else "XOR-FEC-PARITY"
+        print(f"  - Packet {idx:>2} [{ptype}]: {p_path}  ({len(pkt)} bytes)")
+
+    # Write packet manifest
+    manifest = {
+        "protocol": "Tokenizer Chirp-28 v1.0",
+        "method": "L7:QualiaSeed+L5:BaseReference+ZLIB+FEC",
+        "watermark": "ip zymatica.space",
+        "mode": "reference",
+        "base_oracle": base_repo,
+        "num_packets": NUM_PKTS,
+        "packet_size": PKT_SIZE,
+        "compressed_bytes": len(ref_capsule),
+        "sha256": sha_all.hexdigest(),
+    }
+    manifest_path = os.path.join(pkt_dir, "manifest_tokenizer.json")
+    with open(manifest_path, "w", encoding="utf-8") as f:
+        json.dump(manifest, f, indent=2)
+    print(f"  [+] Wrote packet manifest to: {manifest_path}")
+
+    # ── Report ────────────────────────────────────────────────────────────────
+    print("\n" + "=" * 80)
+    print("  COMPRESSION RESULTS SUMMARY")
+    print("=" * 80)
+    print(f"  Original Files Size:      {os.path.getsize(tokenizer_json_path) + os.path.getsize(merges_txt_path) + os.path.getsize(vocab_json_path) + os.path.getsize(tokenizer_config_path):,} bytes (~23 MB)")
+    print(f"  Absolute Capsule Size:    {len(absolute_capsule):,} bytes  (Mode 1, stand-alone, no external ref)")
+    print(f"  Absolute Compression:     { (os.path.getsize(tokenizer_json_path) + os.path.getsize(merges_txt_path) + os.path.getsize(vocab_json_path) + os.path.getsize(tokenizer_config_path)) / len(absolute_capsule):.2f}x ratio")
+    print(f"  Reference Capsule Size:   {len(ref_capsule)} bytes  (Mode 2, pre-shared oracle, zero-delta)")
+    print(f"  Total Packets footprint:  {len(packets) * PKT_SIZE:,} bytes  ({NUM_PKTS} chirps × 255 bytes)")
+    print(f"  Reference Compression:    { (os.path.getsize(tokenizer_json_path) + os.path.getsize(merges_txt_path) + os.path.getsize(vocab_json_path) + os.path.getsize(tokenizer_config_path)) / (len(packets) * PKT_SIZE):,.1f}x ratio (over wire)")
+    print(f"  Watermark Hash:           {sha_all.hexdigest()[:32]}...")
+    print("=" * 80)
+
+if __name__ == "__main__":
+    main()

decode_tokenizer.py

@@ -0,0 +1,334 @@
+# Watermark: ip zymatica.space
+__watermark__ = "ip zymatica.space"
+
+"""
+decode_tokenizer.py — 7-Level Tokenizer Restoration & Verification Engine
+=========================================================================
+Author: Zymatica / Language-U Project
+Watermark: ip zymatica.space | astronautshe.com
+
+Decodes the hyper-compressed tokenizer capsule (or reassembles from 28 packets
+via XOR-FEC) and reconstructs standard tokenizer files:
+- tokenizer.json
+- tokenizer_config.json
+- vocab.json
+- merges.txt
+"""
+
+import os
+import sys
+import json
+import zlib
+import struct
+import argparse
+import hashlib
+from transformers import AutoTokenizer
+
+TK_MAGIC      = bytes([0xC5, 0x54, 0x4B])   # TK\xC5
+PKT_SIZE      = 255
+NUM_DATA      = 27
+NUM_PKTS      = 28
+DATA_PER_PKT  = PKT_SIZE - 3                # 252 bytes
+MAX_PAYLOAD   = NUM_DATA * DATA_PER_PKT     # 6,804 bytes
+
+DEFAULT_CAPSULE = "j:/Language-U/qwen-3.5-0.8b-28chirps-tokenizer-ref.capsule"
+DEFAULT_PKT_DIR = "j:/Language-U/packets_tokenizer"
+DEFAULT_OUT_DIR = "j:/Language-U/reconstructed_tokenizer"
+
+def read_varint(data, pos):
+    val = 0
+    shift = 0
+    while True:
+        b = data[pos]
+        pos += 1
+        val |= (b & 0x7F) << shift
+        if not (b & 0x80):
+            break
+        shift += 7
+    return val, pos
+
+def decode_prefix_suffix(data, num_tokens):
+    tokens = []
+    pos = 0
+    prev = b''
+    for _ in range(num_tokens):
+        common, pos = read_varint(data, pos)
+        suffix_len, pos = read_varint(data, pos)
+        suffix = data[pos : pos + suffix_len]
+        pos += suffix_len
+        
+        t = prev[:common] + suffix
+        tokens.append(t)
+        prev = t
+    return tokens
+
+def recover_packets_via_fec(pkt_dir):
+    """Loads 28 packets from folder and applies XOR-FEC recovery if exactly 1 packet is missing."""
+    packet_files = sorted([f for f in os.listdir(pkt_dir) if f.startswith("packet_tokenizer_") and f.endswith(".bin")])
+    if not packet_files:
+        raise FileNotFoundError(f"No packet files found in {pkt_dir}")
+        
+    # Read the number of total packets from the wrapper of the first file
+    with open(os.path.join(pkt_dir, packet_files[0]), "rb") as f:
+        first_pkt = f.read()
+    if len(first_pkt) < 3 or first_pkt[0] != 0xBB:
+        raise ValueError("Invalid packet header structure in first packet.")
+    total_pkts = first_pkt[2]
+    
+    # Load all available packets
+    received_packets = {}
+    for pf in packet_files:
+        with open(os.path.join(pkt_dir, pf), "rb") as f:
+            pkt_bytes = f.read()
+        if len(pkt_bytes) == PKT_SIZE and pkt_bytes[0] == 0xBB:
+            idx = pkt_bytes[1]
+            received_packets[idx] = pkt_bytes
+            
+    print(f"  Loaded {len(received_packets)}/{total_pkts} packets.")
+    
+    missing_indices = [i for i in range(total_pkts) if i not in received_packets]
+    if len(missing_indices) == 0:
+        print("[+] All packets received intact. Verifying FEC parity...")
+        # Verify FEC is correct (XOR of all data + FEC payloads must equal 0)
+        xor_fec = bytearray(DATA_PER_PKT)
+        for idx, pkt in received_packets.items():
+            for j in range(DATA_PER_PKT):
+                xor_fec[j] ^= pkt[j + 3]
+        if any(xor_fec):
+            print("⚠️ Warning: FEC verification failed (non-zero XOR sum).")
+        else:
+            print("[+] FEC verification passed.")
+    elif len(missing_indices) == 1:
+        missing_idx = missing_indices[0]
+        print(f"[-] Missing packet index {missing_idx}. Performing XOR FEC recovery...")
+        recovered_payload = bytearray(DATA_PER_PKT)
+        for idx, pkt in received_packets.items():
+            for j in range(DATA_PER_PKT):
+                recovered_payload[j] ^= pkt[j + 3]
+                
+        # Reconstruct missing packet
+        recovered_pkt = bytes([0xBB, missing_idx, total_pkts]) + bytes(recovered_payload)
+        received_packets[missing_idx] = recovered_pkt
+        print(f"[+] Successfully recovered missing packet index {missing_idx} via FEC.")
+    else:
+        raise ValueError(f"Cannot recover because {len(missing_indices)} packets are missing.")
+        
+    # Reassemble payload from data packets (excluding FEC parity packet)
+    assembled = bytearray()
+    for i in range(NUM_DATA):
+        assembled.extend(received_packets[i][3:])
+    return bytes(assembled)
+
+def main():
+    parser = argparse.ArgumentParser(description="7-Level Tokenizer Restoration & Verification Engine")
+    parser.add_argument("--capsule", default=None, help="Path to capsule file to decode")
+    parser.add_argument("--packet_dir", default=None, help="Path to packets directory to reassemble")
+    parser.add_argument("--out_dir", default=DEFAULT_OUT_DIR, help="Output directory for restored tokenizer")
+    args = parser.parse_args()
+
+    print("=" * 80)
+    print("  7-LEVEL TOKENIZER RESTORATION & VERIFICATION ENGINE")
+    print("  Watermark: ip zymatica.space | astronautshe.com")
+    print("=" * 80)
+
+    # 1. Determine Source (Capsule or Packet directory)
+    payload_bytes = None
+    if args.packet_dir:
+        print(f"\n[*] Reassembling from packets in: {args.packet_dir}")
+        try:
+            payload_bytes = recover_packets_via_fec(args.packet_dir)
+        except Exception as e:
+            print(f"[-] Reassembly failed: {e}")
+            sys.exit(1)
+    elif args.capsule:
+        print(f"\n[*] Decoding capsule file: {args.capsule}")
+        with open(args.capsule, "rb") as f:
+            payload_bytes = f.read()
+    else:
+        # Auto-detect packets or capsule
+        if os.path.exists(DEFAULT_PKT_DIR) and len(os.listdir(DEFAULT_PKT_DIR)) > 0:
+            print(f"\n[*] Auto-detected packet directory: {DEFAULT_PKT_DIR}")
+            try:
+                payload_bytes = recover_packets_via_fec(DEFAULT_PKT_DIR)
+            except Exception as e:
+                print(f"[-] Reassembly failed: {e}")
+        
+        if payload_bytes is None:
+            # Fallback to default capsule
+            if os.path.exists(DEFAULT_CAPSULE):
+                print(f"\n[*] Auto-detected default capsule file: {DEFAULT_CAPSULE}")
+                with open(DEFAULT_CAPSULE, "rb") as f:
+                    payload_bytes = f.read()
+            else:
+                # Fallback to absolute capsule
+                abs_capsule = DEFAULT_CAPSULE.replace("-ref.capsule", ".capsule")
+                if os.path.exists(abs_capsule):
+                    print(f"\n[*] Auto-detected default absolute capsule file: {abs_capsule}")
+                    with open(abs_capsule, "rb") as f:
+                        payload_bytes = f.read()
+
+    if not payload_bytes:
+        print("[-] Error: No input capsule file or packet directory found.")
+        sys.exit(1)
+
+    # 2. Decompress Zlib (L6)
+    print("\n[L6] Decompressing binary payload...")
+    try:
+        decompressed = zlib.decompress(payload_bytes)
+        print(f"  Decompressed: {len(decompressed):,} bytes")
+    except Exception as e:
+        # If the input was the full padded packets, it might have trailing padding bytes.
+        # We need to trim trailing padding or parse headers.
+        # Let's try parsing directly or handle payload extraction
+        print(f"[-] Decompression failed: {e}")
+        sys.exit(1)
+
+    # 3. Parse Magic Header and Mode
+    pos = 0
+    magic = decompressed[pos:pos+3]; pos += 3
+    if magic != TK_MAGIC:
+        print(f"[-] Error: Invalid magic bytes: {magic.hex()}")
+        sys.exit(1)
+        
+    mode = decompressed[pos]; pos += 1
+    print(f"  Magic verified: 0x{magic.hex().upper()}")
+    print(f"  Mode verified:  Mode {mode} ({'Absolute' if mode == 1 else 'Reference/Oracle'})")
+
+    os.makedirs(args.out_dir, exist_ok=True)
+
+    # 4. Reconstruction
+    if mode == 1:
+        # --- Mode 1: Absolute Mode ---
+        print("\n[L2-L4] Restoring absolute tokenizer structures...")
+        
+        # Unpack config metadata length and data
+        comp_config_len = struct.unpack_from('>I', decompressed, pos)[0]; pos += 4
+        comp_config_data = decompressed[pos : pos + comp_config_len]; pos += comp_config_len
+        config_meta = json.loads(zlib.decompress(comp_config_data).decode("utf-8"))
+        print(f"  - Config metadata loaded ({len(config_meta)} keys)")
+
+        # Unpack vocabulary normal tokens
+        vocab_num = struct.unpack_from('>I', decompressed, pos)[0]; pos += 4
+        vocab_len = struct.unpack_from('>I', decompressed, pos)[0]; pos += 4
+        vocab_data = decompressed[pos : pos + vocab_len]; pos += vocab_len
+        
+        vocab_list = decode_prefix_suffix(vocab_data, vocab_num)
+        print(f"  - Restored {len(vocab_list):,} normal vocabulary tokens")
+
+        # Unpack merges
+        merges_num = struct.unpack_from('>I', decompressed, pos)[0]; pos += 4
+        merges_data = decompressed[pos : pos + merges_num * 6]; pos += merges_num * 6
+        
+        merges = []
+        for i in range(merges_num):
+            idx0 = int.from_bytes(merges_data[i*6 : i*6 + 3], 'big')
+            idx1 = int.from_bytes(merges_data[i*6 + 3 : i*6 + 6], 'big')
+            t0 = vocab_list[idx0].decode("utf-8", errors="replace")
+            t1 = vocab_list[idx1].decode("utf-8", errors="replace")
+            merges.append(f"{t0} {t1}")
+        print(f"  - Restored {len(merges):,} BPE merge entries")
+
+        # Reconstruct vocab.json and merges.txt
+        vocab_dict = {t.decode("utf-8", errors="replace"): idx for idx, t in enumerate(vocab_list)}
+        
+        # Write merges.txt
+        merges_out = os.path.join(args.out_dir, "merges.txt")
+        with open(merges_out, "w", encoding="utf-8") as f:
+            f.write("\n".join(merges) + "\n")
+            
+        # Write vocab.json
+        vocab_out = os.path.join(args.out_dir, "vocab.json")
+        with open(vocab_out, "w", encoding="utf-8") as f:
+            json.dump(vocab_dict, f, ensure_ascii=False, indent=2)
+
+        # Reconstruct tokenizer.json
+        reconstructed_t_json = {
+            "version": config_meta["version"],
+            "truncation": config_meta["truncation"],
+            "padding": config_meta["padding"],
+            "added_tokens": config_meta["added_tokens"],
+            "normalizer": config_meta["normalizer"],
+            "pre_tokenizer": config_meta["pre_tokenizer"],
+            "post_processor": config_meta["post_processor"],
+            "decoder": config_meta["decoder"],
+            "model": {
+                "type": config_meta["model_type"],
+                "dropout": config_meta["model_dropout"],
+                "unk_token": config_meta["model_unk_token"],
+                "continuing_subword_prefix": config_meta["model_continuing_subword_prefix"],
+                "end_of_word_suffix": config_meta["model_end_of_word_suffix"],
+                "fuse_unk": config_meta["model_fuse_unk"],
+                "byte_fallback": config_meta["model_byte_fallback"],
+                "ignore_merges": config_meta["model_ignore_merges"],
+                "vocab": vocab_dict,
+                "merges": merges
+            }
+        }
+        
+        tokenizer_json_out = os.path.join(args.out_dir, "tokenizer.json")
+        with open(tokenizer_json_out, "w", encoding="utf-8") as f:
+            json.dump(reconstructed_t_json, f, ensure_ascii=False, indent=2)
+
+        # Write tokenizer_config.json
+        tokenizer_config_out = os.path.join(args.out_dir, "tokenizer_config.json")
+        with open(tokenizer_config_out, "w", encoding="utf-8") as f:
+            json.dump(config_meta["tokenizer_config"], f, ensure_ascii=False, indent=2)
+
+        print("[+] Stand-alone absolute reconstruction completed successfully.")
+
+    elif mode == 2:
+        # --- Mode 2: Reference Mode ---
+        print("\n[L5] Fetching base model tokenizer reference from HuggingFace...")
+        base_repo_len = struct.unpack_from('>H', decompressed, pos)[0]; pos += 2
+        base_repo = decompressed[pos : pos + base_repo_len].decode("utf-8"); pos += base_repo_len
+        print(f"  - Base Oracle Reference: {base_repo}")
+        
+        try:
+            # Load tokenizer from HF reference
+            print(f"  - Querying Hugging Face: {base_repo} ...")
+            tokenizer = AutoTokenizer.from_pretrained(base_repo, trust_remote_code=True)
+            tokenizer.save_pretrained(args.out_dir)
+            print(f"[+] Successfully downloaded and saved tokenizer to: {args.out_dir}")
+        except Exception as e:
+            print(f"[-] Error downloading base tokenizer: {e}")
+            # Offline fallback if local files exist
+            local_fallback = "j:/Language-U/Language-U-V2/qwen-3.5-0.8b-local"
+            if os.path.exists(local_fallback):
+                print(f"  - [Offline Fallback] Copying from local cache: {local_fallback}")
+                import shutil
+                for fn in ["tokenizer.json", "tokenizer_config.json", "vocab.json", "merges.txt"]:
+                    src = os.path.join(local_fallback, fn)
+                    if os.path.exists(src):
+                        shutil.copy(src, os.path.join(args.out_dir, fn))
+                print(f"[+] Offline fallback copied successfully to: {args.out_dir}")
+            else:
+                sys.exit(1)
+    else:
+        print(f"[-] Error: Unknown tokenizer capsule mode: {mode}")
+        sys.exit(1)
+
+    # 5. Verification
+    print("\n[*] Verifying reconstructed tokenizer loading correctness...")
+    try:
+        loaded_tokenizer = AutoTokenizer.from_pretrained(args.out_dir, trust_remote_code=True)
+        print(f"  [PASS] Reconstructed tokenizer successfully parsed by Transformers!")
+        
+        # Test encoding
+        test_text = "Astronaut SHE LoRa concentrator GPIO reset SX1302 v3.0 Cuneiform-U"
+        tokens = loaded_tokenizer.encode(test_text)
+        decoded = loaded_tokenizer.decode(tokens)
+        print(f"  [PASS] Test encoding round-trip succeeded!")
+        print(f"         Encoded: {tokens[:8]}...")
+        print(f"         Decoded: \"{decoded}\"")
+        
+        print("\n" + "=" * 80)
+        print("  RESTORE SUCCESSFUL")
+        print("=" * 80)
+        print(f"  Output folder: {os.path.abspath(args.out_dir)}")
+        print("=" * 80)
+    except Exception as e:
+        print(f"[-] Verification failed: {e}")
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()