Review fixes: novel-facts gate parity with acceptance harness, CLI defaults, framing scoped to validation

README.md

@@ -17,7 +17,7 @@ tags:
 
 # tessera-compressor
 
-A 1.5B model that compresses English reasoning text into a telegraphic CJK/symbol register under deterministic fidelity gates. It minted the training data for [Tessera-Preview-9B](https://huggingface.co/ZelligeAI/tessera-preview-9b) and replaces the frontier-model teacher that originally produced the register: any English reasoning dataset becomes compressed-register training data at local-inference cost, with no API key and no external dependency.
+A 1.5B model that compresses English reasoning text into a telegraphic CJK/symbol register under deterministic fidelity gates. It minted the training data for [Tessera-Preview-9B](https://huggingface.co/ZelligeAI/tessera-preview-9b) and replaces the frontier-model teacher that originally produced the register: English reasoning text becomes compressed-register training data at local-inference cost, with no API key and no external dependency. Validation covered code-centric reasoning (103 held-out mixed blocks); behavior on distant domains is unmeasured.
 
 Example (real training pair, 85 to 49 tokens):
 
@@ -30,7 +30,7 @@ CJK: Integer(line32),Boolean(line262),BitString(line341),OctetString(line693).
 
 ## How it works
 
-The compressor operates on passages, not whole blocks. A reasoning block is segmented (code fences stay atomic), sentences are grouped into step-sized passages, each passage is classified as fact-dense or narrative, and the model compresses it against the tail of the chain built so far. Every model output then passes a deterministic gate: the sets of numbers, identifiers, and operators must survive, the output must not blow up in length, and it must beat a rules-only compression of the same passage in token count. A passage that fails any check falls back to the rules-only version, so a bad generation costs savings, never content.
+The compressor operates on passages, not whole blocks. A reasoning block is segmented (code fences stay atomic), sentences are grouped into step-sized passages, each passage is classified as fact-dense or narrative, and the model compresses it against the tail of the chain built so far. Every model output then passes a deterministic gate: the passage's novel numbers and identifiers must survive as substrings, the output must not blow up in length, and it must not exceed a rules-only compression of the same passage in token count. A passage that fails any check falls back to the rules-only version, so a bad generation costs savings rather than gated content. The gate is lexical, not semantic: it prevents the loss of numbers and identifiers, and a judged semantic-equivalence check backed it at acceptance (below), but it does not by itself guarantee semantic preservation on arbitrary input.
 
 ## Acceptance record
 
@@ -38,7 +38,7 @@ Measured on 103 held-out reasoning blocks the model never trained on, under crit
 
 | Criterion | Result |
 | --- | --- |
-| Per-passage fidelity gate (numbers, identifiers, operators survive) | 99.0% |
+| Per-passage fidelity gate (numbers and identifiers survive) | 99.0% |
 | Median per-passage compression ratio (output/input tokens) | 0.716 |
 | CJK adoption | 98.9% of compressed passages |
 | Judged semantic equivalence | 103/103 blocks (teacher references on the same blocks: 97.1%) |

scripts/compress.py

@@ -13,7 +13,7 @@ model output costs savings, never content.
 Serve the model first, e.g.:
     vllm serve ZelligeAI/tessera-compressor --port 8001
 or with the GGUF:
-    llama-server -m compressor-v31-q8_0.gguf --port 8001
+    llama-server -m gguf/compressor-v31-q8_0.gguf --port 8001   # from the repo root
 
 Then:
     # one block from a text file
@@ -84,7 +84,7 @@ def compress_block(text, client, model, ntok):
             stats["narr_skipped"] += 1
             seen |= facts(s)
             continue
-        if gate(s, rules_s, out, ntok) is None:
+        if gate(s, rules_s, out, ntok, novel=novel) is None:
             chain.append(out)
             stats["model_ok"] += 1
         else:

scripts/requirements.txt

@@ -1,2 +1,3 @@
 openai>=1.0
 tokenizers>=0.15
+transformers>=4.40  # tokenmax.py standalone CLI only

scripts/segmenting.py

@@ -97,18 +97,25 @@ def classify_passage(seg, seen_facts, ntok):
     return 'narr'
 
 
-def gate(src_seg, rules_seg, out, ntok):
+def gate(src_seg, rules_seg, out, ntok, novel=None):
     """Deterministic per-passage fidelity gate.
     Returns None if the model output is admissible, else a short fail-reason
-    string; on failure the caller uses rules_seg instead."""
+    string; on failure the caller uses rules_seg instead.
+
+    novel: the passage's facts that are NOT already in the accumulated chain.
+    The prompt tells the model never to restate chain content, so only novel
+    facts are required to survive (matching the acceptance harness). Pass None
+    to require every fact of the passage (stricter, for chainless use)."""
     if not out or not out.strip():
         return "empty"
     if '```' in out:
         return "fence"
     if len(out) > 2 * len(src_seg) + 40:           # explanation/blow-up guard
         return "blowup"
-    if facts_preserved(src_seg, out):              # every fact survives (substring check)
+    required = facts(src_seg) if novel is None else novel
+    out_n = out.replace(',', '')
+    if any(f.replace(',', '') not in out_n for f in required):
         return "facts"
-    if ntok(out) > ntok(rules_seg):                # must beat the deterministic version
+    if ntok(out) > ntok(rules_seg):                # must not exceed the rules-only version
         return "tokens"
     return None

scripts/tokenmax.py

@@ -363,8 +363,8 @@ if __name__ == '__main__':
     parser = argparse.ArgumentParser(description='Token-max post-processor for think blocks')
     parser.add_argument('--input', required=True, help='Input JSONL (messages format)')
     parser.add_argument('--output', help='Output JSONL (default: dry run, stats only)')
-    parser.add_argument('--tokenizer', default='assets/omnicoder-tokenizer',
-                        help='Path to tokenizer')
+    parser.add_argument('--tokenizer', default='ZelligeAI/tessera-compressor',
+                        help='HF repo id or local path of the tokenizer to count savings under')
     parser.add_argument('--force-cjk', action='store_true',
                         help='Force CJK substitutions even if total tokens increase. '
                              'Prioritizes CJK adoption over token savings.')