Fix field mapping bug: detect partial JSON, extract by name, add Supabase migration

Critical bug: when users paste terminal output like '"totals": { ... }' (not
valid JSON), parse_four() grabbed numbers positionally — mapping cacheCreation
to input, cacheRead to output, etc. Completely scrambled.

Fix: three-strategy pipeline in ingest_meta():
1. _try_fix_json(): wraps partial fragments in {} to make valid JSON
2. _extract_by_name(): regex extraction by field name (inputTokens,
outputTokens, etc.) when named fields are present in any format
3. parse_four(): fallback for bare numbers only

Also adds _FIELD_ALIASES dict mapping canonical names to all known variants
(camelCase + snake_case). _has_named_fields() detects when Strategy 2 applies.

Codex fragments with named fields correctly route through the two-pathway
parser (Alpha 3:2:1 / Beta 1:9) via the named-field extraction path.

SUPABASE_MIGRATION.md: SQL for adding submitted_at + hf_user columns to
sigrank_operators, creating sigrank_sessions table, and RLS policies.

Co-Authored-By: Deric J. McHenry <deric.mchenry@gmail.com>

SUPABASE_MIGRATION.md

@@ -0,0 +1,97 @@
+# Supabase Migration — SigRank Importer Overhaul
+
+Run these in the Supabase SQL Editor (Dashboard → SQL Editor → New Query).
+
+---
+
+## 1. Add columns to `sigrank_operators`
+
+```sql
+-- Timestamp for when the entry was last submitted/updated
+ALTER TABLE sigrank_operators
+  ADD COLUMN IF NOT EXISTS submitted_at TIMESTAMPTZ DEFAULT now();
+
+-- HuggingFace username — only authenticated users can persist
+ALTER TABLE sigrank_operators
+  ADD COLUMN IF NOT EXISTS hf_user TEXT;
+
+-- Index for fast lookups by HF user
+CREATE INDEX IF NOT EXISTS idx_sigrank_operators_hf_user
+  ON sigrank_operators (hf_user);
+```
+
+---
+
+## 2. Create `sigrank_sessions` table (session history / Greatest Hits)
+
+```sql
+CREATE TABLE IF NOT EXISTS sigrank_sessions (
+  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+  name TEXT NOT NULL,
+  input BIGINT NOT NULL DEFAULT 0,
+  output BIGINT NOT NULL DEFAULT 0,
+  cache_create BIGINT NOT NULL DEFAULT 0,
+  cache_read BIGINT NOT NULL DEFAULT 0,
+  cost_usd DOUBLE PRECISION,
+  source TEXT DEFAULT 'manual',
+  estimated BOOLEAN DEFAULT FALSE,
+  caveat TEXT,
+  hf_user TEXT,
+  submitted_at TIMESTAMPTZ DEFAULT now()
+);
+
+-- Index for loading a user's session history
+CREATE INDEX IF NOT EXISTS idx_sigrank_sessions_name
+  ON sigrank_sessions (name, submitted_at DESC);
+```
+
+---
+
+## 3. RLS policies (keep anon read-only, service key for writes)
+
+```sql
+-- Enable RLS on the new table
+ALTER TABLE sigrank_sessions ENABLE ROW LEVEL SECURITY;
+
+-- Anon can read session history
+CREATE POLICY "anon_read_sessions" ON sigrank_sessions
+  FOR SELECT USING (true);
+
+-- Service role can insert (writes come from the app backend)
+CREATE POLICY "service_insert_sessions" ON sigrank_sessions
+  FOR INSERT WITH CHECK (true);
+
+-- Same pattern for the new columns on sigrank_operators
+-- (existing policies should already cover SELECT/INSERT;
+--  verify the existing INSERT policy allows the new columns)
+```
+
+---
+
+## 4. Verify
+
+After running the above, check:
+
+```sql
+-- Should show submitted_at and hf_user columns
+SELECT column_name, data_type
+FROM information_schema.columns
+WHERE table_name = 'sigrank_operators'
+ORDER BY ordinal_position;
+
+-- Should exist with all columns
+SELECT column_name, data_type
+FROM information_schema.columns
+WHERE table_name = 'sigrank_sessions'
+ORDER BY ordinal_position;
+```
+
+---
+
+## Notes
+
+- `sigrank_operators` still upserts on `name` (one board entry per operator)
+- `sigrank_sessions` is append-only — every submission creates a new row
+- The app reads sessions via `load_session_history(name, limit=5)` for the Greatest Hits display
+- `hf_user` is populated only when the user is authenticated via HuggingFace OAuth on the Space
+- Without the `SUPABASE_SERVICE_KEY` env var, all writes are no-ops (safe for public demo)

ingest.py

@@ -10,6 +10,102 @@ for architectural modeling and are distinct from raw provider API payload logs.
 import json
 import re
 
+# Field names we recognise (ccusage camelCase and snake_case variants)
+_FIELD_ALIASES = {
+    "input":        ("inputTokens", "input_tokens"),
+    "output":       ("outputTokens", "output_tokens"),
+    "cache_create": ("cacheCreationTokens", "cache_creation_input_tokens"),
+    "cache_read":   ("cacheReadTokens", "cache_read_input_tokens",
+                     "cachedInputTokens", "cached_input_tokens"),
+    "cost":         ("totalCost", "costUSD", "cost"),
+    "reasoning":    ("reasoningOutputTokens", "reasoning_output_tokens"),
+}
+
+def _has_named_fields(text):
+    """True if text contains at least two recognised ccusage field names."""
+    hits = 0
+    for aliases in _FIELD_ALIASES.values():
+        for alias in aliases:
+            if alias in text:
+                hits += 1
+                break
+    return hits >= 2
+
+
+def _try_fix_json(text):
+    """Try to turn a partial JSON fragment into valid JSON.
+
+    Common pastes:
+      "totals": { ... }          →  {"totals": { ... }}
+      { "totals": { ... }        →  add missing closing brace
+      "inputTokens": 123, ...    →  { ... }
+    """
+    t = text.strip()
+    # Already valid JSON
+    if t[0] in "{[":
+        try:
+            json.loads(t)
+            return t
+        except json.JSONDecodeError:
+            # Missing closing brace — try adding one
+            try:
+                json.loads(t + "}")
+                return t + "}"
+            except json.JSONDecodeError:
+                try:
+                    json.loads(t + "}}")
+                    return t + "}}"
+                except json.JSONDecodeError:
+                    pass
+    # Starts with a quoted key like "totals": { ... } or "inputTokens": 123
+    if t.startswith('"'):
+        candidate = "{" + t
+        # Try adding closing braces
+        for suffix in ("", "}", "}}"):
+            try:
+                json.loads(candidate + suffix)
+                return candidate + suffix
+            except json.JSONDecodeError:
+                continue
+    return None
+
+
+def _extract_by_name(text):
+    """Extract token values from text containing named fields (any format).
+
+    Works on partial JSON, terminal output, or any text where field names
+    appear next to their values. Returns (i, o, cw, cr, cost) or None.
+    """
+    def grab(aliases):
+        for alias in aliases:
+            # Match "fieldName": 12345 or "fieldName": 12345.67
+            m = re.search(rf'"{re.escape(alias)}"\s*:\s*([\d,.]+)', text)
+            if m:
+                raw = m.group(1).replace(",", "")
+                return int(float(raw))
+        return 0
+
+    i  = grab(_FIELD_ALIASES["input"])
+    o  = grab(_FIELD_ALIASES["output"])
+    cw = grab(_FIELD_ALIASES["cache_create"])
+    cr = grab(_FIELD_ALIASES["cache_read"])
+    cost_val = None
+    for alias in _FIELD_ALIASES["cost"]:
+        m = re.search(rf'"{re.escape(alias)}"\s*:\s*([\d,.]+)', text)
+        if m:
+            raw = m.group(1).replace(",", "")
+            cost_val = float(raw)
+            if cost_val > 0:
+                break
+    reasoning = grab(_FIELD_ALIASES["reasoning"])
+    if reasoning > 0:
+        o += reasoning
+
+    if i + o + cw + cr == 0:
+        return None
+    return i, o, cw, cr, cost_val
+
+
 def parse_ccusage(text):
     """Accept raw `ccusage --json` output (any known shape). Returns (i,o,cw,cr,cost)."""
     d = json.loads(text)
@@ -92,10 +188,10 @@ def parse_codex_submission(payload, operator_profile=None):
 
     Two pathways depending on operator telemetry:
 
-    Pathway Alpha (Standard): No Claude footprint → 3:2:1 baseline.
-      estimated_user_input = outputTokens × 2.0
+    Pathway Alpha (Standard): No Claude footprint -> 3:2:1 baseline.
+      estimated_user_input = outputTokens * 2.0
 
-    Pathway Beta (Claude Engine): Operator has verified Claude profile →
+    Pathway Beta (Claude Engine): Operator has verified Claude profile ->
       dynamic 1:9 transmission velocity extraction.
       estimated_user_input = outputTokens / 9.0
 
@@ -138,6 +234,12 @@ def parse_codex_submission(payload, operator_profile=None):
 def ingest_meta(text, operator_profile=None):
     """Returns (i,o,cw,cr,meta) with estimated/caveat/cost.
 
+    Handles:
+      - Full ccusage JSON (Claude: measured, Codex: two-pathway estimation)
+      - Partial JSON fragments ("totals": { ... } pasted from terminal)
+      - Text with named fields (extracts by field name, not position)
+      - Four bare numbers: input output cache_create cache_read
+
     operator_profile: optional dict with at least {"model_type": "claude"}
     when the submitting user has a verified Claude session profile. This
     switches the Codex parser from the 3:2:1 baseline to the 1:9 closed-loop
@@ -145,11 +247,64 @@ def ingest_meta(text, operator_profile=None):
     """
     text=text.strip()
     if not text: raise ValueError("empty")
-    if text[0] in "{[":
-        d=json.loads(text)
+
+    # --- Strategy 1: Try to parse as valid JSON (or fix partial JSON) ---
+    fixed = None
+    if text[0] in '{["':
+        fixed = _try_fix_json(text)
+    if fixed:
+        d = json.loads(fixed)
         if is_codex_shape(d):
             return parse_codex_submission(d, operator_profile=operator_profile)
-        i,o,cw,cr,cost = parse_ccusage(text)
+        i,o,cw,cr,cost = parse_ccusage(fixed)
         return i,o,cw,cr,{"source":"ccusage","estimated":False,"caveat":None,"cost":cost}
+
+    # --- Strategy 2: Text has named fields — extract by name ---
+    if _has_named_fields(text):
+        result = _extract_by_name(text)
+        if result:
+            i, o, cw, cr, cost_val = result
+            has_codex_fields = any(
+                alias in text
+                for alias in ("cachedInputTokens", "cached_input_tokens",
+                              "reasoningOutputTokens", "reasoning_output_tokens")
+            )
+            if has_codex_fields:
+                # Re-route through Codex pathway with extracted totals
+                raw_cache = 0
+                for alias in _FIELD_ALIASES["cache_read"]:
+                    m = re.search(rf'"{re.escape(alias)}"\s*:\s*([\d,.]+)', text)
+                    if m:
+                        raw_cache = int(float(m.group(1).replace(",", "")))
+                        break
+                raw_in = 0
+                for alias in _FIELD_ALIASES["input"]:
+                    m = re.search(rf'"{re.escape(alias)}"\s*:\s*([\d,.]+)', text)
+                    if m:
+                        raw_in = int(float(m.group(1).replace(",", "")))
+                        break
+                raw_out = o  # already includes reasoning from _extract_by_name
+                if operator_profile and operator_profile.get("model_type") == "claude":
+                    est_input = raw_out / 9.0
+                    parsing_mode = "Claude Closed-Loop Calibration (1:9)"
+                else:
+                    est_input = raw_out * 2.0
+                    parsing_mode = "Standard Open-Loop Baseline (3:2:1)"
+                context_debt = max(0, raw_in - int(est_input))
+                meta = {
+                    "source": "codex", "estimated": True,
+                    "parsing_mode": parsing_mode,
+                    "caveat": f"* {parsing_mode}",
+                    "anchor": parsing_mode, "cost": cost_val,
+                }
+                return int(est_input), raw_out, context_debt, raw_cache, meta
+            else:
+                cost = cost_val if cost_val and cost_val > 0 else None
+                return i, o, cw, cr, {
+                    "source": "ccusage", "estimated": False,
+                    "caveat": None, "cost": cost,
+                }
+
+    # --- Strategy 3: Four bare numbers ---
     i,o,cw,cr = parse_four(text)
     return i,o,cw,cr,{"source":"manual","estimated":False,"caveat":None,"cost":None}