Visualize: ask a dataset a question, get an exact chart with the rows behind every mark (small datasets first: instant glance charts, batched model labels, drill-down, scoped follow-ups)

README.md

@@ -10,15 +10,16 @@ hf_oauth: true
 hf_oauth_scopes:
   - contribute-repos
 hf_oauth_expiration_minutes: 120
-short_description: Invent training rows, or curate your pretraining data
+short_description: Invent training rows, curate data, chart any dataset
 ---
 
 # DataInventor
 
-DataInventor has two modes.
+DataInventor has three modes.
 
 - **Invent rows.** An open-source recreation of the [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) workflow. Describe the data you want, choose a format, domains and languages, preview a blueprint, and get an instruction or preference dataset back as CSV or JSONL.
 - **Curate a corpus.** Clean, score, rephrase and mix the pretraining data you already have. Preview every step here on a sample, then download the recipe and run it on all of your data, on your own machines.
+- **Visualize.** Ask a question about a dataset in plain words and get a chart back, with the rows behind every bar, box or dot one click away. Use the rows you just wrote, a sample, any public Hugging Face dataset, or a CSV/JSONL file.
 
 It works with any OpenAI-compatible endpoint. The default is a free model served through Hugging Face Inference Providers.
 
@@ -52,6 +53,16 @@ It writes `curated/<source>/`, `synthetic/<source>/` and `mixes/<mix>/` as JSONL
 
 Two limits to know. Token counts are estimates at four characters per token. The learned scorer is a hashed n-gram model. In our test it reached a rank agreement of about 0.5 with the grading model after 300 grades, so grade more documents, or set `calibrate.scorer: llm` to grade every document with the model.
 
+## How Visualize works
+
+It is built for small datasets first: the 25 to 200 rows a hosted run writes. Every row stays in memory, so each chart is an exact count over all of them.
+
+- **At a glance.** As soon as data loads you see two or three charts that need no model: rows per category, and how long the texts are in each.
+- **Ask.** A model turns your question into a chart plan. If an existing column answers it (area, language, length), the chart comes back in a couple of seconds. If it needs something no column has, such as tone, intent or whether a reply gives steps, the model reads the rows in small parallel batches and fills a new column. A regex handles literal marks like links or numbered lists without a model.
+- **Drill down.** Click a bar, box, dot or legend entry to read the rows behind it, download them, or ask a follow-up about just those rows.
+- **Reuse.** Columns the model made stay with the dataset, so follow-up questions reuse them instead of reading every row again.
+- **Honest numbers.** The write-up uses only the chart's exact numbers, says when a group is too small to read much into, and the page notes which columns a model assigned.
+
 ## How it compares
 
 | | [Invent a Dataset](https://adaptionlabs.ai/invent-a-dataset) (Adaption) | [Curation Studio](https://www.datologyai.com/) (DatologyAI) | DataInventor |

app/main.py

@@ -264,11 +264,15 @@ from curate_api import router as curate_router  # noqa: E402  (imports make_endp
 
 app.include_router(curate_router)
 
+from visualize_api import router as vis_router  # noqa: E402
+
+app.include_router(vis_router)
+
 
 def _asset_version() -> str:
     import hashlib
     h = hashlib.sha1()
-    for f in ("app.js", "curate.js", "app.css"):
+    for f in ("app.js", "curate.js", "visualize.js", "app.css"):
         h.update((STATIC / f).read_bytes())
     return h.hexdigest()[:10]
 

app/visualize_api.py

@@ -0,0 +1,344 @@
+"""Visualize: load a dataset (a run from this page, a sample, a Hugging Face dataset or a file), ask questions, get charts."""
+from __future__ import annotations
+
+import csv
+import io
+import json
+import math
+import os
+import re
+import threading
+import time
+import uuid
+from concurrent.futures import ThreadPoolExecutor, TimeoutError as FutureTimeout
+from dataclasses import dataclass, field
+from typing import Optional
+
+from fastapi import APIRouter, HTTPException
+from fastapi.responses import Response
+from pydantic import BaseModel
+
+from datainventor import visualize as V
+from datainventor.llm import EndpointError
+from datainventor.sources import _get, inspect
+
+router = APIRouter(prefix="/api/vis")
+MAX_ROWS = int(os.environ.get("VIS_MAX_ROWS", "200"))
+MAX_UPLOAD = 6_000_000
+MAX_RUNNING = int(os.environ.get("VIS_MAX_RUNNING", "4"))
+WRITE_SECONDS = float(os.environ.get("VIS_WRITE_SECONDS", "20"))
+CALL_SECONDS = int(os.environ.get("VIS_CALL_SECONDS", "30"))
+WRITERS = ThreadPoolExecutor(max_workers=4)
+SAMPLES = [("support-general", "Customer support replies"), ("hotel-reviews", "Hotel reviews"), ("medical-qa", "Medical questions"),
+           ("legal-qa", "Legal questions"), ("support-banking", "Banking support"), ("african-qa", "Questions about Africa"),
+           ("hindi-news", "Hindi news requests"), ("serbian-car-ads", "Serbian car ads")]
+JOB_ORDER = ["prompt", "completion", "chosen", "rejected", "area", "subtopic", "topic", "persona", "shape", "difficulty", "language"]
+
+
+@dataclass
+class VisSet:
+    id: str
+    name: str
+    source: str
+    rows: list
+    cols: dict
+    created: float = field(default_factory=time.time)
+    lock: threading.Lock = field(default_factory=threading.Lock)
+    busy: bool = False
+
+
+@dataclass
+class Ask:
+    id: str
+    set_id: str
+    question: str
+    scope: list
+    scope_label: str
+    stage: str = "plan"   # plan | label | chart | write | done | failed
+    done: int = 0
+    total: int = 0
+    result: Optional[dict] = None
+    error: str = ""
+    started: float = field(default_factory=time.time)
+
+
+SETS: dict[str, VisSet] = {}
+ASKS: dict[str, Ask] = {}
+GUARD = threading.Lock()
+
+
+def _ep():
+    from main import make_endpoint  # the host's endpoint; visitors never supply one
+    return make_endpoint()
+
+
+def _token() -> str:
+    return os.environ.get("HF_TOKEN", "")
+
+
+# ------------------------------------------------------------------ loading
+def _hub_rows(dataset: str, n: int) -> list[dict]:
+    """A spread of up to n full rows through the dataset viewer: no download, every column kept."""
+    if not re.fullmatch(r"[\w.-]+/[\w.-]+", dataset):
+        raise HTTPException(400, "Give a dataset id like owner/name.")
+    try:
+        info = inspect(dataset, _token())
+    except Exception as e:
+        raise HTTPException(400, f"Could not open {dataset}: {str(e)[:160]}")
+    total = info.get("num_rows") or 0
+    chunks = 1 if not total or total <= n else min(4, math.ceil(n / 25))
+    per = math.ceil(n / chunks)
+    starts = [0] if chunks == 1 else [max(0, int(total * (i + 0.5) / chunks) - per // 2) for i in range(chunks)]
+    out, seen = [], set()
+    for s in starts:
+        off = s
+        while off < s + per and len(out) < n:
+            try:
+                got = _get("rows", {"dataset": dataset, "config": info["config"], "split": info["split"], "offset": off, "length": min(100, s + per - off)}, _token())
+            except Exception as e:
+                if not out:
+                    raise HTTPException(502, f"The dataset viewer did not return rows for {dataset}: {str(e)[:160]}")
+                break
+            rows = got.get("rows") or []
+            for r in rows:
+                idx = r.get("row_idx", off)
+                if idx not in seen:
+                    seen.add(idx)
+                    out.append(r.get("row") or {})
+            if len(rows) < min(100, s + per - off):
+                break
+            off += len(rows)
+    if not out:
+        raise HTTPException(400, f"{dataset} has no rows the viewer can show.")
+    return out[:n]
+
+
+def _parse_upload(text: str) -> list[dict]:
+    t = text.lstrip("")
+    s = t.lstrip()
+    try:
+        if s.startswith("["):
+            data = json.loads(s)
+        elif s.startswith("{"):
+            try:
+                one = json.loads(s)
+                data = next((v for v in one.values() if isinstance(v, list)), [one]) if isinstance(one, dict) else []
+            except ValueError:
+                data = [json.loads(line) for line in s.splitlines() if line.strip()]
+        else:
+            csv.field_size_limit(10_000_000)
+            data = list(csv.DictReader(io.StringIO(t)))
+    except (ValueError, csv.Error) as e:
+        raise HTTPException(400, f"Could not read that file as CSV, JSON or JSONL: {str(e)[:120]}")
+    rows = [r for r in data if isinstance(r, dict)]
+    if not rows:
+        raise HTTPException(400, "The file has no rows. Use CSV with a header row, a JSON list of objects, or JSONL.")
+    return rows
+
+
+def _job_rows(jid: str) -> tuple[list[dict], str]:
+    from main import JOBS
+    j = JOBS.get(jid)
+    if not j:
+        raise HTTPException(404, "That run is no longer on this host. Download it and upload the file instead.")
+    with j.lock:
+        rows = list(j.rows)
+    if not rows:
+        raise HTTPException(400, "That run has no rows yet.")
+    keys = [k for k in JOB_ORDER if k in rows[0]]
+    req = (j.config.get("request") or "Your run").strip()
+    return [{k: r.get(k) for k in keys} for r in rows], req if len(req) <= 80 else req[:78].rsplit(" ", 1)[0] + "…"
+
+
+class LoadIn(BaseModel):
+    source: str              # job | sample | hub | upload
+    job: str = ""
+    key: str = ""
+    dataset: str = ""
+    text: str = ""
+    filename: str = ""
+    n: int = 50
+
+
+@router.get("/samples")
+def samples():
+    return {"samples": [{"key": k, "title": t, "repo": f"LocalLLaMA/datainventor-{k}"} for k, t in SAMPLES], "max_rows": MAX_ROWS}
+
+
+@router.post("/load")
+def load(body: LoadIn):
+    n = max(5, min(body.n or 50, MAX_ROWS))
+    note = ""
+    if body.source == "job":
+        raw, name = _job_rows(body.job)
+        src = "This page's run"
+    elif body.source == "sample":
+        if body.key not in dict(SAMPLES):
+            raise HTTPException(400, "Unknown sample.")
+        raw, name, src = _hub_rows(f"LocalLLaMA/datainventor-{body.key}", n), dict(SAMPLES)[body.key], f"LocalLLaMA/datainventor-{body.key}"
+    elif body.source == "hub":
+        ds = body.dataset.strip().removeprefix("https://huggingface.co/datasets/").strip("/")
+        raw, name, src = _hub_rows(ds, n), ds.split("/")[-1], ds
+    elif body.source == "upload":
+        if len(body.text) > MAX_UPLOAD:
+            raise HTTPException(413, "That file is over 6 MB. Upload a smaller sample.")
+        raw, name, src = _parse_upload(body.text), (body.filename or "Your file")[:80], "Uploaded file"
+    else:
+        raise HTTPException(400, "Unknown source.")
+    if len(raw) > MAX_ROWS:
+        note = f"Showing the first {MAX_ROWS} of {len(raw)} rows."
+        raw = raw[:MAX_ROWS]
+    rows, cols = V.prepare(raw)
+    if not cols:
+        raise HTTPException(400, "None of the columns hold text, categories or numbers this can chart.")
+    sid = uuid.uuid4().hex[:10]
+    vs = VisSet(id=sid, name=name, source=src, rows=rows, cols=cols)
+    with GUARD:
+        SETS[sid] = vs
+        for old in sorted(SETS, key=lambda k: SETS[k].created)[:-30]:
+            SETS.pop(old, None)
+    return {"id": sid, "name": name, "source": src, "n": len(rows), "note": note, "columns": V.describe(rows, cols),
+            "glance": V.glance(rows, cols), "suggestions": V.suggestions(rows, cols), "rows": rows}
+
+
+# ------------------------------------------------------------------ asking
+class AskIn(BaseModel):
+    question: str
+    scope: list[int] = []
+    scope_label: str = ""
+
+
+def _column(d: dict, vals: dict, filled: set, prev: Optional[dict] = None) -> dict:
+    c = {"type": "num" if d["kind"] in ("score", "count") else "cat", "derived": d["kind"], "from": d["from"],
+         "guide": d.get("guide", ""), "filled": set(filled)}
+    if d.get("labels"):
+        c["labels"] = d["labels"]
+    if d.get("pattern"):
+        c["pattern"] = d["pattern"]
+    if prev:
+        c["filled"] |= prev.get("filled", set())
+    return c
+
+
+def run_ask(a: Ask, vs: VisSet, ep):
+    # short per-call timeouts: a stuck provider is retried instead of holding the chart for minutes
+    chat = lambda prompt, **kw: ep.chat(prompt, retries=3, timeout=CALL_SECONDS, **kw)
+    try:
+        ids = set(a.scope)
+        rows = [r for r in vs.rows if r["_id"] in ids] if ids else list(vs.rows)
+        scope = f" (this question is about a selection of {len(rows)} of {len(vs.rows)} rows: {a.scope_label or 'chosen on a chart'})" if ids else ""
+        plan, err = None, ""
+        for _ in range(2):
+            text = chat(V.plan_prompt(a.question, vs.name, rows, vs.cols, scope, err), max_tokens=700, temperature=0.2)
+            try:
+                plan = V.parse_plan(text, vs.cols)
+                break
+            except V.PlanError as e:
+                err = str(e)
+                print(f"[visualize] plan rejected: {err} | reply: {text[:600]!r}", flush=True)
+        if not plan:
+            plan = V.fallback_plan(a.question, rows, vs.cols)
+        # new columns, plus earlier model-made columns this chart needs that do not cover these rows yet
+        todo = [(d, rows, False) for d in plan["derive"]]
+        for k in ("x", "y", "color"):
+            c = vs.cols.get(plan["chart"].get(k) or "")
+            if c and c.get("derived") in V.KINDS:
+                missing = [r for r in rows if r["_id"] not in c.get("filled", set())]
+                if missing:
+                    todo.append(({"name": plan["chart"][k], "kind": c["derived"], "from": c["from"], "guide": c.get("guide", ""),
+                                  "labels": c.get("labels", []), "pattern": c.get("pattern", "")}, missing, True))
+        touched = {}
+        if todo:
+            a.stage, a.total, a.done = "label", sum(len(rs) for _, rs, _ in todo), 0
+            for d, rs, refill in todo:
+                base = a.done
+                vals = V.fill(d, rs, chat, a.question, progress=lambda k, base=base: setattr(a, "done", base + k))
+                with vs.lock:
+                    for r in vs.rows:
+                        if r["_id"] in vals:
+                            r[d["name"]] = vals[r["_id"]]
+                        elif not refill or d["name"] not in r:  # a new definition: clear values left by an old one
+                            r[d["name"]] = None if d["kind"] in ("score", "count") else ""
+                    vs.cols[d["name"]] = _column(d, vals, set(vals), vs.cols.get(d["name"]) if refill else None)
+                touched[d["name"]] = {str(i): v for i, v in vals.items()}
+        a.stage = "chart"
+        with vs.lock:
+            c = V.chart(rows, vs.cols, V.check_spec(plan["chart"], vs.cols))
+            columns = V.describe(vs.rows, vs.cols)
+        made = [{"name": d["name"], "from": d["from"]} for d, _, _ in todo]
+        made = list({m["name"]: m for m in made}.values())
+        a.result = {"title": plan["title"], "chart": c, "summary": "", "notes": V.notes(c, made, plan.get("fallback", False)),
+                    "facts": V.facts(c), "follow_ups": plan["follow_ups"], "derived": touched, "columns": columns,
+                    "scope": len(rows) if ids else 0}
+        a.stage = "write"
+        try:
+            # the chart is already on screen; a slow write-up gives way to the exact numbers after a while
+            fut = WRITERS.submit(chat, V.summary_prompt(a.question, plan["title"], c, made), max_tokens=320, temperature=0.3)
+            text = fut.result(timeout=WRITE_SECONDS)
+            text = re.sub(r"[*#_`]+", "", re.sub(r"^\s*[-•]\s+", "", text, flags=re.M)).strip()
+            text = re.sub(r"\s*\n\s*", " ", text)
+            a.result["summary"] = " ".join(re.split(r"(?<=[.!?])\s+(?=[A-Z0-9])", text)[:4])[:700]
+        except (EndpointError, FutureTimeout):
+            pass
+        a.stage = "done"
+    except (EndpointError, V.PlanError) as e:
+        a.stage, a.error = "failed", str(e)[:300]
+    except Exception as e:  # never leave the page polling a dead ask
+        a.stage, a.error = "failed", f"Something went wrong drawing that: {str(e)[:200]}"
+    finally:
+        vs.busy = False
+
+
+@router.post("/{sid}/ask")
+def ask(sid: str, body: AskIn):
+    vs = SETS.get(sid)
+    if not vs:
+        raise HTTPException(404, "That dataset is no longer loaded here. Load it again.")
+    q = re.sub(r"\s+", " ", body.question).strip()
+    if not q:
+        raise HTTPException(400, "Type a question first.")
+    ep = _ep()
+    with GUARD:
+        if vs.busy:
+            raise HTTPException(409, "This dataset is still answering the last question.")
+        if sum(1 for a in ASKS.values() if a.stage not in ("done", "failed")) >= MAX_RUNNING:
+            raise HTTPException(429, "Several charts are being drawn right now. Try again in a moment.")
+        aid = uuid.uuid4().hex[:10]
+        a = Ask(id=aid, set_id=sid, question=q[:400], scope=[i for i in body.scope if isinstance(i, int)][:MAX_ROWS], scope_label=body.scope_label[:120])
+        ASKS[aid] = a
+        vs.busy = True
+        for old in sorted(ASKS, key=lambda k: ASKS[k].started)[:-200]:
+            ASKS.pop(old, None)
+    threading.Thread(target=run_ask, args=(a, vs, ep), daemon=True).start()
+    return {"ask": aid}
+
+
+@router.get("/ask/{aid}")
+def ask_status(aid: str):
+    a = ASKS.get(aid)
+    if not a:
+        raise HTTPException(404, "That question is no longer here. Ask it again.")
+    out = {"stage": a.stage, "done": a.done, "total": a.total, "error": a.error, "elapsed": round(time.time() - a.started, 1)}
+    if a.result and a.stage in ("write", "done"):
+        out["result"] = a.result
+    return out
+
+
+@router.get("/{sid}/download")
+def download(sid: str, ids: str = "", format: str = "csv"):
+    vs = SETS.get(sid)
+    if not vs:
+        raise HTTPException(404, "That dataset is no longer loaded here.")
+    want = {int(x) for x in ids.split(",") if x.strip().isdigit()}
+    rows = [r for r in vs.rows if not want or r["_id"] in want]
+    keys = ["_id"] + [k for k, c in vs.cols.items() if c.get("derived") not in ("words", "turns")]
+    name = re.sub(r"[^\w-]+", "-", vs.name.lower()).strip("-")[:40] or "rows"
+    if format == "jsonl":
+        body = "\n".join(json.dumps({("row" if k == "_id" else k): r.get(k) for k in keys}, ensure_ascii=False) for r in rows)
+        return Response(body, media_type="application/jsonl", headers={"Content-Disposition": f'attachment; filename="{name}-{len(rows)}.jsonl"'})
+    buf = io.StringIO()
+    w = csv.writer(buf)
+    w.writerow(["row" if k == "_id" else k for k in keys])
+    for r in rows:
+        w.writerow(["" if r.get(k) is None else r.get(k) for k in keys])
+    return Response(buf.getvalue().encode("utf-8-sig"), media_type="text/csv", headers={"Content-Disposition": f'attachment; filename="{name}-{len(rows)}.csv"'})

datainventor/llm.py

@@ -31,7 +31,7 @@ class Endpoint:
     min_max_tokens: int = 0
 
     def chat(self, prompt: str, max_tokens: int = 1500, temperature: float = 0.8, retries: int = 5,
-             validate: Optional[Callable[[str], bool]] = None, allow_cut: bool = False) -> str:
+             validate: Optional[Callable[[str], bool]] = None, allow_cut: bool = False, timeout: int = 180) -> str:
         reasoning = bool(_REASONING.search(self.model)) and not self.disable_thinking
         body = {"model": self.model,
                 "max_tokens": max(self.min_max_tokens, max_tokens + (REASONING_HEADROOM if reasoning else 0)),
@@ -47,7 +47,7 @@ class Endpoint:
             try:
                 req = urllib.request.Request(url, data, {"Content-Type": "application/json", "User-Agent": "DataInventor/1.0",
                                                          "Authorization": "Bearer " + self.api_key})
-                resp = json.load(urllib.request.urlopen(req, timeout=180))
+                resp = json.load(urllib.request.urlopen(req, timeout=timeout))
                 choice = (resp.get("choices") or [{}])[0] or {}
                 # allow_cut: short answers (a verdict letter) can be read even if the model kept talking
                 if choice.get("finish_reason") == "length" and not (allow_cut and ((choice.get("message") or {}).get("content") or "").strip()):

datainventor/visualize.py

@@ -0,0 +1,738 @@
+"""Visualize: ask a question about a dataset, get a chart and the rows behind every mark.
+
+Built for small datasets first, the 25 to 200 rows a hosted run writes. Every row stays in
+memory, charts are exact counts over all rows (nothing is sampled), and when a question needs
+something no column has (tone, whether a reply gives steps), a model reads the rows in a few
+batched calls and the answer becomes a new column that later questions can reuse.
+"""
+from __future__ import annotations
+
+import json
+import math
+import re
+import statistics
+from concurrent.futures import ThreadPoolExecutor
+from typing import Callable, Optional
+
+ID_NAMES = {"id", "_id", "idx", "index", "row_idx", "uuid", "row_id"}
+CLASS_NAMES = {"label", "labels", "class", "category", "target", "sentiment", "rating", "stars", "grade"}
+MAX_COLUMNS = 30
+MAX_CELL = 4000           # characters kept per cell
+LABEL_CHARS = 600         # characters of each row a labelling call reads
+LABEL_BATCH = 8
+MAX_GROUPS = 12           # categories on one axis before the rest fold into "Other"
+MAX_COLORS = 6
+KINDS = ("label", "score", "pattern", "count")
+CHARTS = ("bar", "stack", "box", "hist", "scatter")
+
+
+class PlanError(ValueError):
+    pass
+
+
+# ------------------------------------------------------------------ rows and columns
+def _turns(v) -> Optional[list]:
+    """Chat-style cells: [{"role", "content"}] or ShareGPT [{"from", "value"}]."""
+    if isinstance(v, list) and v and all(isinstance(x, dict) for x in v):
+        if all("content" in x for x in v):
+            return [(str(x.get("role") or "turn"), x.get("content")) for x in v]
+        if all("value" in x for x in v):
+            return [(str(x.get("from") or "turn"), x.get("value")) for x in v]
+    return None
+
+
+def _cell(v):
+    if v is None:
+        return ""
+    if isinstance(v, bool):
+        return "yes" if v else "no"
+    if isinstance(v, (int, float)):
+        return v if math.isfinite(v) else ""
+    if isinstance(v, str):
+        return v[:MAX_CELL]
+    t = _turns(v)
+    if t:
+        return "\n".join(f"{r}: {c if isinstance(c, str) else json.dumps(c, ensure_ascii=False)}" for r, c in t)[:MAX_CELL]
+    return json.dumps(v, ensure_ascii=False)[:MAX_CELL]
+
+
+def _num(v) -> Optional[float]:
+    if isinstance(v, bool):
+        return None
+    if isinstance(v, (int, float)):
+        return float(v) if math.isfinite(v) else None
+    if isinstance(v, str) and v.strip():
+        try:
+            f = float(v.strip().replace(",", ""))
+            return f if math.isfinite(f) else None
+        except ValueError:
+            return None
+    return None
+
+
+def words(text) -> int:
+    return len(str(text or "").split())
+
+
+def prepare(raw: list[dict]) -> tuple[list[dict], dict]:
+    """Normalize raw rows and type every column: text, category or number.
+
+    Returns (rows, columns). Each row gets `_id` (1-based). Long text columns get a derived
+    "<name> words" number column, and chat columns a "<name> turns" one.
+    """
+    names: list[str] = []
+    for r in raw:
+        for k in r:
+            if k not in names and str(k).lower() not in ID_NAMES and not str(k).startswith("_"):
+                names.append(k)
+    names = names[:MAX_COLUMNS]
+    rows, turns = [], {}
+    for i, r in enumerate(raw, 1):
+        row = {"_id": i}
+        for k in names:
+            v = r.get(k)
+            t = _turns(v)
+            if t:
+                turns.setdefault(k, {})[i] = len(t)
+            row[k] = _cell(v)
+        rows.append(row)
+    cols: dict[str, dict] = {}
+    n = len(rows)
+    for k in names:
+        vals = [row[k] for row in rows if row[k] != ""]
+        if not vals:
+            continue
+        nums = [_num(v) for v in vals]
+        few = all(x is not None for x in nums) and all(float(x).is_integer() for x in nums) and \
+            (len(set(nums)) <= 2 or (str(k).lower() in CLASS_NAMES and len(set(nums)) <= 12))
+        if few:  # 0/1 flags and class ids are categories, not amounts
+            for row in rows:
+                row[k] = str(int(_num(row[k]))) if row[k] != "" else ""
+            cols[k] = {"type": "cat"}
+            continue
+        if all(x is not None for x in nums) and not all(isinstance(v, str) and len(v) > 12 for v in vals):
+            for row in rows:
+                row[k] = _num(row[k]) if row[k] != "" else None
+            cols[k] = {"type": "num"}
+            continue
+        strs = [str(v) for v in vals]
+        avg = sum(len(s) for s in strs) / len(strs)
+        distinct = len(set(strs))
+        if distinct == len(strs) and len(strs) > 1 and not any(" " in x for x in strs) and k not in turns:
+            continue  # hashes, ids, urls: unique and never a phrase
+        if avg > 60 or k in turns:
+            cols[k] = {"type": "text"}
+        elif distinct <= 30 and (distinct <= max(6, 0.6 * len(strs)) if len(strs) > 6 else distinct < len(strs)):
+            cols[k] = {"type": "cat"}
+        elif avg >= 12:
+            cols[k] = {"type": "text"}
+        else:
+            continue  # short and unique: names, ids, codes
+        for row in rows:
+            if not isinstance(row[k], str):
+                row[k] = str(row[k])
+    for k, c in list(cols.items()):
+        if c["type"] != "text":
+            continue
+        if k in turns:
+            name = f"{k} turns"
+            for row in rows:
+                row[name] = float(turns[k].get(row["_id"], 0))
+            cols[name] = {"type": "num", "derived": "turns", "from": k}
+        avg_words = sum(words(row[k]) for row in rows) / max(1, n)
+        if avg_words >= 8:
+            name = f"{k} words"
+            for row in rows:
+                row[name] = float(words(row[k]))
+            cols[name] = {"type": "num", "derived": "words", "from": k}
+    order = {"text": 0, "cat": 1, "num": 2}
+    cols = dict(sorted(cols.items(), key=lambda kv: (order[kv[1]["type"]], bool(kv[1].get("derived")))))
+    return rows, cols
+
+
+def describe(rows: list[dict], cols: dict) -> list[dict]:
+    """Per-column facts for the UI and the planner."""
+    out = []
+    for k, c in cols.items():
+        d = {"name": k, "type": c["type"], "derived": c.get("derived", ""), "from": c.get("from", "")}
+        vals = [r.get(k) for r in rows]
+        if c["type"] == "cat":
+            counts: dict[str, int] = {}
+            for v in vals:
+                counts[v or "(blank)"] = counts.get(v or "(blank)", 0) + 1
+            d["values"] = sorted(counts.items(), key=lambda kv: -kv[1])[:12]
+            d["distinct"] = len(counts)
+        elif c["type"] == "num":
+            xs = [v for v in vals if isinstance(v, (int, float))]
+            if xs:
+                d.update(min=min(xs), max=max(xs), median=statistics.median(xs))
+        else:
+            xs = [words(v) for v in vals if v]
+            d["avg_words"] = round(sum(xs) / max(1, len(xs)))
+            d["example"] = next((str(v)[:140] for v in vals if v), "")
+        if c.get("labels"):
+            d["labels"] = c["labels"]
+        if c.get("guide"):
+            d["guide"] = c["guide"]
+        out.append(d)
+    return out
+
+
+# ------------------------------------------------------------------ charts
+def _q(xs: list[float], p: float) -> float:
+    s = sorted(xs)
+    if len(s) == 1:
+        return s[0]
+    pos = (len(s) - 1) * p
+    lo = int(pos)
+    return s[lo] + (s[min(lo + 1, len(s) - 1)] - s[lo]) * (pos - lo)
+
+
+def _nice(x: float) -> float:
+    if x <= 0:
+        return 1.0
+    e = 10 ** math.floor(math.log10(x))
+    for m in (1, 2, 2.5, 5, 10):
+        if x <= m * e:
+            return m * e
+    return 10 * e
+
+
+def _key(v) -> str:
+    return str(v) if v not in ("", None) else "(blank)"
+
+
+def _fold(groups: dict[str, list[int]], limit: int) -> list[tuple[str, list[int]]]:
+    items = sorted(groups.items(), key=lambda kv: -len(kv[1]))
+    if len(items) <= limit:
+        return items
+    keep, rest = items[:limit - 1], items[limit - 1:]
+    ids = [i for _, g in rest for i in g]
+    return keep + [(f"Other ({len(rest)})", ids)]
+
+
+def chart(rows: list[dict], cols: dict, spec: dict) -> dict:
+    """Exact aggregation of `rows` for one chart spec. Every mark carries the ids of its rows."""
+    t, x, y, color, agg = spec.get("type"), spec.get("x"), spec.get("y"), spec.get("color"), spec.get("agg") or "count"
+    by = {r["_id"]: r for r in rows}
+    out = {"type": t, "n": len(rows), "x": x, "y": y, "color": color, "agg": agg}
+
+    def groups_of(col):
+        g: dict[str, list[int]] = {}
+        for r in rows:
+            g.setdefault(_key(r.get(col)), []).append(r["_id"])
+        return g
+
+    def nums(ids, col):
+        return [(i, by[i][col]) for i in ids if isinstance(by[i].get(col), (int, float))]
+
+    if t == "bar":
+        marks = []
+        for k, ids in _fold(groups_of(x), MAX_GROUPS):
+            if y and agg in ("mean", "median"):
+                pts = nums(ids, y)
+                if not pts:
+                    continue
+                vs = [v for _, v in pts]
+                val = statistics.fmean(vs) if agg == "mean" else statistics.median(vs)
+                marks.append({"key": k, "value": round(val, 1 if abs(val) >= 10 else 2), "n": len(pts), "ids": [i for i, _ in pts]})
+            else:
+                marks.append({"key": k, "value": len(ids), "n": len(ids), "ids": ids})
+        marks.sort(key=lambda m: -m["value"])
+        out["marks"] = marks
+    elif t == "stack":
+        colors = [k for k, _ in _fold(groups_of(color), MAX_COLORS)]
+        color_ids = dict(_fold(groups_of(color), MAX_COLORS))
+        rev = {i: k for k, ids in color_ids.items() for i in ids}
+        marks = []
+        for k, ids in _fold(groups_of(x), MAX_GROUPS):
+            parts = []
+            for c in colors:
+                pids = [i for i in ids if rev.get(i) == c]
+                if pids:
+                    parts.append({"key": c, "value": len(pids), "ids": pids})
+            marks.append({"key": k, "n": len(ids), "ids": ids, "parts": parts})
+        marks.sort(key=lambda m: -m["n"])
+        out.update(marks=marks, colors=colors)
+    elif t == "box":
+        gs = _fold(groups_of(x), MAX_GROUPS) if x else [("All rows", [r["_id"] for r in rows])]
+        marks = []
+        for k, ids in gs:
+            pts = nums(ids, y)
+            if not pts:
+                continue
+            vs = [v for _, v in pts]
+            q1, med, q3 = _q(vs, .25), _q(vs, .5), _q(vs, .75)
+            iqr = q3 - q1
+            lo = min(v for v in vs if v >= q1 - 1.5 * iqr)
+            hi = max(v for v in vs if v <= q3 + 1.5 * iqr)
+            marks.append({"key": k, "n": len(pts), "q1": q1, "median": med, "q3": q3, "lo": lo, "hi": hi,
+                          "ids": [i for i, _ in pts], "points": [[i, v] for i, v in pts]})
+        marks.sort(key=lambda m: -m["median"])
+        out["marks"] = marks
+    elif t == "hist":
+        pts = nums([r["_id"] for r in rows], x)
+        vs = [v for _, v in pts]
+        if vs and all(float(v).is_integer() for v in vs) and max(vs) - min(vs) <= 12:
+            # small whole numbers (scores, counts): one bar per value
+            out["discrete"] = True
+            out["marks"] = [{"x0": v, "x1": v + 1, "value": len(ids), "ids": ids}
+                            for v in range(int(min(vs)), int(max(vs)) + 1) for ids in [[i for i, x in pts if x == v]]]
+        elif vs:
+            lo, hi = min(vs), max(vs)
+            k = 6 if len(vs) <= 30 else min(14, int(math.sqrt(len(vs))) + 2)
+            step = _nice((hi - lo) / k) if hi > lo else 1.0
+            start = math.floor(lo / step) * step
+            nb = max(1, min(40, math.ceil((hi - start) / step + 1e-9)))
+            if start + nb * step <= hi:
+                nb += 1
+            bins = [{"x0": round(start + b * step, 4), "x1": round(start + (b + 1) * step, 4), "value": 0, "ids": []} for b in range(nb)]
+            for i, v in pts:
+                b = min(nb - 1, int((v - start) / step))
+                bins[b]["ids"].append(i)
+                bins[b]["value"] += 1
+            out["marks"] = bins
+        else:
+            out["marks"] = []
+    elif t == "scatter":
+        rev = {}
+        if color:
+            for k, ids in _fold(groups_of(color), MAX_COLORS):
+                for i in ids:
+                    rev[i] = k
+            out["colors"] = [k for k, _ in _fold(groups_of(color), MAX_COLORS)]
+        marks = [{"id": r["_id"], "x": r[x], "y": r[y], "c": rev.get(r["_id"], "")} for r in rows
+                 if isinstance(r.get(x), (int, float)) and isinstance(r.get(y), (int, float))]
+        out["marks"] = marks
+        if len(marks) >= 3:
+            xs, ys = [m["x"] for m in marks], [m["y"] for m in marks]
+            try:
+                out["r"] = round(statistics.correlation(xs, ys), 2)
+            except statistics.StatisticsError:
+                pass
+    else:
+        raise PlanError(f"Unknown chart type {t!r}.")
+    return out
+
+
+def check_spec(spec: dict, cols: dict) -> dict:
+    """Validate a chart spec against the columns, fixing the mistakes that have one obvious fix."""
+    s = {k: spec.get(k) for k in ("type", "x", "y", "color", "agg")}
+    for k in ("x", "y", "color"):
+        if s[k] in ("", "null", "none", "None"):
+            s[k] = None
+    typ = lambda c: (cols.get(c) or {}).get("type")
+    for k in ("x", "y", "color"):
+        if s[k] is not None and s[k] not in cols:
+            raise PlanError(f"There is no column named {s[k]!r}. Columns: {', '.join(cols)}.")
+    t = s["type"]
+    if t not in CHARTS:
+        raise PlanError(f"chart.type must be one of {', '.join(CHARTS)}.")
+    if t == "bar" and typ(s["x"]) == "num":
+        t, s["y"] = "hist", None
+    if t == "box" and s["y"] is None and typ(s["x"]) == "num":
+        s["x"], s["y"] = None, s["x"]
+    if t == "stack" and not s["color"]:
+        t = "bar"
+    if t == "bar":
+        if typ(s["x"]) not in ("cat",):
+            raise PlanError("A bar chart needs x to be a category column.")
+        if s["y"] and typ(s["y"]) != "num":
+            raise PlanError("A bar chart's y must be a number column, or null to count rows.")
+        s["agg"] = (s["agg"] if s["agg"] in ("mean", "median") else "mean") if s["y"] else "count"
+    elif t == "stack":
+        if typ(s["x"]) != "cat" or typ(s["color"]) != "cat":
+            raise PlanError("A stacked chart needs x and color to be category columns.")
+        s["y"], s["agg"] = None, s["agg"] if s["agg"] in ("share", "count") else "share"
+    elif t == "box":
+        if typ(s["y"]) != "num":
+            raise PlanError("A box plot needs y to be a number column.")
+        if s["x"] and typ(s["x"]) != "cat":
+            raise PlanError("A box plot's x must be a category column or null.")
+    elif t == "hist":
+        if typ(s["x"]) != "num":
+            raise PlanError("A histogram needs x to be a number column.")
+    elif t == "scatter":
+        if typ(s["x"]) != "num" or typ(s["y"]) != "num":
+            raise PlanError("A scatter plot needs x and y to be number columns.")
+        if s["color"] and typ(s["color"]) != "cat":
+            s["color"] = None
+    s["type"] = t
+    return s
+
+
+# ------------------------------------------------------------------ instant views (no model)
+def _main_cat(cols: dict, rows: list[dict]) -> Optional[str]:
+    prefer = ("area", "category", "label", "topic", "subtopic", "department", "language")
+    cats = [k for k, c in cols.items() if c["type"] == "cat" and len({r.get(k) for r in rows}) > 1]
+    for p in prefer:
+        if p in cats:
+            return p
+    return min(cats, key=lambda k: abs(len({r.get(k) for r in rows}) - 5)) if cats else None
+
+
+def _main_len(cols: dict) -> Optional[str]:
+    prefer = ("completion words", "chosen words", "response words", "output words", "answer words", "text words")
+    nums = [k for k, c in cols.items() if c.get("derived") == "words"]
+    for p in prefer:
+        if p in nums:
+            return p
+    return nums[-1] if nums else None
+
+
+def glance(rows: list[dict], cols: dict) -> list[dict]:
+    """Two or three charts that need no model, so a dataset shows something the moment it loads."""
+    out = []
+    cat, ln = _main_cat(cols, rows), _main_len(cols)
+    if cat:
+        out.append({"title": f"Rows by {cat}", **chart(rows, cols, {"type": "bar", "x": cat})})
+    if ln:
+        spec = {"type": "box", "x": cat, "y": ln} if cat else {"type": "hist", "x": ln}
+        out.append({"title": f"{ln.capitalize()}" + (f" by {cat}" if cat else ""), **chart(rows, cols, spec)})
+    if not cat:
+        more = [k for k, c in cols.items() if c.get("derived") in ("words", "turns") and k != ln]
+        if more:
+            out.append({"title": more[0].capitalize(), **chart(rows, cols, {"type": "hist", "x": more[0]})})
+    second = [k for k, c in cols.items() if c["type"] == "cat" and k != cat and 1 < len({r.get(k) for r in rows}) <= 8]
+    if second:
+        k = "language" if "language" in second else "difficulty" if "difficulty" in second else second[0]
+        out.append({"title": f"Rows by {k}", **chart(rows, cols, {"type": "bar", "x": k})})
+    return out[:3]
+
+
+def _pl(name: str) -> str:
+    return name if name.endswith("s") else name + "s"
+
+
+def suggestions(rows: list[dict], cols: dict) -> list[str]:
+    cat, ln = _main_cat(cols, rows), _main_len(cols)
+    texts = [k for k, c in cols.items() if c["type"] == "text"]
+    reply = next((k for k in ("completion", "chosen", "response", "output", "answer", "messages", "conversations") if k in texts), None)
+    ask = next((k for k in ("prompt", "instruction", "question", "input", "query") if k in texts), None)
+    other = next((k for k in texts if k not in (reply, ask)), None)
+    out = []
+    if reply:
+        out.append(f"What tone do the {_pl(reply)} take" + (f" in each {cat}?" if cat else "?"))
+    if ask:
+        out.append(f"What do the {_pl(ask)} ask for? Group them into a few themes.")
+    if not ask and not reply and other:
+        out.append(f"What themes come up in the {_pl(other)}?")
+        out.append(f"How positive or negative is each {other}?")
+    if ln and cat:
+        out.append(f"Which {cat} gets the longest {_pl(ln.replace(' words', ''))}?")
+    elif ln:
+        out.append(f"How long are the {_pl(ln.replace(' words', ''))}?")
+    if reply:
+        out.append(f"How many {_pl(reply)} give step-by-step instructions?")
+    seconds = [k for k, c in cols.items() if c["type"] == "cat" and k != cat and 1 < len({r.get(k) for r in rows}) <= 8]
+    if cat and seconds:
+        out.append(f"How does {seconds[0]} vary across {cat}?")
+    return out[:4]
+
+
+# ------------------------------------------------------------------ the model's part
+def first_json(text: str):
+    """The first JSON object or list in a model reply (fences and chatter around it are ignored)."""
+    dec = json.JSONDecoder()
+    for i, ch in enumerate(text):
+        if ch in "{[":
+            try:
+                return dec.raw_decode(text[i:])[0]
+            except ValueError:
+                continue
+    raise ValueError("no JSON in the reply")
+
+
+def _snip(v, n=LABEL_CHARS) -> str:
+    return re.sub(r"\s+", " ", str(v or "")).strip()[:n]
+
+
+def plan_prompt(question: str, name: str, rows: list[dict], cols: dict, scope: str = "", error: str = "") -> str:
+    lines = []
+    for d in describe(rows, cols):
+        if d["type"] == "cat":
+            vals = ", ".join(f"{k} {v}" for k, v in d["values"][:8])
+            more = f" (+{d['distinct'] - 8} more)" if d["distinct"] > 8 else ""
+            lines.append(f'- "{d["name"]}" (category, {d["distinct"]} values: {vals}{more})')
+        elif d["type"] == "num":
+            if "min" in d:
+                lines.append(f'- "{d["name"]}" (number, {d["min"]:g} to {d["max"]:g}, median {d["median"]:g})')
+        else:
+            lines.append(f'- "{d["name"]}" (text, about {d["avg_words"]} words) e.g. "{_snip(d["example"], 100)}"')
+        if d.get("derived") in ("label", "score", "pattern", "count"):
+            lines[-1] += f' [made earlier by reading "{d["from"]}"]'
+    texts = [k for k, c in cols.items() if c["type"] in ("text", "cat")][:5]
+    ex = []
+    for r in rows[:5]:
+        ex.append(f"#{r['_id']} " + " | ".join(f"{k}: {_snip(r.get(k), 110)}" for k in texts))
+    return f"""You turn a question about a dataset into a chart plan.
+
+Dataset: {name}, {len(rows)} rows{scope}.
+Columns:
+{chr(10).join(lines)}
+
+A few rows:
+{chr(10).join(ex)}
+
+Question: "{question}"
+
+Reply with only a JSON object:
+{{"title": "...", "derive": [], "chart": {{"type": "...", "x": "...", "y": null, "color": null, "agg": "count"}}, "follow_ups": ["...", "...", "..."]}}
+
+Rules:
+- Use the columns above when they can answer the question. They are instant.
+- Questions about what the text says, asks for, means or sounds like (themes, intent, tone, quality) need a derived column read from that text, unless a category column already names exactly that.
+- Chart a length or other number column only when the question is about length or amount.
+- Only when no column answers it, add one or two entries to "derive". Each becomes a new column a model fills by reading one column of every row:
+  {{"name": "tone", "kind": "label", "from": "completion", "labels": ["formal", "friendly", "curt"], "guide": "the overall tone of the reply"}}
+  kind "label": one of 2 to 7 short labels per row. Give "labels" yourself when the question or the rows above suggest them (faster); use [] only for open-ended themes, to have them discovered from the rows.
+  kind "score": an integer from 1 to 5; "guide" must say what 1 and 5 mean.
+  kind "pattern": yes or no, whether the case-insensitive regex in "pattern" matches. kind "count": how many times it matches.
+  Use "pattern" or "count" only for literal marks (links, numbered lists, tables, code blocks, digits); judge meaning with "label" or "score".
+  "name" is 1 to 3 lowercase words, "from" is a text or category column.
+- chart.type:
+  "bar": x is a category; y null counts rows, or y is a number column with agg "mean" or "median".
+  "stack": x and color are categories; agg "share" or "count". Use it to compare make-up across groups.
+  "box": y is a number column; x is a category or null. Use it for how a number spreads in each group.
+  "hist": x is a number column.
+  "scatter": x and y are number columns; color is a category or null.
+  A derived column can be used in the chart by its name.
+- "title": what the chart shows, under 60 characters, sentence case, no claims about the result.
+- "follow_ups": three short questions a person exploring this data might ask next.{chr(10) + chr(10) + "Your last plan was rejected: " + error + " Fix it." if error else ""}"""
+
+
+def _clean_name(raw: str) -> str:
+    return re.sub(r"\s+", " ", re.sub(r"[^a-z0-9 ]", "", re.sub(r"[_\-]+", " ", raw.lower()))).strip()[:24]
+
+
+def parse_plan(text: str, cols: dict) -> dict:
+    try:
+        p = first_json(text)
+    except ValueError:
+        raise PlanError("The reply was not JSON.")
+    if not isinstance(p, dict) or not isinstance(p.get("chart"), dict):
+        raise PlanError('The reply needs a "chart" object.')
+    derive = []
+    known = dict(cols)
+    for d in (p.get("derive") or [])[:2]:
+        if not isinstance(d, dict):
+            continue
+        kind = d.get("kind")
+        raw_name = str(d.get("name") or "")
+        name = _clean_name(raw_name)
+        for k in ("x", "y", "color"):  # the chart may refer to the name as the model wrote it
+            if p["chart"].get(k) is not None and p["chart"].get(k) not in cols and _clean_name(str(p["chart"][k])) == name:
+                p["chart"][k] = name
+        src = d.get("from")
+        if kind not in KINDS or not name:
+            raise PlanError(f"derive entries need a name and a kind from {', '.join(KINDS)}.")
+        if src not in cols or cols[src]["type"] not in ("text", "cat"):
+            raise PlanError(f'derive "from" must be a text or category column, not {src!r}.')
+        item = {"name": name, "kind": kind, "from": src, "guide": str(d.get("guide") or "")[:200]}
+        if kind == "label":
+            labels = [str(x).strip()[:40] for x in (d.get("labels") or []) if str(x).strip()]
+            item["labels"] = list(dict.fromkeys(labels))[:7]
+        elif kind in ("pattern", "count"):
+            pat = re.sub(r"^\(\?i\)", "", str(d.get("pattern") or ""))
+            if not pat or len(pat) > 160 or re.search(r"\([^)]*[+*]\)[+*{]", pat):
+                raise PlanError("pattern must be a short regex without nested repeats.")
+            try:
+                re.compile(pat, re.I)
+            except re.error:
+                raise PlanError(f"{pat!r} is not a valid regex.")
+            item["pattern"] = pat
+        if name in cols and cols[name].get("derived") == kind and cols[name].get("from") == src \
+                and (kind != "label" or not item["labels"] or set(item["labels"]) == set(cols[name].get("labels") or [])):
+            continue  # already made by an earlier question: reuse it
+        if name in cols and cols[name].get("derived") not in KINDS:  # an original column: never overwrite it
+            new = f"{name} (model)"
+            for k in ("x", "y", "color"):
+                if p["chart"].get(k) == name:
+                    p["chart"][k] = new
+            item["name"] = name = new
+        derive.append(item)
+        known[name] = {"type": "num" if kind in ("score", "count") else "cat"}
+    spec = check_spec(p["chart"], known)
+    title = re.sub(r"\s+", " ", str(p.get("title") or "")).strip()[:80] or "What the data shows"
+    ups = [re.sub(r"\s+", " ", str(q)).strip()[:140] for q in (p.get("follow_ups") or []) if str(q).strip()][:3]
+    return {"title": title, "derive": derive, "chart": spec, "follow_ups": ups}
+
+
+def fallback_plan(question: str, rows: list[dict], cols: dict) -> dict:
+    """Used when the model cannot produce a valid plan: answer with the closest instant chart."""
+    q = question.lower()
+    cat = next((k for k, c in cols.items() if c["type"] == "cat" and k.lower() in q), None) or _main_cat(cols, rows)
+    num = next((k for k, c in cols.items() if c["type"] == "num" and k.lower().replace(" words", "") in q), None)
+    if not num and re.search(r"\blong|length|short|words?\b", q):
+        num = _main_len(cols)
+    if num:
+        spec = {"type": "box", "x": cat, "y": num} if cat else {"type": "hist", "x": num}
+        title = f"{num.capitalize()}" + (f" by {cat}" if cat else "")
+    elif cat:
+        spec, title = {"type": "bar", "x": cat}, f"Rows by {cat}"
+    else:
+        raise PlanError("This data has no column a chart can use.")
+    return {"title": title, "derive": [], "chart": check_spec(spec, cols), "follow_ups": [], "fallback": True}
+
+
+def discover_prompt(question: str, d: dict, rows: list[dict]) -> str:
+    snips = "\n".join(f"- {_snip(r.get(d['from']), 220)}" for r in rows[:40])
+    return f"""Here are {min(40, len(rows))} entries from the "{d['from']}" column of a dataset. Someone asked: "{question}"
+
+We need a label for every row: "{d['name']}"{f" ({d['guide']})" if d['guide'] else ""}.
+Propose 3 to 7 short labels (1 to 4 words each, lowercase) that together cover these entries, most common first. Use labels a reader would understand without seeing the data.
+
+{snips}
+
+Reply with only a JSON list of strings."""
+
+
+def label_prompt(d: dict, batch: list[dict]) -> str:
+    items = "\n".join(f"[{k}] {_snip(r.get(d['from']))}" for k, r in enumerate(batch, 1))
+    if d["kind"] == "label":
+        ask = f'Pick exactly one label for each item from: {", ".join(json.dumps(x) for x in d["labels"])}.'
+        eg = ", ".join(f'"{k}": {json.dumps(d["labels"][k % len(d["labels"])])}' for k in (1, 2))
+    else:
+        ask = "Give each item an integer score from 1 to 5."
+        eg = '"1": 4, "2": 2'
+    return f"""You are labelling rows of a dataset.
+{d['name'].capitalize()}: {d['guide'] or d['name']}
+{ask}
+
+{items}
+
+Reply with only a JSON object mapping each item number to its {"label" if d["kind"] == "label" else "score"}, like {{{eg}}}."""
+
+
+def _read_labels(text: str, d: dict, k: int) -> Optional[dict]:
+    try:
+        obj = first_json(text)
+    except ValueError:
+        return None
+    if isinstance(obj, list):
+        obj = {str(i): v for i, v in enumerate(obj, 1)}
+    if not isinstance(obj, dict):
+        return None
+    out = {}
+    canon = {x.lower(): x for x in d.get("labels", [])}
+    for key, v in obj.items():
+        m = re.search(r"\d+", str(key))
+        if not m or not 1 <= int(m.group()) <= k:
+            continue
+        i = int(m.group())
+        if d["kind"] == "label":
+            s = str(v).strip().lower()
+            hit = canon.get(s) or next((canon[c] for c in canon if s.startswith(c) or c.startswith(s)), None)
+            if hit:
+                out[i] = hit
+        else:
+            n = _num(v)
+            if n is not None and 1 <= n <= 5:
+                out[i] = float(round(n))
+    return out if len(out) >= max(1, int(0.8 * k)) else None
+
+
+def fill(d: dict, rows: list[dict], chat: Callable[..., str], question: str = "",
+         progress: Optional[Callable[[int], None]] = None, workers: int = 6) -> dict:
+    """Compute one derived column for `rows`. Returns {row id: value} and may set d["labels"]."""
+    if d["kind"] in ("pattern", "count"):
+        rx = re.compile(d["pattern"], re.I)
+        vals = {}
+        for r in rows:
+            hits = len(rx.findall(str(r.get(d["from"]) or "")[:MAX_CELL]))
+            vals[r["_id"]] = ("yes" if hits else "no") if d["kind"] == "pattern" else float(hits)
+        if progress:
+            progress(len(rows))
+        return vals
+    if d["kind"] == "label" and len(d.get("labels") or []) < 2:
+        text = chat(discover_prompt(question, d, rows), max_tokens=200, temperature=0.3)
+        try:
+            labels = [str(x).strip().lower()[:40] for x in first_json(text) if str(x).strip()]
+        except (ValueError, TypeError):
+            labels = []
+        if len(labels) < 2:
+            raise PlanError(f'Could not find labels for "{d["name"]}".')
+        d["labels"] = list(dict.fromkeys(labels))[:7]
+    batches = [rows[i:i + LABEL_BATCH] for i in range(0, len(rows), LABEL_BATCH)]
+    vals: dict[int, object] = {}
+    done = [0]
+
+    def run(batch):
+        got = None
+        for attempt in range(3):
+            text = chat(label_prompt(d, batch), max_tokens=12 * len(batch) + 40, temperature=0.1 if attempt == 0 else 0.4)
+            got = _read_labels(text, d, len(batch))
+            if got:
+                break
+        res = {}
+        for k, r in enumerate(batch, 1):
+            res[r["_id"]] = (got or {}).get(k, "unclear" if d["kind"] == "label" else None)
+        done[0] += len(batch)
+        if progress:
+            progress(done[0])
+        return res
+
+    with ThreadPoolExecutor(max_workers=max(1, min(workers, len(batches)))) as pool:
+        for res in pool.map(run, batches):
+            vals.update(res)
+    return vals
+
+
+def _rows(k) -> str:
+    return f"{k} row" if k == 1 else f"{k} rows"
+
+
+def _unit(col) -> str:
+    col = str(col or "")
+    return " words" if col.endswith(" words") else " turns" if col.endswith(" turns") else ""
+
+
+def facts(c: dict, limit: int = 14) -> list[str]:
+    """The numbers behind a chart in plain lines, for the write-up (and nothing else)."""
+    t, marks, n = c["type"], c.get("marks") or [], max(1, c["n"])
+    u = _unit(c.get("y") if t in ("bar", "box") else c.get("x"))
+    out = []
+    if t == "bar":
+        for m in marks[:limit]:
+            if c["agg"] == "count":
+                out.append(f"{m['key']}: {_rows(m['value'])} ({round(100 * m['value'] / n)}%)")
+            else:
+                out.append(f"{m['key']}: {c['agg']} {c['y']} {m['value']:g}{u} ({_rows(m['n'])})")
+    elif t == "stack":
+        for m in marks[:limit]:
+            out.append(f"{m['key']} ({_rows(m['n'])}): " + ", ".join(f"{p['key']} {p['value']} ({round(100 * p['value'] / m['n'])}%)" for p in m["parts"]))
+    elif t == "box":
+        for m in marks[:limit]:
+            out.append(f"{m['key']} ({_rows(m['n'])}): median {m['median']:g}{u}, middle half {m['q1']:g} to {m['q3']:g}{u}, range {m['lo']:g} to {m['hi']:g}{u}")
+    elif t == "hist":
+        for m in marks:
+            if m["value"]:
+                out.append(f"{c['x']} {m['x0']:g}{u}: {_rows(m['value'])}" if c.get("discrete") else f"{m['x0']:g} to {m['x1']:g}{u}: {_rows(m['value'])}")
+    elif t == "scatter":
+        out.append(f"{len(marks)} rows plotted")
+        if "r" in c:
+            out.append(f"correlation between {c['x']} and {c['y']}: r = {c['r']}")
+    return out
+
+
+def summary_prompt(question: str, title: str, c: dict, derived: list[dict]) -> str:
+    made = "".join(f'\n"{d["name"]}" was assigned by a model reading each row\'s "{d["from"]}" text.' for d in derived)
+    return f"""Someone asked about a dataset: "{question}"
+The chart "{title}" covers {c['n']} rows. Its exact numbers:
+{chr(10).join(facts(c))}{made}
+
+Answer the question in 2 to 4 short sentences of plain prose. Use only these numbers and keep them exact, but write them into sentences; do not paste the lines above. Lead with the clearest difference. Mention groups with fewer than 5 rows at most once, as too small to read much into. No quotation marks, markdown, lists, headings or advice."""
+
+
+def notes(c: dict, derived: list[dict], fallback: bool = False) -> list[str]:
+    out = []
+    if fallback:
+        out.append("The model could not plan a chart for that question, so this is the closest chart the columns give directly.")
+    for d in derived:
+        out.append(f'"{d["name"].capitalize()}" was assigned by a model reading each row\'s {d["from"]}. Open a few rows before relying on it.')
+    if c["n"] < 20:
+        out.append(f"Only {c['n']} rows, so a difference of one or two rows can change the picture.")
+    thin = [m["key"] for m in (c.get("marks") or []) if isinstance(m, dict) and "key" in m and 0 < m.get("n", 99) < 3]
+    if thin and c["type"] in ("bar", "box", "stack") and c.get("agg") != "count":
+        out.append(f"{', '.join(thin[:3])} {'has' if len(thin) == 1 else 'have'} fewer than three rows.")
+    return out

static/app.css

@@ -243,3 +243,89 @@ code { font-size: .92em; background: var(--paper-2); padding: 0 4px; border-radi
   .sctl { justify-content: flex-start; }
 }
 .in.num.wide { width: 11em; }
+
+/* visualize */
+.vh { position: absolute; width: 1px; height: 1px; overflow: hidden; clip: rect(0 0 0 0); white-space: nowrap; }
+.vsrcs { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 22px; margin-top: 30px; }
+.vsrc { background: var(--paper-2); border: 1px solid var(--rule); border-radius: var(--r); padding: 18px 20px 20px; min-width: 0; }
+.vsrc > b { display: block; font-size: 18px; margin-bottom: 4px; }
+.vsrc .actions { margin-top: 16px; }
+.vsrc .seg { margin-top: 12px; }
+.vsrc .addrow { grid-template-columns: minmax(0, 1fr) auto; margin-top: 14px; }
+.vsamples { list-style: none; margin: 10px 0 0; padding: 0; columns: 2 150px; column-gap: 18px; }
+.vsamples li { break-inside: avoid; }
+.vsample { background: none; border: 0; padding: 5px 0; cursor: pointer; text-align: left; font-weight: 600; color: var(--ink); text-decoration: underline; text-decoration-color: var(--rule); text-underline-offset: 4px; text-decoration-thickness: 2px; }
+.vsample:hover:not(:disabled) { color: var(--blue); text-decoration-color: var(--pink); }
+.vsample:disabled { cursor: default; opacity: .6; }
+.vsamples em { color: var(--blue); font-style: normal; font-size: 13px; font-weight: 600; }
+.filebtn { display: inline-block; margin-top: 14px; position: relative; overflow: hidden; }
+.filebtn input { position: absolute; inset: 0; opacity: 0; cursor: pointer; }
+.filebtn:focus-within { outline: 3px solid var(--blue); outline-offset: 2px; }
+.askbar { border-bottom: 1px solid var(--rule); padding-bottom: 22px; }
+textarea.in.vq { min-height: 64px; font-size: 19px; }
+.askbar .actions { gap: 10px 22px; }
+.vz-stages { margin: 0; gap: 4px 18px; }
+.chips { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 16px; }
+.chip { background: none; border: 1px solid #aeb2bf; border-radius: 999px; padding: 6px 13px; cursor: pointer; font-size: 14.5px; text-align: left; }
+.chip:hover:not(:disabled) { border-color: var(--ink); background: var(--paper-2); }
+.chip:disabled { opacity: .5; cursor: default; }
+.scopechip { display: flex; gap: 12px; align-items: baseline; flex-wrap: wrap; margin-bottom: 12px; padding: 8px 12px; background: var(--paper-2); border-left: 4px solid var(--pink); font-size: 14.5px; }
+.answer { margin-top: 28px; }
+.answer h2 { max-width: 40ch; }
+.prose { font-family: var(--doc); font-size: 17.5px; line-height: 1.6; margin-top: 12px; max-width: 64ch; }
+.facts { margin: 12px 0 0; padding-left: 18px; font-family: var(--doc); }
+.chart { margin-top: 22px; min-height: 120px; }
+.chart.small { margin-top: 8px; min-height: 80px; }
+.svgwrap svg { display: block; max-width: 100%; height: auto; overflow: visible; }
+.chart text { font: 13px var(--ui); fill: var(--ink); }
+.chart .tick { font-size: 11.5px; fill: var(--slate); font-variant-numeric: tabular-nums; }
+.chart .val { font-weight: 600; font-variant-numeric: tabular-nums; }
+.chart .ink { fill: var(--blue); }
+.chart .ghost { fill: var(--pink); opacity: .85; }
+.chart .grid { stroke: var(--rule); stroke-width: 1; }
+.chart .hit { fill: transparent; }
+.chart .bx { fill: var(--paper-2); stroke: var(--ink); stroke-width: 1.5; }
+.chart .whisk { stroke: var(--ink); stroke-width: 1.5; }
+.chart .med { stroke: var(--pink); stroke-width: 3.5; pointer-events: none; }
+.chart .dot { fill: var(--blue); opacity: .72; mix-blend-mode: multiply; cursor: pointer; }
+.chart .seg { mix-blend-mode: multiply; }
+.chart .segv { fill: #fff; font-size: 11.5px; font-weight: 600; pointer-events: none; }
+.chart .mk { cursor: pointer; outline: none; }
+.chart .mk:hover .ink { fill: #2a39e6; }
+.chart .mk:hover .lab { font-weight: 600; }
+.chart .mk:focus-visible .hit { stroke: var(--blue); stroke-width: 2; }
+.chart .sel .ink, .chart .sel.seg, .chart .sel .seg { fill: var(--pink); }
+.chart .sel .ghost { fill: var(--blue); }
+.chart .sel .bx { stroke: var(--pink); stroke-width: 2.5; }
+.chart circle.sel { fill: var(--pink); opacity: 1; }
+.chart .sel .lab { font-weight: 700; }
+.chart .dim { opacity: .28; }
+.chart .thin { opacity: .65; }
+.fresh .grow { animation: vzgrow .55s cubic-bezier(.2, .8, .2, 1) both; animation-delay: var(--d, 0ms); }
+.fresh .grow-y { animation: vzgrowy .55s cubic-bezier(.2, .8, .2, 1) both; animation-delay: var(--d, 0ms); }
+.fresh .grow-o { animation: vzfade .5s ease both; animation-delay: var(--d, 0ms); }
+@keyframes vzgrow { from { transform: scaleX(0); } }
+@keyframes vzgrowy { from { transform: scaleY(0); } }
+@keyframes vzfade { from { opacity: 0; } }
+@media (prefers-reduced-motion: reduce) { .fresh .grow, .fresh .grow-y, .fresh .grow-o { animation: none; } }
+.legend { display: flex; flex-wrap: wrap; gap: 6px 16px; margin-bottom: 10px; }
+.lg { display: inline-flex; align-items: center; gap: 7px; background: none; border: 0; padding: 2px 0; cursor: pointer; font-size: 14px; }
+.lg i { width: 12px; height: 12px; border-radius: 2px; }
+.lg.sel { font-weight: 700; text-decoration: underline; text-underline-offset: 3px; }
+.axisline { display: flex; justify-content: space-between; gap: 12px; flex-wrap: wrap; margin-top: 8px; color: var(--slate); font-size: 13.5px; }
+.linkish.sm { font-size: 13.5px; }
+.vnote { margin-top: 8px; max-width: 70ch; }
+.vtools { display: flex; gap: 18px; margin-top: 14px; flex-wrap: wrap; }
+.glance { margin-top: 30px; }
+.glance > h3 { font-size: 15px; font-weight: 600; }
+.gl-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(280px, 1fr)); gap: 22px 30px; margin-top: 10px; }
+.gl { border-top: 1px solid var(--ink); padding-top: 10px; min-width: 0; }
+.gl > b { font-size: 14.5px; }
+.drawer { margin-top: 28px; border-top: 3px solid var(--pink); padding-top: 14px; }
+.drawhead { display: flex; justify-content: space-between; gap: 12px; flex-wrap: wrap; align-items: center; }
+.btn.sm { padding: 7px 13px; font-size: 14px; }
+.tags { display: flex; flex-wrap: wrap; gap: 5px; margin-top: 6px; }
+.tagc { display: inline-block; background: var(--paper-2); border: 1px solid var(--rule); border-radius: 2px; padding: 1px 7px; font-size: 12.5px; color: var(--slate); }
+.tagc.made { border-color: var(--pink); color: var(--ink); }
+.sheet .tagc { margin: 0 2px 4px 0; }
+@media (max-width: 640px) { .vsrcs { grid-template-columns: 1fr; } textarea.in.vq { font-size: 17px; } }

static/app.js

@@ -180,7 +180,7 @@ function headParts() {
     count: `<span class="big">${n.toLocaleString()}</span><span class="of">of ${t.toLocaleString()} rows</span>`, pct,
     state: `${j ? esc(j.message) : "Starting"}${j && !fin ? ` Elapsed ${Math.round(j.elapsed)} s.` : ""}`,
     note: j && j.state === "failed" ? `<p class="note bad">${esc(j.message)}</p>` : j && j.state === "done" && j.count < j.target ? `<p class="note">${esc(j.message)}</p>` : "",
-    actions: fin ? `${j.count ? `<a class="btn go" href="/api/jobs/${j.id}/download?format=csv" style="text-decoration:none">Download CSV</a><a class="btn" href="/api/jobs/${j.id}/download?format=jsonl" style="text-decoration:none">Download JSONL</a>` : ""}<button class="btn" data-act="again">Start another</button>` : `<button class="btn" data-act="stop">Stop and keep what is written</button>`,
+    actions: fin ? `${j.count ? `<a class="btn go" href="/api/jobs/${j.id}/download?format=csv" style="text-decoration:none">Download CSV</a><a class="btn" href="/api/jobs/${j.id}/download?format=jsonl" style="text-decoration:none">Download JSONL</a>` : ""}${j.count ? `<button class="btn" data-act="visualize">Visualize these rows</button>` : ""}<button class="btn" data-act="again">Start another</button>` : `<button class="btn" data-act="stop">Stop and keep what is written</button>`,
     stats: st ? `<div class="stats"><div><b>${st.rows.toLocaleString()}</b><span>rows</span></div><div><b>${st.distinct3_prompts}</b><span>distinct 3-grams in prompts</span></div><div><b>${st.avg_prompt_words}</b><span>words per prompt</span></div><div><b>${st.avg_answer_words}</b><span>words per reply</span></div></div>` : "",
     filtered: j && j.rejected && Object.keys(j.rejected).length ? `<p class="hint" style="margin-top:12px">Filtered out: ${Object.entries(j.rejected).map(([k, v]) => `${v} ${esc(k)}`).join(", ")}.</p>` : "",
   };
@@ -209,7 +209,7 @@ function updateRun() {
 
 /* ------------------------------------------------------------ render + events */
 function render() {
-  if (window.DI_MODE === "curate") return;
+  if (window.DI_MODE !== "invent") return;
   renderLedger(); renderSheet();
   const fn = [stepFormat, stepDomains, stepSubs, stepLangs, stepSummary, stepReview, stepRun][S.step];
   const keep = document.activeElement && document.activeElement.id;
@@ -231,6 +231,7 @@ document.addEventListener("click", async (ev) => {
   if (act === "blueprint") return doBlueprint();
   if (act === "publish") return doPublish();
   if (act === "signout") { await api("/api/signout", {}); S.me = { ...S.me, user: null }; return refreshPublish(); }
+  if (act === "visualize") { if (S.job && window.DI_visualizeJob) window.DI_visualizeJob(S.job); return; }
   if (act === "stop") { if (S.job) await api(`/api/jobs/${S.job}/stop`, {}); return; }
   if (act === "again") { S.job = null; S.jobState = null; S.jobRows = []; S.drawn = 0; S.printed = 0; S.pubShown = false; S.pub = { name: "", priv: true, license: "apache-2.0", ok: false, busy: false, err: "", url: "" }; S.runError = ""; return go(4); }
 });
@@ -264,6 +265,7 @@ async function poll() {
   } catch (e) { console.error(e.stack); S.jobState = { state: "failed", message: e.message, count: S.jobRows.length, target: S.rows, elapsed: 0, id }; if (S.step === 6) render(); }
 }
 window.DI_renderInvent = () => render();
+window.DI_lastJob = () => (S.job && isFinal(S.jobState) && S.jobState.count ? { id: S.job, n: S.jobState.count, request: S.request || "" } : null);
 /* "Start writing" on the review step starts the run */
 const _go = go;
 go = function (i) { const wasReview = S.step === 5; _go(i); if (i === 6 && wasReview && !S.jobState) startRun(); return true; };

static/curate.js

@@ -357,15 +357,23 @@ datainventor run recipe.yaml --out ./curated --procs 32 --workers 64</pre></div>
   document.addEventListener("keydown", (ev) => { if (window.DI_MODE === "curate" && ev.key === "Enter" && ev.target.id === "c-add" && C.add.trim()) addDataset(C.add.trim(), C.addRole); });
 
   /* ------------------------------------------------------------ mode switch */
+  const TAGS = {
+    invent: "Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.",
+    curate: "Clean, score, rephrase and mix the pretraining data you already have. Preview it here on a sample, then run the recipe on your own machines.",
+    visualize: "Ask a question about a dataset and get a chart back, with the rows behind every bar one click away.",
+  };
   function setMode(mode) {
     window.DI_MODE = mode;
     document.querySelectorAll("[data-mode]").forEach((b) => b.setAttribute("aria-pressed", String(b.dataset.mode === mode)));
-    $(".tag").textContent = mode === "curate" ? "Clean, score, rephrase and mix the pretraining data you already have. Preview it here on a sample, then run the recipe on your own machines." : "Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.";
+    $(".tag").textContent = TAGS[mode] || TAGS.invent;
     if (mode === "curate") { if (!C.d) api("/api/curate/defaults").then((d) => { C.d = d; render(); }).catch(() => render()); render(); }
+    else if (mode === "visualize") { if (window.DI_renderVis) window.DI_renderVis(); }
     else if (window.DI_renderInvent) window.DI_renderInvent();
-    if ((location.hash === "#curate") !== (mode === "curate")) history.replaceState(null, "", mode === "curate" ? "#curate" : location.pathname + location.search);
+    const hash = mode === "invent" ? "" : "#" + mode;
+    if (location.hash !== hash) history.replaceState(null, "", hash || location.pathname + location.search);
   }
+  window.DI_setMode = setMode;
   document.addEventListener("click", (ev) => { const b = ev.target.closest("[data-mode]"); if (b) setMode(b.dataset.mode); });
-  window.DI_MODE = location.hash === "#curate" ? "curate" : "invent";
-  if (window.DI_MODE === "curate") document.addEventListener("DOMContentLoaded", () => setMode("curate"));
+  window.DI_MODE = location.hash === "#curate" ? "curate" : location.hash === "#visualize" ? "visualize" : "invent";
+  if (window.DI_MODE !== "invent") document.addEventListener("DOMContentLoaded", () => setMode(window.DI_MODE));
 })();

static/index.html

@@ -4,7 +4,7 @@
 <meta charset="utf-8">
 <meta name="viewport" content="width=device-width, initial-scale=1">
 <title>DataInventor</title>
-<meta name="description" content="Describe a dataset in plain words and get training data back, or clean, score, rephrase and mix the pretraining data you have. Open source, works with any OpenAI-compatible model.">
+<meta name="description" content="Describe a dataset in plain words and get training data back, clean, score, rephrase and mix the pretraining data you have, or ask a dataset a question and get a chart. Open source, works with any OpenAI-compatible model.">
 <link rel="preconnect" href="https://fonts.googleapis.com">
 <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
 <link href="https://fonts.googleapis.com/css2?family=Familjen+Grotesk:ital,wght@0,400..700;1,400..700&family=Literata:ital,opsz,wght@0,7..72,400..600;1,7..72,400..600&display=swap" rel="stylesheet">
@@ -17,7 +17,7 @@
     <span class="mark" aria-hidden="true"><i></i><i></i></span>
     <span class="name">DataInventor</span>
   </div>
-  <nav class="modes" aria-label="Mode"><button data-mode="invent" aria-pressed="true">Invent rows</button><button data-mode="curate" aria-pressed="false">Curate a corpus</button></nav>
+  <nav class="modes" aria-label="Mode"><button data-mode="invent" aria-pressed="true">Invent rows</button><button data-mode="curate" aria-pressed="false">Curate a corpus</button><button data-mode="visualize" aria-pressed="false">Visualize</button></nav>
   <p class="tag">Describe the data you wish existed. Get training rows back. Bring any OpenAI-compatible model.</p>
 </header>
 
@@ -29,5 +29,6 @@
 
 <script src="/static/curate.js"></script>
 <script src="/static/app.js"></script>
+<script src="/static/visualize.js"></script>
 </body>
 </html>

static/visualize.js

@@ -0,0 +1,382 @@
+"use strict";
+/* Visualize: ask a question about a dataset, get a chart, and open the rows behind any mark.
+   Built for the small datasets a hosted run writes, so every chart is exact and instant views need no model. */
+(function () {
+  const INKS = ["#3a4bff", "#ff5c8f", "#00838a", "#ffb000", "#ff6c2f", "#765ba7"];
+  const OTHER = "#8a8fa0";
+  const SIZES = [25, 50, 100, 200];
+  const STAGES = [["plan", "Planning the chart"], ["label", "Reading the rows"], ["chart", "Counting"], ["write", "Writing it up"]];
+  const V = {
+    data: null, rows: [], byId: {}, samples: [], maxRows: 200, n: 50, hub: "", loadBusy: "", loadErr: "",
+    q: "", asking: null, askErr: "", history: [], cur: -1, scope: null, sel: null, selShow: 30, sorts: {}, drawn: {},
+  };
+  const $ = (s, el = document) => el.querySelector(s);
+  const esc = (t) => String(t ?? "").replace(/[&<>"']/g, (c) => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" }[c]));
+  const on = () => window.DI_MODE === "visualize";
+  const plural = (n, w) => `${n.toLocaleString()} ${w}${n === 1 ? "" : "s"}`;
+  const fmt = (x) => { if (x == null || isNaN(x)) return ""; const a = Math.abs(x); return a >= 1000 ? Math.round(x).toLocaleString() : a >= 10 ? String(Math.round(x * 10) / 10) : String(Math.round(x * 100) / 100); };
+  const reduced = () => window.matchMedia && window.matchMedia("(prefers-reduced-motion: reduce)").matches;
+  async function api(path, body) {
+    const r = await fetch(path, { method: body ? "POST" : "GET", headers: body ? { "Content-Type": "application/json" } : {}, body: body ? JSON.stringify(body) : undefined });
+    let data = null; try { data = await r.json(); } catch (e) { /* not json */ }
+    if (!r.ok) throw new Error((data && (data.detail || data.message)) || `Request failed (${r.status})`);
+    return data;
+  }
+  const cols = () => (V.data ? V.data.columns : []);
+  const colType = (name) => (cols().find((c) => c.name === name) || {}).type;
+  const textCols = () => cols().filter((c) => c.type === "text").map((c) => c.name);
+  const catCols = () => cols().filter((c) => c.type === "cat").map((c) => c.name);
+
+  /* ------------------------------------------------------------ charts (hand-set SVG, two inks) */
+  function niceTicks(lo, hi, want = 5) {
+    if (!(hi > lo)) { hi = lo + 1; }
+    const raw = (hi - lo) / want, e = Math.pow(10, Math.floor(Math.log10(raw)));
+    const step = [1, 2, 2.5, 5, 10].map((m) => m * e).find((s) => s >= raw) || 10 * e;
+    const out = []; for (let v = Math.floor(lo / step) * step; v <= hi + step * 1e-9; v += step) out.push(+v.toFixed(10));
+    if (out[out.length - 1] < hi) out.push(out[out.length - 1] + step);
+    return out;
+  }
+  function labelW(keys, w) { const longest = Math.max(4, ...keys.map((k) => String(k).length)); return Math.min(Math.round(w * 0.38), Math.max(70, longest * 7.4 + 16)); }
+  /* narrow charts set each label above its bar instead of clipping it beside the bar */
+  function narrow(marks, w) { const lw = labelW(marks.map((m) => m.key), w); const longest = Math.max(0, ...marks.map((m) => String(m.key).length)); return w < 560 && longest * 7.4 + 16 > lw; }
+  function labText(key, lw, y, nar, w) {
+    return nar ? `<text class="lab" x="0" y="${y + 13}">${esc(clip(key, w - 10))}</text>` : `<text class="lab" x="${lw - 10}" y="${y + 19}" text-anchor="end">${esc(clip(key, lw - 12))}</text>`;
+  }
+  function clip(t, px) { const max = Math.max(4, Math.floor(px / 7.1)); t = String(t); return t.length > max ? t.slice(0, max - 1) + "…" : t; }
+  function colorMap(keys) { const m = {}; let i = 0; keys.forEach((k) => { m[k] = /^Other \(|^\(blank\)$|^unclear$/.test(k) ? OTHER : INKS[i++ % INKS.length]; }); return m; }
+  function mark(attrs, label, ids, inner) {
+    return `<g class="mk" tabindex="0" role="button" aria-label="${esc(label)}" data-vzsel="${esc(label.split(": ")[0])}" data-vzids="${ids.join(",")}" ${attrs}>${inner}</g>`;
+  }
+  function sorted(c, key) {
+    const how = V.sorts[key] || "value";
+    const m = (c.marks || []).slice();
+    if (how === "name") m.sort((a, b) => String(a.key).localeCompare(String(b.key), undefined, { numeric: true }));
+    return m;
+  }
+  function svgBar(c, w, key) {
+    const marks = sorted(c, key), nar = narrow(marks, w), row = nar ? 42 : 30, top = 6, lw = nar ? 0 : labelW(marks.map((m) => m.key), w), vw = 54, by = nar ? 14 : 0;
+    const bw = Math.max(60, w - lw - vw - 12), max = Math.max(...marks.map((m) => m.value), 1), h = top + marks.length * row + 4;
+    const yName = c.agg === "count" ? "rows" : `${c.agg} ${c.y}`;
+    const body = marks.map((m, i) => {
+      const y = top + i * row, len = Math.max(2, (m.value / max) * bw), lab = c.agg === "count" ? plural(m.value, "row") : `${fmt(m.value)} (${plural(m.n, "row")})`;
+      return mark(`style="--d:${i * 40}ms"`, `${c.x} = ${m.key}: ${lab}`, m.ids,
+        `${labText(m.key, lw, y, nar, w)}
+         <g class="grow" style="transform-origin:${lw}px 0"><rect class="ghost" x="${lw + 3}" y="${y + by + 7}" width="${len}" height="19"></rect>
+         <rect class="ink" x="${lw}" y="${y + by + 4}" width="${len}" height="19"></rect></g>
+         <text class="val" x="${lw + len + 8}" y="${y + by + 19}">${fmt(m.value)}</text><rect class="hit" x="0" y="${y}" width="${w}" height="${row}"></rect>`);
+    }).join("");
+    return { svg: `<svg viewBox="0 0 ${w} ${h}" width="${w}" height="${h}" role="img" aria-label="Bar chart of ${esc(yName)} by ${esc(c.x)}">${body}</svg>`, axis: `${c.x} · ${yName}` };
+  }
+  function svgStack(c, w, key) {
+    const marks = sorted(c, key), nar = narrow(marks, w), row = nar ? 42 : 30, top = 6, lw = nar ? 0 : labelW(marks.map((m) => m.key), w), vw = 46, by = nar ? 14 : 0;
+    const bw = Math.max(60, w - lw - vw - 10), share = c.agg !== "count", maxN = Math.max(...marks.map((m) => m.n), 1), cm = colorMap(c.colors || []);
+    const h = top + marks.length * row + 4;
+    const body = marks.map((m, i) => {
+      const y = top + i * row, scale = share ? bw / Math.max(1, m.n) : bw / maxN; let x = lw;
+      const segs = m.parts.map((p) => {
+        const sw = p.value * scale, pct = Math.round((100 * p.value) / m.n);
+        const g = mark("", `${c.x} = ${m.key}, ${c.color} = ${p.key}: ${plural(p.value, "row")} (${pct}%)`, p.ids,
+          `<rect class="seg" x="${x}" y="${y + by + 4}" width="${Math.max(1, sw - 1)}" height="19" fill="${cm[p.key]}"></rect>${sw > 30 ? `<text class="segv" x="${x + 5}" y="${y + by + 17.5}">${share ? pct + "%" : p.value}</text>` : ""}`);
+        x += sw; return g;
+      }).join("");
+      return `<g class="grow" style="--d:${i * 40}ms;transform-origin:${lw}px 0">${mark("", `${c.x} = ${m.key}: ${plural(m.n, "row")}`, m.ids, `${labText(m.key, lw, y, nar, w)}<rect class="hit" x="0" y="${y}" width="${nar ? w : lw}" height="${nar ? 15 : row}"></rect>`)}${segs}<text class="val" x="${lw + (share ? bw : m.n * scale) + 8}" y="${y + by + 19}">${m.n}</text></g>`;
+    }).join("");
+    const legend = (c.colors || []).map((k) => {
+      const ids = marks.flatMap((m) => (m.parts.find((p) => p.key === k) || { ids: [] }).ids);
+      return `<button class="lg" data-vzsel="${esc(`${c.color} = ${k}`)}" data-vzids="${ids.join(",")}"><i style="background:${cm[k]}"></i>${esc(k)}</button>`;
+    }).join("");
+    return { svg: `<svg viewBox="0 0 ${w} ${h}" width="${w}" height="${h}" role="img" aria-label="Stacked bars of ${esc(c.color)} within each ${esc(c.x)}">${body}</svg>`, legend, axis: `${c.x} · ${share ? "share of rows" : "rows"}, split by ${c.color}` };
+  }
+  function jitter(id) { const s = Math.sin(id * 12.9898) * 43758.5453; return s - Math.floor(s) - 0.5; }
+  function svgBox(c, w, key) {
+    const marks = sorted(c, key), nar = c.x && narrow(marks, w), row = nar ? 48 : 38, top = 6, lw = c.x && !nar ? labelW(marks.map((m) => m.key), w) : 8, bw = Math.max(80, w - lw - 16), by = nar ? 8 : 0;
+    const all = marks.flatMap((m) => m.points.map((p) => p[1])), ticks = niceTicks(Math.min(...all), Math.max(...all));
+    const lo = ticks[0], hi = ticks[ticks.length - 1], X = (v) => lw + ((v - lo) / (hi - lo || 1)) * bw, h = top + marks.length * row + 26;
+    const grid = ticks.map((t) => `<line class="grid" x1="${X(t)}" x2="${X(t)}" y1="${top}" y2="${h - 22}"></line><text class="tick" x="${X(t)}" y="${h - 6}" text-anchor="middle">${fmt(t)}</text>`).join("");
+    const body = marks.map((m, i) => {
+      const y = top + i * row, mid = y + row / 2 - 2 + by, thin = m.n < 3 ? " thin" : "";
+      const dots = m.points.map(([id, v]) => `<circle class="dot mk1" cx="${X(v)}" cy="${mid + jitter(id) * (row - 16)}" r="3.4" data-vzsel="row ${id}" data-vzids="${id}" tabindex="-1"><title>Row ${id}: ${fmt(v)}</title></circle>`).join("");
+      return `<g class="grow-o${thin}" style="--d:${i * 50}ms">${mark("", `${c.x ? `${c.x} = ${m.key}: ` : ""}median ${fmt(m.median)}, middle half ${fmt(m.q1)} to ${fmt(m.q3)}, ${plural(m.n, "row")}`, m.ids,
+        `${c.x ? (nar ? labText(m.key, 0, y - 2, true, w) : `<text class="lab" x="${lw - 10}" y="${mid + 4}" text-anchor="end">${esc(clip(m.key, lw - 12))}</text>`) : ""}${m.n >= 4 ? `<line class="whisk" x1="${X(m.lo)}" x2="${X(m.hi)}" y1="${mid}" y2="${mid}"></line>
+        <rect class="bx" x="${X(m.q1)}" y="${mid - 9}" width="${Math.max(2, X(m.q3) - X(m.q1))}" height="18"></rect>` : ""}<rect class="hit" x="0" y="${nar ? y - 2 : y}" width="${nar ? w : lw}" height="${nar ? 14 : row}"></rect>`)}
+        ${dots}<line class="med" x1="${X(m.median)}" x2="${X(m.median)}" y1="${mid - 11}" y2="${mid + 11}"></line></g>`;
+    }).join("");
+    return { svg: `<svg viewBox="0 0 ${w} ${h}" width="${w}" height="${h}" role="img" aria-label="Box plot of ${esc(c.y)}${c.x ? ` by ${esc(c.x)}` : ""}">${grid}${body}</svg>`, axis: `${c.y}${c.x ? ` by ${c.x}` : ""} · each dot is a row` };
+  }
+  function svgHist(c, w) {
+    const marks = c.marks || [], h = 210, left = 34, bottom = 26, top = 8, bw = (w - left - 8) / Math.max(1, marks.length);
+    const max = Math.max(...marks.map((m) => m.value), 1), yt = niceTicks(0, max, 4), Y = (v) => top + (1 - v / yt[yt.length - 1]) * (h - top - bottom);
+    const grid = yt.map((t) => `<line class="grid" x1="${left}" x2="${w - 4}" y1="${Y(t)}" y2="${Y(t)}"></line><text class="tick" x="${left - 6}" y="${Y(t) + 4}" text-anchor="end">${fmt(t)}</text>`).join("");
+    const body = marks.map((m, i) => {
+      const x = left + i * bw, y = Y(m.value);
+      return mark(`style="--d:${i * 35}ms"`, c.discrete ? `${c.x} = ${fmt(m.x0)}: ${plural(m.value, "row")}` : `${fmt(m.x0)} to ${fmt(m.x1)}: ${plural(m.value, "row")}`, m.ids,
+        `<g class="grow-y" style="transform-origin:0 ${h - bottom}px"><rect class="ghost" x="${x + 4}" y="${y + 3}" width="${Math.max(1, bw - 4)}" height="${Math.max(0, h - bottom - y)}"></rect>
+        <rect class="ink" x="${x + 1}" y="${y}" width="${Math.max(1, bw - 4)}" height="${Math.max(0, h - bottom - y)}"></rect></g>
+        <rect class="hit" x="${x}" y="${top}" width="${bw}" height="${h - top - bottom}"></rect>`);
+    }).join("");
+    const every = Math.ceil(marks.length / Math.max(2, Math.floor((w - left) / 52)));
+    const xl = c.discrete ? marks.map((m, i) => `<text class="tick" x="${left + i * bw + bw / 2}" y="${h - 8}" text-anchor="middle">${fmt(m.x0)}</text>`).join("") :
+      marks.map((m, i) => (i % every === 0 ? `<text class="tick" x="${left + i * bw}" y="${h - 8}" text-anchor="middle">${fmt(m.x0)}</text>` : "")).join("") +
+      (marks.length ? `<text class="tick" x="${left + marks.length * bw}" y="${h - 8}" text-anchor="end">${fmt(marks[marks.length - 1].x1)}</text>` : "");
+    return { svg: `<svg viewBox="0 0 ${w} ${h}" width="${w}" height="${h}" role="img" aria-label="Histogram of ${esc(c.x)}">${grid}${body}${xl}</svg>`, axis: `${c.x} · ${c.discrete ? "rows at each value" : "rows in each range"}` };
+  }
+  function svgScatter(c, w) {
+    const pts = c.marks || [], h = Math.min(360, Math.max(240, w * 0.55)), left = 40, bottom = 26, top = 8;
+    const xt = niceTicks(Math.min(...pts.map((p) => p.x)), Math.max(...pts.map((p) => p.x))), yt = niceTicks(Math.min(...pts.map((p) => p.y)), Math.max(...pts.map((p) => p.y)), 4);
+    const X = (v) => left + ((v - xt[0]) / (xt[xt.length - 1] - xt[0] || 1)) * (w - left - 10), Y = (v) => top + (1 - (v - yt[0]) / (yt[yt.length - 1] - yt[0] || 1)) * (h - top - bottom);
+    const cm = colorMap(c.colors || []);
+    const grid = xt.map((t) => `<line class="grid" x1="${X(t)}" x2="${X(t)}" y1="${top}" y2="${h - bottom}"></line><text class="tick" x="${X(t)}" y="${h - 8}" text-anchor="middle">${fmt(t)}</text>`).join("") +
+      yt.map((t) => `<line class="grid" x1="${left}" x2="${w - 10}" y1="${Y(t)}" y2="${Y(t)}"></line><text class="tick" x="${left - 6}" y="${Y(t) + 4}" text-anchor="end">${fmt(t)}</text>`).join("");
+    const dots = pts.map((p) => `<circle class="dot mk1" cx="${X(p.x)}" cy="${Y(p.y)}" r="4.2" style="${p.c ? `fill:${cm[p.c]}` : ""}" data-vzsel="row ${p.id}" data-vzids="${p.id}" tabindex="0" role="button" aria-label="Row ${p.id}: ${esc(c.x)} ${fmt(p.x)}, ${esc(c.y)} ${fmt(p.y)}"><title>Row ${p.id}</title></circle>`).join("");
+    const legend = (c.colors || []).map((k) => `<button class="lg" data-vzsel="${esc(`${c.color} = ${k}`)}" data-vzids="${pts.filter((p) => p.c === k).map((p) => p.id).join(",")}"><i style="background:${cm[k]}"></i>${esc(k)}</button>`).join("");
+    return { svg: `<svg viewBox="0 0 ${w} ${h}" width="${w}" height="${h}" role="img" aria-label="Scatter of ${esc(c.y)} against ${esc(c.x)}">${grid}${dots}</svg>`, legend, axis: `${c.x} → · ${c.y} ↑${c.r !== undefined ? ` · r = ${c.r}` : ""}` };
+  }
+  function chartHtml(c, w, key, fresh) {
+    if (!c || !(c.marks || []).length) return `<p class="hint">Nothing to draw: no rows have values for this.</p>`;
+    const f = { bar: svgBar, stack: svgStack, box: svgBox, hist: svgHist, scatter: svgScatter }[c.type];
+    const out = f(c, Math.max(260, Math.floor(w)), key);
+    const sortable = ["bar", "stack", "box"].includes(c.type) && (c.marks || []).length > 2;
+    return `${out.legend ? `<div class="legend">${out.legend}</div>` : ""}<div class="svgwrap${fresh ? " fresh" : ""}">${out.svg}</div>
+      <div class="axisline"><span>${esc(out.axis)}</span>${sortable ? `<button class="linkish sm" data-vzsort="${esc(key)}">${(V.sorts[key] || "value") === "value" ? "Sort by name" : "Sort by size"}</button>` : ""}</div>`;
+  }
+
+  /* ------------------------------------------------------------ pages */
+  function pickHtml() {
+    const job = window.DI_lastJob && window.DI_lastJob();
+    const card = (title, body) => `<div class="vsrc"><b>${title}</b>${body}</div>`;
+    const sizes = `<div class="seg" role="group" aria-label="Rows to load">${SIZES.filter((n) => n <= V.maxRows).map((n) => `<button data-vzn="${n}" aria-pressed="${V.n === n}">${n}</button>`).join("")}</div>`;
+    return `<h1>Ask your data a question</h1>
+    <p class="lede">Get a chart back, with the rows behind every bar one click away. It is built for the small datasets you write here: every number is exact, and the first charts need no model at all.</p>
+    ${V.loadErr ? `<p class="note bad">${esc(V.loadErr)}</p>` : ""}
+    <div class="vsrcs">
+      ${job ? card("The rows you just wrote", `<p class="hint">${plural(job.n, "row")} from “${esc(job.request.slice(0, 90))}”</p><div class="actions"><button class="btn go" data-vz="load-job" ${V.loadBusy ? "disabled" : ""}>${V.loadBusy === "job" ? "Loading" : "Look at them"}</button></div>`) : ""}
+      ${card("A sample dataset", `<p class="hint">Written with DataInventor and published on Hugging Face. Pick how many rows to load.</p>${sizes}
+        <ul class="vsamples">${V.samples.map((s) => `<li><button class="vsample" data-vzsample="${esc(s.key)}" ${V.loadBusy ? "disabled" : ""}>${esc(s.title)}${V.loadBusy === s.key ? " <em>loading</em>" : ""}</button></li>`).join("")}</ul>`)}
+      ${card("A Hugging Face dataset", `<p class="hint">Any public dataset the viewer can read. A spread of rows is loaded, not the first page.</p>
+        <div class="addrow"><input class="in" id="vz-hub" placeholder="owner/name" value="${esc(V.hub)}" spellcheck="false" autocomplete="off"><button class="btn" data-vz="load-hub" ${V.loadBusy || !V.hub.trim() ? "disabled" : ""}>${V.loadBusy === "hub" ? "Loading" : "Load"}</button></div>`)}
+      ${card("A file", `<p class="hint">CSV with a header row, JSONL, or a JSON list of objects. Up to ${V.maxRows} rows are used.</p>
+        <label class="btn filebtn"><input type="file" id="vz-file" accept=".csv,.jsonl,.json,.txt" ${V.loadBusy ? "disabled" : ""}>${V.loadBusy === "file" ? "Reading" : "Choose a file"}</label>`)}
+    </div>`;
+  }
+  function stagesHtml() {
+    const a = V.asking; if (!a) return "";
+    const idx = STAGES.findIndex(([k]) => k === a.stage);
+    const items = STAGES.filter(([k]) => k !== "label" || a.total).map(([k, name]) => {
+      const i = STAGES.findIndex(([x]) => x === k), cls = i < idx ? "done" : i === idx ? "on" : "";
+      const extra = k === "label" && a.total ? ` ${a.done} of ${a.total}` : "";
+      return `<li class="${cls}">${name}${extra}</li>`;
+    }).join("");
+    return `<ol class="stages vz-stages" aria-live="polite">${items}</ol>`;
+  }
+  function answerHtml() {
+    const h = V.history[V.cur]; if (!h) return "";
+    const r = h.result, key = `a${V.cur}`;
+    return `<article class="answer" id="vz-answer">
+      ${h.scopeLabel ? `<p class="hint">About ${esc(h.scopeLabel)}</p>` : ""}
+      <h2>${esc(r.title)}</h2>
+      ${r.summary ? `<p class="prose">${esc(r.summary)}</p>` : V.asking && V.asking.stage === "write" ? `<p class="prose hint">Writing it up…</p>` : `<ul class="facts">${(r.facts || []).slice(0, 6).map((f) => `<li>${esc(f)}</li>`).join("")}</ul>`}
+      <div class="chart" id="vz-chart" data-vzkey="${key}"></div>
+      ${(r.notes || []).map((n) => `<p class="hint vnote">${esc(n)}</p>`).join("")}
+      <div class="vtools"><button class="linkish sm" data-vz="svg">Download the chart</button>${r.chart.n ? `<a class="linkish sm" href="/api/vis/${V.data.id}/download?ids=${allIds(r.chart).join(",")}">Download its rows</a>` : ""}</div>
+      ${r.follow_ups && r.follow_ups.length ? `<div class="chips">${r.follow_ups.map((q) => `<button class="chip" data-vzq="${esc(q)}">${esc(q)}</button>`).join("")}</div>` : ""}
+    </article>`;
+  }
+  function allIds(c) {
+    if (c.type === "scatter") return c.marks.map((m) => m.id);
+    const s = new Set(); (c.marks || []).forEach((m) => (m.ids || []).forEach((i) => s.add(i))); return [...s];
+  }
+  function glanceHtml() {
+    const g = V.data.glance || []; if (!g.length) return "";
+    return `<section class="glance"><h3>At a glance</h3><div class="gl-grid">${g.map((c, i) => `<div class="gl"><b>${esc(c.title)}</b><div class="chart small" id="vz-g${i}" data-vzkey="g${i}"></div></div>`).join("")}</div></section>`;
+  }
+  function rowHtml(r) {
+    const long = cols().filter((c) => c.type === "text" && (c.avg_words || 0) >= 12).map((c) => c.name);
+    const short = textCols().filter((k) => !long.includes(k));
+    const head = long[0] || short[0], rest = long.slice(1);
+    const chips = short.filter((k) => k !== head).concat(catCols()).map((k) => r[k] ? `<span class="tagc${derivedCol(k) ? " made" : ""}" title="${esc(k)}">${esc(String(r[k]).slice(0, 80))}</span>` : "").join("");
+    const nums = cols().filter((c) => c.type === "num" && derivedCol(c.name) && r[c.name] != null).map((c) => `<span class="tagc made" title="${esc(c.name)}">${esc(c.name)} ${fmt(r[c.name])}</span>`).join("");
+    return `<div class="trow"><span class="id">${r._id}</span><div>${head ? `<div class="p">${esc(String(r[head] || "").slice(0, 320))}${String(r[head] || "").length > 320 ? "…" : ""}</div>` : ""}
+      <div class="tags">${chips}${nums}</div>
+      ${rest.map((k) => `<details><summary>Read the ${esc(k)}</summary><div class="full">${esc(r[k])}</div></details>`).join("")}
+      ${head && String(r[head] || "").length > 320 ? `<details><summary>Read the whole ${esc(head)}</summary><div class="full">${esc(r[head])}</div></details>` : ""}</div></div>`;
+  }
+  const derivedCol = (k) => { const c = cols().find((x) => x.name === k); return c && ["label", "score", "pattern", "count"].includes(c.derived); };
+  function selHtml() {
+    const s = V.sel; if (!s) return "";
+    const rows = s.ids.map((i) => V.byId[i]).filter(Boolean);
+    return `<section class="drawer" id="vz-sel" aria-label="Selected rows">
+      <div class="drawhead"><div><b>${plural(rows.length, "row")}</b> <span class="hint">${esc(s.label)}</span></div>
+        <div class="sctl">${rows.length > 1 ? `<button class="btn sm" data-vz="scope">Ask about these rows</button>` : ""}<a class="btn sm" href="/api/vis/${V.data.id}/download?ids=${s.ids.join(",")}" style="text-decoration:none">Download them</a><button class="linkish sm" data-vz="clear">Close</button></div></div>
+      <div class="table">${rows.slice(0, V.selShow).map(rowHtml).join("")}</div>
+      ${rows.length > V.selShow ? `<p style="margin-top:12px"><button class="linkish" data-vz="more">Show ${Math.min(30, rows.length - V.selShow)} more</button></p>` : ""}
+    </section>`;
+  }
+  function askHtml() {
+    const sug = V.history.length ? [] : V.data.suggestions || [];
+    const busy = !!V.asking;
+    return `<div class="askbar">
+      ${V.scope ? `<div class="scopechip">Asking about ${esc(V.scope.label)} (${plural(V.scope.ids.length, "row")}) <button class="linkish sm" data-vz="unscope" aria-label="Ask about all rows again">Use all rows</button></div>` : ""}
+      <label class="f" style="margin-top:0"><span class="vh">Your question</span>
+        <textarea class="in vq" id="vz-q" rows="2" placeholder="How long are the replies in each area?" ${busy ? "disabled" : ""}>${esc(V.q)}</textarea></label>
+      <div class="actions" style="margin-top:12px"><button class="btn go" data-vz="ask" ${busy || !V.q.trim() ? "disabled" : ""}>${busy ? "Working on it" : "Ask"}</button>${stagesHtml()}</div>
+      ${V.askErr ? `<p class="note bad">${esc(V.askErr)}</p>` : ""}
+      ${sug.length ? `<div class="chips">${sug.map((q) => `<button class="chip" data-vzq="${esc(q)}" ${busy ? "disabled" : ""}>${esc(q)}</button>`).join("")}</div>` : ""}
+    </div>`;
+  }
+  function mainHtml() {
+    return `${askHtml()}<div id="vz-out">${answerHtml()}</div>${V.cur < 0 ? glanceHtml() : ""}<div id="vz-selwrap">${selHtml()}</div>`;
+  }
+  function ledgerHtml() {
+    if (!V.data) return `<li class="on"><button aria-current="step"><span class="n">1</span><span>Pick the data</span></button></li><li><button disabled><span class="n">2</span><span>Ask</span></button></li>`;
+    const items = V.history.map((h, i) => `<li class="${i === V.cur ? "on" : "done"}"><button data-vzh="${i}" ${i === V.cur ? 'aria-current="true"' : ""}><span class="n">${i + 1}</span><span>${esc(h.q.length > 64 ? h.q.slice(0, 62) + "…" : h.q)}</span></button></li>`).join("");
+    return `<li class="${V.cur < 0 ? "on" : "done"}"><button data-vzh="-1"><span class="n">·</span><span>At a glance</span></button></li>${items}`;
+  }
+  function sheetHtml() {
+    if (!V.data) return `<h2>Your data</h2><p class="hint">Nothing loaded yet. Charts are drawn from every row you load, so a small sample is fast and exact.</p>`;
+    const d = V.data, group = (t, f) => { const list = cols().filter(f); return list.length ? `<div class="line"><dt>${t}</dt><dd>${list.map((c) => `<span class="tagc${derivedCol(c.name) ? " made" : ""}">${esc(c.name)}</span>`).join(" ")}</dd></div>` : ""; };
+    return `<h2>Your data</h2><dl>
+      <div class="line"><dt>Name</dt><dd>${esc(d.name)}</dd></div>
+      <div class="line"><dt>From</dt><dd>${/\//.test(d.source) ? `<a href="https://huggingface.co/datasets/${esc(d.source)}" target="_blank" rel="noopener">${esc(d.source)}</a>` : esc(d.source)}</dd></div>
+      <div class="line"><dt>Rows</dt><dd>${d.n.toLocaleString()}</dd></div>
+      ${group("Text", (c) => c.type === "text")}${group("Categories", (c) => c.type === "cat" && !derivedCol(c.name))}
+      ${group("Numbers", (c) => c.type === "num" && !derivedCol(c.name))}${group("Made by the model", (c) => derivedCol(c.name))}
+    </dl>${d.note ? `<p class="hint" style="margin-top:12px">${esc(d.note)}</p>` : ""}
+    <p class="hint" style="margin-top:14px">Columns the model makes stay here, so follow-up questions reuse them instead of reading every row again.</p>
+    <div class="actions" style="margin-top:18px"><button class="btn" data-vz="change">Use different data</button></div>`;
+  }
+  function render() {
+    if (!on()) return;
+    $("#steps").innerHTML = ledgerHtml();
+    $("#sheet").innerHTML = sheetHtml();
+    const keep = document.activeElement && document.activeElement.id;
+    $("#panel").innerHTML = V.data ? mainHtml() : pickHtml();
+    if (V.data) { redrawCharts(); markSelection(); }
+    if (keep) { const el = document.getElementById(keep); if (el) { el.focus(); if (el.setSelectionRange && el.tagName === "TEXTAREA") { const l = el.value.length; el.setSelectionRange(l, l); } } }
+  }
+  /* charts need their container's width, which exists only after the first paint */
+  function redrawCharts() {
+    document.querySelectorAll("[data-vzkey]").forEach((el) => {
+      const key = el.dataset.vzkey, c = key.startsWith("g") ? V.data.glance[+key.slice(1)] : (V.history[+key.slice(1)] || {}).result?.chart;
+      if (c) { el.innerHTML = chartHtml(c, el.clientWidth || 600, key, !V.drawn[key] && !reduced()); V.drawn[key] = true; }
+    });
+    markSelection();
+  }
+  function markSelection() {
+    const ids = V.sel ? new Set(V.sel.ids) : null;
+    document.querySelectorAll("[data-vzids]").forEach((el) => {
+      const mine = el.dataset.vzids.split(",").filter(Boolean).map(Number);
+      const hit = ids && mine.length && mine.every((i) => ids.has(i)) && (V.sel.label === el.dataset.vzsel || mine.length === 1 && ids.has(mine[0]));
+      el.classList.toggle("sel", !!hit);
+      el.classList.toggle("dim", !!ids && !hit && !el.classList.contains("lg") && !mine.some((i) => ids.has(i)));
+    });
+  }
+  function refreshAsk() { const el = $(".askbar"); if (el) { el.outerHTML = askHtml(); } }
+
+  /* ------------------------------------------------------------ actions */
+  async function loadData(body, busyKey) {
+    V.loadBusy = busyKey; V.loadErr = ""; render();
+    try {
+      const d = await api("/api/vis/load", body);
+      V.data = d; V.rows = d.rows; V.byId = Object.fromEntries(d.rows.map((r) => [r._id, r]));
+      V.history = []; V.cur = -1; V.sel = null; V.scope = null; V.q = ""; V.askErr = ""; V.sorts = {}; V.drawn = {};
+    } catch (e) { V.loadErr = e.message; }
+    V.loadBusy = ""; render(); window.scrollTo({ top: 0 });
+  }
+  async function ask(q) {
+    q = (q || "").trim(); if (!q || V.asking || !V.data) return;
+    V.q = q; V.askErr = ""; V.asking = { stage: "plan", done: 0, total: 0 }; refreshAsk();
+    const scope = V.scope;
+    try {
+      const { ask: aid } = await api(`/api/vis/${V.data.id}/ask`, { question: q, scope: scope ? scope.ids : [], scope_label: scope ? scope.label : "" });
+      let shown = false;
+      for (;;) {
+        await new Promise((r) => setTimeout(r, 650));
+        const s = await api(`/api/vis/ask/${aid}`);
+        V.asking = s;
+        if (s.stage === "failed") throw new Error(s.error || "That question could not be answered.");
+        if (s.result && !shown) {
+          shown = true; mergeResult(s.result);
+          V.history.push({ q, result: s.result, scopeLabel: scope ? `${scope.label} (${plural(scope.ids.length, "row")})` : "" });
+          V.cur = V.history.length - 1; V.sel = null; render();
+          const out = $("#vz-out"); if (out && out.getBoundingClientRect().top > window.innerHeight * 0.6) out.scrollIntoView({ behavior: reduced() ? "auto" : "smooth", block: "start" });
+        } else if (shown) { V.history[V.history.length - 1].result = s.result; }
+        if (s.stage === "done") break;
+        if (!shown) refreshAsk(); else { const st = $(".vz-stages"); if (st) st.outerHTML = stagesHtml(); }
+      }
+      V.asking = null; V.q = ""; render();
+    } catch (e) { V.asking = null; V.askErr = e.message; render(); }
+  }
+  function mergeResult(r) {
+    Object.entries(r.derived || {}).forEach(([name, vals]) => Object.entries(vals).forEach(([id, v]) => { if (V.byId[id]) V.byId[id][name] = v; }));
+    if (r.columns) V.data.columns = r.columns;
+  }
+  function select(label, ids) {
+    if (V.sel && V.sel.label === label) { V.sel = null; } else { V.sel = { label, ids }; V.selShow = 30; }
+    const w = $("#vz-selwrap"); if (w) w.innerHTML = selHtml();
+    markSelection();
+    const d = $("#vz-sel"); if (d) d.scrollIntoView({ behavior: reduced() ? "auto" : "smooth", block: "nearest" });
+  }
+  function downloadSvg() {
+    const svg = $("#vz-chart svg"); if (!svg) return;
+    const css = `text{font:13px "Familjen Grotesk",system-ui,sans-serif;fill:#1d2036}.tick{font-size:11.5px;fill:#5d6376}.ink{fill:#3a4bff}.ghost{fill:#ff5c8f;opacity:.85}.grid{stroke:#c5c8d1}.bx{fill:#f1f2f4;stroke:#1d2036;stroke-width:1.5}.whisk{stroke:#1d2036;stroke-width:1.5}.med{stroke:#ff5c8f;stroke-width:3.5}.dot{fill:#3a4bff;opacity:.75}.hit{fill:transparent}.segv{fill:#fff;font-size:11.5px;font-weight:600}.val{font-weight:600}`;
+    const clone = svg.cloneNode(true); clone.setAttribute("xmlns", "http://www.w3.org/2000/svg");
+    clone.insertAdjacentHTML("afterbegin", `<style>${css}</style><rect width="100%" height="100%" fill="#e7e8ea"></rect>`);
+    const a = document.createElement("a"); a.href = URL.createObjectURL(new Blob([clone.outerHTML], { type: "image/svg+xml" }));
+    a.download = (V.history[V.cur]?.result.title || "chart").toLowerCase().replace(/[^a-z0-9]+/g, "-").slice(0, 50) + ".svg"; a.click();
+    setTimeout(() => URL.revokeObjectURL(a.href), 2000);
+  }
+
+  document.addEventListener("click", (ev) => {
+    if (!on()) return;
+    const t = ev.target.closest("[data-vz],[data-vzsel],[data-vzq],[data-vzn],[data-vzsample],[data-vzh],[data-vzsort]"); if (!t) return;
+    if (t.dataset.vzn) { V.n = +t.dataset.vzn; return render(); }
+    if (t.dataset.vzsample) return loadData({ source: "sample", key: t.dataset.vzsample, n: V.n }, t.dataset.vzsample);
+    if (t.dataset.vzq) { V.q = t.dataset.vzq; return ask(V.q); }
+    if (t.dataset.vzh !== undefined) { V.cur = +t.dataset.vzh; V.sel = null; render(); return window.scrollTo({ top: 0 }); }
+    if (t.dataset.vzsort) { const k = t.dataset.vzsort; V.sorts[k] = (V.sorts[k] || "value") === "value" ? "name" : "value"; return redrawCharts(); }
+    if (t.dataset.vzsel !== undefined && t.dataset.vzids !== undefined && !t.dataset.vz) {
+      ev.stopPropagation();
+      return select(t.dataset.vzsel, t.dataset.vzids.split(",").filter(Boolean).map(Number));
+    }
+    const a = t.dataset.vz;
+    if (a === "load-job") { const j = window.DI_lastJob && window.DI_lastJob(); if (j) loadData({ source: "job", job: j.id }, "job"); }
+    else if (a === "load-hub") loadData({ source: "hub", dataset: V.hub.trim(), n: V.n }, "hub");
+    else if (a === "ask") ask(V.q);
+    else if (a === "clear") select(V.sel ? V.sel.label : "", []);
+    else if (a === "more") { V.selShow += 30; const w = $("#vz-selwrap"); if (w) w.innerHTML = selHtml(); }
+    else if (a === "scope" && V.sel) { V.scope = { label: V.sel.label, ids: V.sel.ids.slice() }; V.sel = null; render(); const q = $("#vz-q"); if (q) { q.focus(); q.scrollIntoView({ block: "center" }); } }
+    else if (a === "unscope") { V.scope = null; refreshAsk(); }
+    else if (a === "change") { V.data = null; V.history = []; V.cur = -1; V.sel = null; V.scope = null; render(); }
+    else if (a === "svg") downloadSvg();
+  });
+  /* marks are SVG groups: Enter or Space opens them like a button */
+  document.addEventListener("keydown", (ev) => {
+    if (!on()) return;
+    if ((ev.key === "Enter" || ev.key === " ") && ev.target.matches && ev.target.matches("g.mk, circle.mk1")) { ev.preventDefault(); ev.target.dispatchEvent(new MouseEvent("click", { bubbles: true })); }
+    if (ev.key === "Enter" && !ev.shiftKey && ev.target.id === "vz-q") { ev.preventDefault(); ask(V.q); }
+    if (ev.key === "Enter" && ev.target.id === "vz-hub" && V.hub.trim()) loadData({ source: "hub", dataset: V.hub.trim(), n: V.n }, "hub");
+  });
+  document.addEventListener("input", (ev) => {
+    if (!on()) return;
+    if (ev.target.id === "vz-q") { V.q = ev.target.value; const b = $('[data-vz="ask"]'); if (b) b.disabled = !!V.asking || !V.q.trim(); }
+    if (ev.target.id === "vz-hub") { V.hub = ev.target.value; const b = $('[data-vz="load-hub"]'); if (b) b.disabled = !!V.loadBusy || !V.hub.trim(); }
+  });
+  document.addEventListener("change", (ev) => {
+    if (!on() || ev.target.id !== "vz-file" || !ev.target.files.length) return;
+    const f = ev.target.files[0];
+    if (f.size > 6e6) { V.loadErr = "That file is over 6 MB. Upload a smaller sample."; return render(); }
+    const rd = new FileReader();
+    rd.onload = () => loadData({ source: "upload", text: String(rd.result), filename: f.name }, "file");
+    rd.onerror = () => { V.loadErr = "Could not read that file."; render(); };
+    rd.readAsText(f);
+  });
+  let rz = 0;
+  window.addEventListener("resize", () => { if (!on() || !V.data) return; clearTimeout(rz); rz = setTimeout(redrawCharts, 120); });
+
+  window.DI_renderVis = () => {
+    if (!V.samples.length) api("/api/vis/samples").then((d) => { V.samples = d.samples; V.maxRows = d.max_rows; if (on() && !V.data) render(); }).catch(() => {});
+    render();
+  };
+  /* "Visualize these rows" on a finished run */
+  window.DI_visualizeJob = (id) => { window.DI_setMode && window.DI_setMode("visualize"); loadData({ source: "job", job: id }, "job"); };
+})();

tests/test_visualize.py

@@ -0,0 +1,98 @@
+import unittest
+
+from datainventor import visualize as V
+
+
+def sample(n=12):
+    areas = ["Billing", "Returns", "Shipping"]
+    return [{"id": i, "prompt": f"Question number {i} about an order that arrived late and damaged",
+             "completion": " ".join(["word"] * (20 + 10 * i)), "area": areas[i % 3], "difficulty": "basic" if i % 2 else "advanced",
+             "score": str(i % 5 + 1), "ok": bool(i % 2)} for i in range(n)]
+
+
+class Prepare(unittest.TestCase):
+    def test_types_and_derived_lengths(self):
+        rows, cols = V.prepare(sample())
+        self.assertEqual(cols["prompt"]["type"], "text")
+        self.assertEqual(cols["area"]["type"], "cat")
+        self.assertEqual(cols["score"]["type"], "num")
+        self.assertEqual(cols["ok"]["type"], "cat")
+        self.assertNotIn("id", cols)
+        self.assertEqual(cols["completion words"]["derived"], "words")
+        self.assertEqual(rows[0]["_id"], 1)
+        self.assertEqual(rows[0]["completion words"], 20.0)
+
+    def test_chat_cells_become_text_with_turns(self):
+        rows, cols = V.prepare([{"messages": [{"role": "user", "content": "hi " * 30}, {"role": "assistant", "content": "hello " * 30}]}] * 3)
+        self.assertEqual(cols["messages"]["type"], "text")
+        self.assertEqual(rows[0]["messages turns"], 2.0)
+
+
+class Charts(unittest.TestCase):
+    def setUp(self):
+        self.rows, self.cols = V.prepare(sample(30))
+
+    def test_every_row_lands_in_exactly_one_bin(self):
+        c = V.chart(self.rows, self.cols, {"type": "hist", "x": "completion words"})
+        ids = [i for b in c["marks"] for i in b["ids"]]
+        self.assertEqual(sorted(ids), list(range(1, 31)))
+
+    def test_bar_counts_and_ids(self):
+        c = V.chart(self.rows, self.cols, {"type": "bar", "x": "area", "agg": "count"})
+        self.assertEqual(sum(m["value"] for m in c["marks"]), 30)
+        self.assertTrue(all(len(m["ids"]) == m["value"] for m in c["marks"]))
+
+    def test_box_quartiles_are_ordered(self):
+        c = V.chart(self.rows, self.cols, {"type": "box", "x": "area", "y": "completion words"})
+        for m in c["marks"]:
+            self.assertTrue(m["lo"] <= m["q1"] <= m["median"] <= m["q3"] <= m["hi"])
+
+    def test_check_spec_fixes_and_rejects(self):
+        self.assertEqual(V.check_spec({"type": "bar", "x": "completion words"}, self.cols)["type"], "hist")
+        self.assertEqual(V.check_spec({"type": "bar", "x": "area", "y": "score"}, self.cols)["agg"], "mean")
+        with self.assertRaises(V.PlanError):
+            V.check_spec({"type": "box", "x": "area", "y": "area"}, self.cols)
+        with self.assertRaises(V.PlanError):
+            V.check_spec({"type": "bar", "x": "missing"}, self.cols)
+
+
+class Plans(unittest.TestCase):
+    def setUp(self):
+        self.rows, self.cols = V.prepare(sample())
+
+    def test_parse_plan_with_fences_and_derived_column(self):
+        p = V.parse_plan('Sure!\n```json\n{"title": "Tone by area", "derive": [{"name": "Tone!", "kind": "label", "from": "completion", "labels": ["calm", "curt"]}],'
+                         ' "chart": {"type": "stack", "x": "area", "color": "tone"}, "follow_ups": ["a"]}\n```', self.cols)
+        self.assertEqual(p["derive"][0]["name"], "tone")
+        self.assertEqual(p["chart"]["agg"], "share")
+
+    def test_derived_name_never_overwrites_an_original_column(self):
+        p = V.parse_plan('{"title": "t", "derive": [{"name": "area", "kind": "label", "from": "prompt", "labels": ["a", "b"]}],'
+                         ' "chart": {"type": "bar", "x": "area"}}', self.cols)
+        self.assertEqual(p["derive"][0]["name"], "area (model)")
+        self.assertEqual(p["chart"]["x"], "area (model)")
+
+    def test_snake_case_names_still_match_the_chart(self):
+        p = V.parse_plan('{"derive": [{"name": "step_by_step", "kind": "pattern", "from": "completion", "pattern": "(?i)step \\\\d"}],'
+                         ' "chart": {"type": "bar", "x": "step_by_step"}}', self.cols)
+        self.assertEqual(p["derive"][0]["name"], "step by step")
+        self.assertEqual(p["chart"]["x"], "step by step")
+        self.assertEqual(p["derive"][0]["pattern"], "step \\d")
+
+    def test_dangerous_pattern_rejected(self):
+        with self.assertRaises(V.PlanError):
+            V.parse_plan('{"derive": [{"name": "x", "kind": "pattern", "from": "prompt", "pattern": "(a+)+$"}], "chart": {"type": "bar", "x": "x"}}', self.cols)
+
+    def test_label_reading(self):
+        d = {"kind": "label", "labels": ["Formal", "Friendly"]}
+        self.assertEqual(V._read_labels('{"1": "formal", "2": "friendly tone"}', d, 2), {1: "Formal", 2: "Friendly"})
+        self.assertIsNone(V._read_labels('{"1": "angry"}', d, 2))
+
+    def test_pattern_fill_needs_no_model(self):
+        rows, _ = V.prepare([{"t": "Step 1 do this. Step 2 do that."}, {"t": "Just a plain answer here."}])
+        vals = V.fill({"name": "steps", "kind": "pattern", "from": "t", "pattern": r"step \d"}, rows, chat=None)
+        self.assertEqual(vals, {1: "yes", 2: "no"})
+
+
+if __name__ == "__main__":
+    unittest.main()