README.md

---
title: Datasent's Polygen Tokenization Demo
emoji: 📈
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
---

# Datasent's Polygen Tokenization Demo

An interactive demo of Datasent's Polygen SDK on numeric signals. Pick a sample (or
upload a CSV), and it tokenizes the signal through the real SDK and reports:

- **Lossless round-trip** -- EXACT decode is bit-faithful to the 1/scale grid.
- **COARSE analytics archive** -- coefficients only, tens to hundreds of times
  smaller than raw, and directly queryable (the coefficients are a feature
  vector).
- **Per-segment anomaly signal** (sigma_R) -- a free quality marker; segments
  the model fits poorly stand out.
- **EXACT lossless size** shown honestly against gzip (it sits at parity --
  polygen is a tokenizer, not a byte-compression replacement).

Everything goes through the supported SDK surface: `DataMatrix.from_float` ->
`PolygenTokenizer.tokenize(..., parallel=False)` with `BinaryEncoder` for a
real token, and `tokenizer.decode` for EXACT / COARSE. The in-process
`parallel=False` path means no multiprocessing pool, which keeps the Space
light.

## Running locally

```bash
pip install -r requirements.txt
# plus polygen on the path (editable dev install or the stable wheel)
POLYGEN_DEMO_LOCAL_MOCK=1 python app.py
```

`POLYGEN_DEMO_LOCAL_MOCK=1` generates a self-consistent throwaway license for
local dev (the same pattern `tests/conftest.py` uses). Do not set it in
production.

## Deploying to a HuggingFace Space

1. Install `polygen` via `requirements.txt` (point at the stable wheel URL).
2. Set the `POLYGEN_LICENSE` secret to a real JWT issued for that wheel.
   - Note: this depends on the license keypair being aligned (audit finding
     F8). The bundled public key must match the key that signed the JWT.
3. Leave `POLYGEN_DEMO_LOCAL_MOCK` unset so the SDK validates the real license.

## Notes

- This is a prototype. CPU-only; no GPU / RunPod needed (polygen tokenization
  is CPU-light). RunPod is only warranted for GPU demos such as the VLM Space.
- Decide stable-vs-dev SDK before a customer-facing deploy. This prototype
  uses dev-only surface (`from_float`, `sigma_r`, `DecodeMode`, the
  `parallel=False` path). The frozen stable wheel currently lacks some of it.