README.md

metadata

title: Datasent's Polygen Tokenization Demo
emoji: 📈
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

Datasent's Polygen Tokenization Demo

An interactive demo of Datasent's Polygen SDK on numeric signals. Pick a sample (or upload a CSV), and it tokenizes the signal through the real SDK and reports:

• Lossless round-trip -- EXACT decode is bit-faithful to the 1/scale grid.
• COARSE analytics archive -- coefficients only, tens to hundreds of times smaller than raw, and directly queryable (the coefficients are a feature vector).
• Per-segment anomaly signal (sigma_R) -- a free quality marker; segments the model fits poorly stand out.
• EXACT lossless size shown honestly against gzip (it sits at parity -- polygen is a tokenizer, not a byte-compression replacement).

Everything goes through the supported SDK surface: DataMatrix.from_float -> PolygenTokenizer.tokenize(..., parallel=False) with BinaryEncoder for a real token, and tokenizer.decode for EXACT / COARSE. The in-process parallel=False path means no multiprocessing pool, which keeps the Space light.

Running locally

pip install -r requirements.txt
# plus polygen on the path (editable dev install or the stable wheel)
POLYGEN_DEMO_LOCAL_MOCK=1 python app.py

POLYGEN_DEMO_LOCAL_MOCK=1 generates a self-consistent throwaway license for local dev (the same pattern tests/conftest.py uses). Do not set it in production.

Deploying to a HuggingFace Space

1. Install polygen via requirements.txt (point at the stable wheel URL).
2. Set the POLYGEN_LICENSE secret to a real JWT issued for that wheel.
   • Note: this depends on the license keypair being aligned (audit finding F8). The bundled public key must match the key that signed the JWT.
3. Leave POLYGEN_DEMO_LOCAL_MOCK unset so the SDK validates the real license.

Notes

• This is a prototype. CPU-only; no GPU / RunPod needed (polygen tokenization is CPU-light). RunPod is only warranted for GPU demos such as the VLM Space.
• Decide stable-vs-dev SDK before a customer-facing deploy. This prototype uses dev-only surface (from_float, sigma_r, DecodeMode, the parallel=False path). The frozen stable wheel currently lacks some of it.