README.md

---
license: apache-2.0
language:
- en
- pt
tags:
- chunking
- rag
- text-splitter
- retrieval
- semantic-chunking
library_name: tinyzchunk
pipeline_tag: other
---

# tinyzchunk — weights

Distilled weights for [**tinyzchunk**](https://github.com/cnmoro/tinyzchunk), a
GPU-free, tokenizer-free chunker for RAG pipelines. This repository holds only
the weights; the code lives on PyPI.

```bash
pip install tinyzchunk
```

```python
from tinyzchunk import Chunker

chunker = Chunker()                  # downloads these weights once, then caches
chunks = chunker.chunk(document)     # -> list[str]
```

## What these are

Two small MLPs that read raw characters — no tokenizer, no GPU, numpy only:

| file | model | size |
|---|---|---|
| `line_weights.npz` | line-level unit-start detector (±5 line context, 102 features/char) | ~1.9 MB |
| `weights.npz` | character-level sentence/paragraph boundary model | ~0.2 MB |

They were distilled from an LLM teacher (Qwen) that labelled boundaries offline,
one time. Inference needs neither.

Tuned for **English and Brazilian Portuguese**, and hardened for messy real-world
text: PDF extractions with mid-word wrapping, page numbers and form feeds,
OCR-mangled words, CRLF files, markdown, fenced code, tables, chat logs and legal
enumerations.

## Quality

Evaluated across 95 held-out scenario buckets (`scripts/eval_matrix.py` in the
GitHub repo):

| document family | boundary F1 |
|---|---|
| markdown, code, tables | 0.97 |
| sectioned prose, headings, bios | 0.97 |
| legal articles and enumerations | 0.87 |
| schedules and field blocks | 0.79 |
| Q&A and FAQ | 0.78 |
| wrapped / OCR-noisy prose | 0.72 |

Macro F1 **0.795** overall, **0.77** on noisy-text buckets alone. Fragment chunks
**0.08%**, oversized chunks **0%**. Roughly 26 ms for a 3 kB document on one CPU
core.

## Compatibility

The weights carry a feature-schema digest. If you pair them with a tinyzchunk
build whose feature extractor differs, the library raises a clear error instead
of producing silent garbage — upgrade `tinyzchunk` (>= 0.3.0 for this revision)
or pin the matching weights revision.

Licence: Apache-2.0.