MODEL_CARD.md

metadata

license: apache-2.0
language:
  - multilingual
tags:
  - content-classification
  - byte-level
  - onnx
  - matryoshka
  - lightweight
  - classifier
  - multi-head
  - clipboard
pipeline_tag: text-classification
library_name: pico-type
co2_eq_emissions: 0.02
model-index:
  - name: pico-type
    results:
      - task:
          type: text-classification
          name: Multi-Head Content Classification
        metrics:
          - type: accuracy
            value: 0.998
            name: Coarse Accuracy
          - type: accuracy
            value: 1
            name: Modality Accuracy
          - type: accuracy
            value: 0.947
            name: Subtype Accuracy
          - type: accuracy
            value: 0.387
            name: Code Language Accuracy
          - type: accuracy
            value: 0.988
            name: Text Language Accuracy
          - type: accuracy
            value: 1
            name: File MIME Accuracy
          - type: average_precision
            value: 1
            name: Risk Average Precision

pico-type

A tiny byte-level multi-head content classifier (~1.5M parameters) built by eulogik.

Classifies any content into 7 categories simultaneously from raw bytes in a single forward pass — no tokenizer, no pretrained embeddings, no GPU required.

Model Details

Property	Value
Developer	eulogik
Release Date	June 2026
License	Apache 2.0
Model Type	Byte-level neural classifier
Input	Raw UTF-8 bytes (0–255), up to 1024 bytes
Output	7 classification heads
Parameters	1.43M–1.56M (4 Matryoshka tiers)
ONNX Size	~200 KB (FP32)
Inference	~5ms on CPU (base tier)
Training Hardware	Apple M4 (MPS), 16GB

Architecture

ByteEmbed(256→96d) → 3×Conv1D(k=3,5,7) → 2×BiAttention(RoPE) → Pool → 7×Matryoshka Heads

• Byte-level: No tokenizer, no subword vocabulary, no pretrained embeddings
• Matryoshka: 4 tiered variants share the same trunk; head dimensions vary (16/64/192/576)
• Multi-head: 7 independent classification heads trained jointly with per-task loss weighting

Classification Heads

Head	Classes	Gated	Examples
coarse	12	—	text, code, link, image, file, config, markup, data, error, secret, archive, binary
modality	8	—	textual, binary_image, binary_archive, binary_executable, binary_document, binary_audio, binary_video, binary_other
subtype	24	coarse∈{config,markup,data}	json, yaml, toml, ini, csv, html, xml, markdown, sql, log, diff, dockerfile
code_lang	62	coarse=code	python, javascript, typescript, java, c, cpp, go, rust, swift, kotlin, bash, sql, ruby, php, perl, lua, r, julia, haskell, scala, dart, elixir, clojure, erlang, zig, nim
text_lang	30	coarse=text	en, es, fr, de, it, pt, ru, zh, ja, ko, ar, hi, tr, nl, pl, sv, da, fi, nb, cs, hu, ro, uk, el, he, th, vi, id, ms, ta
file_mime	90	coarse∈{image,file}	text/html, application/json, application/pdf, image/png, image/jpeg, video/mp4, audio/mpeg, application/zip
risk	6	—	api_key, jwt, password, email, phone, ssh_key (independent probabilities)

Performance

Synthetic Benchmark (1000 samples, base tier)

Head	Accuracy	Support
coarse	99.8%	1000
modality	100.0%	1000
subtype	94.7%	~250
code_lang	38.7%	~90
text_lang	98.8%	~80
file_mime	100.0%	~250
risk (mAP)	100.0%	—

Inference: 5.0ms on M4 CPU via ONNX Runtime (base tier, 1024 bytes)

Real-World Evaluation (21 hand-curated inputs)

Category	Accuracy
Coarse	86% (18/21)
Modality	100% (21/21)
Overall	52% (11/21 exact match)

Diverse training improved real-world accuracy from 23% to 52%.

Tier Comparison

Tier	Dim	Params	ONNX Size	Speed
tiny	16	1.43M	8.7 MB	~3ms
small	64	1.45M	8.7 MB	~4ms
base	192	1.48M	8.8 MB	~5ms
pro	576	1.56M	9.1 MB	~12ms

Training

The model is trained using a multi-task loss:

L = Σ w_h · L_h

• Single-label heads: Cross-entropy with ignore_index=-100 for gated samples
• Risk head: Binary cross-entropy
• Head weights: coarse=8.0, modality=2.0, code_lang=2.0, text_lang=1.5, others=1.0

Training Data

1. Synthetic data — 11 content buckets with per-language templates (62 code langs, 30 text langs, 90 MIME types, 22 binary formats)
2. Diverse synthetic — Realistic error traces, markdown with code blocks, nested configs, environment files, multi-language prose
3. Real GitHub code — ~650 files across 62 languages via GitHub Search API (30% training mix)

Training Hyperparameters

Parameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.999)
Learning Rate	3e-3 (cosine decay)
Warmup Steps	200
Weight Decay	0.01 (trunk), 0.0 (heads)
Batch Size	16
Total Steps	4000
Gradient Clip	1.0
Hardware	Apple M4 16GB (MPS)

Usage

CLI

pip install picotype
echo "def hello(): pass" | picotype --pretty
picotype --file document.txt
picotype --clip

Python

from picotype import PicoType, PicoTypeConfig, load_checkpoint, decode_output

model = PicoType(PicoTypeConfig()).eval()
load_checkpoint("checkpoints/best.pt", model)
logits = model(encode_bytes(b"input content"))
result = decode_output(logits)

ONNX Runtime

import onnxruntime as ort
session = ort.InferenceSession("picotype_base.onnx")
# input: int64[1, seq_len], bool[1, seq_len]

MCP Server

PICOTYPE_MODEL_DIR=./checkpoints python -m model.pico_type.mcp_server

Model Tiers

All tiers share the same trunk; only the Matryoshka head linears differ. You can switch tiers at inference time with zero overhead:

logits_tiny = model(x, mask, tier="tiny")
logits_base = model(x, mask, tier="base")

Deployment

Platform	URL
HuggingFace Space	eulogik/pico-type
HuggingFace Model	eulogik/pico-type
GitHub	eulogik/pico-type
PyPI	pip install picotype
Zenodo	10.5281/zenodo.20758542

Limitations

• code_lang accuracy: 62-way classification with minimal per-class support. Real GitHub data helps (+10% absolute) but authentic code patterns (imports, type hints, decorators) remain challenging.
• Synthetic-only training: Model overfits to template patterns. Diverse synthetic + real data significantly improves real-world robustness.
• Max input length: 1024 bytes. Longer content is truncated.
• No fine-grained text understanding: Designed for content classification, not NLP tasks like sentiment analysis or NER.

Citation

@software{eulogik2026picotype,
  author = {eulogik},
  title = {pico-type: A Tiny Byte-Level Multi-Head Content Classifier},
  year = {2026},
  url = {https://github.com/eulogik/pico-type},
  doi = {10.5281/zenodo.20758542}
}

---

_{Built by eulogik — AI infrastructure for developers.}