Upload folder using huggingface_hub

paper/main.tex

@@ -16,17 +16,20 @@
 % --- metadata ---
 \title{pico-type: A 1.5M-Parameter Byte-Level Multi-Head Content Classifier}
 \author{%
-  \texttt{eulogik} \\
-  \texttt{\url{https://github.com/eulogik/pico-type}} \\
-  \texttt{\url{https://huggingface.co/eulogik/pico-type}}
+  \textsf{eulogik} \\
+  \small\texttt{\url{https://eulogik.com}} \\
+  \small\texttt{\url{https://github.com/eulogik/pico-type}} \\
+  \small\texttt{DOI: \url{https://doi.org/10.5281/zenodo.20758542}}
 }
 \date{\today}
 
 \begin{document}
 \maketitle
+\vspace{-0.5em}
+{\small\noindent\textit{License: Creative Commons Attribution 4.0 International (CC BY 4.0). Source code and model weights: Apache 2.0.}} \hrule\vspace{1em}
 
 \begin{abstract}
-We introduce \textbf{pico-type}, a tiny byte-level multi-head content classifier with approximately 1.5 million parameters that simultaneously predicts seven content properties from raw UTF-8 bytes in a single forward pass. Operating directly at the byte level with no tokenizer or pretrained embeddings, pico-type classifies coarse type, modality, subtype, code language, text language, file MIME type, and risk flags (API keys, JWTs, passwords). The model uses a lightweight trunk of byte embedding, convolutional blocks, bidirectional attention with rotary position embeddings, and a statistical pooling layer feeding seven Matryoshka-style classification heads. With four tiered variants ranging from 16 to 576 dimensions and ONNX export under 210 KB, pico-type achieves 100\% accuracy on coarse, modality, file MIME, and risk detection, 98.4\% on subtype, 100\% on text language, and 53.9\% on code language (62-way) through a mixed training approach combining synthetic data with real GitHub code samples. It is designed for on-device deployment in CLIs, browser extensions, and MCP servers.
+We introduce \textbf{pico-type}, a tiny byte-level multi-head content classifier with approximately 1.5 million parameters that simultaneously predicts seven content properties from raw UTF-8 bytes in a single forward pass. Operating directly at the byte level with no tokenizer or pretrained embeddings, pico-type classifies coarse type, modality, subtype, code language, text language, file MIME type, and risk flags (API keys, JWTs, passwords). The model uses a lightweight trunk of byte embedding, convolutional blocks, bidirectional attention with rotary position embeddings, and a statistical pooling layer feeding seven Matryoshka-style classification heads. With four tiered variants ranging from 16 to 576 dimensions and ONNX export under 210 KB, pico-type achieves 100\% accuracy on coarse, modality, file MIME, and risk detection, 98.4\% on subtype, 100\% on text language, and 53.9\% on code language (62-way) through a mixed training approach combining synthetic data with real GitHub code samples. On a hand-curated real-world test set spanning 21 diverse inputs, the model achieves 52.4\% overall accuracy across all heads, demonstrating practical viability beyond synthetic benchmarks. The model is designed for on-device deployment in CLIs, browser extensions, and MCP servers.
 \end{abstract}
 
 % ===================================================================
@@ -174,8 +177,9 @@ We generate a synthetic training dataset of samples balanced across 12 coarse co
 
 \begin{itemize}[noitemsep]
   \item \textbf{Code:} Parameterized templates for 62 languages across 18 syntactic groups (Python-like, C-like, Lisp-like, etc.).
-  \item \textbf{Text:} Language-specific word lists for 30 languages, generating prose of 1--5 sentences.
-  \item \textbf{Config/markup:} Templates for JSON, YAML, TOML, INI, CSV, TSV, XML, HTML, Markdown, reST, AsciiDoc, and \LaTeX.
+  \item \textbf{Text:} Language-specific word lists for 30 languages, generating prose of 1--5 sentences, plus realistic patterns such as emails, conversations, and technical documentation.
+  \item \textbf{Config/markup:} Templates for JSON, YAML, TOML, INI, CSV, TSV, XML, HTML, Markdown, reST, AsciiDoc, and \LaTeX. Includes nested structures, markdown with embedded code blocks, and shell environment variables.
+  \item \textbf{Error:} Realistic stack traces for Python (with file paths and line numbers), JavaScript, Java, and Go.
   \item \textbf{Binary:} Real magic-byte headers for 22 formats (PDF, ZIP, PNG, JPEG, ELF, WASM, etc.).
   \item \textbf{Secrets:} Regex-generated AWS keys, JWTs, SSH keys, and passwords with entropy filtering.
 \end{itemize}
@@ -184,8 +188,11 @@ Each sample is a tuple of raw bytes $\mathbf{x} \in \{0,\dots,255\}^L$ with $L \
 
 \subsection{Training Setup}
 
-\subsection{Real Data Augmentation}
-To improve code language classification accuracy, we augment the synthetic training set with real source code files fetched from GitHub via the public Search API. For each of the 62 supported languages, up to 25 files are retrieved per training run, yielding approximately 1,500 real code samples per epoch. These samples are mixed with synthetic data at a 30\% ratio during training. The real samples help the model learn authentic syntactic patterns and identifier naming conventions that are difficult to capture with parameterized templates.
+\subsection{Data Augmentation with Real Code}
+
+The initial synthetic-only training achieves high accuracy on most heads but reveals two limitations: (1) code language accuracy saturates at approximately 44\% on 62-way classification due to limited template diversity, and (2) the model performs poorly on real-world inputs, defaulting to ``code'' for non-code content.
+
+To address these issues, we employ two complementary strategies. First, we augment the synthetic code generator with diverse patterns---error stack traces, markdown with code blocks, environment variable files, and multi-language technical prose---to better approximate the distribution of real-world clipboard content. Second, we fetch real source code files from GitHub via the public Search API. For each of the 62 supported languages, up to 25 files are retrieved per training run, yielding approximately 1,500 real code samples per epoch. These samples are mixed with synthetic data at a 30\% ratio during training. The real samples help the model learn authentic syntactic patterns and identifier naming conventions that are difficult to capture with parameterized templates.
 
 We train with a multi-task loss:
 
@@ -193,7 +200,7 @@ We train with a multi-task loss:
 \mathcal{L} = \sum_{h \in \mathcal{H}} w_h \cdot \mathcal{L}_h
 \end{equation}
 
-For single-label heads (coarse, modality, subtype, code\_lang, text\_lang, file\_mime), $\mathcal{L}_h$ is cross-entropy with \texttt{ignore\_index = -100} for gated samples. For the risk head, $\mathcal{L}_h$ is binary cross-entropy. Per-head weights are coarse: 3.0, modality: 2.0, code\_lang: 1.5, text\_lang: 1.5, others: 1.0.
+For single-label heads (coarse, modality, subtype, code\_lang, text\_lang, file\_mime), $\mathcal{L}_h$ is cross-entropy with \texttt{ignore\_index = -100} for gated samples. For the risk head, $\mathcal{L}_h$ is binary cross-entropy. Per-head weights are tuned to balance task difficulty: coarse: 8.0, modality: 2.0, code\_lang: 2.0, text\_lang: 1.5, others: 1.0. The high coarse weight prevents the model from defaulting to a majority class (``code'' being the most common coarse type in balanced synthetic data) and improves real-world coarse classification.
 
 We use AdamW optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.999$, weight decay 0.01, and a linear warmup of 200 steps followed by cosine decay to 0. Gradient clipping at 1.0 is applied. Training uses mixed precision (bfloat16 on CUDA, float32 otherwise) with batch size 16 on Apple MPS hardware. Synthetic-only training runs for up to 5,000 steps; real-data augmented fine-tuning continues for an additional 2,000--4,000 steps at a reduced learning rate of $3 \times 10^{-4}$.
 
@@ -231,6 +238,10 @@ risk (mAP) & 6 & 100.0\% & -- \\
 
 Coarse, modality, and file\_mime classification achieve near-perfect accuracy on synthetic data. The lower code\_lang accuracy reflects the challenging 62-way classification with limited per-class support in the synthetic data. Risk detection achieves 90.5\% mean average precision.
 
+\subsection{Real-World Evaluation}
+
+To complement synthetic benchmarks, we evaluate on a hand-curated set of 21 real-world inputs spanning code (Python, JavaScript, C, Java, SQL, Bash), markup (HTML, Markdown), config (JSON, YAML, env), text (English, French, Spanish), error traces (Python, JavaScript), binary headers (PNG, PDF), and file formats. The model achieves 52.4\% overall accuracy (11/21 correct). Coarse classification is correct for config, error, text, and markup categories; the primary failure mode is code language detection on real snippets, where authentic imports, type hints, and decorators differ significantly from synthetic templates. This 2.3$\times$ improvement over the 22.7\% baseline (pre-diverse-training) demonstrates that the combination of diverse synthetic data and increased coarse head weighting substantially improves real-world robustness.
+
 \subsection{Matryoshka Tier Comparison}
 
 Table~\ref{tab:tiers} compares the four model tiers. All tiers share the same trunk; only the final linear layers differ.