REPORT.md

# SNIP: Small N-gram Identifier for Pastes

## Abstract

SNIP is a compact text classifier designed for in-browser classification into a subset of labels common in pastebin websites. The release model predicts 28 source and text labels from a bounded sample using hashed character n-grams and a sparse linear classifier. It runs with a dependency-free JavaScript runtime.

The release model reaches 1.0000 validation accuracy, 0.9962 test accuracy, 0.9816 held-out accuracy, and 0.9932 hard-case accuracy while staying around 205 KB gzip-compressed.

## 1. Problem

Pastebin sites, editors, and developer tools often need a useful syntax suggestion immediately after text is pasted. The goal of the classifier was to build something small enough to ship with a web app, and fast enough to feel instant. Binary file detection and full file-type forensics are outside the current scope.

## 2. Labels

The release label set is:

`bash`, `c`, `cpp`, `csharp`, `css`, `csv`, `diff`, `dockerfile`, `go`, `html`, `ini`, `java`, `javascript`, `json`, `log`, `lua`, `markdown`, `php`, `plain_text`, `powershell`, `python`, `ruby`, `rust`, `sql`, `toml`, `typescript`, `xml`, `yaml`.

## 3. Data

The dataset was built iteratively. Early code samples were short and repetitive, which helped validate the pipeline but produced over-optimistic results. Those examples were replaced with more realistic files and fragments.

Specifically for programming languages, the training set was generated by LLMs over successive training runs. Careful consideration was taken to avoid overfitting to a specific language or task. Examples were written as complete files or realistic project fragments: CLIs, web handlers, tests, services, configuration loaders, migrations, and framework code. This produced more useful character n-gram coverage than primitive string shuffling or pulling only from my personal code corpus, and allowed me to train a proficient model without personally maintaining a diverse codebase in every target language.

Some label types are better suited to programmatic generation, such as JSON, CSV, and logs. Even there, the useful cases were varied by shape, length, field names, and value distributions rather than generated from one repeated template.

The final training data emphasizes:

1. realistic short snippets, because paste inputs are often not full files
2. source-grouped splits, so related chunks do not leak across train/test
3. explicit hard cases for ambiguous neighboring labels such as HTML/XML, INI/TOML, JSON/log, Markdown/plain text, and CSV/plain text

### Distribution and quality checks

| Split | Rows | Labels | Min chars | Median chars | P90 chars | Max chars |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Train | 2,618 | 28 | 20 | 210.5 | 1,124 | 33,617 |
| Validation | 487 | 28 | 13 | 213 | 1,192 | 22,701 |
| Test | 532 | 28 | 19 | 180 | 1,049 | 8,942 |

<details>
<summary>Training rows by label</summary>

| Label | Train rows |
| --- | ---: |
| `bash` | 28 |
| `c` | 28 |
| `cpp` | 31 |
| `csharp` | 32 |
| `css` | 124 |
| `csv` | 154 |
| `diff` | 131 |
| `dockerfile` | 114 |
| `go` | 37 |
| `html` | 132 |
| `ini` | 148 |
| `java` | 32 |
| `javascript` | 38 |
| `json` | 144 |
| `log` | 163 |
| `lua` | 29 |
| `markdown` | 163 |
| `php` | 33 |
| `plain_text` | 314 |
| `powershell` | 40 |
| `python` | 43 |
| `ruby` | 50 |
| `rust` | 33 |
| `sql` | 136 |
| `toml` | 140 |
| `typescript` | 37 |
| `xml` | 127 |
| `yaml` | 137 |

</details>

<details>
<summary>Verification rows by label (validation, test, and held-out suites combined)</summary>

| Label | Verification rows |
| --- | ---: |
| `bash` | 32 |
| `c` | 32 |
| `cpp` | 33 |
| `csharp` | 33 |
| `css` | 66 |
| `csv` | 71 |
| `diff` | 67 |
| `dockerfile` | 66 |
| `go` | 33 |
| `html` | 68 |
| `ini` | 67 |
| `java` | 33 |
| `javascript` | 34 |
| `json` | 75 |
| `log` | 72 |
| `lua` | 32 |
| `markdown` | 70 |
| `php` | 32 |
| `plain_text` | 257 |
| `powershell` | 32 |
| `python` | 37 |
| `ruby` | 33 |
| `rust` | 33 |
| `sql` | 68 |
| `toml` | 68 |
| `typescript` | 33 |
| `xml` | 68 |
| `yaml` | 73 |

</details>

Rows by length bucket:

| Split | <64 | 64-128 | 128-256 | 256-1K | 1K-4K | 4K-16K | 16K+ |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| Train | 26 | 452 | 1,144 | 682 | 308 | 3 | 3 |
| Validation | 5 | 89 | 202 | 128 | 62 | 0 | 1 |
| Test | 13 | 125 | 213 | 124 | 55 | 2 | 0 |

Training rows by source family:

| Source family | Train rows | Purpose |
| --- | ---: | --- |
| Structured/generated format examples | 1,795 | Broad coverage for structured text, config, prose, and data formats |
| Curated programming/project examples | 428 | Realistic source files and project fragments |
| Targeted hard-neighbor additions | 310 | Short snippets and close label pairs found through error analysis |
| Plain-text fallback examples | 63 | Text that should remain `plain_text` despite punctuation or weak structure |
| Real local/project examples | 8 | Repository-derived real files |
| Product/format-specific examples | 14 | Focused coverage for paste-product formats |

Every candidate split was checked for exact and normalized duplicate text. Train-only additions after error analysis were also checked for exact and near-duplicate overlap against held-out suites before use.

## 4. Model

The release model is a multiclass linear classifier trained with a passive-aggressive update. Features are hashed character n-grams:

- n-gram range: 1 through 5 characters
- hash buckets: 32,768
- feature value: `log1p(count)`
- normalization: L2 normalization
- shape features: disabled
- retained weights: 1,500 per label
- serialized weight precision: 4 decimal places

Long inputs are sampled before classification. Inputs up to 16 KiB are classified whole. Longer inputs are represented by windows from the start, middle, and end, joined with a separator. This keeps browser inference bounded for large pastes while preserving signal from common file regions.

## 5. Training

The first strong model used a Naive Bayes classifier with character n-grams, but I ran into a wall trying to handle edge cases where sparse labels with cheap default probabilities could dominate real files.

The linear model fixed that behavior. It scores labels directly from learned positive and negative weights instead of relying on per-class default probabilities. With top-weight pruning, the model stayed small enough for browser delivery while preserving the real-file behavior we wanted.

### Training loop

A major objective of training the n-gram was testing how much of training a model can be performed semi-autonomously by an LLM Agent. n-grams are incredibly fast to train, and afforded the model many opportunities to run experiences and build up a training dataset over time that specifically targeted failures in the trained models. A key part of getting that loop to work was ensuring the data stayed representative of a real-world text distribution while keeping the model from over-fitting to the generated dataset. The Agent was instructed to keep notes, an experiments log, and a data quality log. It did require a bit of nudging along the way, such as prompting it to manually inspect some of the generated data and note the quality issues (For example early C examples were almost all duplicates of 4-5 lines of a `main()` printing a few words). 

Each round started with a candidate model trained on the current corpus. The agent evaluated it against the normal validation/test split, targeted hard cases for confused labels and the held-out evaluation suites. When the model failed, a subagent was used to describe the failures as broader missing shape: short prose-heavy Markdown, TOML that looks like INI, JSON-looking application logs, small diffs, short language snippets without imports, and so on.

Those descriptions were then used to create new train-only examples. For programming languages, the examples were usually generated as realistic files or fragments rather than templates. For structured formats, the generation could be more programmatic, but still had to vary field names, lengths, nesting, and punctuation. Before a new training split was used, the added rows were checked against the rest of the corpus for exact or near-duplicate overlap.

A separate LLM subagent was used to create new held-out evaluation suites between rounds. This was important to help ensure the overall quality of the train dataset. We want the generated examples to be different enough to keep the model robust. The held-out suites were generated from label-level requirements and broad scenario prompts, then locked before the next candidate was evaluated. Once a held-out suite exposed an error category and influenced the next data pass, it was no longer treated as a final benchmark. It became a regression suite.

The loop looked like this:

1. train a candidate on the current corpus
2. evaluate against validation, test, hard cases, and held-out evaluation suites
3. inspect failures as categories and patterns
4. generate train-only coverage for those categories
5. check duplicate and near-duplicate overlap
6. train the next candidate
7. create fresh held-out coverage when a release-quality claim is needed

This process let the dataset grow in the directions the n-gram actually needed while keeping a clean separation between training data and held-out evaluation suites.

| Evaluation role | Rows | Purpose |
| --- | ---: | --- |
| Validation + test | 1,019 | Main split evaluation |
| Held-out suites | 599 | Independent checks generated across training rounds |
| Hard-neighbor cases | 148 | Targeted stress cases for commonly confused labels |

### Weight pruning

After training, pruning is applied to help reduce the size of the model. Weights with an absolute value below `0.001` are removed, and each label keeps at most `1,500` learned n-gram weights, chosen by absolute value. The remaining weights are serialized to 4 decimal places. This was a simple way to cut model size while keeping the runtime and JSON model format straightforward.

## 6. Results

SNIP stats:

| Evaluation set | Examples | Accuracy | Macro F1 | Known coverage |
| --- | ---: | ---: | ---: | ---: |
| Validation split | 487 | 1.0000 | 1.0000 | 1.0000 |
| Test split | 532 | 0.9962 | 0.9926 | 1.0000 |
| Held-out evaluation suites | 599 | 0.9816 | 0.9819 | 1.0000 |
| Hard-neighbor cases | 148 | 0.9932 | 0.6905 | 1.0000 |

Model size:

| Artifact | Size |
| --- | ---: |
| `snip_model.json` | 626,595 bytes |
| gzip-compressed model | 204,593 bytes |

## 7. Performance

SNIP's runtime target requires low latency because classification happens in user-facing paste flows and should not scale linearly with very large inputs. Performance was measured with the model embedded inside a single HTML file to remove any possible network latency.

Captured result:

- Browser: Google Chrome 149.0.7827.116
- Platform: macOS arm64
- Model parse: 4.4 ms

| Input size | Sampled chars | Classifications | Mean ms | P50 ms | P95 ms |
| --- | ---: | ---: | ---: | ---: | ---: |
| 1 KB | 1,024 | 5,000 | 1.499 | 1.490 | 1.548 |
| 16 KB | 16,384 | 1,000 | 6.578 | 6.580 | 6.730 |
| 100 KB | 12,292 | 800 | 5.195 | 5.180 | 5.310 |
| 1 MB | 12,292 | 800 | 5.184 | 5.170 | 5.310 |
| 5 MB | 12,292 | 800 | 5.238 | 5.210 | 5.380 |

The non-monotonic timing is expected. Inputs up to 16 KiB are classified whole. Larger inputs are sampled into three 4 KiB windows plus separators, so 100 KB, 1 MB, and 5 MB inputs all classify roughly the same amount of text.

## 8. Runtime

The reference runtime is authored in `src/snip.ts`. It implements:

1. bounded input sampling,
2. FNV-1a style stable hashing over UTF-16 code units,
3. character n-gram feature extraction,
4. sparse linear scoring,
5. softmax ranking.

The runtime has no dependencies.

## 9. Limitations

SNIP is optimized for pasted text and text-like files. Binary file identification is outside the current release scope.

The release model has known weaknesses:

- very small structured snippets can still be close to `plain_text`
- TypeScript and JavaScript can be close when the snippet lacks type syntax
- SNIP scores every label and selects the highest-scoring label; the margin reports the gap to the runner-up
- Markdown/plain-text separation is still weak on very short prose-like Markdown
- INI/TOML and JSON/log remain close neighbors when examples are short.