REPORT.md

SNIP: Small N-gram Identifier for Pastes

Abstract

SNIP is a compact text classifier designed for in-browser classification into a subset of labels common in pastebin websites. The release model predicts 28 source and text labels from a bounded sample using hashed character n-grams and a sparse linear classifier. It runs with a dependency-free JavaScript runtime.

The release model reaches 1.0000 validation accuracy, 0.9962 test accuracy, 0.9816 held-out accuracy, and 0.9932 hard-case accuracy while staying around 205 KB gzip-compressed.

1. Problem

Pastebin sites, editors, and developer tools often need a useful syntax suggestion immediately after text is pasted. The goal of the classifier was to build something small enough to ship with a web app, and fast enough to feel instant. Binary file detection and full file-type forensics are outside the current scope.

2. Labels

The release label set is:

bash, c, cpp, csharp, css, csv, diff, dockerfile, go, html, ini, java, javascript, json, log, lua, markdown, php, plain_text, powershell, python, ruby, rust, sql, toml, typescript, xml, yaml.

3. Data

The dataset was built iteratively. Early code samples were short and repetitive, which helped validate the pipeline but produced over-optimistic results. Those examples were replaced with more realistic files and fragments.

Specifically for programming languages, the training set was generated by LLMs over successive training runs. Careful consideration was taken to avoid overfitting to a specific language or task. Examples were written as complete files or realistic project fragments: CLIs, web handlers, tests, services, configuration loaders, migrations, and framework code. This produced more useful character n-gram coverage than primitive string shuffling or pulling only from my personal code corpus, and allowed me to train a proficient model without personally maintaining a diverse codebase in every target language.

Some label types are better suited to programmatic generation, such as JSON, CSV, and logs. Even there, the useful cases were varied by shape, length, field names, and value distributions rather than generated from one repeated template.

The final training data emphasizes:

1. realistic short snippets, because paste inputs are often not full files
2. source-grouped splits, so related chunks do not leak across train/test
3. explicit hard cases for ambiguous neighboring labels such as HTML/XML, INI/TOML, JSON/log, Markdown/plain text, and CSV/plain text

Distribution and quality checks

Split	Rows	Labels	Min chars	Median chars	P90 chars	Max chars
Train	2,618	28	20	210.5	1,124	33,617
Validation	487	28	13	213	1,192	22,701
Test	532	28	19	180	1,049	8,942

Training rows by label

Label	Train rows
bash	28
c	28
cpp	31
csharp	32
css	124
csv	154
diff	131
dockerfile	114
go	37
html	132
ini	148
java	32
javascript	38
json	144
log	163
lua	29
markdown	163
php	33
plain_text	314
powershell	40
python	43
ruby	50
rust	33
sql	136
toml	140
typescript	37
xml	127
yaml	137

Verification rows by label (validation, test, and held-out suites combined)

Label	Verification rows
bash	32
c	32
cpp	33
csharp	33
css	66
csv	71
diff	67
dockerfile	66
go	33
html	68
ini	67
java	33
javascript	34
json	75
log	72
lua	32
markdown	70
php	32
plain_text	257
powershell	32
python	37
ruby	33
rust	33
sql	68
toml	68
typescript	33
xml	68
yaml	73

Rows by length bucket:

Split	<64	64-128	128-256	256-1K	1K-4K	4K-16K	16K+
Train	26	452	1,144	682	308	3	3
Validation	5	89	202	128	62	0	1
Test	13	125	213	124	55	2	0

Training rows by source family:

Source family	Train rows	Purpose
Structured/generated format examples	1,795	Broad coverage for structured text, config, prose, and data formats
Curated programming/project examples	428	Realistic source files and project fragments
Targeted hard-neighbor additions	310	Short snippets and close label pairs found through error analysis
Plain-text fallback examples	63	Text that should remain plain_text despite punctuation or weak structure
Real local/project examples	8	Repository-derived real files
Product/format-specific examples	14	Focused coverage for paste-product formats

Every candidate split was checked for exact and normalized duplicate text. Train-only additions after error analysis were also checked for exact and near-duplicate overlap against held-out suites before use.

4. Model

The release model is a multiclass linear classifier trained with a passive-aggressive update. Features are hashed character n-grams:

• n-gram range: 1 through 5 characters
• hash buckets: 32,768
• feature value: log1p(count)
• normalization: L2 normalization
• shape features: disabled
• retained weights: 1,500 per label
• serialized weight precision: 4 decimal places

Long inputs are sampled before classification. Inputs up to 16 KiB are classified whole. Longer inputs are represented by windows from the start, middle, and end, joined with a separator. This keeps browser inference bounded for large pastes while preserving signal from common file regions.

5. Training

The first strong model used a Naive Bayes classifier with character n-grams, but I ran into a wall trying to handle edge cases where sparse labels with cheap default probabilities could dominate real files.

The linear model fixed that behavior. It scores labels directly from learned positive and negative weights instead of relying on per-class default probabilities. With top-weight pruning, the model stayed small enough for browser delivery while preserving the real-file behavior we wanted.

Training loop

A major objective of training the n-gram was testing how much of training a model can be performed semi-autonomously by an LLM Agent. n-grams are incredibly fast to train, and afforded the model many opportunities to run experiences and build up a training dataset over time that specifically targeted failures in the trained models. A key part of getting that loop to work was ensuring the data stayed representative of a real-world text distribution while keeping the model from over-fitting to the generated dataset. The Agent was instructed to keep notes, an experiments log, and a data quality log. It did require a bit of nudging along the way, such as prompting it to manually inspect some of the generated data and note the quality issues (For example early C examples were almost all duplicates of 4-5 lines of a main() printing a few words).

Each round started with a candidate model trained on the current corpus. The agent evaluated it against the normal validation/test split, targeted hard cases for confused labels and the held-out evaluation suites. When the model failed, a subagent was used to describe the failures as broader missing shape: short prose-heavy Markdown, TOML that looks like INI, JSON-looking application logs, small diffs, short language snippets without imports, and so on.

Those descriptions were then used to create new train-only examples. For programming languages, the examples were usually generated as realistic files or fragments rather than templates. For structured formats, the generation could be more programmatic, but still had to vary field names, lengths, nesting, and punctuation. Before a new training split was used, the added rows were checked against the rest of the corpus for exact or near-duplicate overlap.

A separate LLM subagent was used to create new held-out evaluation suites between rounds. This was important to help ensure the overall quality of the train dataset. We want the generated examples to be different enough to keep the model robust. The held-out suites were generated from label-level requirements and broad scenario prompts, then locked before the next candidate was evaluated. Once a held-out suite exposed an error category and influenced the next data pass, it was no longer treated as a final benchmark. It became a regression suite.

The loop looked like this:

1. train a candidate on the current corpus
2. evaluate against validation, test, hard cases, and held-out evaluation suites
3. inspect failures as categories and patterns
4. generate train-only coverage for those categories
5. check duplicate and near-duplicate overlap
6. train the next candidate
7. create fresh held-out coverage when a release-quality claim is needed

This process let the dataset grow in the directions the n-gram actually needed while keeping a clean separation between training data and held-out evaluation suites.

Evaluation role	Rows	Purpose
Validation + test	1,019	Main split evaluation
Held-out suites	599	Independent checks generated across training rounds
Hard-neighbor cases	148	Targeted stress cases for commonly confused labels

Weight pruning

After training, pruning is applied to help reduce the size of the model. Weights with an absolute value below 0.001 are removed, and each label keeps at most 1,500 learned n-gram weights, chosen by absolute value. The remaining weights are serialized to 4 decimal places. This was a simple way to cut model size while keeping the runtime and JSON model format straightforward.

6. Results

SNIP stats:

Evaluation set	Examples	Accuracy	Macro F1	Known coverage
Validation split	487	1.0000	1.0000	1.0000
Test split	532	0.9962	0.9926	1.0000
Held-out evaluation suites	599	0.9816	0.9819	1.0000
Hard-neighbor cases	148	0.9932	0.6905	1.0000

Model size:

Artifact	Size
snip_model.json	626,595 bytes
gzip-compressed model	204,593 bytes

7. Performance

SNIP's runtime target requires low latency because classification happens in user-facing paste flows and should not scale linearly with very large inputs. Performance was measured with the model embedded inside a single HTML file to remove any possible network latency.

Captured result:

• Browser: Google Chrome 149.0.7827.116
• Platform: macOS arm64
• Model parse: 4.4 ms

Input size	Sampled chars	Classifications	Mean ms	P50 ms	P95 ms
1 KB	1,024	5,000	1.499	1.490	1.548
16 KB	16,384	1,000	6.578	6.580	6.730
100 KB	12,292	800	5.195	5.180	5.310
1 MB	12,292	800	5.184	5.170	5.310
5 MB	12,292	800	5.238	5.210	5.380

The non-monotonic timing is expected. Inputs up to 16 KiB are classified whole. Larger inputs are sampled into three 4 KiB windows plus separators, so 100 KB, 1 MB, and 5 MB inputs all classify roughly the same amount of text.

8. Runtime

The reference runtime is authored in src/snip.ts. It implements:

1. bounded input sampling,
2. FNV-1a style stable hashing over UTF-16 code units,
3. character n-gram feature extraction,
4. sparse linear scoring,
5. softmax ranking.

The runtime has no dependencies.

9. Limitations

SNIP is optimized for pasted text and text-like files. Binary file identification is outside the current release scope.

The release model has known weaknesses:

• very small structured snippets can still be close to plain_text
• TypeScript and JavaScript can be close when the snippet lacks type syntax
• SNIP scores every label and selects the highest-scoring label; the margin reports the gap to the runner-up
• Markdown/plain-text separation is still weak on very short prose-like Markdown
• INI/TOML and JSON/log remain close neighbors when examples are short.