marketeam/Fineweb-Classifier-Marketing

FineWeb-Marketing Classifier

A high-performance regression model for assessing marketing content quality on a 0–5 scale. Trained on 495k marketing documents annotated by Gemma-3-27B-it, optimized for data curation and pretraining dataset filtering.

Use Case & Applications

This model is built for large-scale content quality filtering, not single-document review. Typical applications:

• Pretraining corpus curation — score every document in a FineWeb-scale crawl and keep only the top percentile of marketing content, the same way FineWeb-Edu uses an educational-quality classifier to filter Common Crawl.
• Dataset construction for domain-specific LLMs — build filtered marketing-domain pretraining or fine-tuning corpora.
• Marketing content quality scoring — rank or threshold marketing web pages, blog posts, or landing pages by practitioner-quality writing, as opposed to spam/SEO filler.

It is not intended as a fine-grained per-document editorial tool — see Intended Use for scope and Safety & Bias for known limitations.

Model Description

This model predicts marketing content quality by scoring web pages against a five-criterion rubric (Relevance, Competence, Professional, Expert, Exceptional). It uses a frozen Snowflake Arctic Embed v2.0 encoder (305M params) with a trainable regression head (591k params).

The primary model is BMse (Balanced MSE), which dominates alternative training approaches on Spearman correlation, F1@3, and recall@3 simultaneously. It achieves Spearman 0.7953 on a held-out 50k-document evaluation set.

Model Details

Architecture

Component	Details
Encoder	Snowflake/snowflake-arctic-embed-m-v2.0 (frozen)
Encoder params	305M (not trainable)
Head	Linear(768→768) + ReLU + Linear(768→1)
Head params	591,361 (trainable)
Input	[CLS] token embedding (dim=768)
Output	Scalar regression score, 0–5
Max token length	2048

Intended Use

In scope:

• Batch/offline scoring of English-language web documents for marketing-content quality, at FineWeb scale.
• Percentile- or threshold-based filtering for building pretraining or fine-tuning corpora.
• Relative ranking of documents by quality within a corpus.

Out of scope:

• Non-English content (untested).
• Fine-grained editorial feedback on a single document — the model gives one scalar score, not the per-criterion (C1–C5) breakdown; use the annotation model (Gemma-3-27B-it with the rubric in prompts/marketing_annotation.txt) directly if you need that.
• Any use presented as an absolute, universal notion of "quality" — the score reflects Gemma-3-27B-it's annotation patterns on this rubric, not an objective ground truth.

Training

Process: sample documents from FineWeb → annotate with Gemma-3-27B-it (3 independent samples per document, majority-vote aggregation) → cache frozen-encoder [CLS] embeddings → train the regression head with Balanced MSE → select the checkpoint with the best held-out Spearman correlation.

Training Details

Parameter	Value
Loss function	Balanced MSE (noise_var=1.0)
Optimizer	Adam, lr=3e-4 (no weight decay)
Batch size	32
Train set	445,116 docs (natural distribution)
Eval set	50,000 docs, held out from training (fixed split, seed 42)
Epochs	5 (early stopping patience=3 on Spearman)
Best checkpoint	Epoch 2
Seed	42
Framework versions	PyTorch 2.11.0, transformers 4.46.3

Annotation Details

Rubric: Five criteria (binary each, sum 0–5)

• C1 Relevance — Genuinely about marketing
• C2 Competence — Specific, not spam/boilerplate/SEO filler
• C3 Professional — Coherent, practitioner-quality writing
• C4 Expert — Correct frameworks, real depth, concrete data
• C5 Exceptional — Best-in-class within its dimension

Annotation model: Gemma-3-27B-it with 3 independent samples per document, majority-vote aggregation.

Annotation quality:

• 495,116 documents with valid scores (99.02% retention)
• Mean score: 1.52/5 (left skew is expected for random web crawl)
• Unanimous agreement (3/3): 55.0%
• Majority agreement (≥2/3): 94.3%
• Mean std across 3 samples: 0.253

Training Data

Source: FineWeb dataset

Sample: 500k documents from Common Crawl

Annotated: 495,116 documents with Gemma-3-27B-it — the full annotated set (including per-sample raw scores) is published at marketeam/FineWeb-Marketing-Annotations

Split:

• Training: 445,116 docs
• Evaluation: 50,000 docs — held out from training (fixed split, seed 42)

Distribution: Heavily left-skewed (mean 1.52/5), typical of web crawl data. Top ~20% score ≥3; top ~6% score ≥4.

How to Use

All usage paths below require transformers in the 4.46.x line (the frozen encoder's own custom code is not compatible with transformers 5.x): pip install "transformers==4.46.3".

Pipeline (recommended)

from transformers import pipeline

pipe = pipeline(
    "text-classification",
    model="marketeam/Fineweb-Classifier-Marketing",
    trust_remote_code=True,
)

document = (
    """
    Marketeam.ai is more than a tool; it’s your strategic partner.
    Powered by a proprietary marketing LLM and autonomous AI agents,
    it integrates seamlessly into your workflow to boost precision,
    efficiency, and impact. Whether you're a solo marketer or part of a larger team,
    Marketeam.ai expands your capabilities and helps you tackle any challenge with confidence.
    """
)
result = pipe(document)
print(round(result["score"]))

Output:

2

trust_remote_code=True is required because this repo ships a custom PreTrainedModel/pipeline (frozen encoder + regression head is not a stock transformers architecture). Review modeling_marketing_classifier.py, configuration_marketing_classifier.py, and pipeline_marketing_classifier.py in this repo before enabling it, per standard trust_remote_code practice.

Manual (no trust_remote_code)

import torch
from transformers import AutoTokenizer, AutoModel
from model import MarketingClassifier

# Load the model and encoder
model = MarketingClassifier()
state_dict = torch.load("pytorch_model.bin", map_location="cpu", weights_only=True)
model.load_state_dict(state_dict)
model.eval()

# Load the embedding encoder
tokenizer = AutoTokenizer.from_pretrained("Snowflake/snowflake-arctic-embed-m-v2.0")
encoder = AutoModel.from_pretrained("Snowflake/snowflake-arctic-embed-m-v2.0")

# Score a document
text = "Marketeam.ai is more than a tool..."
inputs = tokenizer(text, return_tensors="pt", max_length=2048, truncation=True)
with torch.no_grad():
    embeddings = encoder(**inputs).last_hidden_state[:, 0, :]  # [CLS] token
    score = model(embeddings).item()

print(f"Quality score: {score:.2f}")

Using Pre-Computed Embeddings

If you already have [CLS] embeddings, skip the encoder:

from model import load_head_only, HIDDEN_SIZE

head = load_head_only("pytorch_model.bin", HIDDEN_SIZE)
head.eval()

with torch.no_grad():
    scores = head(embeddings_tensor).squeeze(-1)

Safety & Bias

Known limitations

1. Score 5 is sparse. Only 1 document in the training set reached score 5. Percentile-based filtering is robust to this; absolute thresholds may need adjustment.

2. Gemma-calibrated. This classifier reflects Gemma-3-27B's annotation patterns. Different annotators produce different distributions.

3. English-only. Trained on English marketing content from Common Crawl. Performance on other languages is unknown.

4. Single score output. No per-criterion breakdown. To get C1–C5 scores, use the annotation model directly.

5. Frozen encoder. Only the MLP head (591k params) is trained. The embedding encoder is fixed.

Potential biases

• Longer, more formal text tends to score higher due to annotation model preferences
• Mainstream marketing sources may be over-represented
• Recent content may have different quality distributions than older web pages

Performance & Benchmarking

Evaluated on a held-out 50k-document split. Best checkpoint selected by eval Spearman.

BMse (Balanced MSE) — Primary

Metric	Value
Spearman	0.7953
F1@3	0.691
Precision@3	0.717
Recall@3	0.666
Pred std	1.88
% predicted ≥3	18.7% (true ≥3 = 20.2%)

Why Spearman?

Spearman correlation directly measures how well the model ranks documents by quality, which is critical for data filtering. F1@3 is order-invariant and unsuitable for ordinal (0–5) scoring. BMse's 0.7953 Spearman indicates strong correlation between predicted and ground-truth quality scores.

Deployment & Integration

For scoring at FineWeb scale (all CC-MAIN dumps or a specific one), use the batch inference path in infer.py, which streams a dump, scores in batches, and writes (id, dump, score, int_score, percentile) parquet shards per dump — see scripts/run_inference.sh for the CLI invocation. Percentile rank is computed per-dump so it stays stable when dumps are processed independently.

For ad hoc or low-volume scoring, the pipeline(...) one-liner above is sufficient.

License & Attribution

• License: Apache 2.0 — this checkpoint's weights are majority-composed of the frozen base encoder below, redistributed unchanged, so this repo is licensed to match that encoder's own license rather than a separately-chosen license for just the new head/wrapper code.
• Base encoder: Snowflake/snowflake-arctic-embed-m-v2.0 (frozen, unmodified, Apache 2.0)
• Training data: HuggingFaceFW/fineweb
• Annotation model: Gemma-3-27B-it

Dependencies

See requirements.txt for all dependencies. Key packages:

• torch >= 2.0
• transformers >= 4.46
• numpy
• scipy