A high-performance regression model for assessing marketing content quality on a 0–5 scale. Trained on 495k marketing documents annotated by Gemma-3-27B-it, optimized for data curation and pretraining dataset filtering.
Use Case & Applications
This model is built for large-scale content quality filtering, not single-document review. Typical applications:
Pretraining corpus curation — score every document in a FineWeb-scale crawl and keep only the top percentile of marketing content, the same way FineWeb-Edu uses an educational-quality classifier to filter Common Crawl.
Dataset construction for domain-specific LLMs — build filtered marketing-domain pretraining or fine-tuning corpora.
Marketing content quality scoring — rank or threshold marketing web pages, blog posts, or landing pages by practitioner-quality writing, as opposed to spam/SEO filler.
Model Description
This model predicts marketing content quality by scoring web pages against a five-criterion rubric (Relevance, Competence, Professional, Expert, Exceptional). It uses a frozen Snowflake Arctic Embed v2.0 encoder (305M params) with a trainable regression head (591k params).
The primary model is BMse (Balanced MSE), which dominates alternative training approaches on Spearman correlation, F1@3, and recall@3 simultaneously. It achieves Spearman 0.7953 on a held-out 50k-document evaluation set.
Model Details
Architecture
Component
Details
Encoder
Snowflake/snowflake-arctic-embed-m-v2.0 (frozen)
Encoder params
305M (not trainable)
Head
Linear(768→768) + ReLU + Linear(768→1)
Head params
591,361 (trainable)
Input
[CLS] token embedding (dim=768)
Output
Scalar regression score, 0–5
Max token length
2048
Training
Process: sample documents from FineWeb → annotate with Gemma-3-27B-it (3 independent samples per document, majority-vote aggregation) → cache frozen-encoder [CLS] embeddings → train the regression head with Balanced MSE → select the checkpoint with the best held-out Spearman correlation.
Training Details
Parameter
Value
Loss function
Balanced MSE (noise_var=1.0)
Optimizer
Adam, lr=3e-4 (no weight decay)
Batch size
32
Train set
445,116 docs (natural distribution)
Eval set
50,000 docs, held out from training (fixed split, seed 42)
Annotated: 495,116 documents with Gemma-3-27B-it — the full annotated set (including per-sample raw scores) is published at marketeam/FineWeb-Marketing-Annotations
Split:
Training: 445,116 docs
Evaluation: 50,000 docs — held out from training (fixed split, seed 42)
Distribution: Heavily left-skewed (mean 1.52/5), typical of web crawl data. Top ~20% score ≥3; top ~6% score ≥4.
How to Use
All usage paths below require transformers in the 4.46.x line (the frozen encoder's own custom code is not compatible with transformers 5.x): pip install "transformers==4.46.3".
Pipeline (recommended)
python
1from transformers import pipeline
23pipe = pipeline(4"text-classification",5 model="marketeam/Fineweb-Classifier-Marketing",6 trust_remote_code=True,7)89document =(10"""
11 Marketeam.ai is more than a tool; it’s your strategic partner.
12 Powered by a proprietary marketing LLM and autonomous AI agents,
13 it integrates seamlessly into your workflow to boost precision,
14 efficiency, and impact. Whether you're a solo marketer or part of a larger team,
15 Marketeam.ai expands your capabilities and helps you tackle any challenge with confidence.
16 """17)18result = pipe(document)19print(round(result["score"]))
Output:
2
trust_remote_code=True is required because this repo ships a custom PreTrainedModel/pipeline (frozen encoder + regression head is not a stock transformers architecture). Review modeling_marketing_classifier.py, configuration_marketing_classifier.py, and pipeline_marketing_classifier.py in this repo before enabling it, per standard trust_remote_code practice.
Manual (no trust_remote_code)
python
1import torch
2from transformers import AutoTokenizer, AutoModel
3from model import MarketingClassifier
45# Load the model and encoder6model = MarketingClassifier()7state_dict = torch.load("pytorch_model.bin", map_location="cpu", weights_only=True)8model.load_state_dict(state_dict)9model.eval()1011# Load the embedding encoder12tokenizer = AutoTokenizer.from_pretrained("Snowflake/snowflake-arctic-embed-m-v2.0")13encoder = AutoModel.from_pretrained("Snowflake/snowflake-arctic-embed-m-v2.0")1415# Score a document16text ="Marketeam.ai is more than a tool..."17inputs = tokenizer(text, return_tensors="pt", max_length=2048, truncation=True)18with torch.no_grad():19 embeddings = encoder(**inputs).last_hidden_state[:,0,:]# [CLS] token20 score = model(embeddings).item()2122print(f"Quality score: {score:.2f}")
Using Pre-Computed Embeddings
If you already have [CLS] embeddings, skip the encoder:
Score 5 is sparse. Only 1 document in the training set reached score 5. Percentile-based filtering is robust to this; absolute thresholds may need adjustment.
Gemma-calibrated. This classifier reflects Gemma-3-27B's annotation patterns. Different annotators produce different distributions.
English-only. Trained on English marketing content from Common Crawl. Performance on other languages is unknown.
Single score output. No per-criterion breakdown. To get C1–C5 scores, use the annotation model directly.
Frozen encoder. Only the MLP head (591k params) is trained. The embedding encoder is fixed.
Potential biases
Longer, more formal text tends to score higher due to annotation model preferences
Mainstream marketing sources may be over-represented
Recent content may have different quality distributions than older web pages
Performance & Benchmarking
Evaluated on a held-out 50k-document split. Best checkpoint selected by eval Spearman.
BMse (Balanced MSE) — Primary
Metric
Value
Spearman
0.7953
F1@3
0.691
Precision@3
0.717
Recall@3
0.666
Pred std
1.88
% predicted ≥3
18.7% (true ≥3 = 20.2%)
Why Spearman?
Spearman correlation directly measures how well the model ranks documents by quality, which is critical for data filtering. F1@3 is order-invariant and unsuitable for ordinal (0–5) scoring. BMse's 0.7953 Spearman indicates strong correlation between predicted and ground-truth quality scores.
Deployment & Integration
For scoring at FineWeb scale (all CC-MAIN dumps or a specific one), use the batch inference path in infer.py, which streams a dump, scores in batches, and writes (id, dump, score, int_score, percentile) parquet shards per dump — see scripts/run_inference.sh for the CLI invocation. Percentile rank is computed per-dump so it stays stable when dumps are processed independently.
For ad hoc or low-volume scoring, the pipeline(...) one-liner above is sufficient.
License & Attribution
License: Apache 2.0 — this checkpoint's weights are majority-composed of the frozen base encoder below, redistributed unchanged, so this repo is licensed to match that encoder's own license rather than a separately-chosen license for just the new head/wrapper code.
