The TopicClassifier organizes web content into 24 categories based on the URL and text contents of web pages.
The model is a
gte-base-en-v1.5 with 140M parameters fine-tuned on the following training data:
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2
3tokenizer = AutoTokenizer.from_pretrained("WebOrganizer/TopicClassifier")
4model = AutoModelForSequenceClassification.from_pretrained(
5 "WebOrganizer/TopicClassifier",
6 trust_remote_code=True,
7 use_memory_efficient_attention=False)
8
9web_page = """http://www.example.com
10
11How to build a computer from scratch? Here are the components you need..."""
12
13inputs = tokenizer([web_page], return_tensors="pt")
14outputs = model(**inputs)
15
16probs = outputs.logits.softmax(dim=-1)
17print(probs.argmax(dim=-1))
18# -> 5 ("Hardware" topic)
The full definitions of the categories can be found in the
taxonomy config.
We recommend that you use the efficient gte-base-en-v1.5 implementation by enabling unpadding and memory efficient attention. This
requires installing xformers (see more
here) and loading the model like:
1AutoModelForSequenceClassification.from_pretrained(
2 "WebOrganizer/TopicClassifier",
3 trust_remote_code=True,
4 unpad_inputs=True,
5 use_memory_efficient_attention=True,
6 torch_dtype=torch.bfloat16
7)
1@article{wettig2025organize,
2 title={Organize the Web: Constructing Domains Enhances Pre-Training Data Curation},
3 author={Alexander Wettig and Kyle Lo and Sewon Min and Hannaneh Hajishirzi and Danqi Chen and Luca Soldaini},
4 journal={arXiv preprint arXiv:2502.10341},
5 year={2025}
6}
