token-counts – Dataset by marin-community | AlphaNeural AI

You can deploy this model and start earning money today!

marin-community/
token-counts

like

0

text-generation

Views

No views yet

Marin Token Counts

Token counts for all datasets used in Marin pretraining runs.

	Schema

Column Type Description

dataset string Dataset identifier

marin_tokens int Number of tokens after tokenization

category string Content domain (web, code, math, academic, books, etc.)

synthetic bool Whether the data is LLM-generated or LLM-translated

Categories

web — Quality-classified Common Crawl text (Nemotron-CC) code — Source code and… See the full description on the dataset page: https://huggingface.co/datasets/marin-community/token-counts.