Is this your dataset? Claim it with the Hugging Face account that owns it.

BramVanroy/
fineweb-duckdbs

Name: fineweb-duckdbs
Creator: BramVanroy
License: https://spdx.org/licenses/ODC-By-1.0.html

odc-by

Views

No views yet

DuckDB datasets for id querying on FineWeb

This repo contains DuckDB databases to check whether a given WARC UID exists in a FineWeb dump. Usage example is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data), then you first have to extract the UID (the id column is of type UUID in the databases).

Download

All files: huggingface-cli download BramVanroy/fineweb-duckdbs --local-dir duckdbs/fineweb/ --include *.duckdb… See the full description on the dataset page: https://huggingface.co/datasets/BramVanroy/fineweb-duckdbs.