2.5B GPT-2 tokens staged for extending harims95/hobbylm-1b-hf's
context window from 1024 tokens to 4096, then 8192.
Format
Each _NNNNNN.bin file is the same flat-binary format used throughout
HobbyLM training: a 256 x int32 header (magic=20240520, version=1,
num_tokens), followed by the token stream as uint16 GPT-2 token ids.
The flat .bin files do not encode document boundaries on their own. Each
source also ships… See the full description on the dataset page: https://huggingface.co/datasets/harims95/hobbylm-longctx-2p5b.
