Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
mosaic-madlad-400-ms – Dataset by malaysia-ai | AlphaNeural AI
You can deploy this model and start earning money today!
malaysia-ai
/
mosaic-madlad-400-ms
like
0
ms
n<1K
json
text
datasets
pandas
mlcroissant
polars
us
Views
No views yet
Model card
Files and Versions
Community
API
Mosaic format for extra dataset to train Malaysian LLM
This repository is to store dataset shards using mosaic format.
prepared at
https://github.com/malaysia-ai/dedup-text-dataset/blob/main/pretrain-llm/combine-madlad-400-ms.ipynb
using tokenizer
https://huggingface.co/malaysia-ai/bpe-tokenizer
4096 context length.
how-to
git clone,
git lfs clone
https://huggingface.co/datasets/malaysia-ai/mosaic-madlad-400-ms
load it,
from streaming import LocalDataset import… See the full description on the dataset page:
https://huggingface.co/datasets/malaysia-ai/mosaic-madlad-400-ms
.
