The RyokoAI/ShareGPT52K dataset, converted to Markdown and labeled with the language used.
vinta/pangu.js — To insert whitespace between CJK (Chinese, Japanese, Korean) and half-width characters (alphabetical letters, numerical digits and symbols).
matthewwithanm/python-markdownify — Provides a starting point to convert HTML to Markdown.
BYVoid/OpenCC — Conversions between Traditional Chinese and Simplified Chinese.
aboSamoor/polyglot… See the full description on the dataset page:
https://huggingface.co/datasets/zetavg/ShareGPT-Processed.
