README.md - You are here!
source/ - Original Mediawiki xml file, as well as a cleaned version. The cleaned version is the one used to generate documents/ and chromadb/.
documents/ - Converted into markdown files.
chromadb/ - Converted into a chromadb to eliminate the markdown conversion and embedding steps to use the dataset for RAG. It is highly recommended that the embedding model used for retrieval is the same as the one used to generate the embeddings. The… See the full description on the dataset page:
https://huggingface.co/datasets/xarical/wof-wiki.
