raw.zip is the ~5m posts that i initially pulled that were full of duplicates, where i removed the duplicates and compiled those into 2.6 million posts in final_posts.jsonl
the rest of the .jsonl files in the data folder, and not inside the raw.zip are unchecked and may have duplicates. feel free to write your own script to remove them, but this should contain around 3 million unique posts.
