You can deploy this model and start earning money today!

PRLM/
oasst2

Views

No views yet

from datasets import load_dataset, Dataset import re

Script to filter and process the OpenAssistant dataset (oasst2).

Based on the conversion script from the open-instruct repo -> https://github.com/allenai/open-instruct/blob/main/scripts/data/sft/utils.py#L1

def should_be_filtered_by_keyword(example, verbose=False): # we filter out conversations that contain some specific strings filter_strings = [ "OpenAI", "Open AI", "ChatGPT", "Chat GPT"… See the full description on the dataset page: https://huggingface.co/datasets/PRLM/oasst2.