This model is based on GPT2-Medium finetuned on chat logs from twitch.tv/MOONMOON.
The data consists of ~3.8GB of plaintext across 632 days of logs, ranging from 2021-01-01 to 2022-09-26. They were sourced from
https://logs.ivr.fi/. The logs were cleaned by dropping
-
bots: messages from a (manually determined, non-exhaustive) list of bots
-
links: messages matching the regex
r"[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_\+.~#?&//=]*)"
-
long messages: messages longer than 300 characters
-
short messages: messages shorter than 2 characters
-
caps spam: messages containing more than 100 characters that are more than 80% capital letters
-
commands: messages starting with !
The data was batched into groups of up to 512 tokens, preferring to end on a newline (\n) rather than start another line and truncate it. The batches were then padded to 512 tokens using a pad token added to the model and tokenizer.
10% of the data was set aside for validation.
Training was done on a system with a 6800XT (16GB of VRAM) and 32GB of RAM. The following hyperparameters were used:
Evaluation was performed 10 times throughout training. Accuracy and perplexity were calculated.
