License: Use of this model is governed by the Meta license. In order to download the model weights and tokenizer, please visit the website and accept their License before downloading the model weights.
Datasets Details
Scraped posts of a particular subject within an image board.
The dataset was heavily augmented with various types of filtering to improve coherency and relevency to the origin and our goals.
For our Cesspit model, it contains 272,637 entries.
Prompt Template
The model was not trained in an instructional or chat-style format, please ensure your inference program does not attempt to inject anything more than your sole input when inferencing, simply type whatever comes to mind and the model will attempt to complete it.
Hardware and Software
Hardware: We utilized 3.8 Nvidia RTX 4090 hours for training our model.
Training Factors: We created this model using Axolotl
Training details
The rank and alpha we used was 128 and 16.
Our learning rate was 4e-4 with 10 warmups steps with a cosine scheduler for 3 epoch.
Our batch size was 5 microbatch
Sample packing was used.
Limitations
It is strongly recommend to not deploy this model into a real-world environment unless its behavior is well-understood and explicit and strict limitations on the scope, impact, and duration of the deployment are enforced.
