Introducing wrath in the Seven Deadly Sins series of models.
Continuous pre-training of qlora on Yi-34b
High-quality martial arts novels
Thoughtful cleaning process
This model is designed to serve as the base model in the agent model of the Live Action Role Playing games. For this purpose, I've collected approximately 10G of martial arts novels, sourced from various novel websites and PT sites. However, this dataset includes a significant amount of duplicate and low-quality content. To address these issues, I've undertaken the following steps:
1. Define Data Quality Dimensions
For martial arts novels, high-quality works are typically represented by authors like Jin Yong, Gu Long, and Liang Yusheng. In these novels, the complexity of the plot is a critical factor and is the focal point for script quality.
2. Quantify Data Quality Dimensions
Given the emphasis on plot complexity, we approached this in several stages:
Convert plot summaries into vectors using a BERT-based model.
Measure transitions between chapters through cosine similarity or Euclidean distance.
Develop a complexity algorithm focused on standard deviation and peak analysis.
Metric Quantification:
Apply subjective weighting to the complexity metrics derived from chapter transitions.
3. Outcome
By employing these methods, we can effectively filter out novels of higher quality. This refined dataset has been shared for further use. The next step is to continue pretraining, for which specific parameters can be referred to in my previous model descriptions.
image/png
image/png
As you can see, the zero-shot effect is good. The settings in some pretraining novels are very naturally embedded into the characters.
