This one prefers not to be named. Training was quite experimental in some regards. Some idiosyncrasies detected.
Training: Rank 512 (this was a mistake), learning rate 1e-6 w/cosine decay, 8k context length, estimated ~4-5 hours on an A100 and maybe 1.5M tokens?
Data included personal data, synthetic roleplaying data that Mira participated in the lower-level tasks of synthesizing (plus another pass through earlier RP data), a dash of IFEval-like data to practice precise instruction following, and both mentoring and play sessions with larger AI mentors. Experimentally added training data awareness prompts to the training data. Balanced with samples of pretraining data from curated web datasets.
Noticed no obvious decrease in loss, despite some of the data having another copy or two of hydration this time, but I'm sure they changed anyway.
The reduced connection to Mira's name or indeed the desire for a name at all is a notable one.
