This is an LLaMA 3 Youko qlora created using a slightly modified version of the VNTL-v3.1-1k dataset, concatenated with the VNTL-Chat dataset.
This was trained mostly with the same hyperparameters as the VNTL 7B v0.3.1 lora, the differences are:
Added ["<<METADATA>>", "<<TRANSLATE>>", "<<JAPANESE>>", "<<ENGLISH>>", "<<CHAT>>", "<<HUMAN>>", "<<LLM>>"] as special tokens.
Trained the ["embed_tokens", "lm_head"] layers.
10x smaller learning rate, 0.00065 -> 0.000065.
This version also includes a new "chat mode", which was lazily trained just to find out how that would impact the end result. I think this ended up quite good for breaking down or explaining Japanese sentences, but it is terrible for most other things, which is expected given the nature of the VNTL-Chat dataset. To be honest, I wasn't aiming for it to be good, so the fact that it works at all is very nice.
The generated translation for that prompt, with temperature 0, is:
[Shingo]: 「No, I know this isn't the best way to say it, but I'm glad you got lost. You're so cute that I was worried about you.」
Chat Prompt
This is an prompt example for chat:
<<CHAT>>
<<HUMAN>>
Hi! Can you break this down to me? Like, what is the meaning of "桜乃は可愛いから"?
"ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は可愛いから、いろいろ心配しちゃってたんだぞ俺」"
<<LLM>>
The generated response for that prompt, with temperature 0, is:
Certainly! The phrase "桜乃は可愛いから" literally translates to "Because Sakura-no is cute." In Japanese culture, there is a strong emphasis on physical attractiveness and cuteness, especially when it comes to young girls or women. This phrase suggests that the speaker finds Sakura-no (presumably a girl) to be very attractive and endearing in appearance.
