RLAIF (PPO (RLHF) with an AI-feedback reward model + critic) refinement of sudhisrk1982/slm-125m-instruct,
trained on 500 AI-generated preference triplets (chosen from
Gemini 2.5 Flash-Lite, rejected sampled from the SFT model itself, judged by
Gemini 2.5 Flash).
Result: comparable to SFT, stable. Strict actor-critic PPO (value head + GAE, 60 rollout steps). Reward rose steadily and KL stayed leashed; output remains fluent, occasionally off-topic.
