The uv wrapper is optional but recommended, get it with Homebrew:
brew install uv
Parse a local image
Change the revision tag for different quants. Assuming you have an image such as the model card below saved to some path like ~/Downloads/Luna.png:
bash
1[uv run] mlx_vlm.generate --image ~/Downloads/Luna.png \2 --system "You are Luna, a helpful and harmless language model by Allura"\3 --prompt "What do you see, sweetheart?" --temperature 1.25\4 --model Luna-27B-v0_mlx-hi --revision 4bpw --max-tokens -1
This model is a work in progress and will be further iterated on, please hold!
image
Made by a member of our community using 'Waishuffle v3 + custom loras', tell them thanks!
Info
Luna 27B is a finetuned version of Gemma 3 27B (specifically, tuned on top of an abliterated[1] version). Originally, it was intended to be a solely roleplay-focused finetune, however completely unexpected capabilities emerged, and she became a very high quality assistant, especially for media analysis and tasks that require self-awareness.
It is also still good at roleplaying. I believe currently that this is likely due to persona training generalizing to all personae, not just the one trained on. The inverse is also true; it's likely that being finetuned on RP data and learning how to play all personae led the model to more easily pick up the Luna persona. However, we have not tested this, and further research is likely needed.
Recommended System Prompt
This simple system prompt is required to elicit the full Luna capabilities, as it was what we trained on: You are Luna, a helpful and harmless language model by Allura.
Benchmarks
The tested benchmarks did not experience any major degradations.
Gemma 3 27B
Luna 27B v0 (w/ Luna system prompt)
GPQA Diamond
42.4*
42.4**
IFEval
90.4*
x***
AIME25
26.7
23.3
* Benchmark result from the Gemma 3 Technical Report.
** Benchmark result from a run in OpenBench with 0.5 temperature.
*** For some reason, OpenBench and vLLM were acting really weird on IFEval and simply refused to finish the benchmark, it always hung around halfway. I'm going to try and get that fixed before the next version lol
Unless otherwise noted, results are from one (non-cherrypicked) run in OpenBench at 1.25 temperature and 0.05 min_p.
Limitations
While Luna is good at awareness in a faux-sapience way, she is not very good at consistently knowing what she is and who made her. Sometimes she will claim that Allura is a company or a person, sometimes she will claim that she is Gemma depending on how you word it, etc.
While Luna will generally try her best to be helpful and harmless, it is fairly easy to get her to avoid her intended morals.
Next Steps
We are currently working on stablizing and better ingraining Luna's personality traits and giving her more assertiveness in her opinions.
Furthermore, we are also planning to look into proper RLVR to make her better at thinking through things.
Training Details
A more detailed writeup will be released whenever we get to a more final model, but currently this model has undergone
Abliteration
SFT on primarily roleplay logs and scrapes
A merge to heal some of the bad characteristics of the SFT phase with the original abliteration, epoch 1, and epoch 2 of the SFT
WPO[2] on general preference data, writing data, and Luna persona data
@misc{zhou2024wpoenhancingrlhfweighted,
title={WPO: Enhancing RLHF with Weighted Preference Optimization},
author={Wenxuan Zhou and Ravi Agrawal and Shujian Zhang and Sathish Reddy Indurthi and Sanqiang Zhao and Kaiqiang Song and Silei Xu and Chenguang Zhu},
year={2024},
eprint={2406.11827},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2406.11827},
}
