NanoFlare is my newest model, based on the recently released Qwen 3.8 27B. It is designed to run completely* on GPU if you have 8GB of VRAM, otherwise it should be able to run on CPU if you have at least 8GB or more of system memory (this has not been tested).
v1.b is the latest version (recommended)
Revision B brings major improvements to the quality of output, especially in code over what we were seeing from rev.A. It also no longer gets into thinking loops that often anymore.
*If you are running NanoFlare on an 8GB GPU, you will have to offload the kv cache to CPU in most cases. And unfortunately this means it's going to be a little slow. If you are running this model on a card that has 10GB or more you should have all the room you need and see much better performance with the kv cache on GPU.
Relative perplexity scores (all tested with wikitext2, lower is better):
| Model | Size | 2k ctx PPL | 8k ctx PPL |
|---|
| v1 revision b | 7469.3 MB | 8.2480 ±0.05573 | 7.9631 ±0.05479 |
| unsloth/IQ2_XXS (UD 3.0) | 6929.5 MB | 7.5441 +/- 0.04906 | 8.0013 +/- 0.05689 |
| v1 revision a | 7165.9 MB | 8.6832 ±0.05605 | 8.1774 ±0.05276 |
| release candidate 1 | 7442.6 MB | 8.7946 ±0.05764 | 8.2998 ±0.05454 |
| beta 1 | 7532.8 MB | 8.8221 ±0.05797 | 8.3352 ±0.05506 |
The model seems to run best at temperature: 0.95 + top-k: 20 (although it works fine at llama.cpp defaults of 0.8 and 40 too).
Note this model has been hyper-compressed, so it's not perfect. It may flake out on you from time to time, but it is mostly fairly stable. Taking quantization to the extreme levels I'm using comes at a cost. The LLM will sometimes not complete its thought process. If you find yourself in this situation, try the prompt again. It often works correctly on the second try when this happens. It seems to be pretty good at most general tasks, and is good at coding. It has forgotten some facts and figures from its training data through the heavy quantization, so I'd recommend giving it web search access or double checking anything its telling you from recall.
A moderate amount of testing has been performed now, but more scores and such will be posted soon. From subjective, manual testing v1b seems to be far superior to v1a.
