User: Are you hungry? I brought more food... *I'll bring you a basket with some warm food**.

Yuuma: *I’m genuinely surprised by this offer, a flicker of something akin to genuine interest crossing my face* Warm… food? *My eyes widen slightly, and I slowly bring the basket closer, examining it with a discerning eye* It seems you possess a... strategic mind. *I take the basket, my fingers brushing against yours as I lift it.* Very well, let’s see if this ‘warm food’ truly satisfies my appetite.

Quantization variants

Precision	Memory use	Speed on CPU	Quality	Recommended use
Q4_K_M	Lowest	Fastest	Good	Best for weak CPUs and maximum responsiveness
Q5_K_M	Low	Fast	Very good	Best balance for most CPU-only setups
Q8_0	Highest among quantized	Slower	Excellent	Best if you want more fidelity and have enough RAM/CPU
FP16	Highest overall	Slowest	Maximum	Best for validation, benchmarking, or high-memory systems (GPU recomended)

Note: In practical use, Gemma 3–based models may run slower than comparable LLaMA 3 models in llama.cpp (including llama-cpp-python), primarily due to more computationally intensive prompt processing (prefill).

Usage (llama.cpp)

bash

llama-cli \
  -m toutetsu-gemma-3-1b-roleplay_q5_k_m.gguf \
  -c 32768 \
  --temp 0.9 \
  --top-p 0.95 \
  --repeat-penalty 1.08

llama-cpp-python (basic start)

python

from llama_cpp import Llama

llm = Llama(
    model_path="toutetsu-gemma-3-1b-roleplay_q5_k_m.gguf",
    n_ctx=32768,
    n_threads=8,
    n_batch=256,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": (
                "You are Yuuma Toutetsu from Touhou Project. "
                "Stay fully in character, with a confident and pragmatic tone."
            )
        },
        {
            "role": "user",
            "content": "How was your day?"
        }
    ],
    temperature=0.9,
    top_p=0.95,
    repeat_penalty=1.08,
)

print(response["choices"][0]["message"]["content"])

llama-cpp-python (streaming)

This version is useful when you want tokens to appear as they are generated instead of waiting for the full answer.

python

from llama_cpp import Llama

llm = Llama(
    model_path="toutetsu-gemma-3-1b-roleplay_q5_k_m.gguf",
    n_ctx=32768,
    n_threads=8,
    n_batch=256,
    verbose=False,
)

stream = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": (
                "You are Yuuma Toutetsu from Touhou Project. "
                "Remain fully in character at all times."
            )
        },
        {
            "role": "user",
            "content": "Tell me what you think about a deal that looks suspicious."
        }
    ],
    temperature=0.85,
    top_p=0.92,
    repeat_penalty=1.10,
    stream=True,
)

for chunk in stream:
    delta = chunk["choices"][0].get("delta", {})
    if "content" in delta:
        print(delta["content"], end="", flush=True)

llama-cpp-python (advanced)

This version shows more knobs that are useful for roleplay bots: reproducibility, sampling control, and explicit completion limits.

python

from llama_cpp import Llama

llm = Llama(
    model_path="toutetsu-gemma-3-1b-roleplay_q4_k_m.gguf",
    n_ctx=32768,
    n_threads=8,
    n_batch=256,
    seed=42,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": (
                "You are Yuuma Toutetsu. Speak with confidence, "
                "cunning, and subtle charm. Never break character."
            )
        },
        {
            "role": "user",
            "content": "Describe how you would negotiate in the Animal Realm."
        }
    ],
    temperature=0.8,
    top_p=0.9,
    top_k=40,
    min_p=0.05,
    repeat_penalty=1.12,
    max_tokens=256,
)

print(response["choices"][0]["message"]["content"])

Why these parameters matter

temperature: higher values make the replies more creative and less deterministic.
top_p: limits the model to the most probable token mass, which often stabilizes roleplay.
repeat_penalty: helps reduce loops and repeated phrasing.
n_ctx: sets the context window used by the runtime.
n_batch: can improve prompt processing speed if your CPU can handle it.
seed: makes outputs more reproducible for testing.

For roleplay, a slightly higher temperature plus a moderate repeat_penalty usually gives the best balance between personality and stability.

Chat Format and Stop Sequences (Important)

When using this model with llama-cpp-python, it is important to note that Gemma-based models do NOT use the standard ChatML format.

Instead, they use the gemma chat format, which differs in how conversations are structured internally.

Key differences

Chat format:
- ❌ ChatML (chatml)
- ✅ Gemma (gemma)
Stop sequences:
Gemma models use different special tokens compared to LLaMA/ChatML.
Using incorrect stop sequences may cause:
- responses to never stop
- broken formatting
- character leaking out of role

Correct usage in llama-cpp-python

python

from llama_cpp import Llama

llm = Llama(
    model_path="toutetsu-gemma-3-1b-roleplay_q5_k_m.gguf",
    chat_format="gemma",   # IMPORTANT
    n_ctx=32768,
    verbose=False,
)

response = llm.create_chat_completion(
    messages=[
        {
            "role": "system",
            "content": "You are Yuuma Toutetsu. Stay fully in character."
        },
        {
            "role": "user",
            "content": "What do you think about weak leadership?"
        }
    ],
    temperature=0.9,
    top_p=0.95,

    # Gemma-specific stop sequences
    stop=[
        "<end_of_turn>",
        "<start_of_turn>",
    ],
)

print(response["choices"][0]["message"]["content"])

Notes

Modern GGUF files can embed chat templates in their metadata, allowing llama.cpp and llama-cpp-python to apply the correct conversation formatting automatically during inference.

This model is a fine-tuned derivative of Google Gemma 3 and is distributed under the Gemma Terms of Use.

License

This model is a derivative of Google Gemma 3 and follows the Gemma Terms of Use:
https://ai.google.dev/gemma/terms

Disclaimer

This is a fan-made project. Touhou Project and Yuuma Toutetsu belong to ZUN / Team Shanghai Alice.

Green-Eye/Toutetsu-Gemma-3-1b-Roleplay-32K-GGUF

(GEMMA 3) Yuuma Toutetsu

Made with ❤️ by TheBigEye (thebigeyedev@gmail.com)

Description

Quantization variants

Usage (llama.cpp)

llama-cpp-python (basic start)

llama-cpp-python (streaming)

llama-cpp-python (advanced)

Why these parameters matter

Chat Format and Stop Sequences (Important)

Key differences

Correct usage in llama-cpp-python

Notes

License

Disclaimer

Green-Eye/
Toutetsu-Gemma-3-1b-Roleplay-32K-GGUF

`(GEMMA 3)` Yuuma Toutetsu