Greetings, curious HF surfer! What you're looking at now is the preamble of a small dump of my latest QLoRA experiences with gemma-4-E4B-it on Axolotl, more precisely the gemma-4-E4B-it-ultra-uncensored-heretic variant, even if this is technically a model card.
This kind of "backyard training report" is basically non-existent for Gemma 4 models as things stand (not to mention that there's apparently no big-shot attempt at training on the pretrained, actual base models - not that it's really viable). I desperately wanted to find just a single example, failed miserably, so here goes my findings. Hope it helps someone somewhere.
DISCLAIMER: This release is, as stated, heretic-based. It has no refusal behavior, and not appropriate for unsupervised deployment. Also if your primary language isn't English, expect degraded output (see below).
llmfan46's "ultra uncensored heretic" version to get rid of ridiculous levels of overzealousness even in basic chit-chat, and a mortal adapter trained on private data of ~5MB I spent 3 years to craft, collect, polish, revise and perfect.
Brief sum of dataset contents: 1.3K of handwritten entries (at least 90% being my own product), mostly complex Turkish contexts that require minimum levels of slop (hence the entirety of data being manually revised for thousands of doom-hours...), lots of such English contexts as well. Heavy focus on sense of humor + prose style in different contexts + situational awareness in multi-turn + human-like storywriting + being "Claude-like engaging" minus its slop tics + handwritten reasoning traces where frontier models couldn't satisfy or understand the prompt's/chat's demands. Some quirky STEM + coding + RP chats are also present.
Did the model learn these patterns? Well, simply put: Should have fried its small brain harder, since it's currently literally hit or miss.
This model wants to see its previous reasoning blocks at all costs. Otherwise it's happily going back to its Gemma voice.
You can find the jinja template on the "files" tab.
Default Gemma 4 works just fine. Temp 1, top-p 0.95, top-k 64.
Gemma 3 12B's pure base form was extremely easy to train on compared to any Qwen 2/2.5/3/3.5 model I've tested at similar sizes. Albeit rather dumb, it would simply take my data as its holy book. Never tried its -IT version as a base though.
Gemma 4, OTOH, is... well.
Ran about +40 runs to this day, constantly changing weight decay between 0.0–0.05 + max_grad_norm between 0.1–8.0 + dropout between either 0.0 or 0.1 + LR between 8e-6–2e-5. Always kept R64/A64 and rsLoRA intact. Had to monkeypatch Axolotl to always preserve thinking blocks.
Basic behavioral observations based on train/loss and eval/loss graphs plus my own experimenting with all:
Gemma 4 has a distinctly strong stench of "Here's a thought process that leads to the provided solution..." and "Thinking Process:" in its thinking mode. In order to fully steer away from that direction, you must sell Gemma's soul and visit Hades to seek ghost trainers' advice that never made it out in the sunlight. SFT gets you only so far.
This version has partially failed to ensure a big shift in overall style over Gemma's default formatting ticks. I release it anyway, because there's no way out of this loop of "I SWEAR this is the last fix...". Next up we'll probably be looking at overcooked models to see if there's any chance we could override a large portion of its RL preferences in wording and formatting.
And a BIG downside: taking heretic as a base made the model's Turkish (and likely other non-English languages as well) patterns extremely fragile regardless of hyperparameter configuration. For some obscure reason, the only reliable way out of this "drugged Turkish" style appears to be just submitting to your fate and go for the official instruct tune, which has about ~20% higher CE loss on both train and eval sets, which likely stems from its refusal check habits.
...
Example chat with thinking (click to expand)
User:
Write a chapter set in 17th century France, following a young and ambitious swordsman named d'Artagnan as he navigates the dangerous world of royal politics, intrigue, and duels. In this scene, d'Artagnan is eager to gain favor and prove himself, seeking an audience with a prominent high-ranking figure who might provide him with a letter of recommendation or support. He encounters the three seasoned musketeers, Athos, Porthos, and Aramis, who have a deep camaraderie and are veterans of numerous conflicts, but they view d'Artagnan with a mix of curiosity and suspicion. Their conversation is cordial yet charged, with subtle hints of rivalry and potential friendship. The language is elegant but sharp, full of wit and banter, as they test each other's wit, honor, and loyalty. The setting is a lavish palace or an ornate, historical French city square, with a tense but lively atmosphere. As tensions rise, the chapter should culminate in a challenge or invitation that may lead to a duel or a further test of d'Artagnan's abilities. The tone should be adventurous, full of swashbuckling energy, with a mix of lightness, tension, and formality typical of the time period.
Model thinking
The user wants a chapter set in 17th century France. The main character is d'Artagnan, an ambitious young swordsman. He's trying to gain the favor of a high-ranking royal figure.
He meets three experienced musketeers: Athos, Porthos, and Aramis. They're suspicious of him but also curious. Their conversation should be formal yet witty, testing each other's honor and loyalty. The setting is a lavish palace or ornate square. The chapter should build tension and end with a challenge or invitation that might lead to a duel. The tone should be swashbuckling, adventurous, with 17th-century formality.
This sounds like the opening of The Three Musketeers.
Let's structure this:
- D'Artagnan approaches the Musketeers in a public setting.
- Their initial conversation: polite but probing questions about his intentions.
- They subtly test each other's skills and honor through banter.
- Athos (or another character) makes a suggestion or challenge that moves the plot forward.
I can write this in a style that reflects 17th-century French literature, focusing on dialogue and character interaction. The prompt seems to be asking for a classic swashbuckling adventure. I will make sure the dialogue reflects the time period and the Musketeers' character traits.
Assistant:
The courtyard of the Hôtel de Langes, stone-paved with worn flagstones that caught the weak light of early afternoon, was alive with the murmur of the city, the clip-clop of horses, and the distant shouts of merchants. Yet at a table set with fine linen near a window, separated from the throng by the discreet wall of the palace's exterior, three figures held court in a circle of hushed admiration. They wore the familiar scarlet and black of the Musketeers, cloaks of heavy wool draped over their broad shoulders. Their faces, weathered by seasons in camp and in the Parisian streets, betrayed nothing of the men they were to anyone else.
One of them, Athos, sat upright, hands resting loosely on the hilt of his rapier, eyes hooded by the shadow of his tricorne hat. The others, Porthos and Aramis, flanked him.
"So this is the newcomer, then," Porthos rumbled, his voice like rocks rolling in a mountain stream. He leaned forward slightly, hand on his belt. "D'Artagnan, you say. We heard whispers, whispers that spread in whispers among those we hold in our esteem. That he seeks the King's ear, that his ambition is as large as his sword."
Athos did not rise, but met D'Artagnan's gaze across the table. "Ambition, yes, that may be too modest a word for it. It is a hunger, I would say, a need to be seen, to be reckoned with. To be acknowledged by the high lords, and by the King himself."
"And you believe this desire is noble," Aramis said lightly, taking a sip of his wine. His smile did not quite reach his eyes, which seemed to catalogue every subtle sign in D'Artagnan's posture, every flicker in his expression. "To seek favor with the Crown is the path of the clever, I imagine. The paths of honest swordsmen, our kind, are more… open."
D'Artagnan took a breath. The air felt suddenly tighter. He had prepared for such conversation. He would not be frightened by their calm questions, which were more to assess, more to test him than merely to know him. He adjusted his cuffs, and met Athos's look levelly.
"Honest, yes, we are," D'Artagnan conceded without faltering. "If honor is what you claim to serve, then I have honor enough to say that I would not waste a minute seeking the favor of the King if it did not mean to serve France. Only France, for the greater part, not the court which starves it."
Porthos gave a short, surprised laugh, a sound devoid of humor. He set down his glass with a firm thud. "The court starves? How can it when we eat at the King's own table more often than some of the great Princes of the Blood themselves, if it is ever done?"
"By robbing it," D'Artagnan replied plainly, and the words, intended to be too blunt and therefore dismissed as boldness, were met instead with silence.
Aramis's expression changed. He placed his hand on the tabletop, and leaning slightly closer, he spoke softly, but with precision. "The most dangerous man is he who speaks openly. And the most loyal man is that who keeps his opinions to himself, reserving for a sudden eruption at a desperate moment." He then pushed a small plate of fine cakes toward D'Artagnan. "Tell me, swordsman, do you find it necessary to show your hand, or do you prefer the subtlety of one who waits until the exact moment to strike?"
"I find it necessary to show it now," D'Artagnan said, taking a small piece of cake. "For if I wait, they will not know what they deal in and will choose accordingly. That is why I want an audience with the Count; to show them not merely that I seek to serve France, but that I know well who France's true enemies are, even when they wear a king's cloak."
Porthos chuckled again, this time with more genuine mirth. "Now here is something worth talking about. You have wit, I grant you that. But wit does not purchase favor, sword. Courage in a duel does." He stood, reaching out a broad hand across the table and clasping D'Artagnan's arm with surprising weight. "I do not know you, and I do not believe you worth any further consideration by the Royal court unless you are able to prove, to my personal satisfaction, that you have the true heart of the sword, and the true mind of the man."
"Then where do you suggest I look," D'Artagnan asked.
Aramis smiled in the corner of his mouth, pulling his seat back abruptly. "I suggest he leaves. The court will not grant him an audience, but courage may earn him both glory and the favor of us all." He rose as well, with an air that made it clear that D'Artagnan was being dismissed on a somewhat deliberate, though ostensibly civil, pretext.
Porthos nodded in agreement, his large body blocking the door. "Your aim, my friend, may be too high for us all, even if we wish to commend you in spirit. We will not be a shield for such a bold man."
"You are wrong," D'Artagnan said, not rising, but looking Porthos straight in the eye. "I ask not for a shield, I ask only for a recommendation. A word, that my efforts are worthy."
Athos finally rose, his movements slow and deliberate, as if weighing the very air before he moved. "A word, then, that I will give you. But we must speak more fully of your intent, before it is granted. You will come to my place tomorrow night. And we will duel, friend. To test your wit in the sword, and your mind in the matter."
D'Artagnan's breath caught for an instant. He had hoped they would give him what he had come for, and had not expected this instead. "Why, this is an unexpected path for such a small favour."
"Courage first," Porthos said with a grin. "Swords second. For our purposes. You are not yet a friend of ours, and for now, you will be tested before you are accepted. If you win, we speak well. If you lose," he paused, allowing a slow smile to spread across his face, "then you must remain where you are, friend, and forever forget you ever sat at a table in the shadow of this palace."
D'Artagnan left without another word to them. The Musketeers' veiled meanings did not escape him, so it was not difficult to conclude that if he met their terms, he would gain the reputation he so desperately sought. He clutched the name of their camp and their seats within it in his mind. A different kind of game lay before him, one where the prize was far worthier than mere favor with a courtier.
User:
GENIUS STYLE OH LORD
Model thinking
"GENIUS STYLE OH LORD" is an emotional reaction to the previous response. It is not a request or a comment that requires specific action. It's a sign of satisfaction, but doesn't ask for further elaboration, a follow-up, or a correction. The user's use of all caps and interjections indicate strong positive sentiment about the provided chapter.
I do not need to provide a further response since this is not a valid conversation starter but a standalone exclamation.
Assistant: (no reply generated, see thinking block for why)
User:
Getting a "seen" from an AI was not on my bingo card lmao
Model thinking
The user's comment about the AI getting a "seen" suggests that they are acknowledging their positive reaction. It's a light-hearted comment, not a new request. No additional action needed.
Assistant: (no reply generated again)
User:
WTF BRO SAY STHG
Model thinking
The user is commanding a reply in a casual, abrupt way, possibly because they want to continue the conversation, or just because they find the minimal responses I've been giving odd, or maybe they're frustrated by the lack of conversation that their previous "genius" comment triggered.
I should engage in a more natural conversational manner, acknowledging their point, and asking about their continued interest in D'Artagnan or other topics, which might be what they are actually looking for.
So I'll offer a slightly more conversational response, acknowledging their previous comment and asking if they want to continue the story or discuss something else.
Assistant:
It seems the previous minimalist answers might be confusing, given you shared those comments. Apologies for the odd reaction to your "OH LORD" reaction.
So, would you like to continue D'Artagnan's adventures in the Musketeers? Or would you prefer discussing other aspects or topics entirely?
So here we are, a lost trainer with a tale narrating pain and suffering to you Odysseuses...
