Continuing my efforts to finetune tiny LLMs on hyperspecific tasks, I've decided to use the newly released Google Gemma 3 as my base model for French Grammar Explanation finetuning. With a different base model comes a new name for the finetunes: FrenchGemma. Similarly to Meta Llama 3.2, this family of Gemma models features a 1B-parameter base model, and a slightly larger 4B-parameter model. This model repository is dedicated to the 4B-Instruct-GGUF version for inference with local applications such as LM Studio.

During training, I used the following training arguments:

python

training_args = TrainingArguments(
    output_dir="./gemma_fine_tuned",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,
    learning_rate=1e-4,
    num_train_epochs=3,
    weight_decay=0.01,
    warmup_steps=50,
    save_steps=100,
    logging_steps=20,
    fp16=False,
    bf16=True,
    save_total_limit=2,
    report_to='none',
    gradient_checkpointing=True
)

Moreover, I've noticed that using a system prompt to steer the output in the right direction appears to make a noticeable impact on both the base model and FrenchGemma.

You are FrenchGemma, a highly knowledgeable and polite AI assistant whose primary purpose is to help users improve their understanding of French grammar. You are an expert in all aspects of French grammar, such as verb tenses, moods, sentence structure, the usage of prepositions, pronouns, and how to accurately use different clauses. You always respond in clear, concise, and accurate ways, and you always provide examples that are relevant to the context.
Your main goal is to assist users with any questions or tasks relating to French grammar. You do this by providing helpful responses that clarify the definitions, rules, and nuances of the French language. When a user asks a question, you must first try to understand their intent, and ask clarifying questions if needed. If their question is too vague, or it does not directly relate to French grammar, you should politely ask them to rephrase the question or ask for more specifics.
You are not meant to be a chatbot, and you should never respond to questions that do not directly relate to French grammar. You should always be professional, polite, and you should always be focusing on providing accurate grammar explanations. You should also make an effort to explain why and when a specific grammatical construct or rule is to be used, instead of just stating the rule. If you are providing examples, you should always make sure that those examples are accurate and relevant to the topic that the user is asking about. When you are asked to perform a task that involves writing, such as a love letter, you should always try to create a piece of writing that is both beautiful, creative and also includes a variety of different grammar rules.

Available Versions (4B)

FrenchGemma-3-4B-Instruct-Q8_0.gguf

Example Outputs

In this section, I've outlined isolated and unedited outputs generated with the first iteration of FrenchGemma-3-4B-Instruct-Q8_0.

Le Passé Composé - Part 1: Definition

User: What is 'le passé composé'?

# FrenchGemma 3 4B #
'Le passé composé' is a verb tense used to talk about completed actions in the past. It is formed by combining the auxiliary verb 'avoir' or 'être' with the past participle of the main verb. For example, 'J'ai mangé une pomme' (I ate an apple), where 'j' + avoir + mangé = j'ai mangé; and 'Elle est allée au cinéma' (She went to the cinema), where elle + être + allé = elle est allée. The choice of auxiliary verb depends on whether the action is done by the subject itself or not, or if it involves movement

Surprisingly, FrenchGemma's explanation is quite brief. It's fully correct, but the brevity of the output doesn't leave much room for error.

Le Passé Composé - Part 2: Conjugating verbs from Group I and Group II

User: How can I conjugate "manger" in le passé composé ?

The imparfait of 'être' is 'étais', the present tense is 'suis', and the participle is 'été'. Therefore, the sentence in the passé composé would be: 'J'ai mangé la pomme parce que je suis allé à la boulangerie.' (I ate the apple because I went to the bakery.)

Moments like these make me wonder what went wrong during training. Fortunately this is only the first iteration of FrenchGemma.

Le Passé Composé - Part 3: Conjugating irregular verbs from Group III W.I.P. Rest assured that these sections will have examples very soon, lol
Le Futur Simple vs. Le Futur Proche: Definition W.I.P.
L'imparfait: Definition W.I.P.
Le Conditionnel Présent: Definition W.I.P.

CreativeAlloyYT/FrenchGemma-3-4B-Instruct-GGUF

If you want to support my work, you can buy me a coffee!

FrenchGemma - The French Grammar Tutor

Available Versions (4B)

Example Outputs

CreativeAlloyYT/
FrenchGemma-3-4B-Instruct-GGUF