The model was fine-tuned with the following script using Lora (no, qlora). I also included a inference script to make sure it works since there were some issues with Gemma. Results of the inference test are

bash

prompt:
 What is the capital of Germany? Explain why thats the case and if it was different in the past?
response:
 Berlin is the capital of Germany.  It was the capital of Prussia until 1918, when the monarchy was abolished.  It was also the capital of the Weimar Republic.  It was the capital of the Third Reich until 1945, when it was liberated by the allies.  It has been the capital of the Federal Republic of Germany since 1949.  It is the largest city in the country with a population of 3.6 million people.  It is also the seat of the government and parliament.

prompt:
 In a town, 60% of the population are adults. Among the adults, 30% have a pet dog and 40% have a pet cat. What percentage of the total population has a pet dog?
response:
 60% of the total population have a pet dog. The calculation is 30% of adults multiplied by 60% of the total population. 30% of adults is 18% of the total population and 18% multiplied by 60% is 10.8% or 60% of the total population.

Run inference

python

import torch
from peft import AutoPeftModelForCausalLM
from transformers import  AutoTokenizer, pipeline

peft_model_id = "philschmid/gemma-7b-dolly-chatml"

# Load Model with PEFT adapter
tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
model = AutoPeftModelForCausalLM.from_pretrained(peft_model_id, device_map="auto", torch_dtype=torch.float16)
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer)
eos_token = tokenizer("<|im_end|>",add_special_tokens=False)["input_ids"][0]
print(f"eos_token: {eos_token}")

# run inference
messages = [
    {
        "role": "user",
        "content": "What is the capital of Germany? Explain why thats the case and if it was different in the past?"
    }
]

prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=1024, do_sample=True, temperature=0.7, top_k=50, top_p=0.95, eos_token_id=eos_token)
print(outputs[0]['generated_text'][len(prompt):])
#  Berlin is the capital of Germany.  It was the capital of Prussia until 1918, when the monarchy was abolished.  It was also the capital of the Weimar Republic.  It was the capital of the Third Reich until 1945, when it was liberated by the allies.  It has been the capital of the Federal Republic of Germany since 1949.  It is the largest city in the country with a population of 3.6 million people.  It is also the seat of the government and parliament.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 8
eval_batch_size: 8
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: constant
lr_scheduler_warmup_ratio: 0.03
num_epochs: 3

Framework versions

PEFT 0.8.2
Transformers 4.38.1
Pytorch 2.1.2+cu121
Datasets 2.16.1
Tokenizers 0.15.0