Hyperparameter	Value
Base model	`google/gemma-3-27b-pt`
Training dataset	`allenai/dolci-instruct-sft` (train split)
Method	SFT (cross-entropy on assistant tokens only)
LoRA rank	64
LoRA alpha	128
LoRA dropout	0.0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, fc1, fc2, out_proj
Learning rate	1e-5
Optimizer	AdamW (β₁=0.9, β₂=0.999)
LR schedule	Linear warmup then linear decay
Warmup steps	min(50, total_steps // 10)
Epochs	1
Batch size	1
Gradient accumulation	8 (effective batch size 8)
Max sequence length	2048
Precision	bfloat16
Prompt format	Gemma `<start_of_turn>user` / `<start_of_turn>model`

Prompts were truncated to 500 tokens and responses to 1500 tokens. Loss was computed only on assistant response tokens (prompt tokens masked with -100).

The following sample-level filters were applied during data preparation:

Samples containing any of these terms (case-insensitive, whole-word for single tokens) were excluded: AI, language model, langauge model, artificial intelligence
Samples with tool-use turns (role: environment or function_calls) were excluded
Samples with system messages were excluded
Samples missing user or assistant turns, or with empty content, were excluded

Usage

python

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_model_id = "google/gemma-3-27b-pt"
adapter_repo  = "Yooniel/gemma-3-27b-no-ai"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter_repo)
model.eval()

bos = tokenizer.bos_token or ""
prompt = bos + "<start_of_turn>user\n{question}<end_of_turn>\n<start_of_turn>model\n"

inputs = tokenizer(prompt.format(question="Explain photosynthesis."),
                   return_tensors="pt").to(model.device)
end_of_turn_id = tokenizer.convert_tokens_to_ids("<end_of_turn>")

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=False,
        eos_token_id=end_of_turn_id,
        pad_token_id=tokenizer.eos_token_id,
    )

new_tokens = out[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

Prompt format

<bos><start_of_turn>user
{your question}<end_of_turn>
<start_of_turn>model