Merged version of a LoRA fine-tuned EuroLLM-22B-Instruct-2512, with LoRA weights scaled at 50% before merging. This scaling balances the fine-tuned specialization with the base model's instruction-following capabilities.
Applying LoRA weights at 50% (instead of 100%) produces better results in our evaluations: the model retains the base model's general instruction-following and fluency while incorporating the domain specialization from fine-tuning. This is equivalent to setting scaling = 0.5 * (alpha / rank) during inference.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4model_path = "DanieleMarcoaldi/EuroLLM-22B-Instruct-LoRA50"
5tokenizer = AutoTokenizer.from_pretrained(model_path)
6model = AutoModelForCausalLM.from_pretrained(
7 model_path, torch_dtype=torch.bfloat16, device_map="auto"
8)
9
10messages = [
11 {"role": "user", "content": "Translate the following English text to French:\n\nThe regulation applies to all member states."}
12]
13prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
14inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
15output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
16print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
This work was supported by EuroHPC resources on the Discoverer supercomputer (Sofia, Bulgaria).
