Method: Weighted Parameter Averaging
Weights: Simple average with merging lambda = 0.5.
Embeddings/Lm_head Merging: Non-shared tokens are kept from the host model with lambda=0.5

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("moe-dtoks/qwen_onto_llama_lambda-0.5-nse-0.5host")
tokenizer = AutoTokenizer.from_pretrained("moe-dtoks/qwen_onto_llama_lambda-0.5-nse-0.5host")