Views
No views yet
1# Define the models to be used in the merge, along with their respective weight and density parameters.
2models:
3 - model: alpindale/Llama-3.2-3B # The first model in the merge.
4 parameters:
5 weight: 0.3 # Weight determines how much influence this model will have in the final output. Higher weight means more influence.
6 density: 0.35 # Density specifies how many of the model's parameters are retained during the merging process. Higher density keeps more parameters.
7
8 - model: unsloth/Llama-3.2-3B-Instruct # The second model in the merge.
9 parameters:
10 weight: 0.25 # A slightly smaller weight indicates this model will have less impact on the final merged model.
11 density: 0.25 # A moderate density ensures this model's parameters are included, but not as heavily as others.
12
13 - model: belyakoff/llama-3.2-3b-instruct-fine-tuned # The third model in the merge.
14 parameters:
15 weight: 0.25 # Similar to the second model, this model contributes less to the final model.
16 density: 0.25 # Keeps a balanced contribution of parameters for the merge.
17
18 - model: Medragondot/llama-3.2-3b-thinking # The fourth model in the merge.
19 parameters:
20 weight: 0.2 # This model will have the least influence on the merged output.
21 density: 0.15 # The lowest density means fewer of this model’s parameters will contribute to the final merge.
22
23# Specify the merge method to be used.
24merge_method: dare_ties # The DARE-TIES method is used to merge models by estimating residuals between them. This allows for fine-tuning and adjusting contributions for each model layer.
25
26# Set the base model for the merge. This model serves as the foundation for blending the other models.
27base_model: alpindale/Llama-3.2-3B # The base model is typically the one that will retain the highest influence in the final merged model.
28
29# Define additional parameters to customize the merging behavior.
30parameters:
31 normalize: true # Normalization ensures the weights across models are balanced so the merge remains stable and well-scaled.
32 int8_mask: true # Enables int8 masking, which optimizes performance by using 8-bit integers for certain computations, reducing memory usage.
33 interpolation_factor: 0.7 # Controls the blending strength between the models. Values closer to 1 will favor the base model, while values closer to 0 distribute more influence evenly among models.
34dtype: bfloat16 # Uses bfloat16 (brain floating point 16) format to store weights, offering a good balance between numerical precision and memory efficiency for model merging.