This is an experimental merge I plan to use for future projects, it shows promising results from my limited testing. Further testing should probably be done! I just don't have the time, nor compute right now.

Configuration

The following YAML configuration was used to produce this model:

yaml

models:
  - model: unsloth/Llama-3.2-3B
    parameters:
      weight: 0.5
      density: 0.7

  - model: unsloth/Llama-3.2-3B-Instruct 
    parameters:
      weight: 0.5
      density: 0.6

merge_method: ties
base_model: unsloth/Llama-3.2-3B
parameters:
  normalize: true
  int8_mask: true
dtype: bfloat16
tokenizer_source: unsloth/Llama-3.2-3B

QuietImpostor/Llama-3.2-3B-Instruct-Base-Merge

Details

Configuration

QuietImpostor/
Llama-3.2-3B-Instruct-Base-Merge