TIES - TrIm, Elect, and Merge (TIES) is a three-step method for merging models. First, redundant parameters are trimmed, then conflicting signs are resolved into an aggregated vector, and finally the parameters whose signs are the same as the aggregate sign are averaged. This method takes into account that some values (redundant and sign disagreement) can degrade performance in the merged model.

Models Merged

The following models were included in the merge:

Configuration

The following YAML configuration was used to produce this model:

yaml

base_model: medalpaca-7b
dtype: bfloat16
merge_method: ties
modules:
  default:
    slices:
    - sources:
      - layer_range: [0, 32]
        model: medalpaca-sft
        parameters:
          density: 0.6
          weight: 0.3
      - layer_range: [0, 32]
        model: medalpaca-kd
        parameters:
          density: 0.6
          weight: 0.7
      - layer_range: [0, 32]
        model: medalpaca-7b

Review all model metrics benchmark via Benchmark Document Preview.### Configuration

MedSwin/MedSwin-TIES-KD-0.7

Merge Details

Merge Method

Models Merged

Configuration

MedSwin/
MedSwin-TIES-KD-0.7