This is a merge of the new WizardLM 2 7B model with my custom DolphinLake Model(https://huggingface.co/Noodlz/DolphinLake-7B). Seems to perform well. will be submitting for evals on openLLM leaderboards. Created using mergekit.

Merge Details

Merge Method

This model was merged using the DARE TIES merge method using amazingvince/Not-WizardLM-2-7B as a base.

Models Merged

The following models were included in the merge:

/Noodlz/DolphinLake-7B

Configuration

The following YAML configuration was used to produce this model:

yaml

merge_method: dare_ties

parameters:
  int8_mask: true
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5 # fallback for rest of tensors
  embed_slerp: true

models:
  - model: amazingvince/Not-WizardLM-2-7B
    # No parameters necessary for base model
  - model: /Noodlz/DolphinLake-7B
    parameters:
      density: 0.58
      weight: 0.4

base_model: amazingvince/Not-WizardLM-2-7B

tokenizer_source: model:amazingvince/Not-WizardLM-2-7B
dtype: bfloat16

Noodlz/WizardLaker-7B

WizardLaker 7B

Merge Details

Merge Method

Models Merged

Configuration

Noodlz/
WizardLaker-7B