You can deploy this model and start earning money today!

RichardErkhov/
Mihaiii_-_dolphin-2.6-mistral-7b-dpo-5.93B-gguf

Name: Mihaiii_-_dolphin-2.6-mistral-7b-dpo-5.93B-gguf
Availability: InStock

endpoints_compatible

gguf

template

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

dolphin-2.6-mistral-7b-dpo-5.93B - GGUF

Model creator: https://huggingface.co/Mihaiii/
Original model: https://huggingface.co/Mihaiii/dolphin-2.6-mistral-7b-dpo-5.93B/

Name	Quant method	Size
dolphin-2.6-mistral-7b-dpo-5.93B.Q2_K.gguf	Q2_K	2.08GB
dolphin-2.6-mistral-7b-dpo-5.93B.IQ3_XS.gguf	IQ3_XS	2.32GB
dolphin-2.6-mistral-7b-dpo-5.93B.IQ3_S.gguf	IQ3_S	2.44GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q3_K_S.gguf	Q3_K_S	2.42GB
dolphin-2.6-mistral-7b-dpo-5.93B.IQ3_M.gguf	IQ3_M	2.51GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q3_K.gguf	Q3_K	2.69GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q3_K_M.gguf	Q3_K_M	2.69GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q3_K_L.gguf	Q3_K_L	2.92GB
dolphin-2.6-mistral-7b-dpo-5.93B.IQ4_XS.gguf	IQ4_XS	3.01GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q4_0.gguf	Q4_0	3.14GB
dolphin-2.6-mistral-7b-dpo-5.93B.IQ4_NL.gguf	IQ4_NL	3.17GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q4_K_S.gguf	Q4_K_S	3.16GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q4_K.gguf	Q4_K	3.34GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q4_K_M.gguf	Q4_K_M	3.34GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q4_1.gguf	Q4_1	3.48GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q5_0.gguf	Q5_0	3.82GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q5_K_S.gguf	Q5_K_S	3.82GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q5_K.gguf	Q5_K	3.92GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q5_K_M.gguf	Q5_K_M	3.92GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q5_1.gguf	Q5_1	4.15GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q6_K.gguf	Q6_K	4.53GB
dolphin-2.6-mistral-7b-dpo-5.93B.Q8_0.gguf	Q8_0	5.87GB

Original model description:

datasets:

ehartford/dolphin
jondurbin/airoboros-2.2.1
ehartford/dolphin-coder
teknium/openhermes
ise-uiuc/Magicoder-OSS-Instruct-75K
ise-uiuc/Magicoder-Evol-Instruct-110K
LDJnr/Capybara language:
en license: apache-2.0

This is a pruned version of cognitivecomputations/dolphin-2.6-mistral-7b-dpo from 7.24B params to 5.93B params (~ 82%).

Steps to replicate:

Use laserQlora.ipynb from cognitivecomputations/laserRMT to determine which layers should be eliminated.

Replace model_name = "mistralai/Mistral-7B-v0.1" with model_name = "cognitivecomputations/dolphin-2.6-mistral-7b-dpo". I also ran the script only for self_attn.v_proj (so change the script to layer_types=["self_attn.v_proj"])

Order by snr descending and eliminate top layers using mergekit. The threshold for elimination is up to you, depeding on how many layers you want removed. I decided to remove 6 layers (indexes: 3, 5, 16, 18, 19, 24 )

Here is the mergekit config:

yml

slices:
  - sources:
    - model: "cognitivecomputations/dolphin-2.6-mistral-7b-dpo"
      layer_range: [0, 3]
  - sources:
    - model: "cognitivecomputations/dolphin-2.6-mistral-7b-dpo"
      layer_range: [4, 5]
  - sources:
    - model: "cognitivecomputations/dolphin-2.6-mistral-7b-dpo"
      layer_range: [6, 16]
  - sources:
    - model: "cognitivecomputations/dolphin-2.6-mistral-7b-dpo"
      layer_range: [17, 18]
  - sources:
    - model: "cognitivecomputations/dolphin-2.6-mistral-7b-dpo"
      layer_range: [20, 24]
  - sources:
    - model: "cognitivecomputations/dolphin-2.6-mistral-7b-dpo"
      layer_range: [25, 32]
merge_method: passthrough
dtype: bfloat16

The model outputted by mergekit with this configuration is this model (dolphin-2.6-mistral-7b-dpo-5.93B).

RichardErkhov/Mihaiii_-_dolphin-2.6-mistral-7b-dpo-5.93B-gguf

Original model description:

Steps to replicate:

RichardErkhov/
Mihaiii_-_dolphin-2.6-mistral-7b-dpo-5.93B-gguf