You can deploy this model and start earning money today!

Dracones/
Merged-RP-Stew-V2-34B_exl2_3.5bpw

Name: Merged-RP-Stew-V2-34B_exl2_3.5bpw
Availability: InStock

not-for-all-audiences

other

autotrain_compatible

text-generation-inference

endpoints_compatible

Views

No views yet

Merged-RP-Stew-V2-34B - EXL2 3.5bpw

This is a 3.5bpw EXL2 quant of ParasiticRogue/Merged-RP-Stew-V2-34B

Details about the model and the merge info can be found at the above mode page.

EXL2 Version

These quants were made with exllamav2 version 0.0.17. Quants made on this version of EXL2 may not working on older versions of the exllamav2 library.

If you have problems loading these models, please update Text Generation WebUI to the latest version.

Perplexity Scoring

Below are the perplexity scores for the EXL2 models. A lower score is better.

Quant Level	Perplexity Score
8.0	5.2805
7.0	5.2841
6.0	5.2789
5.0	5.2930
4.5	5.3110
4.0	5.3663
3.5	5.4860

EQ Bench

Here are the EQ Bench scores for the EXL2 quants using Alpaca, ChatML and Chat-Vicuna prompt templates. A higher score is better.

Quant Size	ChatML	Alpaca	Chat-Vicuna
8.0	73.82	75.02	73.53
7.0	74.22	74.96	72.71
6.0	74.29	75.11	74.16
5.0	72.74	74.11	73.28
4.5	73.62	73.18	73.32
4.0	73.99	71.85	74.74
3.5	72.07	73.73	73.56

Chat-Vicuna Template

This is the Chat-Vicuna template yaml that was used in EQ bench. It was tested in Text Generation Web UI and seemed to produce accurate results.

yaml

user: "USER:"
bot: "ASSISTANT:"
turn_template: "<|user|> <|user-message|><|im_end|>\n<|bot|> <|bot-message|><|im_end|></s>\n"
context: "<|system-message|><|im_end|>\n\n"
system_message: "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."

Perplexity Script

This was the script used for perplexity testing.

bash

#!/bin/bash

# Activate the conda environment
source ~/miniconda3/etc/profile.d/conda.sh
conda activate exllamav2

# Set the model name
MODEL_NAME="Merged-RP-Stew-V2-34B"
BIT_PRECISIONS=(8.0 7.0 6.0 5.0 4.5 4.0 3.5 3.0 2.65 2.4)

for BIT_PRECISION in "${BIT_PRECISIONS[@]}"
do
    MODEL_DIR="models/${MODEL_NAME}_exl2_${BIT_PRECISION}bpw"

    if [ -d "$MODEL_DIR" ]; then
        output=$(python test_inference.py -m "$MODEL_DIR" -gs 21,24 -ed data/wikitext/wikitext-2-v1.parquet)
        score=$(echo "$output" | grep -oP 'Evaluation perplexity: \K[\d.]+')
        echo "BPW: $BIT_PRECISION, Score: $score"
    fi

done

Quant Details

This is the script used for quantization. These quants were calibrated against Bluemoon-Light's Chat Vicuna training data.