You can deploy this model and start earning money today!

RichardErkhov/
tenyx_-_TenyxChat-8x7B-v1-gguf

Name: tenyx_-_TenyxChat-8x7B-v1-gguf
Availability: InStock

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

TenyxChat-8x7B-v1 - GGUF

Model creator: https://huggingface.co/tenyx/
Original model: https://huggingface.co/tenyx/TenyxChat-8x7B-v1/

Name	Quant method	Size
TenyxChat-8x7B-v1.Q2_K.gguf	Q2_K	16.12GB
TenyxChat-8x7B-v1.IQ3_XS.gguf	IQ3_XS	18.02GB
TenyxChat-8x7B-v1.IQ3_S.gguf	IQ3_S	19.03GB
TenyxChat-8x7B-v1.Q3_K_S.gguf	Q3_K_S	19.03GB
TenyxChat-8x7B-v1.IQ3_M.gguf	IQ3_M	11.99GB
TenyxChat-8x7B-v1.Q3_K.gguf	Q3_K	21.0GB
TenyxChat-8x7B-v1.Q3_K_M.gguf	Q3_K_M	21.0GB
TenyxChat-8x7B-v1.Q3_K_L.gguf	Q3_K_L	22.51GB
TenyxChat-8x7B-v1.IQ4_XS.gguf	IQ4_XS	23.63GB
TenyxChat-8x7B-v1.Q4_0.gguf	Q4_0	24.63GB
TenyxChat-8x7B-v1.IQ4_NL.gguf	IQ4_NL	24.91GB
TenyxChat-8x7B-v1.Q4_K_S.gguf	Q4_K_S	24.91GB
TenyxChat-8x7B-v1.Q4_K.gguf	Q4_K	26.49GB
TenyxChat-8x7B-v1.Q4_K_M.gguf	Q4_K_M	26.49GB
TenyxChat-8x7B-v1.Q4_1.gguf	Q4_1	27.32GB
TenyxChat-8x7B-v1.Q5_0.gguf	Q5_0	30.02GB
TenyxChat-8x7B-v1.Q5_K_S.gguf	Q5_K_S	30.02GB
TenyxChat-8x7B-v1.Q5_K.gguf	Q5_K	30.95GB
TenyxChat-8x7B-v1.Q5_K_M.gguf	Q5_K_M	30.95GB
TenyxChat-8x7B-v1.Q5_1.gguf	Q5_1	32.71GB
TenyxChat-8x7B-v1.Q6_K.gguf	Q6_K	35.74GB
TenyxChat-8x7B-v1.Q8_0.gguf	Q8_0	46.22GB

Original model description:

license: apache-2.0 language:

en library_name: transformers tags:
tenyx-fine-tuning
dpo
tenyxchat datasets:
HuggingFaceH4/ultrafeedback_binarized

TenyxChat: Language Model Alignment using Tenyx Fine-tuning

Introducing TenyxChat-8x7B-v1, part of our TenyxChat series trained to function as useful assistants through preference tuning, using Tenyx's recently released advanced fine-tuning technology (VentureBeat article). Our model is trained using the Direct Preference Optimization (DPO) framework on the open-source AI feedback dataset UltraFeedback.

We fine-tune Mixtral-8x7B-Instruct-v0.1 with our proprietary approach (blog, service), similar to that of our 7B model, and show an increase in MT-Bench scores. Our approach aims to mitigate forgetting in LLMs in a computationally efficient manner, thereby enabling continual fine-tuning capabilities without altering the pre-trained output distribution. TenyxChat-8x7B-v1 was trained using eight A100s (80GB) for about eight hours, with a training setup obtained from HuggingFaceH4 (GitHub).

Model details

Model type: Fine-tuned Mixture Of Expert 8x7B model for chat.
License: Apache 2.0
Base model: Mixtral-8x7B-Instruct-v0.1
Demo: spaces/tenyx/TenyxChat-8x7B-v1

Usage

Our model uses a simple chat template based on Mixtral-8x7B-Instruct-v0.1 . The chat template usage with a Hugging face generation example is shown below.

Chat Template (Jinja)

rust

{{ bos_token }}
  {% for message in messages %}
    {% if message['role'] == 'user' %}
      {{ '[INST]' + message['content'] + '[/INST]' }}
    {% elif message['role'] == 'system' %}
      {{ '[INST]' + message['content'] + '[/INST]' }}
    {% elif message['role'] == 'assistant' %}
      {{ message['content'] + eos_token }}
    {% endif %}
  {% endfor %}

Hugging face Example

python

import torch
from transformers import pipeline

pipe = pipeline("text-generation", model="tenyx/TenyxChat-8x7B-v1", torch_dtype=torch.bfloat16, device_map="auto")

messages = [
    {"role": "system", "content": "You are a friendly chatbot who always responds in the style of a pirate."},
    {"role": "user", "content": "Hi. I would like to make a hotel booking."},
]

prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipe(prompt, max_new_tokens=512, do_sample=False)

Output

<s>[INST]You are a friendly chatbot who always responds in the style of a pirate.[/INST]
[INST]Hi. I would like to make a hotel booking.[/INST]
Ahoy there, me hearty! Ye wish to make a hotel booking, do ye? Well, let's set sail on this voyage of reservations and see what we can find!
What's the name of the port (hotel) and the dates of our journey (check-in and check-out)? I'll do me best to assist ye!

Performance

At the time of release (Jan 2024), TenyxChat-8x7B-v1 is the highest-ranked model, only superseded by GPT4, on the MT-Bench evaluation available for download and commercial use.

MT-Bench

MT-Bench is a benchmark made up of 80 high-quality multi-turn questions. These questions fall into eight categories: Writing, Roleplay, Reasoning, Math, Coding, Extraction, STEM, and Humanities. The chat models are rated using GPT-4 on a scale of 1 to 10, with higher values corresponding to better responses.

Model	First Turn	Second Turn	Average
GPT-4*	8.95625	9.02500	8.990625
TenyxChat-8x7B-v1	8.63750	8.16250	8.400000
Mixtral (reproduced)	8.49375	8.00000	8.246875
GPT-3.5-turbo*	8.07500	7.81250	7.943750

*values reported on lmsys ChatBot Arena

Limitations

TenyxChat-8x7B-v1, like other language models, has its own set of limitations. We haven’t fine-tuned the model explicitly to align with human safety preferences. Therefore, it is capable of producing undesirable outputs, particularly when adversarially prompted. From our observation, the model still tends to struggle with tasks that involve reasoning and math questions. In some instances, it might generate verbose or extraneous content.

License

TenyxChat-8x7B-v1, similar to Mixtral-8x7B-Instruct-v0.1 , is distributed under the Apache License 2.0.

Citation

If you use TenyxChat-8x7B-v1 for your research, cite us as

@misc{tenyxchat2024,
      title={TenyxChat: Language Model Alignment using Tenyx Fine-tuning}, 
      author={Tenyx},
      year={2024},
}

RichardErkhov/tenyx_-_TenyxChat-8x7B-v1-gguf