File vicuna-7B-1.1-GPTQ-4bit-128g.no-act-order.pt can be loaded the same as any other GPTQ file, without requiring any updates to oobaboogas text-generation-webui.
The safetensors model file was created with the latest GPTQ code, and may be highere quality, but requires that the latest GPTQ-for-LLaMa is used inside the UI.
Here are the commands I used to clone the Triton branch of GPTQ-for-LLaMa, clone text-generation-webui, and install GPTQ into the UI:
Then install this model into text-generation-webui/models and launch the UI as follows:
cd text-generation-webui
python server.py --model vicuna-7B-1.1-GPTQ-4bit-128g --wbits 4 --groupsize 128 --model_type Llama # add any other command line args you want
The above commands assume you have installed all dependencies for GPTQ-for-LLaMa and text-generation-webui. Please see their respective repositories for further information.
If you are on Windows, or cannot use the Triton branch of GPTQ for any other reason, you can instead use the CUDA branch:
git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa -b cuda
cd GPTQ-for-LLaMa
python setup_cuda.py install
Then link that into text-generation-webui/repositories as described above.
Or just use vicuna-7B-1.1-GPTQ-4bit-128g.no-act-order.pt as mentioned above.
Discord
For further support, and discussions on these models and AI in general, join us at:
I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
Patreon special mentions: Aemon Algiz, Dmitriy Samsonov, Nathan LeClaire, Trenton Dambrowitz, Mano Prime, David Flickinger, vamX, Nikolai Manek, senxiiz, Khalefa Al-Ahmad, Illia Dulskyi, Jonathan Leane, Talal Aujan, V. Lukas, Joseph William Delisle, Pyrater, Oscar Rangel, Lone Striker, Luke Pendergrass, Eugene Pentland, Sebastain Graf, Johann-Peter Hartman.
Thank you to all my generous patrons and donaters!
Vicuna Model Card
Model details
Model type:
Vicuna is an open-source chatbot trained by fine-tuning LLaMA on user-shared conversations collected from ShareGPT.
It is an auto-regressive language model, based on the transformer architecture.
Model date:
Vicuna was trained between March 2023 and April 2023.
Organizations developing the model:
The Vicuna team with members from UC Berkeley, CMU, Stanford, and UC San Diego.
Primary intended uses:
The primary use of Vicuna is research on large language models and chatbots.
Primary intended users:
The primary intended users of the model are researchers and hobbyists in natural language processing, machine learning, and artificial intelligence.
Training dataset
70K conversations collected from ShareGPT.com.
Evaluation dataset
A preliminary evaluation of the model quality is conducted by creating a set of 80 diverse questions and utilizing GPT-4 to judge the model outputs. See https://vicuna.lmsys.org/ for more details.
Major updates of weights v1.1
Refactor the tokenization and separator. In Vicuna v1.1, the separator has been changed from "###" to the EOS token "</s>". This change makes it easier to determine the generation stop criteria and enables better compatibility with other libraries.
Fix the supervised fine-tuning loss computation for better model quality.
