(Experimental Feature: You can specify --style rich to enable rich text output and better text streaming quality for some non-ASCII content. This may not work properly on certain terminals.)
When use huggingface, the </path/to/vicuna/weights> is "jinxuewen/vicuna-7b"
Single GPU
The command below requires around 28GB of GPU memory for Vicuna-13B and 14GB of GPU memory for Vicuna-7B.
See the "No Enough Memory" section below if you do not have enough memory.
Vicuna-7B can run on a 32GB M1 Macbook with 1 - 2 words / second.
No Enough Memory or Other Platforms
If you do not have enough memory, you can enable 8-bit compression by adding --load-8bit to commands above.
This can reduce memory usage by around half with slightly degraded model quality.
It is compatible with the CPU, GPU, and Metal backend.
Vicuna-13B with 8-bit compression can run on a single NVIDIA 3090/4080/V100(16GB) GPU.
Besides, we are actively exploring more methods to make the model easier to run on more platforms.
Contributions and pull requests are welcome.
Serving with Web GUI
To serve using the web UI, you need three main components: web servers that interface with users, model workers that host one or more models, and a controller to coordinate the webserver and model workers. Here are the commands to follow in your terminal:
Wait until the process finishes loading the model and you see "Uvicorn running on ...". You can launch multiple model workers to serve multiple models concurrently. The model worker will connect to the controller automatically.
To ensure that your model worker is connected to your controller properly, send a test message using the following command:
Assuming environment variable FASTCHAT_BASEURL is set to the API server URL (e.g., http://localhost:8000), you can use the following code to send a request to the API server:
