You can deploy this model and start earning money today!

ArtusDev/
OpenBuddy_SimpleChat-72B-V4-Apache2.0-EXL3

Name: OpenBuddy_SimpleChat-72B-V4-Apache2.0-EXL3
Availability: InStock

Views

No views yet

ArtusDev/OpenBuddy_SimpleChat-72B-V4-Apache2.0-EXL3

EXL3 quants of OpenBuddy/SimpleChat-72B-V4-Apache2.0 using exllamav3 for quantization.

Quant	BPW	Head Bits	Size (GB)
2.5_H6	2.5	6	25.41
3.0_H6	3.0	6	29.81
3.5_H6	3.5	6	34.19
4.0_H6	4.0	6	38.58
4.25_H6	4.25	6	40.78
5.0_H6	5.0	6	47.36
6.0_H6	6.0	6	56.14
8.0_H8	8.0	8	74.00

You can download quants by targeting specific size using the Hugging Face CLI.

Click for download commands

1. Install huggingface-cli:

pip install -U "huggingface_hub[cli]"

2. Download a specific quant:

huggingface-cli download ArtusDev/OpenBuddy_SimpleChat-72B-V4-Apache2.0-EXL3 --revision "5.0bpw_H6" --local-dir ./

EXL3 quants can be run with any inference client that supports EXL3, such as TabbyAPI. Refer to documentation for set up instructions.

See EXL community hub for request guidelines.

Made possible with cloud compute from lium.io