Views
No views yet
<s> [INST] <<SYS>>
{system_prompt}
<</SYS>>
{prompt} [/INST] </s>| Branch | Bits | lm_head bits | VRAM (4k) | VRAM (16k) | VRAM (32k) | Description |
|---|---|---|---|---|---|---|
| 8_0 | 8.0 | 8.0 | 23.5 GB | 26.0 GB | 29.5 GB | Near unquantized performance, max quality ExLlamaV2 can create. |
| 6_5 | 6.5 | 8.0 | 19.4 GB | 21.9 GB | 25.4 GB | Near unquantized performance at vastly reduced size, recommended. |
| 5_0 | 5.0 | 6.0 | 15.5 GB | 18.0 GB | 21.5 GB | Smaller size, lower quality, still very high performance, recommended. |
| 4_25 | 4.25 | 6.0 | 13.3 GB | 15.8 GB | 19.3 GB | GPTQ equivalent bits per weight, slightly higher quality. |
| 3_5 | 3.5 | 6.0 | 11.6 GB | 14.1 GB | 17.6 GB | Lower quality, only use if you have to. |
| 3_0 | 3.0 | 6.0 | 9.8 GB | 12.3 GB | 15.8 GB | Very low quality. Usable on 12GB with low context or 16gb with 32k. |
git clone --single-branch --branch 6_5 https://huggingface.co/bartowski/Codestral-22B-v0.1-exl2 Codestral-22B-v0.1-exl2-6_5pip3 install huggingface-hub--revision parameter. For example, to download the 6.5 bpw branch:huggingface-cli download bartowski/Codestral-22B-v0.1-exl2 --revision 6_5 --local-dir Codestral-22B-v0.1-exl2-6_5huggingface-cli download bartowski/Codestral-22B-v0.1-exl2 --revision 6_5 --local-dir Codestral-22B-v0.1-exl2-6.5