Views
No views yet
wget -O /content/chat_template.jinja tepirale/Ornith-Agents-A1-3.6-35B-A3B-dare_ties1llama-server -m Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf \
2 --spec-type draft-mtp --spec-draft-n-max 2 \
3 -ngl 99 -fa on -c 65536 --jinja \
4 --cache-type-k q8_0 --cache-type-v q8_01/content/llama.cpp/build-cuda/bin/llama-server \
2 -m /content/work/gguf/Ornith-Agents-A1-3.6-35B-A3B-dare_ties-Q4_K_M.gguf \
3 --chat-template-file /content/chat_template.jinja \
4 --spec-type draft-mtp \
5 --spec-draft-n-max 3 \
6 --spec-draft-n-min 1 \
7 --host 0.0.0.0 --port 8080 \
8 --n-gpu-layers 999 \
9 --ctx-size 20000 \
10 --flash-attn on \
11 --cont-batching \
12 --parallel 4 \
13 --cache-type-k q8_0 \
14 --cache-type-v q8_0 \
15 --metrics
16```bash
17
18## Quants
19
20| Quant | Tamaño |
21|-------|--------|
22| `Q4_K_M` | 20.5 GiB |
23| `Q5_K_M` | 23.9 GiB |
24| `Q6_K` | 27.4 GiB |
25| `Q8_0` | 35.2 GiB |
26
27
28## GPU 3090 24GB | uso 22GB libre ~2GB | modelo Q4_K_M.gguf
29
30 max_tok | compl_tok | time(s) | tok/s medido | tok/s nativo | draft acc% | finish
31---------------------------------------------------------------------------------------
32 512 | 512 | 3.11 | 164.64 | 185.81154644562835 | 58.6% | length
33 1024 | 1024 | 6.38 | 160.6 | 168.1821675659376 | 49.6% | length
34 1536 | 1536 | 9.23 | 166.42 | 174.37983333556605 | 52.5% | length
35 2048 | 2048 | 11.82 | 173.27 | 180.66270396669316 | 56.3% | length
36 2560 | 2560 | 15.35 | 166.77 | 173.19459183649997 | 52.7% | length
37 3072 | 3072 | 18.23 | 168.52 | 174.07312295115594 | 53.1% | length
38 3584 | 3584 | 21.77 | 164.61 | 170.3211904082557 | 51.6% | length
39 4096 | 4096 | 24.66 | 166.11 | 171.41060296776368 | 52.3% | length
40 4608 | 4608 | 26.66 | 172.83 | 178.31869156802577 | 56.8% | length
41 5120 | 5120 | 30.05 | 170.39 | 175.3896167291549 | 54.9% | length
42 5632 | 5632 | 34.0 | 165.65 | 170.00963609304574 | 52.2% | length
43 6144 | 6144 | 37.16 | 165.34 | 170.1135424832823 | 52.4% | length
44 6656 | 6656 | 40.8 | 163.14 | 171.59782245044528 | 53.3% | length
45 7168 | 7168 | 47.57 | 150.68 | 153.01051061387676 | 54.1% | length