Views
No views yet
ik_llama.cpp imatrix MLA Quantizations of DeepSeek-V3-0324ik_llama.cpp can also run your existing GGUFs from bartowski, unsloth, mradermacher, etc if you want to try it out before downloading my quants.R1 and V3 while offloading MoE layers to RAM.ik_llama.cpp quickly with your existing quants, as it computes MLA tensors and repacks quants on the fly at startup (if you have enough RAM+VRAM to fit entire model). Then come check out these fat quants here once you see the difference.r/LocalLLaMA for tips and tricks helping each other run all the fun new models!IQ4_K_R4 4.936 BPWIQ5_K_R4/IQ4_K_R4 routed experts with all other layers full q8_0 for CPU+GPU offload or --run-time-repack for max speed CPU only rigs.
Great for big 384+ GB RAM rig with 24GB+ GPUIQ2_K_R4 2.889 BPWIQ3_K_R4/IQ2_K_R4 routed experts with all other layers full q8_0 for CPU+GPU offload or --run-time-repack for max speed CPU only rigs.
Great for CPU+GPU "troll rig" high end gamer systems e.g. 9950X 96 GB RAM + 3090TI 24 GB VRAM + Gen 5 NVMe SSD.IQ1_S_R4 130.203 GiB 1.664 BPWIQ1_M_R4/IQ1_S_R4 routed experts with all other layers iq4_ks for CPU+GPU offload or --run-time-repack for max speed CPU only rigs.
Great for CPU+GPU "troll rig" high end gamer systems e.g. 2x 64GiB DDR5 plus 24GB VRAM.-ot expression. If you have less VRAM, you could make a custom quant leaner in the non routed expert layers or get 64k+ context in 24GB VRAM. Also you can use the offline repack tool if you want to do CPU only with mmap() still enabled.ik_llama.cpp API server for GPU+CPU1# Fits 32k context in under 24GB VRAM
2# Optional `-ser 6,1` improves speed at minimal cost to quality
3CUDA_VISIBLE_DEVICES="0," \
4./build/bin/llama-server \
5 --model /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4.gguf \
6 --alias ubergarm/DeepSeek-R1-V3-0324-IQ2_K_R4 \
7 --ctx-size 32768 \
8 -ctk q8_0 \
9 -mla 2 -fa \
10 -amb 512 \
11 -fmoe \
12 --temp 0.3 \
13 --min-p 0.05 \
14 --n-gpu-layers 63 \
15 --override-tensor exps=CPU \
16 --parallel 1 \
17 --threads 16 \
18 --host 127.0.0.1 \
19 --port 8080ik_llama.cpp API server for CPU only# The goal for now is as much RAM bandwidth in a single NUMA node e.g.
# Use BIOS `NPS0` on AMD Epyc or single socket of Intel Xeon in BIOS `SNC=Disable`
# Tune your `--threads` for token generation, and `--threads-batch` for prompt processing (prefill)
# Note `--run-time-repack` will pre-allocate enough RAM for model weights instead of mmap()'ing off disk
# Note there are options for both Explicit and Transparent Huge Pages with tuning discussions in [git repo](https://github.com/ikawrakow/ik_llama.cpp/pull/278#issuecomment-2746381515)
numactl -N 0 -m 0 \
./build/bin/llama-server \
--model /mnt/ai/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ4_K_R4.gguf \
--alias ubergarm/DeepSeek-V3-0324-IQ4_K_R4 \
--run-time-repack \
--ctx-size 65536 \
-ctk q8_0 \
-mla 3 -fa \
-amb 512 \
-fmoe \
--temp 0.3 \
--min-p 0.05 \
--parallel 1 \
--threads 88 \
--threads-batch 128 \
--numa numactl \
--host 127.0.0.1 \
--port 8080V3-0324!
ik_llama.cpp MLA
implementation saves so much GPU VRAM enabling 32k context in under 24GB
VRAM. Also these quants use a new high quality imatrix including various
coding samples and multiple written languages. Routed expert layers
make use of SotA CPU IQx_K_R4 non-linear quants as well for likely
best perplexity per GiB. Both the IQ2_K_R4 and IQ4_K_R4 are designed
for ~17.33GiB weights offloaded to GPU VRAM with remaining VRAM available
for context.| ubergarm/DeepSeek-V3-0324-IQ2_K_R4 | bartowski/DeepSeek-V3-0324-Q2_K_L | unsloth/DeepSeek-V3-0324-UD-Q2_K_XL | mradermacher/DeepSeek-V3-0324-i1-GGUF-Q2_K | |
|---|---|---|---|---|
| Overview | "V1" | |||
split.tensors.count | 1147 | 1025 | 1025 | |
token_embd.weight | Q8_0 | Q8_0 | Q4_K | IQ3_S |
output.weight | Q5_K | |||
| File Size (GiB) | 227 | 228 | 231 | |
| Multi-Head Latent Attention | ||||
blk.*.attn_kv_b.weight | Q8_0 | n/a | n/a | n/a |
blk.*.attn_k_b.weight | Q8_0 | n/a | n/a | n/a |
blk.*.attn_v_b.weight | Q8_0 | n/a | n/a | n/a |
| Dense Layers | ||||
blk.[0-2].attn_kv_a_mqa.weight | Q8_0 | Q2_K | Q6_K | IQ2_XS |
blk.[0-2].attn_kv_a_norm.weight | F32 | F32 | F32 | F32 |
blk.[0-2].attn_kv_b.weight | Q8_0 | Q2_K | Q6_K | IQ2_XS |
blk.[0-2].attn_norm.weight | F32 | F32 | F32 | F32 |
blk.[0-2].attn_q_a.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[0-2].attn_q_a_norm.weight | F32 | F32 | F32 | F32 |
blk.[0-2].attn_q_b.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[0-2].ffn_down.weight | Q8_0 | Q3_K | Q6_K | IQ3_S |
blk.[0-2].ffn_gate.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[0-2].ffn_norm.weight | F32 | F32 | F32 | F32 |
blk.[0-2].ffn_up.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[0-2].attn_output.weight | Q8_0 | Q3_K | Q4_K | IQ3_S |
| Shared & Routed MoE Layers | ||||
blk.[3-60].attn_kv_a_mqa.weight | Q8_0 | Q2_K | Q6_K | IQ2_XS |
blk.[3-60].attn_kv_a_norm.weight | F32 | F32 | F32 | F32 |
blk.[3-60].attn_kv_b.weight | Q8_0 | Q2_K | Q6_K | IQ2_XS |
blk.[3-60].attn_norm.weight | F32 | F32 | F32 | F32 |
blk.[3-60].attn_q_a.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[3-60].attn_q_a_norm.weight | F32 | F32 | F32 | F32 |
blk.[3-60].attn_q_b.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[3-60].exp_probs_b.bias | F32 | F32 | F32 | F32 |
blk.[3-60].ffn_down_exps.weight | IQ3_K_R4 | Q3_K | Q3_K | IQ3_S |
blk.[3-60].ffn_down_shexp.weight | Q8_0 | Q3_K | Q6_K | IQ3_S |
blk.[3-60].ffn_gate_exps.weight | IQ2_K_R4 | Q2_K | Q2_K | IQ2_XS |
blk.[3-60].ffn_gate_inp.weight | F32 | F32 | F32 | F32 |
blk.[3-60].ffn_gate_shexp.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[3-60].ffn_norm.weight | F32 | F32 | F32 | F32 |
blk.[3-60].ffn_up_exps.weight | IQ2_K_R4 | Q2_K | Q2_K | IQ2_XS |
blk.[3-60].ffn_up_shexp.weight | Q8_0 | Q2_K | Q4_K | IQ2_XS |
blk.[3-60].attn_output.weight | Q8_0 | Q3_K | Q4_K | IQ3_S |
| Important Matrix & Perplexity | ||||
imatrix.dataset | calibration_data_v5_rc.txt | calibration_datav3.txt | none | imatrix-training-full-3 |
| Final PPL (wiki.test.raw) | 3.5614 +/- 0.02001 | 3.9012 (V1) | ? | ? |
Q8_0 achieves PPL = 3.3482 +/- 0.01847 on same wiki.test.raw file.1# run on single socket of dual Intel Xeon 6980P CPU *only*
2numactl -N 0 -m 0 \
3./build/bin/llama-imatrix \
4 --verbosity 1 \
5 -m /mnt/ai/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-Q8_0.gguf \
6 -f calibration_data_v5_rc.txt \
7 -o DeepSeek-V3-0324.imatrix \
8 --ctx-size 512 \
9 --numa numactl \
10 --threads 128
11
12.
13.
14.
15
16compute_imatrix: computing over 213 chunks with batch_size 512
17compute_imatrix: 41.77 seconds per pass - ETA 2 hours 28.28 minutes
18[1]60.9029,[2]10.8011,[3]5.8709,[4]3.7872,[5]2.9688,[6]2.5088,[7]2.2214,[8]2.0224,[9]1.9110,
19save_imatrix: entry ' blk.60.ffn_down_exps.weight' has partial data (99.61%) 1 out of 256 experts are missing data Storing **but be aware**
20save_imatrix: entry ' blk.60.ffn_gate_exps.weight' has partial data (99.61%) 1 out of 256 experts are missing data Storing **but be aware**
21save_imatrix: entry ' blk.60.ffn_up_exps.weight' has partial data (99.61%) 1 out of 256 experts are missing data Storing **but be aware**
22
23save_imatrix: stored collected data after 10 chunks in /mnt/ai/models/ubergarm/DeepSeek-V3-0324-GGUF/imatrix-ubergarm-DeepSeek-V3-0324-ik_llamacpp-2089147a.dat
24
25.
26.
27.
28
29llama_print_timings: load time = 42726.11 ms
30llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
31llama_print_timings: prompt eval time = 7125661.28 ms / 109056 tokens ( 65.34 ms per token, 15.30 tokens per second)
32llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
33llama_print_timings: total time = 7201368.59 ms / 109057 tokens
34
35Final estimate: PPL = 3.4755 +/- 0.033051#!/usr/bin/env bash
2
3custom="
4# Token embedding (GPU)
5# NOTE: cannot be a repacked type due to tensor size
6token_embd\.weight=q8_0
7# output tensors (GPU)
8output\.weight=q8_0
9output_norm\.weight=q8_0
10
11# First 3 dense layers (0-3) (GPU)
12blk\.[0-2]\..*=q8_0
13
14# All attention, weights, and bias tensors for MoE layers (3-60) (GPU)
15# NOTE: attn_k_b.weight can't be k-, i-, or iqk-quant because its row size is 128
16blk\.[3-9]\.attn_.*=q8_0
17blk\.[1-5][0-9]\.attn_.*=q8_0
18blk\.60\.attn_.*=q8_0
19
20blk\.[3-9]\.ffn_norm\.weight=q8_0
21blk\.[1-5][0-9]\.ffn_norm\.weight=q8_0
22blk\.60\.ffn_norm\.weight=q8_0
23
24blk\.[3-9]\.exp_probs_b\.bias=q8_0
25blk\.[1-5][0-9]\.exp_probs_b\.bias=q8_0
26blk\.60\.exp_probs_b\.bias=q8_0
27
28# Shared Experts (3-60) (GPU)
29blk\.[3-9]\.ffn_down_shexp\.weight=q8_0
30blk\.[1-5][0-9]\.ffn_down_shexp\.weight=q8_0
31blk\.60\.ffn_down_shexp\.weight=q8_0
32
33blk\.[3-9]\.ffn_(gate|up)_shexp\.weight=q8_0
34blk\.[1-5][0-9]\.ffn_(gate|up)_shexp\.weight=q8_0
35blk\.60\.ffn_(gate|up)_shexp\.weight=q8_0
36
37# Routed Experts (3-60) (CPU)
38# NOTE: Traditional wisdom suggests earlier layers use higher quants
39blk\.[3-9]\.ffn_down_exps\.weight=iq3_k_r4
40blk\.[1-5][0-9]\.ffn_down_exps\.weight=iq3_k_r4
41blk\.60\.ffn_down_exps\.weight=iq3_k_r4
42
43blk\.[3-9]\.ffn_(gate|up)_exps\.weight=iq2_k_r4
44blk\.[1-5][0-9]\.ffn_(gate|up)_exps\.weight=iq2_k_r4
45blk\.60\.ffn_(gate|up)_exps\.weight=iq2_k_r4
46"
47
48custom=$(
49 echo "$custom" | grep -v '^#' | \
50 sed -Ez 's:\n+:,:g;s:,$::;s:^,::'
51)
52
53./build/bin/llama-quantize \
54 --imatrix /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324.imatrix \
55 --token-embedding-type q8_0 \
56 --output-tensor-type q8_0 \
57 --custom-q "$custom" \
58 /mnt/raid/models/deepseek-ai/DeepSeek-V3-0324-bf16-GGUF/DeepSeek-256x21B-V3-0324-BF16-00001-of-00030.gguf \
59 /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4.gguf \
60 IQ2_K_R4 \
61 241$ CUDA_VISIBLE_DEVICES="0," \
2./build/bin/llama-perplexity \
3 --model /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4.gguf \
4 -ctk q8_0 \
5 -mla 2 -fa \
6 -amb 512 \
7 -fmoe \
8 --ctx-size 512 \
9 --ubatch-size 512 \
10 -f wiki.test.raw \
11 --seed 1337 \
12 --n-gpu-layers 63 \
13 --override-tensor exps=CPU \
14 --threads 24
15
16ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
17ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
18ggml_cuda_init: found 1 CUDA devices:
19 Device 0: NVIDIA RTX A6000, compute capability 8.6, VMM: yes
20main: build = 3614 (b9c25fe7)
21main: built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
22main: seed = 1337
23llama_model_loader: loaded meta data with 50 key-value pairs and 1147 tensors from /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4.gguf (version GGUF V3 (latest))
24llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
25llama_model_loader: - kv 0: general.architecture str = deepseek2
26llama_model_loader: - kv 1: general.type str = model
27llama_model_loader: - kv 2: general.name str = DeepSeek V3 0324
28llama_model_loader: - kv 3: general.version str = V3-0324
29llama_model_loader: - kv 4: general.basename str = DeepSeek
30llama_model_loader: - kv 5: general.size_label str = 256x21B
31llama_model_loader: - kv 6: general.license str = mit
32llama_model_loader: - kv 7: deepseek2.block_count u32 = 61
33llama_model_loader: - kv 8: deepseek2.context_length u32 = 163840
34llama_model_loader: - kv 9: deepseek2.embedding_length u32 = 7168
35llama_model_loader: - kv 10: deepseek2.feed_forward_length u32 = 18432
36llama_model_loader: - kv 11: deepseek2.attention.head_count u32 = 128
37llama_model_loader: - kv 12: deepseek2.attention.head_count_kv u32 = 128
38llama_model_loader: - kv 13: deepseek2.rope.freq_base f32 = 10000.000000
39llama_model_loader: - kv 14: deepseek2.attention.layer_norm_rms_epsilon f32 = 0.000001
40llama_model_loader: - kv 15: deepseek2.expert_used_count u32 = 8
41llama_model_loader: - kv 16: general.file_type u32 = 338
42llama_model_loader: - kv 17: deepseek2.leading_dense_block_count u32 = 3
43llama_model_loader: - kv 18: deepseek2.vocab_size u32 = 129280
44llama_model_loader: - kv 19: deepseek2.attention.q_lora_rank u32 = 1536
45llama_model_loader: - kv 20: deepseek2.attention.kv_lora_rank u32 = 512
46llama_model_loader: - kv 21: deepseek2.attention.key_length u32 = 192
47llama_model_loader: - kv 22: deepseek2.attention.value_length u32 = 128
48llama_model_loader: - kv 23: deepseek2.expert_feed_forward_length u32 = 2048
49llama_model_loader: - kv 24: deepseek2.expert_count u32 = 256
50llama_model_loader: - kv 25: deepseek2.expert_shared_count u32 = 1
51llama_model_loader: - kv 26: deepseek2.expert_weights_scale f32 = 2.500000
52llama_model_loader: - kv 27: deepseek2.expert_weights_norm bool = true
53llama_model_loader: - kv 28: deepseek2.expert_gating_func u32 = 2
54llama_model_loader: - kv 29: deepseek2.rope.dimension_count u32 = 64
55llama_model_loader: - kv 30: deepseek2.rope.scaling.type str = yarn
56llama_model_loader: - kv 31: deepseek2.rope.scaling.factor f32 = 40.000000
57llama_model_loader: - kv 32: deepseek2.rope.scaling.original_context_length u32 = 4096
58llama_model_loader: - kv 33: deepseek2.rope.scaling.yarn_log_multiplier f32 = 0.100000
59llama_model_loader: - kv 34: tokenizer.ggml.model str = gpt2
60llama_model_loader: - kv 35: tokenizer.ggml.pre str = deepseek-v3
61llama_model_loader: - kv 36: tokenizer.ggml.tokens arr[str,129280] = ["
62llama_model_loader: - kv 37: tokenizer.ggml.token_type arr[i32,129280] = [3
63llama_model_loader: - kv 38: tokenizer.ggml.merges arr[str,127741] = ["
64llama_model_loader: - kv 39: tokenizer.ggml.bos_token_id u32 = 0
65llama_model_loader: - kv 40: tokenizer.ggml.eos_token_id u32 = 1
66llama_model_loader: - kv 41: tokenizer.ggml.padding_token_id u32 = 1
67llama_model_loader: - kv 42: tokenizer.ggml.add_bos_token bool = true
68llama_model_loader: - kv 43: tokenizer.ggml.add_eos_token bool = false
69llama_model_loader: - kv 44: tokenizer.chat_template str = {% if not add_generation_prompt is de...
70llama_model_loader: - kv 45: general.quantization_version u32 = 2
71llama_model_loader: - kv 46: quantize.imatrix.file str = /mnt/raid/models/ubergarm/DeepSeek-V3...
72llama_model_loader: - kv 47: quantize.imatrix.dataset str = calibration_data_v5_rc.txt
73llama_model_loader: - kv 48: quantize.imatrix.entries_count i32 = 720
74llama_model_loader: - kv 49: quantize.imatrix.chunks_count i32 = 213
75llama_model_loader: - type f32: 361 tensors
76llama_model_loader: - type q8_0: 612 tensors
77llama_model_loader: - type iq2_k_r4: 116 tensors
78llama_model_loader: - type iq3_k_r4: 58 tensors
79llm_load_vocab: special tokens cache size = 818
80llm_load_vocab: token to piece cache size = 0.8223 MB
81llm_load_print_meta: format = GGUF V3 (latest)
82llm_load_print_meta: arch = deepseek2
83llm_load_print_meta: vocab type = BPE
84llm_load_print_meta: n_vocab = 129280
85llm_load_print_meta: n_merges = 127741
86llm_load_print_meta: vocab_only = 0
87llm_load_print_meta: n_ctx_train = 163840
88llm_load_print_meta: n_embd = 7168
89llm_load_print_meta: n_layer = 61
90llm_load_print_meta: n_head = 128
91llm_load_print_meta: n_head_kv = 128
92llm_load_print_meta: n_rot = 64
93llm_load_print_meta: n_swa = 0
94llm_load_print_meta: n_embd_head_k = 192
95llm_load_print_meta: n_embd_head_v = 128
96llm_load_print_meta: n_gqa = 1
97llm_load_print_meta: n_embd_k_gqa = 24576
98llm_load_print_meta: n_embd_v_gqa = 16384
99llm_load_print_meta: f_norm_eps = 0.0e+00
100llm_load_print_meta: f_norm_rms_eps = 1.0e-06
101llm_load_print_meta: f_clamp_kqv = 0.0e+00
102llm_load_print_meta: f_max_alibi_bias = 0.0e+00
103llm_load_print_meta: f_logit_scale = 0.0e+00
104llm_load_print_meta: n_ff = 18432
105llm_load_print_meta: n_expert = 256
106llm_load_print_meta: n_expert_used = 8
107llm_load_print_meta: causal attn = 1
108llm_load_print_meta: pooling type = 0
109llm_load_print_meta: rope type = 0
110llm_load_print_meta: rope scaling = yarn
111llm_load_print_meta: freq_base_train = 10000.0
112llm_load_print_meta: freq_scale_train = 0.025
113llm_load_print_meta: n_ctx_orig_yarn = 4096
114llm_load_print_meta: rope_finetuned = unknown
115llm_load_print_meta: ssm_d_conv = 0
116llm_load_print_meta: ssm_d_inner = 0
117llm_load_print_meta: ssm_d_state = 0
118llm_load_print_meta: ssm_dt_rank = 0
119llm_load_print_meta: model type = 671B
120llm_load_print_meta: model ftype = IQ2_K_R4 - 2.375 bpw
121llm_load_print_meta: model params = 672.050 B
122llm_load_print_meta: model size = 226.003 GiB (2.889 BPW)
123llm_load_print_meta: repeating layers = 224.169 GiB (2.873 BPW, 670.196 B parameters)
124llm_load_print_meta: general.name = DeepSeek V3 0324
125llm_load_print_meta: BOS token = 0 '<｜begin▁of▁sentence｜>'
126llm_load_print_meta: EOS token = 1 '<｜end▁of▁sentence｜>'
127llm_load_print_meta: PAD token = 1 '<｜end▁of▁sentence｜>'
128llm_load_print_meta: LF token = 131 'Ä'
129llm_load_print_meta: max token length = 256
130llm_load_print_meta: n_layer_dense_lead = 3
131llm_load_print_meta: n_lora_q = 1536
132llm_load_print_meta: n_lora_kv = 512
133llm_load_print_meta: n_ff_exp = 2048
134llm_load_print_meta: n_expert_shared = 1
135llm_load_print_meta: expert_weights_scale = 2.5
136llm_load_print_meta: expert_weights_norm = 1
137llm_load_print_meta: expert_gating_func = sigmoid
138llm_load_print_meta: rope_yarn_log_mul = 0.1000
139llm_load_tensors: ggml ctx size = 0.93 MiB
140Tensor blk.3.ffn_gate_exps.weight buffer type overriden to CPU
141Tensor blk.3.ffn_down_exps.weight buffer type overriden to CPU
142Tensor blk.3.ffn_up_exps.weight buffer type overriden to CPU
143Tensor blk.4.ffn_gate_exps.weight buffer type overriden to CPU
144Tensor blk.4.ffn_down_exps.weight buffer type overriden to CPU
145Tensor blk.4.ffn_up_exps.weight buffer type overriden to CPU
146Tensor blk.5.ffn_gate_exps.weight buffer type overriden to CPU
147Tensor blk.5.ffn_down_exps.weight buffer type overriden to CPU
148Tensor blk.5.ffn_up_exps.weight buffer type overriden to CPU
149Tensor blk.6.ffn_gate_exps.weight buffer type overriden to CPU
150Tensor blk.6.ffn_down_exps.weight buffer type overriden to CPU
151Tensor blk.6.ffn_up_exps.weight buffer type overriden to CPU
152Tensor blk.7.ffn_gate_exps.weight buffer type overriden to CPU
153Tensor blk.7.ffn_down_exps.weight buffer type overriden to CPU
154Tensor blk.7.ffn_up_exps.weight buffer type overriden to CPU
155Tensor blk.8.ffn_gate_exps.weight buffer type overriden to CPU
156Tensor blk.8.ffn_down_exps.weight buffer type overriden to CPU
157Tensor blk.8.ffn_up_exps.weight buffer type overriden to CPU
158Tensor blk.9.ffn_gate_exps.weight buffer type overriden to CPU
159Tensor blk.9.ffn_down_exps.weight buffer type overriden to CPU
160Tensor blk.9.ffn_up_exps.weight buffer type overriden to CPU
161Tensor blk.10.ffn_gate_exps.weight buffer type overriden to CPU
162Tensor blk.10.ffn_down_exps.weight buffer type overriden to CPU
163Tensor blk.10.ffn_up_exps.weight buffer type overriden to CPU
164Tensor blk.11.ffn_gate_exps.weight buffer type overriden to CPU
165Tensor blk.11.ffn_down_exps.weight buffer type overriden to CPU
166Tensor blk.11.ffn_up_exps.weight buffer type overriden to CPU
167Tensor blk.12.ffn_gate_exps.weight buffer type overriden to CPU
168Tensor blk.12.ffn_down_exps.weight buffer type overriden to CPU
169Tensor blk.12.ffn_up_exps.weight buffer type overriden to CPU
170Tensor blk.13.ffn_gate_exps.weight buffer type overriden to CPU
171Tensor blk.13.ffn_down_exps.weight buffer type overriden to CPU
172Tensor blk.13.ffn_up_exps.weight buffer type overriden to CPU
173Tensor blk.14.ffn_gate_exps.weight buffer type overriden to CPU
174Tensor blk.14.ffn_down_exps.weight buffer type overriden to CPU
175Tensor blk.14.ffn_up_exps.weight buffer type overriden to CPU
176Tensor blk.15.ffn_gate_exps.weight buffer type overriden to CPU
177Tensor blk.15.ffn_down_exps.weight buffer type overriden to CPU
178Tensor blk.15.ffn_up_exps.weight buffer type overriden to CPU
179Tensor blk.16.ffn_gate_exps.weight buffer type overriden to CPU
180Tensor blk.16.ffn_down_exps.weight buffer type overriden to CPU
181Tensor blk.16.ffn_up_exps.weight buffer type overriden to CPU
182Tensor blk.17.ffn_gate_exps.weight buffer type overriden to CPU
183Tensor blk.17.ffn_down_exps.weight buffer type overriden to CPU
184Tensor blk.17.ffn_up_exps.weight buffer type overriden to CPU
185Tensor blk.18.ffn_gate_exps.weight buffer type overriden to CPU
186Tensor blk.18.ffn_down_exps.weight buffer type overriden to CPU
187Tensor blk.18.ffn_up_exps.weight buffer type overriden to CPU
188Tensor blk.19.ffn_gate_exps.weight buffer type overriden to CPU
189Tensor blk.19.ffn_down_exps.weight buffer type overriden to CPU
190Tensor blk.19.ffn_up_exps.weight buffer type overriden to CPU
191Tensor blk.20.ffn_gate_exps.weight buffer type overriden to CPU
192Tensor blk.20.ffn_down_exps.weight buffer type overriden to CPU
193Tensor blk.20.ffn_up_exps.weight buffer type overriden to CPU
194Tensor blk.21.ffn_gate_exps.weight buffer type overriden to CPU
195Tensor blk.21.ffn_down_exps.weight buffer type overriden to CPU
196Tensor blk.21.ffn_up_exps.weight buffer type overriden to CPU
197Tensor blk.22.ffn_gate_exps.weight buffer type overriden to CPU
198Tensor blk.22.ffn_down_exps.weight buffer type overriden to CPU
199Tensor blk.22.ffn_up_exps.weight buffer type overriden to CPU
200Tensor blk.23.ffn_gate_exps.weight buffer type overriden to CPU
201Tensor blk.23.ffn_down_exps.weight buffer type overriden to CPU
202Tensor blk.23.ffn_up_exps.weight buffer type overriden to CPU
203Tensor blk.24.ffn_gate_exps.weight buffer type overriden to CPU
204Tensor blk.24.ffn_down_exps.weight buffer type overriden to CPU
205Tensor blk.24.ffn_up_exps.weight buffer type overriden to CPU
206Tensor blk.25.ffn_gate_exps.weight buffer type overriden to CPU
207Tensor blk.25.ffn_down_exps.weight buffer type overriden to CPU
208Tensor blk.25.ffn_up_exps.weight buffer type overriden to CPU
209Tensor blk.26.ffn_gate_exps.weight buffer type overriden to CPU
210Tensor blk.26.ffn_down_exps.weight buffer type overriden to CPU
211Tensor blk.26.ffn_up_exps.weight buffer type overriden to CPU
212Tensor blk.27.ffn_gate_exps.weight buffer type overriden to CPU
213Tensor blk.27.ffn_down_exps.weight buffer type overriden to CPU
214Tensor blk.27.ffn_up_exps.weight buffer type overriden to CPU
215Tensor blk.28.ffn_gate_exps.weight buffer type overriden to CPU
216Tensor blk.28.ffn_down_exps.weight buffer type overriden to CPU
217Tensor blk.28.ffn_up_exps.weight buffer type overriden to CPU
218Tensor blk.29.ffn_gate_exps.weight buffer type overriden to CPU
219Tensor blk.29.ffn_down_exps.weight buffer type overriden to CPU
220Tensor blk.29.ffn_up_exps.weight buffer type overriden to CPU
221Tensor blk.30.ffn_gate_exps.weight buffer type overriden to CPU
222Tensor blk.30.ffn_down_exps.weight buffer type overriden to CPU
223Tensor blk.30.ffn_up_exps.weight buffer type overriden to CPU
224Tensor blk.31.ffn_gate_exps.weight buffer type overriden to CPU
225Tensor blk.31.ffn_down_exps.weight buffer type overriden to CPU
226Tensor blk.31.ffn_up_exps.weight buffer type overriden to CPU
227Tensor blk.32.ffn_gate_exps.weight buffer type overriden to CPU
228Tensor blk.32.ffn_down_exps.weight buffer type overriden to CPU
229Tensor blk.32.ffn_up_exps.weight buffer type overriden to CPU
230Tensor blk.33.ffn_gate_exps.weight buffer type overriden to CPU
231Tensor blk.33.ffn_down_exps.weight buffer type overriden to CPU
232Tensor blk.33.ffn_up_exps.weight buffer type overriden to CPU
233Tensor blk.34.ffn_gate_exps.weight buffer type overriden to CPU
234Tensor blk.34.ffn_down_exps.weight buffer type overriden to CPU
235Tensor blk.34.ffn_up_exps.weight buffer type overriden to CPU
236Tensor blk.35.ffn_gate_exps.weight buffer type overriden to CPU
237Tensor blk.35.ffn_down_exps.weight buffer type overriden to CPU
238Tensor blk.35.ffn_up_exps.weight buffer type overriden to CPU
239Tensor blk.36.ffn_gate_exps.weight buffer type overriden to CPU
240Tensor blk.36.ffn_down_exps.weight buffer type overriden to CPU
241Tensor blk.36.ffn_up_exps.weight buffer type overriden to CPU
242Tensor blk.37.ffn_gate_exps.weight buffer type overriden to CPU
243Tensor blk.37.ffn_down_exps.weight buffer type overriden to CPU
244Tensor blk.37.ffn_up_exps.weight buffer type overriden to CPU
245Tensor blk.38.ffn_gate_exps.weight buffer type overriden to CPU
246Tensor blk.38.ffn_down_exps.weight buffer type overriden to CPU
247Tensor blk.38.ffn_up_exps.weight buffer type overriden to CPU
248Tensor blk.39.ffn_gate_exps.weight buffer type overriden to CPU
249Tensor blk.39.ffn_down_exps.weight buffer type overriden to CPU
250Tensor blk.39.ffn_up_exps.weight buffer type overriden to CPU
251Tensor blk.40.ffn_gate_exps.weight buffer type overriden to CPU
252Tensor blk.40.ffn_down_exps.weight buffer type overriden to CPU
253Tensor blk.40.ffn_up_exps.weight buffer type overriden to CPU
254Tensor blk.41.ffn_gate_exps.weight buffer type overriden to CPU
255Tensor blk.41.ffn_down_exps.weight buffer type overriden to CPU
256Tensor blk.41.ffn_up_exps.weight buffer type overriden to CPU
257Tensor blk.42.ffn_gate_exps.weight buffer type overriden to CPU
258Tensor blk.42.ffn_down_exps.weight buffer type overriden to CPU
259Tensor blk.42.ffn_up_exps.weight buffer type overriden to CPU
260Tensor blk.43.ffn_gate_exps.weight buffer type overriden to CPU
261Tensor blk.43.ffn_down_exps.weight buffer type overriden to CPU
262Tensor blk.43.ffn_up_exps.weight buffer type overriden to CPU
263Tensor blk.44.ffn_gate_exps.weight buffer type overriden to CPU
264Tensor blk.44.ffn_down_exps.weight buffer type overriden to CPU
265Tensor blk.44.ffn_up_exps.weight buffer type overriden to CPU
266Tensor blk.45.ffn_gate_exps.weight buffer type overriden to CPU
267Tensor blk.45.ffn_down_exps.weight buffer type overriden to CPU
268Tensor blk.45.ffn_up_exps.weight buffer type overriden to CPU
269Tensor blk.46.ffn_gate_exps.weight buffer type overriden to CPU
270Tensor blk.46.ffn_down_exps.weight buffer type overriden to CPU
271Tensor blk.46.ffn_up_exps.weight buffer type overriden to CPU
272Tensor blk.47.ffn_gate_exps.weight buffer type overriden to CPU
273Tensor blk.47.ffn_down_exps.weight buffer type overriden to CPU
274Tensor blk.47.ffn_up_exps.weight buffer type overriden to CPU
275Tensor blk.48.ffn_gate_exps.weight buffer type overriden to CPU
276Tensor blk.48.ffn_down_exps.weight buffer type overriden to CPU
277Tensor blk.48.ffn_up_exps.weight buffer type overriden to CPU
278Tensor blk.49.ffn_gate_exps.weight buffer type overriden to CPU
279Tensor blk.49.ffn_down_exps.weight buffer type overriden to CPU
280Tensor blk.49.ffn_up_exps.weight buffer type overriden to CPU
281Tensor blk.50.ffn_gate_exps.weight buffer type overriden to CPU
282Tensor blk.50.ffn_down_exps.weight buffer type overriden to CPU
283Tensor blk.50.ffn_up_exps.weight buffer type overriden to CPU
284Tensor blk.51.ffn_gate_exps.weight buffer type overriden to CPU
285Tensor blk.51.ffn_down_exps.weight buffer type overriden to CPU
286Tensor blk.51.ffn_up_exps.weight buffer type overriden to CPU
287Tensor blk.52.ffn_gate_exps.weight buffer type overriden to CPU
288Tensor blk.52.ffn_down_exps.weight buffer type overriden to CPU
289Tensor blk.52.ffn_up_exps.weight buffer type overriden to CPU
290Tensor blk.53.ffn_gate_exps.weight buffer type overriden to CPU
291Tensor blk.53.ffn_down_exps.weight buffer type overriden to CPU
292Tensor blk.53.ffn_up_exps.weight buffer type overriden to CPU
293Tensor blk.54.ffn_gate_exps.weight buffer type overriden to CPU
294Tensor blk.54.ffn_down_exps.weight buffer type overriden to CPU
295Tensor blk.54.ffn_up_exps.weight buffer type overriden to CPU
296Tensor blk.55.ffn_gate_exps.weight buffer type overriden to CPU
297Tensor blk.55.ffn_down_exps.weight buffer type overriden to CPU
298Tensor blk.55.ffn_up_exps.weight buffer type overriden to CPU
299Tensor blk.56.ffn_gate_exps.weight buffer type overriden to CPU
300Tensor blk.56.ffn_down_exps.weight buffer type overriden to CPU
301Tensor blk.56.ffn_up_exps.weight buffer type overriden to CPU
302Tensor blk.57.ffn_gate_exps.weight buffer type overriden to CPU
303Tensor blk.57.ffn_down_exps.weight buffer type overriden to CPU
304Tensor blk.57.ffn_up_exps.weight buffer type overriden to CPU
305Tensor blk.58.ffn_gate_exps.weight buffer type overriden to CPU
306Tensor blk.58.ffn_down_exps.weight buffer type overriden to CPU
307Tensor blk.58.ffn_up_exps.weight buffer type overriden to CPU
308Tensor blk.59.ffn_gate_exps.weight buffer type overriden to CPU
309Tensor blk.59.ffn_down_exps.weight buffer type overriden to CPU
310Tensor blk.59.ffn_up_exps.weight buffer type overriden to CPU
311Tensor blk.60.ffn_gate_exps.weight buffer type overriden to CPU
312Tensor blk.60.ffn_down_exps.weight buffer type overriden to CPU
313Tensor blk.60.ffn_up_exps.weight buffer type overriden to CPU
314llm_load_tensors: offloading 61 repeating layers to GPU
315llm_load_tensors: offloading non-repeating layers to GPU
316llm_load_tensors: offloaded 62/62 layers to GPU
317llm_load_tensors: CPU buffer size = 228404.85 MiB
318llm_load_tensors: CPU buffer size = 938.98 MiB
319llm_load_tensors: CUDA0 buffer size = 17744.02 MiB
320....................................................................................................
321llama_new_context_with_model: n_ctx = 2048
322llama_new_context_with_model: n_batch = 2048
323llama_new_context_with_model: n_ubatch = 512
324llama_new_context_with_model: flash_attn = 1
325llama_new_context_with_model: mla_attn = 2
326llama_new_context_with_model: attn_max_b = 512
327llama_new_context_with_model: fused_moe = 1
328llama_new_context_with_model: ser = -1, 0
329llama_new_context_with_model: freq_base = 10000.0
330llama_new_context_with_model: freq_scale = 0.025
331llama_kv_cache_init: layer 0: n_embd_head_qk_rope = 64, kv_lora_rank = 512
332llama_kv_cache_init: layer 1: n_embd_head_qk_rope = 64, kv_lora_rank = 512
333llama_kv_cache_init: layer 2: n_embd_head_qk_rope = 64, kv_lora_rank = 512
334llama_kv_cache_init: layer 3: n_embd_head_qk_rope = 64, kv_lora_rank = 512
335llama_kv_cache_init: layer 4: n_embd_head_qk_rope = 64, kv_lora_rank = 512
336llama_kv_cache_init: layer 5: n_embd_head_qk_rope = 64, kv_lora_rank = 512
337llama_kv_cache_init: layer 6: n_embd_head_qk_rope = 64, kv_lora_rank = 512
338llama_kv_cache_init: layer 7: n_embd_head_qk_rope = 64, kv_lora_rank = 512
339llama_kv_cache_init: layer 8: n_embd_head_qk_rope = 64, kv_lora_rank = 512
340llama_kv_cache_init: layer 9: n_embd_head_qk_rope = 64, kv_lora_rank = 512
341llama_kv_cache_init: layer 10: n_embd_head_qk_rope = 64, kv_lora_rank = 512
342llama_kv_cache_init: layer 11: n_embd_head_qk_rope = 64, kv_lora_rank = 512
343llama_kv_cache_init: layer 12: n_embd_head_qk_rope = 64, kv_lora_rank = 512
344llama_kv_cache_init: layer 13: n_embd_head_qk_rope = 64, kv_lora_rank = 512
345llama_kv_cache_init: layer 14: n_embd_head_qk_rope = 64, kv_lora_rank = 512
346llama_kv_cache_init: layer 15: n_embd_head_qk_rope = 64, kv_lora_rank = 512
347llama_kv_cache_init: layer 16: n_embd_head_qk_rope = 64, kv_lora_rank = 512
348llama_kv_cache_init: layer 17: n_embd_head_qk_rope = 64, kv_lora_rank = 512
349llama_kv_cache_init: layer 18: n_embd_head_qk_rope = 64, kv_lora_rank = 512
350llama_kv_cache_init: layer 19: n_embd_head_qk_rope = 64, kv_lora_rank = 512
351llama_kv_cache_init: layer 20: n_embd_head_qk_rope = 64, kv_lora_rank = 512
352llama_kv_cache_init: layer 21: n_embd_head_qk_rope = 64, kv_lora_rank = 512
353llama_kv_cache_init: layer 22: n_embd_head_qk_rope = 64, kv_lora_rank = 512
354llama_kv_cache_init: layer 23: n_embd_head_qk_rope = 64, kv_lora_rank = 512
355llama_kv_cache_init: layer 24: n_embd_head_qk_rope = 64, kv_lora_rank = 512
356llama_kv_cache_init: layer 25: n_embd_head_qk_rope = 64, kv_lora_rank = 512
357llama_kv_cache_init: layer 26: n_embd_head_qk_rope = 64, kv_lora_rank = 512
358llama_kv_cache_init: layer 27: n_embd_head_qk_rope = 64, kv_lora_rank = 512
359llama_kv_cache_init: layer 28: n_embd_head_qk_rope = 64, kv_lora_rank = 512
360llama_kv_cache_init: layer 29: n_embd_head_qk_rope = 64, kv_lora_rank = 512
361llama_kv_cache_init: layer 30: n_embd_head_qk_rope = 64, kv_lora_rank = 512
362llama_kv_cache_init: layer 31: n_embd_head_qk_rope = 64, kv_lora_rank = 512
363llama_kv_cache_init: layer 32: n_embd_head_qk_rope = 64, kv_lora_rank = 512
364llama_kv_cache_init: layer 33: n_embd_head_qk_rope = 64, kv_lora_rank = 512
365llama_kv_cache_init: layer 34: n_embd_head_qk_rope = 64, kv_lora_rank = 512
366llama_kv_cache_init: layer 35: n_embd_head_qk_rope = 64, kv_lora_rank = 512
367llama_kv_cache_init: layer 36: n_embd_head_qk_rope = 64, kv_lora_rank = 512
368llama_kv_cache_init: layer 37: n_embd_head_qk_rope = 64, kv_lora_rank = 512
369llama_kv_cache_init: layer 38: n_embd_head_qk_rope = 64, kv_lora_rank = 512
370llama_kv_cache_init: layer 39: n_embd_head_qk_rope = 64, kv_lora_rank = 512
371llama_kv_cache_init: layer 40: n_embd_head_qk_rope = 64, kv_lora_rank = 512
372llama_kv_cache_init: layer 41: n_embd_head_qk_rope = 64, kv_lora_rank = 512
373llama_kv_cache_init: layer 42: n_embd_head_qk_rope = 64, kv_lora_rank = 512
374llama_kv_cache_init: layer 43: n_embd_head_qk_rope = 64, kv_lora_rank = 512
375llama_kv_cache_init: layer 44: n_embd_head_qk_rope = 64, kv_lora_rank = 512
376llama_kv_cache_init: layer 45: n_embd_head_qk_rope = 64, kv_lora_rank = 512
377llama_kv_cache_init: layer 46: n_embd_head_qk_rope = 64, kv_lora_rank = 512
378llama_kv_cache_init: layer 47: n_embd_head_qk_rope = 64, kv_lora_rank = 512
379llama_kv_cache_init: layer 48: n_embd_head_qk_rope = 64, kv_lora_rank = 512
380llama_kv_cache_init: layer 49: n_embd_head_qk_rope = 64, kv_lora_rank = 512
381llama_kv_cache_init: layer 50: n_embd_head_qk_rope = 64, kv_lora_rank = 512
382llama_kv_cache_init: layer 51: n_embd_head_qk_rope = 64, kv_lora_rank = 512
383llama_kv_cache_init: layer 52: n_embd_head_qk_rope = 64, kv_lora_rank = 512
384llama_kv_cache_init: layer 53: n_embd_head_qk_rope = 64, kv_lora_rank = 512
385llama_kv_cache_init: layer 54: n_embd_head_qk_rope = 64, kv_lora_rank = 512
386llama_kv_cache_init: layer 55: n_embd_head_qk_rope = 64, kv_lora_rank = 512
387llama_kv_cache_init: layer 56: n_embd_head_qk_rope = 64, kv_lora_rank = 512
388llama_kv_cache_init: layer 57: n_embd_head_qk_rope = 64, kv_lora_rank = 512
389llama_kv_cache_init: layer 58: n_embd_head_qk_rope = 64, kv_lora_rank = 512
390llama_kv_cache_init: layer 59: n_embd_head_qk_rope = 64, kv_lora_rank = 512
391llama_kv_cache_init: layer 60: n_embd_head_qk_rope = 64, kv_lora_rank = 512
392llama_kv_cache_init: CUDA0 KV buffer size = 72.94 MiB
393llama_new_context_with_model: KV self size = 72.91 MiB, c^KV (q8_0): 72.91 MiB, kv^T: not used
394llama_new_context_with_model: CUDA_Host output buffer size = 1.97 MiB
395llama_new_context_with_model: CUDA0 compute buffer size = 503.00 MiB
396llama_new_context_with_model: CUDA_Host compute buffer size = 162.01 MiB
397llama_new_context_with_model: graph nodes = 3548
398llama_new_context_with_model: graph splits = 118
399
400system_info: n_threads = 24 / 48 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
401perplexity: tokenizing the input ..
402perplexity: tokenization took 617.314 ms
403perplexity: calculating perplexity over 561 chunks, n_ctx=512, batch_size=2048, n_seq=4
404perplexity: 20.43 seconds per pass - ETA 47.75 minutes
405[1]2.7687,[2]3.5402,[3]2.5152,[4]2.1223,[5]1.9022,[6]1.7765,[7]1.6869,[8]1.6282,[9]1.5856,[10]1.5431,[11]1.5379,[12]1.5781,[13]1.5947,[14]1.7232,[15]1.8539,[16]1.9054,[17]2.0733,[18]2.1998,[19]2.1545,[20]2.1438,[21]2.2433,[22]2.2143,[23]2.1801,[24]2.1951,[25]2.1614,[26]2.1344,[27]2.1796,[28]2.1874,[29]2.2426,[30]2.2752,[31]2.3094,[32]2.3270,[33]2.3654,[34]2.4142,[35]2.4636,[36]2.5196,[37]2.5536,[38]2.6038,[39]2.6420,[40]2.7049,[41]2.7453,[42]2.7591,[43]2.8110,[44]2.8248,[45]2.9060,[46]2.9548,[47]2.9179,[48]2.8736,[49]2.8574,[50]2.8805,[51]2.9241,[52]2.9352,[53]2.9930,[54]3.0074,[55]3.0389,[56]3.0729,[57]3.0906,[58]3.1289,[59]3.1400,[60]3.1872,[61]3.2286,[62]3.2827,[63]3.3128,[64]3.3584,[65]3.3660,[66]3.3592,[67]3.3366,[68]3.3659,[69]3.3658,[70]3.3835,[71]3.4002,[72]3.4111,[73]3.4239,[74]3.4442,[75]3.4233,[76]3.3758,[77]3.3326,[78]3.3303,[79]3.3130,[80]3.2983,[81]3.2648,[82]3.2686,[83]3.2416,[84]3.2081,[85]3.1745,[86]3.1530,[87]3.1529,[88]3.1288,[89]3.1175,[90]3.0963,[91]3.0711,[92]3.0476,[93]3.0211,[94]2.9999,[95]2.9825,[96]2.9833,[97]2.9927,[98]2.9829,[99]2.9668,[100]2.9680,[101]2.9599,[102]2.9779,[103]3.0038,[104]3.0240,[105]3.0204,[106]3.0449,[107]3.0697,[108]3.0899,[109]3.1240,[110]3.1566,[111]3.1776,[112]3.1502,[113]3.1382,[114]3.1169,[115]3.1009,[116]3.0955,[117]3.0743,[118]3.0530,[119]3.0316,[120]3.0096,[121]2.9932,[122]2.9731,[123]2.9560,[124]2.9358,[125]2.9177,[126]2.9016,[127]2.8884,[128]2.8807,[129]2.8715,[130]2.8593,[131]2.8514,[132]2.8569,[133]2.8658,[134]2.8729,[135]2.8841,[136]2.8996,[137]2.9132,[138]2.9206,[139]2.9316,[140]2.9307,[141]2.9306,[142]2.9272,[143]2.9268,[144]2.9227,[145]2.9142,[146]2.9109,[147]2.9136,[148]2.9117,[149]2.9118,[150]2.9045,[151]2.9008,[152]2.8968,[153]2.8916,[154]2.8899,[155]2.8930,[156]2.8931,[157]2.8978,[158]2.9057,[159]2.9076,[160]2.9166,[161]2.9246,[162]2.9344,[163]2.9420,[164]2.9627,[165]2.9857,[166]3.0037,[167]3.0164,[168]3.0418,[169]3.0653,[170]3.0882,[171]3.1102,[172]3.0929,[173]3.0748,[174]3.0617,[175]3.0500,[176]3.0377,[177]3.0267,[178]3.0141,[179]3.0021,[180]3.0052,[181]3.0197,[182]3.0349,[183]3.0493,[184]3.0624,[185]3.0716,[186]3.0873,[187]3.1029,[188]3.1162,[189]3.1261,[190]3.1263,[191]3.1330,[192]3.1349,[193]3.1393,[194]3.1599,[195]3.1691,[196]3.1822,[197]3.1921,[198]3.1960,[199]3.2012,[200]3.1993,[201]3.2141,[202]3.2085,[203]3.2130,[204]3.2147,[205]3.2140,[206]3.2168,[207]3.2250,[208]3.2347,[209]3.2436,[210]3.2429,[211]3.2373,[212]3.2379,[213]3.2456,[214]3.2470,[215]3.2523,[216]3.2521,[217]3.2455,[218]3.2449,[219]3.2453,[220]3.2444,[221]3.2445,[222]3.2436,[223]3.2447,[224]3.2492,[225]3.2506,[226]3.2411,[227]3.2394,[228]3.2405,[229]3.2439,[230]3.2492,[231]3.2556,[232]3.2476,[233]3.2406,[234]3.2426,[235]3.2421,[236]3.2507,[237]3.2588,[238]3.2680,[239]3.2784,[240]3.2870,[241]3.2980,[242]3.3129,[243]3.3258,[244]3.3340,[245]3.3461,[246]3.3567,[247]3.3546,[248]3.3498,[249]3.3471,[250]3.3394,[251]3.3362,[252]3.3375,[253]3.3407,[254]3.3468,[255]3.3525,[256]3.3556,[257]3.3577,[258]3.3586,[259]3.3614,[260]3.3636,[261]3.3641,[262]3.3625,[263]3.3674,[264]3.3696,[265]3.3698,[266]3.3714,[267]3.3732,[268]3.3765,[269]3.3796,[270]3.3775,[271]3.3755,[272]3.3686,[273]3.3688,[274]3.3615,[275]3.3506,[276]3.3398,[277]3.3414,[278]3.3514,[279]3.3572,[280]3.3649,[281]3.3719,[282]3.3773,[283]3.3837,[284]3.3896,[285]3.4035,[286]3.4052,[287]3.4076,[288]3.4123,[289]3.4143,[290]3.4063,[291]3.3989,[292]3.4003,[293]3.4002,[294]3.3985,[295]3.3973,[296]3.3992,[297]3.4007,[298]3.4059,[299]3.4120,[300]3.4151,[301]3.4186,[302]3.4206,[303]3.4218,[304]3.4201,[305]3.4319,[306]3.4388,[307]3.4495,[308]3.4378,[309]3.4322,[310]3.4227,[311]3.4258,[312]3.4287,[313]3.4348,[314]3.4368,[315]3.4397,[316]3.4407,[317]3.4420,[318]3.4426,[319]3.4432,[320]3.4472,[321]3.4472,[322]3.4485,[323]3.4545,[324]3.4549,[325]3.4602,[326]3.4643,[327]3.4680,[328]3.4700,[329]3.4715,[330]3.4776,[331]3.4810,[332]3.4845,[333]3.4828,[334]3.4827,[335]3.4826,[336]3.4819,[337]3.4828,[338]3.4829,[339]3.4850,[340]3.4883,[341]3.4937,[342]3.5027,[343]3.5120,[344]3.5171,[345]3.5093,[346]3.5022,[347]3.4991,[348]3.4918,[349]3.4879,[350]3.4865,[351]3.4911,[352]3.5060,[353]3.5152,[354]3.5281,[355]3.5370,[356]3.5432,[357]3.5550,[358]3.5655,[359]3.5685,[360]3.5745,[361]3.5839,[362]3.5924,[363]3.5975,[364]3.6038,[365]3.6091,[366]3.6193,[367]3.6280,[368]3.6342,[369]3.6417,[370]3.6499,[371]3.6633,[372]3.6725,[373]3.6756,[374]3.6789,[375]3.6833,[376]3.6959,[377]3.7072,[378]3.7094,[379]3.7088,[380]3.7053,[381]3.7103,[382]3.7160,[383]3.7190,[384]3.7229,[385]3.7265,[386]3.7323,[387]3.7382,[388]3.7409,[389]3.7299,[390]3.7200,[391]3.7099,[392]3.7042,[393]3.6954,[394]3.6870,[395]3.6790,[396]3.6692,[397]3.6606,[398]3.6507,[399]3.6408,[400]3.6318,[401]3.6214,[402]3.6109,[403]3.6021,[404]3.5911,[405]3.5808,[406]3.5703,[407]3.5604,[408]3.5516,[409]3.5429,[410]3.5373,[411]3.5385,[412]3.5339,[413]3.5373,[414]3.5406,[415]3.5383,[416]3.5384,[417]3.5403,[418]3.5345,[419]3.5357,[420]3.5327,[421]3.5318,[422]3.5329,[423]3.5331,[424]3.5372,[425]3.5368,[426]3.5376,[427]3.5367,[428]3.5397,[429]3.5405,[430]3.5433,[431]3.5445,[432]3.5429,[433]3.5394,[434]3.5394,[435]3.5334,[436]3.5277,[437]3.5235,[438]3.5219,[439]3.5200,[440]3.5245,[441]3.5297,[442]3.5373,[443]3.5347,[444]3.5352,[445]3.5360,[446]3.5408,[447]3.5439,[448]3.5462,[449]3.5488,[450]3.5524,[451]3.5560,[452]3.5581,[453]3.5595,[454]3.5578,[455]3.5603,[456]3.5603,[457]3.5624,[458]3.5674,[459]3.5675,[460]3.5673,[461]3.5639,[462]3.5674,[463]3.5746,[464]3.5799,[465]3.5733,[466]3.5721,[467]3.5710,[468]3.5729,[469]3.5704,[470]3.5678,[471]3.5682,[472]3.5695,[473]3.5684,[474]3.5670,[475]3.5680,[476]3.5669,[477]3.5660,[478]3.5668,[479]3.5685,[480]3.5709,[481]3.5669,[482]3.5704,[483]3.5695,[484]3.5728,[485]3.5790,[486]3.5820,[487]3.5853,[488]3.5908,[489]3.5930,[490]3.5979,[491]3.6040,[492]3.6085,[493]3.6080,[494]3.6086,[495]3.6109,[496]3.6127,[497]3.6156,[498]3.6160,[499]3.6153,[500]3.6191,[501]3.6235,[502]3.6223,[503]3.6207,[504]3.6229,[505]3.6260,[506]3.6343,[507]3.6372,[508]3.6405,[509]3.6328,[510]3.6289,[511]3.6229,[512]3.6191,[513]3.6132,[514]3.6120,[515]3.6142,[516]3.6099,[517]3.6103,[518]3.6094,[519]3.6100,[520]3.6145,[521]3.6133,[522]3.6114,[523]3.6172,[524]3.6158,[525]3.6143,[526]3.6100,[527]3.6046,[528]3.6021,[529]3.5989,[530]3.5958,[531]3.5925,[532]3.5864,[533]3.5800,[534]3.5762,[535]3.5767,[536]3.5794,[537]3.5827,[538]3.5859,[539]3.5887,[540]3.5944,[541]3.5979,[542]3.6007,[543]3.5961,[544]3.5918,[545]3.5918,[546]3.5850,[547]3.5790,[548]3.5722,[549]3.5657,[550]3.5601,[551]3.5546,[552]3.5491,[553]3.5435,[554]3.5431,[555]3.5418,[556]3.5443,[557]3.5479,[558]3.5539,[559]3.5582,[560]3.5635,[561]3.5614,
406llama_print_timings: load time = 39774.62 ms
407llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
408llama_print_timings: prompt eval time = 2837909.33 ms / 287232 tokens ( 9.88 ms per token, 101.21 tokens per second)
409llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
410llama_print_timings: total time = 2841519.57 ms / 287233 tokens
411
412Final estimate: PPL = 3.5614 +/- 0.020011$ ./build/bin/llama-gguf-split \
2 --dry-run \
3 --split \
4 --split-max-size 50G \
5 /mnt/raid/models/ubergarm/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4.gguf \
6 /mnt/raid/hf/DeepSeek-V3-0324-GGUF/DeepSeek-V3-0324-IQ2_K_R4/DeepSeek-V3-0324-IQ2_K_R4