Views
No views yet
1$ python run_inference.py -m models/Falcon3-10B-Instruct-1.58bit/ggml-model-i2_s.gguf -p "how many letters 'a' do we have in the word 'Ananas'"
2
3build: 3954 (957b59d2) with cc (Debian 12.2.0-14) 12.2.0 for x86_64-linux-gnu
4
5llama_model_loader: - kv 0: general.architecture str = llama
6llama_model_loader: - kv 1: general.name str = Falcon3-10B-Instruct-1.58bit
7llama_model_loader: - kv 2: llama.block_count u32 = 40
8llama_model_loader: - kv 3: llama.context_length u32 = 32768
9llama_model_loader: - kv 4: llama.embedding_length u32 = 3072
10llama_model_loader: - kv 5: llama.feed_forward_length u32 = 23040
11llama_model_loader: - kv 6: llama.attention.head_count u32 = 12
12llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 4
13llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000042.000000
14llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000001
15llama_model_loader: - kv 10: general.file_type u32 = 40
16llama_model_loader: - kv 11: llama.vocab_size u32 = 131072
17llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 256
18llama_model_loader: - kv 13: tokenizer.ggml.add_space_prefix bool = false
19llama_model_loader: - kv 14: tokenizer.ggml.model str = gpt2
20llama_model_loader: - kv 15: tokenizer.ggml.pre str = falcon3
21llama_model_loader: - kv 16: tokenizer.ggml.tokens arr[str,131072] = [">>TITLE<<", ">>ABSTRACT<<", ">>INTR...
22llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,131072] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
23llama_model_loader: - kv 18: tokenizer.ggml.merges arr[str,128810] = ["N E", "Ġ Ġ", "Ġ t", "Ġ a", "> >...
24llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 11
25llama_model_loader: - kv 20: tokenizer.ggml.padding_token_id u32 = 2023
26llama_model_loader: - kv 21: tokenizer.chat_template str = {% if tools %}{% for message in messa...
27llama_model_loader: - kv 22: general.quantization_version u32 = 2
28llama_model_loader: - type f32: 81 tensors
29llama_model_loader: - type f16: 2 tensors
30llama_model_loader: - type i2_s: 280 tensors
31# other stuff
32llm_load_vocab: special tokens cache size = 2024
33llm_load_vocab: token to piece cache size = 0.8741 MB
34llm_load_print_meta: format = GGUF V3 (latest)
35llm_load_print_meta: arch = llama
36llm_load_print_meta: vocab type = BPE
37llm_load_print_meta: n_vocab = 131072
38llm_load_print_meta: n_merges = 128810
39llm_load_print_meta: vocab_only = 0
40llm_load_print_meta: n_ctx_train = 32768
41llm_load_print_meta: n_embd = 3072
42llm_load_print_meta: n_layer = 40
43llm_load_print_meta: n_head = 12
44llm_load_print_meta: n_head_kv = 4
45llm_load_print_meta: n_rot = 256
46llm_load_print_meta: n_swa = 0
47llm_load_print_meta: n_embd_head_k = 256
48llm_load_print_meta: n_embd_head_v = 256
49llm_load_print_meta: n_gqa = 3
50llm_load_print_meta: n_embd_k_gqa = 1024
51llm_load_print_meta: n_embd_v_gqa = 1024
52llm_load_print_meta: f_norm_eps = 0.0e+00
53llm_load_print_meta: f_norm_rms_eps = 1.0e-06
54llm_load_print_meta: f_clamp_kqv = 0.0e+00
55llm_load_print_meta: f_max_alibi_bias = 0.0e+00
56llm_load_print_meta: f_logit_scale = 0.0e+00
57llm_load_print_meta: n_ff = 23040
58llm_load_print_meta: n_expert = 0
59llm_load_print_meta: n_expert_used = 0
60llm_load_print_meta: causal attn = 1
61llm_load_print_meta: pooling type = 0
62llm_load_print_meta: rope type = 0
63llm_load_print_meta: rope scaling = linear
64llm_load_print_meta: freq_base_train = 1000042.0
65llm_load_print_meta: freq_scale_train = 1
66llm_load_print_meta: n_ctx_orig_yarn = 32768
67llm_load_print_meta: rope_finetuned = unknown
68llm_load_print_meta: ssm_d_conv = 0
69llm_load_print_meta: ssm_d_inner = 0
70llm_load_print_meta: ssm_d_state = 0
71llm_load_print_meta: ssm_dt_rank = 0
72llm_load_print_meta: ssm_dt_b_c_rms = 0
73llm_load_print_meta: model type = 13B
74llm_load_print_meta: model ftype = I2_S - 2 bpw ternary
75llm_load_print_meta: model params = 10.31 B
76llm_load_print_meta: model size = 3.71 GiB (3.09 BPW)
77llm_load_print_meta: general.name = Falcon3-10B-Instruct-1.58bit
78llm_load_print_meta: BOS token = 11 '<|endoftext|>'
79llm_load_print_meta: EOS token = 11 '<|endoftext|>'
80llm_load_print_meta: EOT token = 11 '<|endoftext|>'
81llm_load_print_meta: PAD token = 2023 '<|pad|>'
82llm_load_print_meta: LF token = 2150 'Ä'
83llm_load_print_meta: EOG token = 11 '<|endoftext|>'
84llm_load_print_meta: max token length = 256
85llm_load_tensors: ggml ctx size = 0.17 MiB
86llm_load_tensors: CPU buffer size = 3801.96 MiB
87..............................................................
88llama_new_context_with_model: n_batch is less than GGML_KQ_MASK_PAD - increasing to 32
89llama_new_context_with_model: n_ctx = 2048
90llama_new_context_with_model: n_batch = 32
91llama_new_context_with_model: n_ubatch = 32
92llama_new_context_with_model: flash_attn = 0
93llama_new_context_with_model: freq_base = 1000042.0
94llama_new_context_with_model: freq_scale = 1
95llama_kv_cache_init: CPU KV buffer size = 320.00 MiB
96llama_new_context_with_model: KV self size = 320.00 MiB, K (f16): 160.00 MiB, V (f16): 160.00 MiB
97llama_new_context_with_model: CPU output buffer size = 0.50 MiB
98llama_new_context_with_model: CPU compute buffer size = 16.38 MiB
99llama_new_context_with_model: graph nodes = 1286
100llama_new_context_with_model: graph splits = 1
101common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
102main: llama threadpool init, n_threads = 2
103
104system_info: n_threads = 2 (n_threads_batch = 2) / 24 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
105
106sampler seed: 4226296889
107sampler params:
108 repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
109 top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
110 mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
111sampler chain: logits -> logit-bias -> penalties -> top-k -> tail-free -> typical -> top-p -> min-p -> temp-ext -> softmax -> dist
112generate: n_ctx = 2048, n_batch = 1, n_predict = 128, n_keep = 0
113
114how many letters 'a' do we have in the word 'Ananas' in Python?
115<|assistant|>
116In Python, you can use the built-in count() function in the str class to count the number of occurrences of a specific character in a string.
117
118Here is a code snippet that accomplishes this:
119
120``python
121word = 'Ananas'
122count = word.count('a')
123print(count)
124``
125
126When you run this code, it will output:
127
128``
1293
130``
131
132This means that in the word 'Ananas', there are 3 occurrences of the letter 'a'. [end of text]
133
134
135llama_perf_sampler_print: sampling time = 5.12 ms / 144 runs ( 0.04 ms per token, 28130.49 tokens per second)
136llama_perf_context_print: load time = 336.46 ms
137llama_perf_context_print: prompt eval time = 1775.89 ms / 18 tokens ( 98.66 ms per token, 10.14 tokens per second)
138llama_perf_context_print: eval time = 12286.29 ms / 125 runs ( 98.29 ms per token, 10.17 tokens per second)
139llama_perf_context_print: total time = 14075.18 ms / 143 tokens
1401python run_inference.py -m models/Falcon3-10B-Instruct-1.58bit/ggml-model-i2_s.gguf -p "In our production code, we're building a utility function for generating totals with default values for missing parameters. However, the behavior is not as expected. Here's a simplified version of the code:\n\n```js\nexport const calculateTotal = (base, multiplier = 2, increment = 2) => {\n return base * multiplier + increment;\n};\n\nconst data = {\n baseAmount: 10,\n extraParams: [3, 5] // These should map to multiplier and increment\n};\n\nconst total = calculateTotal(data.baseAmount, ...data.extraParams);\nconsole.log(total); // Expecting 12, but the output seems wrong\n```\n\nCan you help figure out why this is happening and how to fix it?"
2# Same as above, just changed the prompt
3sampler seed: 2694760223
4sampler params:
5 repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
6 top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
7 mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
8sampler chain: logits -> logit-bias -> penalties -> top-k -> tail-free -> typical -> top-p -> min-p -> temp-ext -> softmax -> dist
9generate: n_ctx = 2048, n_batch = 1, n_predict = 128, n_keep = 0
10
11In our production code, we're building a utility function for generating totals with default values for missing parameters. However, the behavior is not as expected. Here's a simplified version of the code:
12
13Can you help figure out why this is happening and how to fix it?
14
15I'm using Visual Studio 2005 (2003.01).
16<|assistant|>
17This error is caused by the fact that you're trying to call a virtual method that does not exist in the base class.
18
19You're trying to call `GetCount` in the `IMyDataContext` class, but this method does not exist in the base class `IMyDataContext`. This is because `GetCount` is a virtual function in `IMyDataContext`, and it's meant to be overridden in the
20
21llama_perf_sampler_print: sampling time = 5.50 ms / 188 runs ( 0.03 ms per token, 34212.92 tokens per second)
22llama_perf_context_print: load time = 318.08 ms
23llama_perf_context_print: prompt eval time = 5935.47 ms / 60 tokens ( 98.92 ms per token, 10.11 tokens per second)
24llama_perf_context_print: eval time = 12688.14 ms / 127 runs ( 99.91 ms per token, 10.01 tokens per second)
25llama_perf_context_print: total time = 18638.24 ms / 187 tokens1# Same as above, just changed the prompt
2sampler seed: 4052516490
3sampler params:
4 repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
5 top_k = 40, tfs_z = 1.000, top_p = 0.950, min_p = 0.050, typical_p = 1.000, temp = 0.800
6 mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
7sampler chain: logits -> logit-bias -> penalties -> top-k -> tail-free -> typical -> top-p -> min-p -> temp-ext -> softmax -> dist
8generate: n_ctx = 2048, n_batch = 1, n_predict = 128, n_keep = 0
9
10Provide a summary with about two sentences for the following article:
11The efficient-market hypothesis (EMH) is a hypothesis in financial economics that states that asset prices reflect all available information. A direct implication is that it is impossible to “beat the market” consistently on a risk-adjusted basis since market prices should only react to new information. Because the EMH is formulated in terms of risk adjustment, it only makes testable predictions when coupled with a particular model of risk. As a result, research in financial economics since at least the 1990s has focused on market anomalies, that is, deviations from specific models of risk. The idea that financial market returns are difficult to predict goes back to Bachelier, Mandelbrot, and Samuelson, but is closely associated with Eugene Fama, in part due to his influential 1970 review of the theoretical and empirical research. The EMH provides the basic logic for modern risk-based theories of asset prices, and frameworks such as consumption-based asset pricing and intermediary asset pricing can be thought of as the combination of a model of risk with the EMH. Many decades of empirical research on return predictability has found mixed evidence. Research in the 1950s and 1960s often found a lack of predictability (e.g. Ball and Brown 1968; Fama, Fisher, Jensen, and Roll 1969), yet the 1980s-2000s saw an explosion of discovered return predictors (e.g. Rosenberg, Reid, and Lanstein 1985; Campbell and Shiller 1988; Jegadeesh and Titman 1993). Since the 2010s, studies have often found that return predictability has become more elusive, as predictability fails to work out-of-sample (Goyal and Welch 2008), or has been weakened by advances in trading technology and investor learning (Chordia, Subrahmanyam, and Tong 2014; McLean and Pontiff 2016; Martineau 2021).
12Summary:
13<|assistant|>
14The Efficient Market Hypothesis (EMH) is a financial economics theory that suggests that asset prices reflect all available information. Since the EMH is formulated in terms of risk adjustment, it only makes testable predictions when coupled with a particular model of risk. Today, research in financial economics since the 1990s has focused on market anomalies, that is, deviations from specific models of risk. [end of text]
15
16
17llama_perf_sampler_print: sampling time = 3.88 ms / 539 runs ( 0.01 ms per token, 139060.89 tokens per second)
18llama_perf_context_print: load time = 320.65 ms
19llama_perf_context_print: prompt eval time = 45040.32 ms / 452 tokens ( 99.65 ms per token, 10.04 tokens per second)
20llama_perf_context_print: eval time = 8845.43 ms / 86 runs ( 102.85 ms per token, 9.72 tokens per second)
21llama_perf_context_print: total time = 53896.92 ms / 538 tokens
221git clone https://github.com/microsoft/BitNet.git
2cd BitNet
3// They recommend conda but I tried venv and worked fine in debian 12
4python3 -m venv venv
5source venv/bin/activate
6pip install -r requirements.txt
7// python setup_env.py --hf-repo your_hf_username/Falcon3-10B-Instruct-1.58bit -q i2_s // You can skip this one
8// move the model to the folder models, then
9python run_inference.py -m models/Falcon3-10B-Instruct-1.58bit/ggml-model-i2_s.gguf -p "What is 1.58bit quantization in LLM and why its iteresting for gpu poor people?" -cnv
10# 1.58-bit quantization is a method in which floating-point numbers in a neural network are represented using fewer bits, specifically 1.58 bits. This technique reduces the number of bits needed to represent a number, which can lead to improved performance and lower memory usage.| Benchmark | Llama-3.2-1B | Qwen2.5-1.5B | Llama3-8B-1.58-100B-tokens | Falcon3-10B-Instruct-1.58bit |
|---|---|---|---|---|
| IFEval | 55.8 | 44.4 | 17.91 | 54.37 |
| MUSR | 32.4 | 36.8 | 4.87 | 2.57 |
| GPQA | 25.3 | 29.6 | 1.83 | 4.27 |
| BBH | 30.3 | 38.5 | 5.36 | 6.59 |
| MMLU-PRO | 11.3 | 21.3 | 2.78 | 6.62 |
| MATH | 3.9 | 0.2 | 0.26 | 2.44 |