This is the Q4_1 quant optimized for older cards from
Bartowski patched with the missing MTP tensors.
The modified conversion script from user
buzz can be found
here
For more information see the
PR or the
/r/locallama discussion
NOTE: This file only work with the
am17an:mtp-clean branch mentioned in the above PR or other MTP supported branches until official support is merged.
The 27B dense version can be found
here
Dual Mi50@150W with -sm tensor and -ngram mod performance compressing a 120K token context:
prompt eval time = 168364.79 ms / 121534 tokens ( 1.39 ms per token, 721.85 tokens per second)
eval time = 7194.32 ms / 410 tokens ( 17.55 ms per token, 56.99 tokens per second)
total time = 175559.11 ms / 121944 tokens
slot release: id 2 | task 12770 | stop processing: n_tokens = 121943, truncated = 0
