This repo contains the GGUF and quantized version of
talkie-1930-13b-yarn-32k-tf It is based off of the base version of Talkie-1930, and therefore is not meant for actual chat conversations.
It is recommended to use llama.cpp to load this model.
ffn5_0 refer to those files having their ffn_down forced to be q5_0 rather than letting the usually determined 5_1, this has to do with the fact that Talkie's ffn_down is not divisble by 256 meaning the base llama-quantize chooses 5_1, although when I first quantized with 5_1, the models output were completely broken; repeating the same character over and over rather than generating text, irregardless of input. I am pretty sure the fact the ffn_down cannot be divided by 256 is just a architectural design decision made by the upstream talkie-1930-base model.
i have found this issue with the talkie-1930-13b-it-hf-GGUF models as well? Not necessarily the q5_1 issue, as I haven't verified it yet. But the exact mode of failure with a repeating '>' token or other control token with their Q5_K_M & K_S models.
