Talkie 1930 13b Yarn32k Tf GGUF

--- license: apache-2.0 language: - en base_model: - xlr8harder/talkie-1930-13b-yarn-32k-tf pipeline_tag: text-generation tags: - talkie --- This repo contains the GGUF and quantized version of [talkie-1930-13b-yarn-32k-tf](https://huggingface.co/xlr8harder/talkie-1930-13b-yarn-32k-tf) It is based off of the base version of Talkie-1930, and therefore is not meant for actual chat conversations. It is recommended to use llama.cpp to load this model. Included files: | Quant File | Size | |------------|------| | talkie-1930-13b-yarn32k-Q2_K.gguf | 5.1G | | talkie-1930-13b-yarn32k-Q3_K_S.gguf | 5.8G | | talkie-1930-13b-yarn32k-Q3_K_M.gguf | 6.4G | | talkie-1930-13b-yarn32k-Q4_K_S.gguf | 7.4G | | talkie-1930-13b-yarn32k-Q4_K_M.gguf | 8.0G | | talkie-1930-13b-yarn32k-Q5_K_S-ffn5_0.gguf | 9.2G | | talkie-1930-13b-yarn32k-Q5_K_M-ffn5_0.gguf | 9.2G | | talkie-1930-13b-yarn32k-Q6_K.gguf | 11G | | talkie-1930-13b-yarn32k-Q8_0.gguf | 14G | | talkie-1930-13b-yarn32k-f16.gguf | 25G | Note: ffn5_0 refer to those files having their ffn_down forced to be q5_0 rather than letting the usually determined 5_1, this has to do with the fact that Talkie's ffn_down is not divisble by 256 meaning the base llama-quantize chooses 5_1, although when I first quantized with 5_1, the models output were completely broken; repeating the same character over and over rather than generating text, irregardless of input. I am pretty sure the fact the ffn_down cannot be divided by 256 is just a architectural design decision made by the upstream talkie-1930-base model. i have found this issue with the talkie-1930-13b-it-hf-GGUF models as well? Not necessarily the q5_1 issue, as I haven't verified it yet. But the exact mode of failure with a repeating '>' token or other control token with their Q5_K_M & K_S models.

Source context: 134 downloads · 0 likes · Pipeline text-generation · Repo Raydev/talkie-1930-13b-yarn32k-tf-GGUF