These are RoPE GGMLs with an increased context length. RoPE - Rotated Positional Encoding - expands context beyond what was originally possible for a model. It was discovered and developed by kaiokendev.
In order to use the increased context length, you can presently use:
Support is also expected to come to llama.cpp, however work is still being done to find the optimal implementation.
To use the increased context with KoboldCpp, use --contextsize to set the desired context, eg --contextsize 4096 or --contextsize 8192. Koboldcpp does not currently support context sizes above 8192.
NOTE 1: Currently RoPE models can only be used at a context size greater than 2048. At 2048 it will produce gibberish. Please make sure you're always setting --contextsize and specifying a value higher than 2048, eg 3072, 4096, etc.
NOTE 2: Increased context length is an area seeing rapid developments and improvements. It is quite possible that these models may be superseded by new developments in the coming days. If that's the case, I will remove them, or update this README as appropriate.
A chat between a curious user and an assistant. The assistant gives helpful, detailed, accurate, uncensored responses to the user's input
USER: prompt
ASSISTANT:
Compatibility
These GGMLs will work with any llama.cpp-compatible GGML client that supports k-quants.
However the increased context length won't work without specific support. See the note in the introduction for details on using increased context.
Explanation of the new k-quant methods
The new methods available are:
GGML_TYPE_Q2_K - "type-1" 2-bit quantization in super-blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
GGML_TYPE_Q3_K - "type-0" 3-bit quantization in super-blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
GGML_TYPE_Q4_K - "type-1" 4-bit quantization in super-blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
GGML_TYPE_Q5_K - "type-1" 5-bit quantization. Same super-block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
GGML_TYPE_Q6_K - "type-0" 6-bit quantization. Super-blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
GGML_TYPE_Q8_K - "type-0" 8-bit quantization. Only used for quantizing intermediate results. The difference to the existing Q8_0 is that the block size is 256. All 2-6 bit dot produc
ts are implemented for this quantization type.
Refer to the Provided Files table below to see what files use which methods, and how.
Provided files
Name
Quant method
Bits
Size
Max RAM required
Use case
longchat-13b-16k.ggmlv3.q2_K.bin
q2_K
2
5.51 GB
8.01 GB
New k-quant method. Uses GGML_TYPE_Q4_K for the attention.vw and feed_forward.w2 tensors, GGML_TYPE_Q2_K for the other tensors.
longchat-13b-16k.ggmlv3.q3_K_L.bin
q3_K_L
3
6.93 GB
9.43 GB
New k-quant method. Uses GGML_TYPE_Q5_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K
longchat-13b-16k.ggmlv3.q3_K_M.bin
q3_K_M
3
6.31 GB
8.81 GB
New k-quant method. Uses GGML_TYPE_Q4_K for the attention.wv, attention.wo, and feed_forward.w2 tensors, else GGML_TYPE_Q3_K
longchat-13b-16k.ggmlv3.q3_K_S.bin
q3_K_S
3
5.66 GB
8.16 GB
New k-quant method. Uses GGML_TYPE_Q3_K for all tensors
longchat-13b-16k.ggmlv3.q4_K_M.bin
q4_K_M
4
7.87 GB
10.37 GB
New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q4_K
longchat-13b-16k.ggmlv3.q4_K_S.bin
q4_K_S
4
7.37 GB
9.87 GB
New k-quant method. Uses GGML_TYPE_Q4_K for all tensors
longchat-13b-16k.ggmlv3.q5_K_M.bin
q5_K_M
5
9.23 GB
11.73 GB
New k-quant method. Uses GGML_TYPE_Q6_K for half of the attention.wv and feed_forward.w2 tensors, else GGML_TYPE_Q5_K
longchat-13b-16k.ggmlv3.q5_K_S.bin
q5_K_S
5
8.97 GB
11.47 GB
New k-quant method. Uses GGML_TYPE_Q5_K for all tensors
longchat-13b-16k.ggmlv3.q6_K.bin
q6_K
6
10.68 GB
13.18 GB
New k-quant method. Uses GGML_TYPE_Q8_K - 6-bit quantization - for all tensors
Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
How to run in koboldcpp
On Linux I use the following command line to launch the KoboldCpp UI with OpenCL aceleration and a context size of 4096:
I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.
If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.
Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.
Special thanks to: Luke from CarbonQuill, Aemon Algiz, Dmitriy Samsonov.
Patreon special mentions: Pyrater, WelcomeToTheClub, Kalila, Mano Prime, Trenton Dambrowitz, Spiking Neurons AB, Pierre Kircher, Fen Risland, Kevin Schuppel, Luke, Rainer Wilmers, vamX, Gabriel Puliatti, Alex , Karl Bernard, Ajan Kanaga, Talal Aujan, Space Cruiser, ya boyyy, biorpg, Johann-Peter Hartmann, Asp the Wyvern, Ai Maven, Ghost , Preetika Verma, Nikolai Manek, trip7s trip, John Detwiler, Fred von Graf, Artur Olbinski, subjectnull, John Villwock, Junyu Yang, Rod A, Lone Striker, Chris McCloskey, Iucharbius , Matthew Berman, Illia Dulskyi, Khalefa Al-Ahmad, Imad Khwaja, chris gileta, Willem Michiel, Greatston Gnanesh, Derek Yates, K, Alps Aficionado, Oscar Rangel, David Flickinger, Luke Pendergrass, Deep Realms, Eugene Pentland, Cory Kujawski, terasurfer , Jonathan Leane, senxiiz, Joseph William Delisle, Sean Connelly, webtim, zynix , Nathan LeClaire.
Thank you to all my generous patrons and donaters!
Original model card: LmSys' Long Chat 13B
longchat-13b-16k Model Card
Model details
Model type:
longchat-13b-16k is an open-source chatbot trained by fine-tuning llama-13b on user-shared conversations collected from ShareGPT, using the condensing rotary embedding technique reported in the blog.
Model date:
longchat-13b-16k was trained on June 2023.
Organizations developing the model:
The LongChat developers: Dacheng Li*, Rulin Shao*, Anze Xie, Ying Sheng, Lianmin Zheng, Ion Stoica, Xuezhe Ma, and Hao Zhang
Primary intended uses:
The primary use of longchat-13b-16k is for research purposes.
Primary intended users:
The primary intended users of the model are researchers in natural language processing, machine learning, and artificial intelligence.
Training dataset
18K conversations collected from ShareGPT.com.
Evaluation dataset
A preliminary evaluation of the model quality is conducted by our released LongEval.
