You can deploy this model and start earning money today!

kaiokendev/
superhot-13b-16k-no-rlhf-test

Name: superhot-13b-16k-no-rlhf-test
Availability: InStock

mit

Views

No views yet

SuperHOT Prototype 2 w/ 16K Context

This is a second prototype of SuperHOT, a NSFW focused LoRA, this time with 16K context and no RLHF, using the same technique described in the github blog. Tests have shown that the model does indeed leverage the extended context at 8K, so naturally, let's try going even further.

Looking for Merged & Quantized Models?

13B 16K GGML: tmpupload/superhot-13b-16k-no-rlhf-test-GGML
13B 16K CUDA (no groupsize): tmpupload/superhot-13b-16k-no-rlhf-test-GPTQ

Using the monkey-patch?

You will need to use either the monkeypatch or, if you are already using the monkeypatch, change the scaling factor to 0.125 and the maximum sequence length to 16384

Using Oobabooga with Exllama?

python server.py --max_seq_len 16384 --compress_pos_emb 8 --loader exllama_hf

I trained the LoRA with the following configuration:

1200 samples (~400 samples over 2048 sequence length)
learning rate of 3e-4
3 epochs
The exported modules are:
- q_proj
- k_proj
- v_proj
- o_proj
- no bias
Rank = 4
Alpha = 8
no dropout
weight decay of 0.1
AdamW beta1 of 0.9 and beta2 0.99, epsilon of 1e-5
Trained on 4-bit base model
Cutoff of 4096

kaiokendev/superhot-13b-16k-no-rlhf-test

SuperHOT Prototype 2 w/ 16K Context

Looking for Merged & Quantized Models?

Using the monkey-patch?

Using Oobabooga with Exllama?

kaiokendev/
superhot-13b-16k-no-rlhf-test