Views
No views yet
RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). It's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
convert_rwkv_checkpoint_to_hf.py script by specifying the repo_id of the original weights, the filename and the output directory. You can also optionally directly push the converted model on the Hub by passing --push_to_hub flag and --model_name argument to specify where to push the converted weights.python convert_rwkv_checkpoint_to_hf.py --repo_id RAW_HUB_REPO --checkpoint_file RAW_FILE --output_dir OUTPUT_DIR --push_to_hub --model_name dummy_user/converted-rwkvAutoModelForCausalLM and AutoTokenizer classes to generate texts from the model. Expand the sections below to understand how to run the model in different scenarios:
The "Raven" models needs to be prompted in a specific way, learn more about that in the integration blogpost.1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5")
4tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")
5
6prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."
7
8inputs = tokenizer(prompt, return_tensors="pt")
9output = model.generate(inputs["input_ids"], max_new_tokens=40)
10print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))1from transformers import AutoModelForCausalLM, AutoTokenizer
2
3model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5").to(0)
4tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")
5
6prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."
7
8inputs = tokenizer(prompt, return_tensors="pt").to(0)
9output = model.generate(inputs["input_ids"], max_new_tokens=40)
10print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5", torch_dtype=torch.float16).to(0)
5tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")
6
7prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."
8
9inputs = tokenizer(prompt, return_tensors="pt").to(0)
10output = model.generate(inputs["input_ids"], max_new_tokens=40)
11print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))1# pip install accelerate
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model = AutoModelForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5", device_map="auto")
5tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")
6
7prompt = "\nIn a shocking finding, scientist discovered a herd of dragons living in a remote, previously unexplored valley, in Tibet. Even more surprising to the researchers was the fact that the dragons spoke perfect Chinese."
8
9inputs = tokenizer(prompt, return_tensors="pt").to(0)
10output = model.generate(inputs["input_ids"], max_new_tokens=40)
11print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))