The Mobius-12B-base-m1 Large Language Model (LLM) is a pretrained model based on RWKV v5 arch.
We utilized 0.01 billion tokens to conduct post-training on this model for alignment benchmarks, excluding the utilization of
DPO and SFT. The process took approximately 10 hours, employing 4 * a800.
This repo contains weights that are not compatible with Hugging Face
transformers library yet. But you can try this
PR as well.
RWKV runner or
AI00 server also work.
This format must be strictly respected, otherwise the model will generate sub-optimal outputs.
The template used to build a prompt for the Instruct model is defined as follows:
Need to install this
PR
pip install -e git://github.com/BBuf/transformers.git
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model = AutoModelForCausalLM.from_pretrained("TimeMobius/Mobius-12B-base-m1", torch_dtype=torch.float16).to(0)
5tokenizer = AutoTokenizer.from_pretrained("TimeMobius/Mobius-12B-base-m1", trust_remote_code=True)
6
7text = "x"
8prompt = f'Question: {text.strip()}\n\nAnswer:'
9
10inputs = tokenizer(prompt, return_tensors="pt").to(0)
11output = model.generate(inputs["input_ids"], max_new_tokens=40)
12print(tokenizer.decode(output[0].tolist(), skip_special_tokens=True))
The Mobius base m1 is the base model can be easily fine-tuned to achieve compelling performance.
if you wanna better benchmark results use
DPO and SFT ,details in
readme
