This model is a fine-tuned version of
OuteAI/Lite-Oute-1-300M-Instruct on the
cardiffnlp/tweet_eval dataset to determine tweets tonality in one of the three classes: positive, neutral or negative.
It was finetuned with DoRA to make training more memory and time efficient. Low-rank finetuning was applied only to V and K matrices of attention layers.
This model was trained with batch_size=32, rank = 16, alpha = 32, learning_rate = 1e-5 on cardiffnlp/tweet_eval for three epochs.
The model achieved 0.53 f1-score on the test dataset.
Sorry bout the stream last night I crashed out but will be on tonight for sure. Then back to Minecraft in pc tomorrow night. -> "positive"
Sorry bout the stream last night I crashed out but will be on tonight for sure. Then back to Minecraft in pc tomorrow night. -> "neutral. GCSE English \n"
Although there are redundant tokens, the output is considered correct.
1
2from safetensors.torch import load_file
3from huggingface_hub import hf_hub_download
4
5
6model = AutoModelForCausalLM.from_pretrained(f"{REPO_NAME}-dora", device_map="auto")
7tokenizer = AutoTokenizer.from_pretrained(f"{REPO_NAME}-dora")
8tokenizer.pad_token = tokenizer.eos_token
9tokenizer.padding_side = "left"
10
11apply_peft_to_module(model, LinearWithDoRA, r=16, alpha=32, target_submodules=["v_proj", "k_proj"])
12model.to(DEVICE)
13
14path = hf_hub_download(f"{REPO_NAME}-dora", "model.safetensors")
15state_dict = load_file(path)
16
17model.load_state_dict(state_dict, strict=False)
18
19DoRA_saved_model_accuracy = eval(model, dataset["test"], tokenizer)
20print(f"Accuracy after DoRA training: {DoRA_saved_model_accuracy:.2f}")
