This SpaRTA adapter spacializes the
google/gemma-2b-it instruction-following model to do sentiment classification of English text sentences.
-
PEFT method: SpaRTA @ 99.8% sparsity
-
Base Model: google/gemma-2b-it
-
Task: Text Classification (Sentiment Analysis)
-
Language: English
-
Input:
Text string representing a sentence to be classified as having positive or negative sentiment,
wrapped within an instruction and formated with the model (google/gemma-2b-it) chat template as follows:
1
2input_template = ("<start_of_turn>user\n"
3 "Determine the sentiment of the following sentence about a movie. "
4 "The sentiment can only be classified as positive or negative.\n"
5 "Sentence: {sentence}"
6 "<end_of_turn>\n<start_of_turn>model\n"
7 "The sentiment of the sentence is")
8
9sentence = "I loved the movie. It was great."
10
11model_input = input_template.format(sentence=sentence)
12
13print(model_input)
14
1<start_of_turn>user
2Determine the sentiment of the following sentence about a movie. The sentiment can only be classified as positive or negative.
3Sentence: I loved the movie. It was great.<end_of_turn>
4<start_of_turn>model
5The sentiment of the sentence is
We neeed to use this input template since the adapted model was trained with it.
-
Output:
One of two tokens represeting the sentiment class of the input: with token id 0 for negative sentiment, and 1 for positive.
For instructions on how to load and use this adapter to classify input sentences, see
https://pypi.org/project/peft-sparta/.
The adapter was trained on the
SST-2 dataset, using a 99.8% sparsity, that is, freezing 99.8% of the Gemma-2B-IT model parameters and training only on the remaining 0.2%.
The trainable parameters were chosen randomly from the self-attention
value (Wv) and
output (Wo) projection metrices.
This resulted in a total of approx. 5 million trainable parameters.
We used a handcrafted instruction and the model (
google/gemma-2b-it) chat template to process the raw inputs (text sentences) in the training set.
See model input template for details.
Binary sentiment classification (positive/negative).
