You can deploy this model and start earning money today!

Jeesup/
llama32-1B-sst2-nf4-lora-seed43

meta-llama/Llama-3.2-1B

adapter

Views

No views yet

llama32-1B-sst2-nf4-lora-seed43

LoRA adapter trained on GLUE SST2 on top of a nf4 backbone of meta-llama/Llama-3.2-1B.

Part of a controlled study of whether the backbone bit-width changes what a LoRA adapter learns. For a given (model size, seed) the adapter initialisation is identical across the bf16 / int8 / nf4 arms, and the data order, optimiser, schedule and LoRA hyperparameters are held fixed — so any difference in the learned update is attributable to the backbone.

Result

metric	validation	test
accuracy	0.9630	0.9644
macro-F1	0.9624	0.9644
loss	0.1486	0.1591

Test-set majority-class baseline: 0.5092

peak GPU memory: 2.53 GiB
training time: 27.6 min (936 steps)
GPU: NVIDIA GeForce RTX 4090

Setup

seed: 43 · adapter init: shared:lora_init_1B_seed43.pt:128tensors
LoRA: r=16, alpha=32, dropout=0.0, bias=none, target_modules=['q_proj', 'k_proj', 'v_proj', 'o_proj']
trainable params: 3,407,872
epochs 3, lr 0.0002, max_len 256, batch 4 x grad_accum 16, cosine schedule, warmup 0.03

Prompt format

Trained as causal LM with the loss on the answer letter only (prompt tokens masked to -100):

Sentence: ...

Is the sentiment of this sentence positive or negative?

A. Negative
B. Positive

Answer:

Evaluated by conditional likelihood over the answer letters (Negative, Positive).

GLUE test is unlabeled, so the official validation split is used as TEST and the validation set is carved from train (disjoint).

Jeesup/llama32-1B-sst2-nf4-lora-seed43

llama32-1B-sst2-nf4-lora-seed43

Result

Setup

Prompt format

Jeesup/
llama32-1B-sst2-nf4-lora-seed43