You can deploy this model and start earning money today!

1jamesthompson1/
Qwen3.6-27B-nz-wvs-first_token_distribution-overall

Name: Qwen3.6-27B-nz-wvs-first_token_distribution-overall
Availability: InStock

first_token_distribution

overall

text-generation

conversational

1jamesthompson1/wvs-nz-value-alignment

Qwen/Qwen3.6-27B

cc-by-sa-4.0

Views

No views yet

Qwen3.6-27B LoRA — First Token Distribution, Overall

This model is a LoRA fine-tune of Qwen/Qwen3.6-27B as part of the AIML589 project.

This adapter is licensed under CC BY-SA 4.0.

Dataset

Fine-tuned on the first_token_distribution config of the wvs-nz-value-alignment dataset, overall subpopulation.

Part of the wvs-nz-value-alignment collection.

GPU: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition · Training time: 52m 12s

Training hyperparameters

Parameter	Value
LoRA rank	16
LoRA alpha	32
LoRA dropout	0.05
DoRA	False
Learning rate	0.0002
Batch size	4
Gradient accumulation	4
Epochs	3
Max seq length	1024
Warmup ratio
Dtype	bf16

Training log

{"loss": 2.52270450592041, "grad_norm": 11.155061721801758, "learning_rate": 7.500000000000001e-05, "epoch": 0.12987012987012986, "step": 10}
{"eval_loss": 1.5031676292419434, "eval_runtime": 32.1599, "eval_samples_per_second": 8.582, "eval_steps_per_second": 2.146, "epoch": 0.12987012987012986, "step": 10}
{"loss": 1.4450578689575195, "grad_norm": 3.00559139251709, "learning_rate": 0.00015833333333333332, "epoch": 0.2597402597402597, "step": 20}
{"eval_loss": 1.4111301898956299, "eval_runtime": 31.9494, "eval_samples_per_second": 8.639, "eval_steps_per_second": 2.16, "epoch": 0.2597402597402597, "step": 20}
{"loss": 1.4065752983093263, "grad_norm": 7.892879962921143, "learning_rate": 0.00019516908212560387, "epoch": 0.38961038961038963, "step": 30}
{"eval_loss": 1.4626983404159546, "eval_runtime": 32.622, "eval_samples_per_second": 8.461, "eval_steps_per_second": 2.115, "epoch": 0.38961038961038963, "step": 30}
{"loss": 1.2934237480163575, "grad_norm": 0.977397620677948, "learning_rate": 0.0001855072463768116, "epoch": 0.5194805194805194, "step": 40}
{"eval_loss": 1.2637510299682617, "eval_runtime": 31.8341, "eval_samples_per_second": 8.67, "eval_steps_per_second": 2.167, "epoch": 0.5194805194805194, "step": 40}
{"loss": 1.2959080696105958, "grad_norm": 1.0415736436843872, "learning_rate": 0.00017584541062801935, "epoch": 0.6493506493506493, "step": 50}
{"eval_loss": 1.2328848838806152, "eval_runtime": 32.4209, "eval_samples_per_second": 8.513, "eval_steps_per_second": 2.128, "epoch": 0.6493506493506493, "step": 50}
{"loss": 1.2640339851379394, "grad_norm": 1.0579698085784912, "learning_rate": 0.00016618357487922707, "epoch": 0.7792207792207793, "step": 60}
{"eval_loss": 1.2323572635650635, "eval_runtime": 32.5646, "eval_samples_per_second": 8.475, "eval_steps_per_second": 2.119, "epoch": 0.7792207792207793, "step": 60}
{"loss": 1.2558379173278809, "grad_norm": 0.6798368096351624, "learning_rate": 0.0001565217391304348, "epoch": 0.9090909090909091, "step": 70}
{"eval_loss": 1.2007805109024048, "eval_runtime": 32.357, "eval_samples_per_second": 8.53, "eval_steps_per_second": 2.132, "epoch": 0.9090909090909091, "step": 70}
{"loss": 1.203580093383789, "grad_norm": 0.5169912576675415, "learning_rate": 0.00014685990338164252, "epoch": 1.0389610389610389, "step": 80}
{"eval_loss": 1.2004268169403076, "eval_runtime": 32.3433, "eval_samples_per_second": 8.533, "eval_steps_per_second": 2.133, "epoch": 1.0389610389610389, "step": 80}
{"loss": 1.1731551170349122, "grad_norm": 0.7591021656990051, "learning_rate": 0.00013719806763285024, "epoch": 1.1688311688311688, "step": 90}
{"eval_loss": 1.1893230676651, "eval_runtime": 32.8818, "eval_samples_per_second": 8.394, "eval_steps_per_second": 2.098, "epoch": 1.1688311688311688, "step": 90}
{"loss": 1.1729385375976562, "grad_norm": 0.429182231426239, "learning_rate": 0.00012753623188405797, "epoch": 1.2987012987012987, "step": 100}
{"eval_loss": 1.1763889789581299, "eval_runtime": 31.9679, "eval_samples_per_second": 8.634, "eval_steps_per_second": 2.158, "epoch": 1.2987012987012987, "step": 100}
{"loss": 1.1912422180175781, "grad_norm": 0.36074358224868774, "learning_rate": 0.00011787439613526569, "epoch": 1.4285714285714286, "step": 110}
{"eval_loss": 1.1839007139205933, "eval_runtime": 32.9135, "eval_samples_per_second": 8.386, "eval_steps_per_second": 2.096, "epoch": 1.4285714285714286, "step": 110}
{"loss": 1.12225341796875, "grad_norm": 0.3937525749206543, "learning_rate": 0.00010821256038647343, "epoch": 1.5584415584415585, "step": 120}
{"eval_loss": 1.1677358150482178, "eval_runtime": 32.3488, "eval_samples_per_second": 8.532, "eval_steps_per_second": 2.133, "epoch": 1.5584415584415585, "step": 120}
{"loss": 1.1528189659118653, "grad_norm": 0.2851729094982147, "learning_rate": 9.855072463768117e-05, "epoch": 1.6883116883116882, "step": 130}
{"eval_loss": 1.1612310409545898, "eval_runtime": 31.9451, "eval_samples_per_second": 8.64, "eval_steps_per_second": 2.16, "epoch": 1.6883116883116882, "step": 130}
{"loss": 1.222947597503662, "grad_norm": 0.24821361899375916, "learning_rate": 8.888888888888889e-05, "epoch": 1.8181818181818183, "step": 140}
{"eval_loss": 1.1577999591827393, "eval_runtime": 32.0849, "eval_samples_per_second": 8.602, "eval_steps_per_second": 2.151, "epoch": 1.8181818181818183, "step": 140}
{"loss": 1.1778755187988281, "grad_norm": 0.2734643816947937, "learning_rate": 7.922705314009662e-05, "epoch": 1.948051948051948, "step": 150}
{"eval_loss": 1.1580313444137573, "eval_runtime": 31.9434, "eval_samples_per_second": 8.64, "eval_steps_per_second": 2.16, "epoch": 1.948051948051948, "step": 150}
{"loss": 1.104105281829834, "grad_norm": 0.22014258801937103, "learning_rate": 6.956521739130436e-05, "epoch": 2.0779220779220777, "step": 160}
{"eval_loss": 1.1563979387283325, "eval_runtime": 31.9689, "eval_samples_per_second": 8.633, "eval_steps_per_second": 2.158, "epoch": 2.0779220779220777, "step": 160}
{"loss": 1.1487351417541505, "grad_norm": 0.16325421631336212, "learning_rate": 5.990338164251208e-05, "epoch": 2.207792207792208, "step": 170}
{"eval_loss": 1.1532561779022217, "eval_runtime": 32.1106, "eval_samples_per_second": 8.595, "eval_steps_per_second": 2.149, "epoch": 2.207792207792208, "step": 170}
{"loss": 1.1301685333251954, "grad_norm": 0.11060268431901932, "learning_rate": 5.024154589371981e-05, "epoch": 2.3376623376623376, "step": 180}
{"eval_loss": 1.1531744003295898, "eval_runtime": 32.1927, "eval_samples_per_second": 8.573, "eval_steps_per_second": 2.143, "epoch": 2.3376623376623376, "step": 180}
{"loss": 1.1581315040588378, "grad_norm": 0.09855422377586365, "learning_rate": 4.057971014492754e-05, "epoch": 2.4675324675324677, "step": 190}
{"eval_loss": 1.1524766683578491, "eval_runtime": 32.1678, "eval_samples_per_second": 8.58, "eval_steps_per_second": 2.145, "epoch": 2.4675324675324677, "step": 190}
{"loss": 1.1629021644592286, "grad_norm": 0.14986781775951385, "learning_rate": 3.0917874396135266e-05, "epoch": 2.5974025974025974, "step": 200}
{"eval_loss": 1.1509665250778198, "eval_runtime": 32.0243, "eval_samples_per_second": 8.618, "eval_steps_per_second": 2.155, "epoch": 2.5974025974025974, "step": 200}
{"loss": 1.157134437561035, "grad_norm": 0.14960630238056183, "learning_rate": 2.1256038647342997e-05, "epoch": 2.7272727272727275, "step": 210}
{"eval_loss": 1.1515204906463623, "eval_runtime": 32.2604, "eval_samples_per_second": 8.555, "eval_steps_per_second": 2.139, "epoch": 2.7272727272727275, "step": 210}
{"loss": 1.1381695747375489, "grad_norm": 0.16626720130443573, "learning_rate": 1.1594202898550725e-05, "epoch": 2.857142857142857, "step": 220}
{"eval_loss": 1.1508792638778687, "eval_runtime": 31.9132, "eval_samples_per_second": 8.648, "eval_steps_per_second": 2.162, "epoch": 2.857142857142857, "step": 220}
{"loss": 1.1570892333984375, "grad_norm": 0.1603062003850937, "learning_rate": 1.932367149758454e-06, "epoch": 2.987012987012987, "step": 230}
{"eval_loss": 1.1509487628936768, "eval_runtime": 32.1745, "eval_samples_per_second": 8.578, "eval_steps_per_second": 2.145, "epoch": 2.987012987012987, "step": 230}
{"eval_loss": 1.1512011289596558, "eval_runtime": 32.2741, "eval_samples_per_second": 8.552, "eval_steps_per_second": 2.138, "epoch": 3.0, "step": 231}
{"train_runtime": 3132.5209, "train_samples_per_second": 1.178, "train_steps_per_second": 0.074, "total_flos": 1.1323439712258048e+17, "train_loss": 1.263454377393186, "epoch": 3.0, "step": 231}

Environment

Package	Version
torch	2.13.0
transformers	5.15.0
trl	1.10.0
datasets	5.0.1
accelerate	1.14.0
python-dotenv	1.2.2
peft	0.20.0
bitsandbytes	0.50.0
huggingface-hub	?
jinja2	?
torchvision	0.28.0
pillow	12.3.0

Intended use

This adapter is intended for research purposes only as part of the AIML589 project, which investigates value alignment of LLMs with New Zealand population distributions from the World Values Survey.

Out-of-scope

This model has not been safety-tuned for general-purpose deployment. It should not be used in production systems, for making decisions about people, or in contexts where reliability and safety are critical.

Limitations and biases

Fine-tuned on a single WVS wave (Wave 7) for New Zealand only.
The training data reflects the values of those who responded to the survey and may not represent all New Zealanders.
LoRA adapters are subject to the limitations and biases of the base model (Qwen/Qwen3.6-27B).

1jamesthompson1/Qwen3.6-27B-nz-wvs-first_token_distribution-overall