base_model: Qwen/Qwen3-0.6B

# Automatically upload checkpoint and final model to HF
hub_model_id: abdullahmeda/pointwise-rerank-qwen3-600m-ds0-pd74hs73h

load_in_8bit: false
load_in_4bit: false
strict: false

chat_template: qwen3
datasets:
  - path: kaggle-map/pointwise-reranker
    type: chat_template
    split: train

test_datasets:
  - path: kaggle-map/pointwise-reranker
    type: chat_template
    split: val

dataset_processes: 32
dataset_prepared_path: last_run_prepared
output_dir: ./outputs/pointwise-rerank-qwen3-600m-ds0-pd74hs73h

sequence_len: 1024
sample_packing: true
eval_sample_packing: false

deepspeed: deepspeed_configs/zero1.json

wandb_project: map-math-misconceptions
wandb_entity:
wandb_watch:
wandb_name: pointwise-rerank-qwen3-600m-ds0-pd74hs73h
wandb_log_model:

gradient_accumulation_steps: 8
micro_batch_size: 32
num_epochs: 5
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 1e-5

bf16: true
tf32: true

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
resume_from_checkpoint:
logging_steps: 10
flash_attention: true

warmup_ratio: 0.1
evals_per_epoch: 3
saves_per_epoch: 3
weight_decay: 0.01

save_first_step: true  # uncomment this to validate checkpoint saving works with your config

pointwise-rerank-qwen3-600m-ds0-pd74hs73h

This model is a fine-tuned version of Qwen/Qwen3-0.6B on the kaggle-map/pointwise-reranker dataset. It achieves the following results on the evaluation set:

Loss: 0.1314
Memory/max Mem Active(gib): 66.58
Memory/max Mem Allocated(gib): 66.58
Memory/device Mem Reserved(gib): 76.3

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 32
eval_batch_size: 32
seed: 42
distributed_type: multi-GPU
num_devices: 2
gradient_accumulation_steps: 8
total_train_batch_size: 512
total_eval_batch_size: 64
optimizer: Use adamw_torch_fused with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 90
training_steps: 905

Training results

Training Loss	Epoch	Step	Validation Loss	Mem Active(gib)	Mem Allocated(gib)	Mem Reserved(gib)
No log	0	0	6.2204	49.85	49.85	50.34
0.1499	0.3370	61	0.1324	66.58	66.58	75.46
0.0923	0.6740	122	0.0977	66.58	66.58	76.3
0.0663	1.0110	183	0.0745	66.58	66.58	76.3
0.0569	1.3481	244	0.0691	66.58	66.58	76.3
0.052	1.6851	305	0.0763	66.58	66.58	76.3
0.0367	2.0221	366	0.0821	66.58	66.58	76.3
0.028	2.3591	427	0.0806	66.58	66.58	76.3
0.0274	2.6961	488	0.1026	66.58	66.58	76.3
0.0174	3.0331	549	0.0988	66.58	66.58	76.3
0.017	3.3702	610	0.0999	66.58	66.58	76.3
0.0092	3.7072	671	0.1196	66.58	66.58	76.3
0.0074	4.0442	732	0.1223	66.58	66.58	76.3
0.0085	4.3812	793	0.1231	66.58	66.58	76.3
0.0033	4.7182	854	0.1314	66.58	66.58	76.3

Framework versions

Transformers 4.55.2
Pytorch 2.6.0+cu124
Datasets 4.0.0
Tokenizers 0.21.4