wandb: - 0.003 MB of 0.003 MB uploaded
wandb: \ 0.003 MB of 0.003 MB uploaded
wandb:                                                                                
wandb: 
wandb: Run history:
wandb:               eval/loss █▅▅▄▃▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
wandb:            eval/runtime ▆▃▃▁▂▅█▅▄▄▄▄▄▄▅▄▆▄▂▅▃▄▆▆▂▆▅
wandb: eval/samples_per_second ▃▆▆█▇▄▁▄▅▅▅▅▅▅▄▅▃▅▇▄▆▅▃▃▇▃▄
wandb:   eval/steps_per_second ▃▆▆█▇▄▁▄▄▄▅▅▄▅▄▅▃▆▇▃▆▅▃▃▇▃▄
wandb:             train/epoch ▁▁▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▅▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇███
wandb:       train/global_step ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▇▇▇▇▇███
wandb:         train/grad_norm ▇▂▂▂▁▃█▃▂▁▃▂▃▂▁▂▃▃▄▃▂▃▃▃▄▂▃▃▄▃▂▃▃▃▃▄▄▅▄▃
wandb:     train/learning_rate ███▇▇▇▇▇▇▆▆▆▆▆▅▅▅▅▅▅▄▄▄▄▄▄▃▃▃▃▃▂▂▂▂▂▂▁▁▁
wandb:              train/loss █▃▃▃▁▄▂▂▃▁▃▂▂▂▁▃▂▂▁▃▂▂▂▁▃▂▂▂▁▃▂▂▂▃▁▂▂▁▃▁
wandb: 
wandb: Run summary:
wandb:                eval/loss 0.92221
wandb:             eval/runtime 93.6611
wandb:  eval/samples_per_second 3.587
wandb:    eval/steps_per_second 1.196
wandb:               total_flos 2.952274602780672e+16
wandb:              train/epoch 2.46201
wandb:        train/global_step 810
wandb:          train/grad_norm 0.81067
wandb:      train/learning_rate 3e-05
wandb:               train/loss 0.7747
wandb:               train_loss 1.05936
wandb:            train_runtime 8326.639
wandb: train_samples_per_second 1.58
wandb:   train_steps_per_second 0.198

training_arguments = SFTConfig(
    output_dir=new_model,
    run_name="fine_tune_ocr_correction",
    per_device_train_batch_size=4,
    per_device_eval_batch_size=3,
    gradient_accumulation_steps=4,
    optim="paged_adamw_32bit",
    num_train_epochs=5, 
    eval_strategy="steps",
    eval_steps=30,  # normally 10 steps, but our dataset is small
    save_steps=30,
    logging_steps=20,  # Log progress every 20 steps
    warmup_steps=10,
    logging_strategy="steps",
    learning_rate=5e-5,
    fp16=use_fp16, 
    bf16=use_bf16,  
    group_by_length=True,
    report_to="wandb",
    max_seq_length=1220,
    save_strategy="steps",
    dataset_text_field="text",
    load_best_model_at_end = True
)

Llama-3.2-post-ocr-synthetic-data-2 (test on synth data, training on full corpus).json
Average PCIS: -0.06044562
Average Dataset CER: 0.09836092
Average Model CER: 0.15258657
Average Dataset WER: 0.21986217
Average Model WER: 0.80281940

Llama-3.2-post-ocr-synthetic-data-2 (test on real data, training on full corpus).json
Average PCIS: -0.00842250
Average Dataset CER: 0.01391665
Average Model CER: 0.02204702
Average Dataset WER: 0.06207812
Average Model WER: 0.10260357



Dataset complet