A longer 1-epoch local run was attempted, but Apple MPS training became very slow and was interrupted after 112 optimizer steps. That interrupted run showed loss decreasing from about 2.423 to 2.125, but it did not reach a checkpoint save. The saved adapter in this repo is therefore the completed 20-step smoke run.

Base Model

Base model: Qwen/Qwen3-0.6B
Adapter method: LoRA / PEFT
Training library: TRL SFTTrainer

Training Data

Dataset source:

yaobojun/arxiv-ml-paper-intelligence

Local SFT task:

work/arxiv_dataset/processed/tasks/paper_profile_sft

SFT split sizes:

split	rows
train	4,955
validation	4,827
test	9,677

The target JSON contains fields such as:

primary_category
paper_type
topic_labels
method_keywords
task_keywords
mentioned_models
mentioned_datasets
mentioned_benchmarks

Training Settings

Smoke run settings:

setting	value
max_steps	20
max_length	512
per_device_train_batch_size	1
gradient_accumulation_steps	8
LoRA rank	8
LoRA alpha	16
learning rate	2e-4
evaluation during training	disabled
device	Apple MPS

Final smoke run training metrics:

metric	value
train_loss	2.9
train_runtime_seconds	69.62
train_steps_per_second	0.287

Smoke Evaluation

A small validation generation check was run on 5 examples.

metric	value
total	5
parsed JSON outputs	3
JSON parse rate	0.60
primary category accuracy	0.00
paper type accuracy	0.00

The eval outputs are included in:

text

eval/metrics.json
eval/predictions.jsonl

This low score is expected for a 20-step smoke adapter. The purpose of this artifact is to prove the end-to-end SFT pipeline works: data construction, chat-template rendering, LoRA training, adapter saving, generation, evaluation, and Hub upload.

Local Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = "Qwen/Qwen3-0.6B"
adapter = "yaobojun/qwen3-0.6b-arxiv-paper-profile-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter)

Recommended Next Run

For a better adapter, run a shorter controlled training job that saves frequently, for example:

bash

PYTORCH_ENABLE_MPS_FALLBACK=1 uv run arxiv-lab-train-lora-sft \
  --max-steps 100 \
  --max-length 512 \
  --gradient-accumulation-steps 1 \
  --save-steps 50 \
  --no-do-eval \
  --output-dir work/arxiv_dataset/models/qwen3_0_6b_paper_profile_lora_100step

For a proper training run, prefer a CUDA GPU or Hugging Face Jobs.