Views
No views yet
1from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
2
3model_name = "mia-project-2025/T5-base-adapter-natural-questions-shortQA"
4tokenizer = AutoTokenizer.from_pretrained(model_name)
5model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
6
7question = "Who wrote the novel 1984?"
8context = "1984 is a novel written by George Orwell in 1949."
9
10inputs = tokenizer(
11 f"question: {question} context: {context}",
12 return_tensors="pt",
13 truncation=True,
14 padding=True,
15)
16
17outputs = model.generate(**inputs, max_length=64)
18answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
19print("Answer:", answer)
20
21---
22
23
24
25
26Training Details
27
28Dataset: lucadiliello/naturalquestionsshortqa
29
30Split: 80% train / 20% validation
31
32Hyperparameters
33
34Learning rate: 5e-5
35
36Batch size: 16 (per device)
37
38Gradient accumulation: 2
39
40Epochs: 10
41
42Optimizer: adamw_torch_fused
43
44Mixed precision: fp16 / bf16 (hardware dependent)
45
46Max input length: 512
47
48Max target length: 128
49
50AdaLoRA Config
51
52r = 8
53
54lora_alpha = 64
55
56target_modules = ["q", "v"]
57
58lora_dropout = 0.05
59
60Orthogonal regularization = 0.5
61
62Dynamic rank allocation enabled
63
64
65
66---
67
68## Evaluation Results
69
70The model was evaluated on the NaturalQuestionsShortQA validation set.
71
72Final Evaluation (Epoch 10)
73Metric Score
74Eval Loss 0.374
75Exact Match (EM) 51.70%
76F1 Score 64.52%
77Eval Runtime 2559.84s
78Samples per Second 8.13
79Steps per Second 0.127
80Training Summary
81Metric Value
82Train Loss 0.611
83Train Runtime 36,367.54s (~10 hours on 1 GPU)
84Samples per Second 22.89
85Steps per Second 0.179
86