Finetuned using axolotl, using the following configuration
1base_model: Qwen/Qwen2.5-VL-3B-Instruct
2processor_type: AutoProcessor
3
4# these 3 lines are needed for now to handle vision chat templates w images
5skip_prepare_dataset: true
6remove_unused_columns: false
7sample_packing: false
8
9chat_template: qwen2_vl
10datasets:
11 - path: e-zorzi/reasoning_distractors_choice_chat
12 type: chat_template
13 split: train
14 revision: small
15
16test_datasets:
17 - path: e-zorzi/reasoning_distractors_choice_chat
18 type: chat_template
19 split: val_seen[:10%]
20 revision: small
21 - path: e-zorzi/reasoning_distractors_choice_chat
22 type: chat_template
23 split: val_unseen[:10%]
24 revision: small
25
26load_in_8bit: True
27adapter: lora
28lora_model_dir:
29
30sequence_len: 2048 #8192
31pad_to_sequence_len: false
32
33lora_r: 128
34lora_alpha: 16
35lora_dropout: 0.05
36lora_target_modules: 'model.language_model.layers.[\d]+.(mlp|cross_attn|self_attn).(up|down|gate|q|k|v|o)_proj'
37
38gradient_accumulation_steps: 2
39micro_batch_size: 32
40num_epochs: 15
41optimizer: adamw_bnb_8bit
42lr_scheduler: cosine
43learning_rate: 0.001
44
45bf16: true
46fp16:
47tf32: true
48
49gradient_checkpointing: true
50logging_steps: 1
51flash_attention: true
52eager_attention:
53
54warmup_ratio: 0.0067
55evals_per_epoch: 3
56saves_per_epoch: 1
57save_strategy: epoch
58weight_decay: 0.0
59
60# save_first_step: true # uncomment this to validate checkpoint saving works with your config
Model Card for Model ID
This modelcard aims to be a base template for new models. It has been generated using
this raw template.
Model Details
Model Description
- Developed by: [More Information Needed]
- Funded by [optional]: [More Information Needed]
- Shared by [optional]: [More Information Needed]
- Model type: [More Information Needed]
- Language(s) (NLP): [More Information Needed]
- License: [More Information Needed]
- Finetuned from model [optional]: [More Information Needed]
Model Sources [optional]
- Repository: [More Information Needed]
- Paper [optional]: [More Information Needed]
- Demo [optional]: [More Information Needed]
Uses
Direct Use
[More Information Needed]
Downstream Use [optional]
[More Information Needed]
Out-of-Scope Use
[More Information Needed]
Bias, Risks, and Limitations
[More Information Needed]
Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
Training Details
Training Data
[More Information Needed]
Training Procedure
Preprocessing [optional]
[More Information Needed]
Training Hyperparameters
- Training regime: [More Information Needed]
Speeds, Sizes, Times [optional]
[More Information Needed]
Evaluation
Testing Data, Factors & Metrics
Testing Data
[More Information Needed]
Factors
[More Information Needed]
Metrics
[More Information Needed]
Results
[More Information Needed]
Summary
Model Examination [optional]
[More Information Needed]
Environmental Impact
Carbon emissions can be estimated using the
Machine Learning Impact calculator presented in
Lacoste et al. (2019).
- Hardware Type: [More Information Needed]
- Hours used: [More Information Needed]
- Cloud Provider: [More Information Needed]
- Compute Region: [More Information Needed]
- Carbon Emitted: [More Information Needed]
Technical Specifications [optional]
Model Architecture and Objective
[More Information Needed]
Compute Infrastructure
[More Information Needed]
Hardware
[More Information Needed]
Software
[More Information Needed]
Citation [optional]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
Glossary [optional]
[More Information Needed]
More Information [optional]
[More Information Needed]
Model Card Authors [optional]
[More Information Needed]
Model Card Contact
[More Information Needed]
