A PEFT LoRA adapter trained on 17,070 scientific chart questions, where 14.5% of the questions cannot be answered from the figure and the correct response is to say so.
It did not beat its base model. The head-to-head result is 50 wins to 50, a null result. This card explains why, using the weights themselves rather than a guess. The short version: the adapter never touched the vision tower, and a chart QA task lives in the vision tower.
Publishing a null result with the diagnosis attached is more useful than publishing nothing, so the weights, the dataset, the training state and the failure analysis are all here.
What it is: a rank-8 LoRA adapter on google/gemma-3-27b-it, 8,380,416 trainable parameters, 33.5 MB. Text-side attention only.
What it was for: answering questions about scientific figures, including the ones that cannot be answered, where "cannot be determined" is the correct answer.
The number: 50 wins for the adapted model, 50 for the base. No improvement. Held-out eval loss did fall, 1.699 to 1.427, which turned out not to matter.
The reason: the adapter has zero vision tensors. All 27 layers of the SigLIP vision tower and the multimodal projector are named in exclude_modules and were frozen. Only q_proj and v_proj in the 62 language-model decoder layers were trained.
Base model pointer, corrected:adapter_config.json was exported declaring togethercomputer/gemma-3-27b-it-VLM, which does not exist. I changed it to google/gemma-3-27b-it on 2026-08-13 and kept the original string verbatim in the same file. See Quickstart.
Built by: MANIFESTA (Aivaras Navardauskas) for the Adaption AutoScientist Challenge, Data Visualization.
Quickstart
The base model string was wrong. I fixed it, and the original is on the record
The export pipeline wrote this into adapter_config.json, and it shipped that way from 2026-08-06 to 2026-08-13:
That repository does not exist. It returns HTTP 401 to an anonymous client, which is what the Hub returns for a repo that is not there, and togethercomputer publishes no gemma-3 repo under any name. google/gemma-3-27b-it-VLM, which the original auto-generated card advertised, does not exist either. Nobody typed that string by hand. The pipeline invented it, and I did not catch it before publishing.
The consequence was real: anything resolving the base from the config, including AutoPeftModel...from_pretrained on the adapter alone, could not find a base to load.
No weight changed and nothing else in the config changed. The original exported value stays in the same file so the artifact remains auditable: you can see what shipped first, what it was changed to, and decide for yourself whether the change was right.
The real base is google/gemma-3-27b-it. The config.json shipped in this repo matches it exactly: 62 decoder layers, hidden size 5376, 32 attention heads, 16 key-value heads, head dim 128, vocab 262,208, and a 27-layer SigLIP tower at 896 px with patch size 14. The LoRA shapes agree too, q_proj projects to 4096 (32 x 128) and v_proj to 2048 (16 x 128).
Working load snippet
Name the base explicitly. Note that this is a vision language model, so use the image-text-to-text auto class, not AutoModelForCausalLM.
python
1import torch
2from transformers import AutoModelForImageTextToText, AutoProcessor
3from peft import PeftModel
45BASE ="google/gemma-3-27b-it"# gated, request access on the Hub first6ADAPTER ="manifesta/adaption_scientific_chart_qa_17k"78processor = AutoProcessor.from_pretrained(BASE)9base = AutoModelForImageTextToText.from_pretrained(10 BASE, torch_dtype=torch.bfloat16, device_map="auto"11)12model = PeftModel.from_pretrained(base, ADAPTER)13model.eval()1415messages =[{16"role":"user",17"content":[18{"type":"image","url":"https://upload.wikimedia.org/wikipedia/commons/a/a4/Line_chart_example.png"},19{"type":"text","text":"What is the value of the blue series at x = 4? If the figure does not show it, say 'cannot be determined'."},20],21}]2223inputs = processor.apply_chat_template(24 messages, add_generation_prompt=True, tokenize=True,25 return_dict=True, return_tensors="pt",26).to(model.device)2728with torch.inference_mode():29 out = model.generate(**inputs, max_new_tokens=256, do_sample=False)30print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Naming the base explicitly is still the snippet I would use, because google/gemma-3-27b-it is gated and you need to accept the licence on the Hub before anything can pull it.
Merging is optional and only touches the language model:
model = model.merge_and_unload()
What is actually in these weights
Read straight out of adapter_model.safetensors and adapter_config.json. You can reproduce every row of this table from the repo.
Property
Value
PEFT type
LoRA, task_type: CAUSAL_LM
Rank r
8
lora_alpha
8 (so the scaling factor alpha/r is exactly 1.0)
lora_dropout
0.0
bias
none
use_rslora / use_dora
false / false
Target modules
q_proj, v_proj
Tensors in the file
248
Decoder layers touched
62 of 62 (every layer, both modules, no gaps)
Vision tensors
0
Trainable parameters
8,380,416 (float32, 33.5 MB)
Share of the 27B base
roughly 0.031%
exclude_modules
334 entries, all of them vision
Every tensor key in the file starts with base_model.model.model.language_model.layers.. Not one starts with vision_tower, multi_modal_projector, or anything else on the image path.
The exclusion is explicit rather than accidental. exclude_modules contains 334 module names and every single one is part of the image pathway:
all 27 SigLIP encoder layers, each with self_attn.{q,k,v,out}_proj, mlp.fc1, mlp.fc2, both layer norms and the activation
So the vision tower and the projector that feeds image tokens into the language model were both frozen by configuration, before a single gradient was computed.
Evaluation
Judged by Adaption's AutoScientist evaluator on a held-out split the adapter never saw, against the same base model it was trained from. Head-to-head, blind.
Metric
Base
Adapted
Change
Win rate
50
50
0
Held-out eval loss (first eval)
1.6992
Held-out eval loss (final)
1.4272
-0.272
Training loss
1.871 (step 1)
1.420 (step 34)
-0.451
Eval loss across the five checkpoints: 1.6992, 1.5371, 1.4683, 1.4370, 1.4272. Monotonic, textbook, and it bought nothing.
Win rates, adapted versus base
A 50 to 50 split is a tie, and a tie against your own base model is a null result. The adapter did not make Gemma 3 27B better at reading scientific figures. It also did not make it worse. Loss went down the whole way, which is exactly the trap: a clean loss curve tells you the model learned the answer format in the training data, not that it learned to see.
Why the null result
The task and the trained parameters do not overlap.
Answering a question about a scientific figure means doing four things: locating an axis and reading its scale, matching a plotted series to a legend entry, interpolating a value at a point, and deciding whether the thing being asked about is present in the figure at all. All four are perception. All four depend on the SigLIP tower and on the multimodal projector that turns 256 image tokens into something the decoder can attend over.
Neither of those was trained. What was trained is q_proj and v_proj inside the text decoder, which can change how the model phrases an answer, how it structures a derivation, and how readily it says "cannot be determined". It cannot change what the model sees.
So the honest reading is that this run had a ceiling built into it from the start. The dataset was not the bottleneck, and neither, mostly, was the step count. The trainable surface was.
A secondary factor worth naming: the run took 34 optimizer steps. With 16,684 rows in the training split and batch_size: "max", that is roughly 491 rows per optimizer step. Adding data widened the batch instead of adding steps, which is the subject of the next section.
What would need to change for a real result: put LoRA on the vision tower and the multimodal projector, or at minimum on multi_modal_projector, and force more optimizer steps by capping batch size. Until the image path is trainable, more chart data is not going to move the number.
The cross-run finding
Three completed AutoScientist runs, three different base models. Win rate tracked how weak the base already was on the domain, and did not track dataset size.
Run
Base
Params
Rows ingested
Steps
Rows per step
Win rate (adapted vs base)
Chart QA, first build
Qwen3.5 9B
9B
6,976
21
332
51 vs 49
Chart QA 17k (this model)
Gemma 3 27B
27B
17,070
34
502
50 vs 50
Verified math and code
Gemma 4 31B
31B
17,586
59
298
46 vs 54
Two things fall out of that table.
The bigger the base, the smaller the win. 9B scored 51, 27B scored 50, 31B scored 46. A stronger base has less room above it, and a rank-8 adapter on two projections is not enough to add capability the base does not already have. It can only redistribute what is there.
Scaling the corpus 2.4x did nothing. The chart corpus went from 6,976 rows to 17,070 rows, and the win rate went from 51 to 50. Because batch_size is set to max, the extra rows were absorbed into wider batches: step count only went 21 to 34 while rows per step went 332 to 502. More data per step is not more learning, it is a smoother gradient estimate for the same small number of updates.
Base model size against win rate across three runs
Caveat, stated plainly: this is three data points from three different base models, three different corpora and three different domains. Base size, base family and task difficulty all move together here. It is not a controlled experiment and it should not be read as one. It is a pattern worth testing properly, which would mean holding the corpus fixed and varying only the base.
Training
Trained with Adaption AutoScientist, SFT with LoRA, on Adaptive Data output.
Hyperparameter
Value
Base model
google/gemma-3-27b-it, declared correctly in adapter_config.json since 2026-08-13. The original export value togethercomputer/gemma-3-27b-it-VLM is preserved in the same file as original_base_model_name_or_path
Method
SFT, LoRA (PEFT), chat format
LoRA rank / alpha / dropout
8 / 8 / 0.0
Trainable modules
q_proj, v_proj
Learning rate
1e-4
Scheduler
cosine, 0.5 cycles, warmup ratio 0.1, min LR ratio 0.1
Weight decay
0
Max gradient norm
2
Epochs
1
Batch size
max (per-device train batch size 1 with accumulation)
Optimizer steps
34
Evaluations
5, every 6 steps
Train on inputs
false
Precision
bfloat16 base, float32 adapter weights
Total FLOPs
1.389e18
Adaption job ID
de678a1e-4280-4a33-954e-8096e018f987
Training experiment ID
d2500abf-4185-4437-bc6d-3a1f85e8e02d
Adaption dataset ID
3f347c8b-5724-4f96-9417-251623d8aaa5
Training metrics
Numerical stability
This run was clean. Gradient norm peaked at 0.72 against a clipping threshold of 2, and not one of the 34 steps was clipped. Range across the run was 0.243 to 0.720.
That matters because it rules out the easy explanation. Nothing here blew up. The optimizer did what it was told and the loss came down smoothly. The null result is not a broken run, it is a run that trained the wrong parameters. Compare with the math and code adapter, where 29 of 59 steps exceeded the clip threshold and the peak norm was 677.
(The Adaption run dashboard reports the peak as 0.79. The value above, 0.72, is the maximum in trainer_state.json in this repo, which is the number you can check yourself.)
Intended use and limits
Use it for
Reproducing and inspecting this null result. That is the honest primary use.
Studying what a language-side-only LoRA does to a vision language model's answer behaviour, particularly refusal phrasing.
A starting point for a run that actually adapts the vision tower.
Research on chart question answering and on unanswerable-question handling.
Do not use it for
Anything that assumes it reads figures better than stock Gemma 3 27B. It does not, and the evaluation says so.
Extracting numbers from figures in medical, financial, safety or regulatory work. A misread axis is a wrong number delivered confidently.
Presenting model-read values from a chart as measured data.
Any use that violates the Gemma Terms of Use or the Gemma Prohibited Use Policy, which apply through the base model.
Technical limits
Vision tower and multimodal projector are frozen and untrained. This is the headline limitation.
Rank 8 on two projections, roughly 0.031% of base parameters, one epoch, 34 optimizer steps. A very light touch.
English only.
The declared base string was wrong until 2026-08-13. It resolves now, and the original is recorded in the config.
Adapter weights are float32 while the base is bfloat16. PEFT casts on load, but expect the memory footprint of the adapter to be double what a bf16 export would be.
One judge, one held-out split. Reported as-is, no reruns, no best-of selection.
14.5% of questions are unanswerable by design, where the correct output is a refusal
58.2% are exactly gradable against a known numeric answer
6,426 real figures (CC-BY 6,220, CC-BY-SA 164, CC0 42) plus 10,644 synthetic figures
Adaptive Data lifted the corpus quality before training:
Adaptive Data metric
Before
After
Quality score
6.0
7.1 (+18.3%)
Grade
C
B
Percentile
8.2
9.3
Task mix across 17,070 questions
Worth separating two numbers that are easy to conflate: +18.3% is the improvement in the data, not in the model. The data got better. The model did not. Those are different measurements and only one of them is the headline.
Live interface
Ground Truth runs at manifestavisual.adaptionlabs.app. Drop a figure in, ask a question, and use the "Try to trick it" button to ask for a series that was never plotted.
Ground Truth correctly refusing a question about a series that is not in the figure
The green "Correctly refused" card above is the behaviour the dataset was built around: asked about a series called Zephyr-net that does not appear in the grouped bar chart, the answer is "cannot be determined" rather than a confident invention.
Related models
manifesta/scientific-chart-qa-lora-qwen3.5-9b, the same task on a 9B base with the first 6,976-row build. 51 vs 49, 21 optimizer steps. Its LoRA landed on only 8 of 32 decoder layers (indices 3, 7, 11, 15, 19, 23, 27, 31) and, like this one, on zero vision tensors. Its adapter_config.json carried the same export defect, togethercomputer/Qwen3.5-9B, and was corrected to Qwen/Qwen3.5-9B on the same day with the original preserved the same way.
Standard library only, no install step and no account. 16 checks, and the same 16 run on a daily
schedule in GitHub Actions, so the badge above goes red if any claim on this card stops being true.
The tensor facts in this card are among the checks: layer coverage is re-derived from the published
adapter_model.safetensors by name, not copied from the config.
Adapter weights: released under the Gemma Terms of Use. Derivatives of Gemma inherit Gemma's terms, so the adapter travels with the same conditions as the base.
Base model:google/gemma-3-27b-it is gated. Accept the licence and request access on the Hub before loading. The Gemma Prohibited Use Policy applies.
Training dataset:manifesta/scientific-chart-qa-17k. Source figures carry their own licences (CC-BY, CC-BY-SA, CC0) and are attributed per row in the dataset.
Citation
bibtex
1@misc{groundtruth_chartqa_2026,
2 title = {Ground Truth: a scientific chart QA adapter for Gemma 3 27B, and why it produced a null result},
3 author = {Navardauskas, Aivaras},
4 year = {2026},
5 note = {MANIFESTA. Adapted with Adaptive Data by Adaption, AutoScientist Challenge, Data Visualization. Win rate 50 vs 50. Vision tower frozen.},
6 howpublished = {\url{https://huggingface.co/manifesta/adaption_scientific_chart_qa_17k}}
7}
Built with Adaptive Data by Adaption. Platform documentation at docs.adaptionlabs.ai. Dataset, adapter and analysis by MANIFESTA (Aivaras Navardauskas) for the AutoScientist Challenge.
