Qwen2.5-7B-GrapheneContact-QDoRA-CoT
A domain-specific language model fine-tuned for condensed matter physics, specifically graphene–metal contact resistance research. This 7B model, trained on only 64 question–answer pairs, outperforms a 235B-parameter model with open-book access under closed-book evaluation conditions.
Key Result
| Model | Parameters | Condition | Correct (✓) | Partial (△) | Wrong (✗) | Accuracy |
|---|
| Base 7B | 7B | Open-book | 9 | 17 | 4 | 30% |
| FT 7B (this model) | 7B | Closed-book | 22 | 5 | 3 | 73% |
| Qwen3-235B | 235B | Open-book | 18 | 9 | 3 | 60% |
A fine-tuned 7B model with no access to the source paper outperforms a 33× larger model that receives the paper's content as a prompt.
Model Details
Model Description
- Developed by: Junsu Park
- Affiliation: Department of Physics and Photon Science, Gwangju Institute of Science and Technology (GIST), Korea
- Base model: Qwen/Qwen2.5-7B-Instruct
- Fine-tuning method: QDoRA (Quantized Weight-Decomposed Low-Rank Adaptation) + Chain-of-Thought reasoning
- Language: English
- License: Apache 2.0
- Domain: Condensed matter physics — graphene–metal contact resistance (TLCD/TECD devices)
Model Sources
- Repository: GitHub (coming soon)
- Research Article: "Small Data, Small Model, Superior Domain Accuracy: Fine-Tuning a 7B Language Model as a Condensed Matter Physics Expert with 64 Examples"(https://www.preprints.org/manuscript/202603.1691)
Uses
Direct Use
This model is designed to answer domain-specific questions about graphene–metal contact resistance, including device structure (TLCD/TECD), fabrication processes, theoretical models (Landauer, BTH), and experimental results. It serves as a personalized research assistant for this specific domain.
Out-of-Scope Use
This model is trained on a single research paper and should not be used for general scientific reasoning, other physics domains, or any task outside graphene–metal contact resistance research.
Training Details
Training Data
64 expert-crafted question–answer pairs from a single condensed matter physics paper, organized into five categories:
| Category | Description |
|---|
| Structure / Geometry | TLCD vs TECD device structure, contact dimensions |
| Theory / Mechanism | Landauer model, BTH model, current crowding |
| Numerical / Experimental | Contact resistance values, TLM measurements |
| Fabrication | Metal-on-bottom process, Cr/Au deposition |
| Novelty / Contribution | First experimental verification of area-dependent Rc |
Each answer includes Chain-of-Thought (CoT) reasoning with <think> and <final> tags.
Training Hyperparameters
| Parameter | Value |
|---|
| Adapter | QDoRA (use_dora=True) |
| Rank (r) | 64 |
| Alpha (α) | 128 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Quantization | 4-bit NF4, double quantization, bfloat16 compute |
| Max steps | 80 |
| Learning rate | 5×10⁻⁴ |
| Batch size | 1 (gradient accumulation = 8, effective batch = 8) |
| Max sequence length | 2048 |
| Optimizer | paged_adamw_8bit |
| Training time | ~20 minutes |
| Training regime | bf16 mixed precision |
Hardware
| Component | Specification |
|---|
| GPU | NVIDIA RTX 5090 (32 GB VRAM) |
| CPU | AMD Threadripper PRO 5975WX (32 cores) |
| RAM | 256 GB DDR4-3200 ECC |
| OS | Ubuntu |
Evaluation
30 expert-designed questions (distinct from training set) evaluated by the domain expert (first author of the source paper) using a three-tier rubric: ✓ (correct), △ (partial), ✗ (wrong).
Results by Category
| Category | Questions | Base 7B | FT 7B | 235B |
|---|
| Structure / Geometry | 4 | 1 | 3 | 3 |
| Theory / Mechanism | 10 | 4 | 8 | 7 |
| Numerical / Experimental | 6 | 2 | 5 | 4 |
| Fabrication | 6 | 1 | 4 | 2 |
| Novelty / Contribution | 4 | 1 | 2 | 2 |
Error Pattern Analysis
Each model exhibits qualitatively distinct failure modes:
- Base 7B: Dominated by partial coverage (PC) — generates relevant but insufficiently specific responses
- 235B: Dominated by over-extrapolation (OE) — fills knowledge gaps with plausible but incorrect scientific details
- FT 7B (this model): Occasional directional reversal (DR) — correctly identifies relevant concepts but sometimes inverts their relationships
How to Get Started with the Model
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3
4# Load base model
5base_model = AutoModelForCausalLM.from_pretrained(
6 "Qwen/Qwen2.5-7B-Instruct",
7 torch_dtype="auto",
8 device_map="auto"
9)
10tokenizer = AutoTokenizer.from_pretrained(
11 "pjspjs0987/Qwen2.5-7B-GrapheneContact-QDoRA-CoT"
12)
13
14# Load fine-tuned adapter
15model = PeftModel.from_pretrained(
16 base_model,
17 "pjspjs0987/Qwen2.5-7B-GrapheneContact-QDoRA-CoT"
18)
19
20# Inference
21messages = [
22 {"role": "user", "content": "Why does increasing contact area reduce contact resistance in graphene-metal interfaces?"}
23]
24text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
25inputs = tokenizer(text, return_tensors="pt").to(model.device)
26outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.7, top_p=0.9)
27print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Citation
1@article{park2026small,
2 title={Small Data, Small Model, Superior Domain Accuracy: Fine-Tuning a 7B Language Model as a Condensed Matter Physics Expert with 64 Examples},
3 author={Park, Junsu},
4 year={2026},
5 institution={Gwangju Institute of Science and Technology (GIST)}
6}
Contact
- Author: Junsu Park
- Affiliation: GIST, Department of Physics and Photon Science
- Hugging Face: pjspjs0987
Framework versions
