library_name: transformers tags: [scientific, instruction-following, gemma, lora, gemma3-270m, science]
Model Card for ps2program/gemma3-270m-scisinstruct
This is a LoRA-fine-tuned version of Gemma 3 (270M), specialized for scientific instruction-following and reasoning tasks. The model has been trained on the zd21/SciInstruct dataset to excel at generating explanations, summaries, and scientific reasoning outputs in response to natural language prompts.
Model Details
Model Description
This model extends the Google-developed Gemma 3 base language model using LoRA (Low-Rank Adaptation) adapters. This technique allows for efficient fine-tuning on a specific task without modifying the entire model. The resulting model is designed to handle a variety of scientific questions and instructions, making it a valuable tool for academic and research-oriented applications.
- Developed by: ps2program
- Model type: Causal Language Model (LoRA-finetuned)
- Language(s) (NLP): English
- License: Apache-2.0
- Finetuned from model:
gemma3-270m
Model Sources
Uses
Direct Use
This model is intended for direct use in applications requiring scientific text generation. Examples of its use include:
- Answering questions about scientific principles.
- Generating summaries of scientific papers or concepts.
- Assisting in educational contexts for students and researchers.
Downstream Use
This model can serve as a base for further fine-tuning on highly specialized, domain-specific scientific corpora (e.g., specific fields like biochemistry or astrophysics) to improve performance on those particular tasks.
Out-of-Scope Use
The model is not intended for:
- General-purpose conversation or casual chat.
- Providing medical, legal, or financial advice.
- Generating content in non-scientific domains where it may produce inaccurate or nonsensical outputs.
- Applications where factual accuracy is critical without human verification.
Bias, Risks, and Limitations
- Factual Inaccuracies: The model may generate factually incorrect or outdated information. Users should always verify outputs, especially in academic or research contexts.
- Data Bias: The model's performance and outputs are limited by the quality and content of its training data (
SciInstruct). It may reflect any biases present in the original dataset.
- Limited Scope: The model is specialized for scientific reasoning and may perform poorly on tasks outside of this domain.
Recommendations
- Always verify the generated scientific content with reliable sources.
- Do not use the model for high-stakes decision-making.
- Be aware of the model's limitations and potential for generating incorrect information.
How to Get Started with the Model
To use this model, you'll need to load the base gemma3-270m model first, and then load the LoRA adapters on top of it.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2from peft import PeftModel
3import torch
4
5# Load the base model (you must have access to gemma3-270m)
6base_model_id = "google/gemma-3-270m"
7model_id = "ps2program/gemma3-270m-scisinstruct"
8
9base_model = AutoModelForCausalLM.from_pretrained(
10 base_model_id,
11 torch_dtype=torch.bfloat16,
12 device_map="auto"
13)
14
15# Load the LoRA adapters from your repo
16model = PeftModel.from_pretrained(base_model, model_id)
17
18# Load the tokenizer
19tokenizer = AutoTokenizer.from_pretrained(model_id)
20
21# Prepare a scientific prompt
22prompt = "Explain the significance of CRISPR-Cas9 technology in genetic engineering."
23inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
24
25# Generate the output
26outputs = model.generate(
27 **inputs,
28 max_new_tokens=256,
29 do_sample=True,
30 temperature=0.7,
31 top_p=0.95
32)
33
34print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Training Details
Training Data
The model was fine-tuned on the zd21/SciInstruct dataset, which contains high-quality, scientifically-grounded instruction-following examples.
Training Procedure
Preprocessing
The dataset was formatted into instruction-following prompts and tokenized using the gemma3-270m tokenizer.
Training Hyperparameters
- Training regime: Mixed precision (bfloat16)
- LoRA parameters:
r=8, lora_alpha=16
- Optimizer:
adamw_torch_fused
- Learning rate:
2e-5
- Batch size:
4
- Epochs:
5
- Gradient accumulation steps:
4
Evaluation
Testing Data, Factors & Metrics
The model's performance was evaluated qualitatively by observing its ability to generate coherent and scientifically accurate responses to a diverse set of prompts. Standard metrics were not used in this initial fine-tuning.
Results
The model demonstrates an improved ability to follow scientific instructions compared to the base gemma3-270m model. It can provide well-structured explanations and summaries.
Environmental Impact
Carbon emissions were not calculated for this fine-tuning process. However, as a small-parameter model fine-tuned using LoRA, the training was computationally efficient and had a significantly lower environmental impact than training a large model from scratch.
- Hardware Type: (e.g., 1x NVIDIA A100 GPU)
- Hours used: (e.g., ~2 hours)
- Cloud Provider: (e.g., Google Cloud, AWS, etc.)
- Compute Region: (e.g., us-east-1)
- Carbon Emitted: [More Information Needed]
Citation
BibTeX:
1@misc{ps2program2025gemma3,
2 title={Gemma 3 270M SciInstruct LoRA},
3 author={ps2program},
4 year={2025},
5 howpublished={\url{https://huggingface.co/ps2program/gemma3-270m-scisinstruct}},
6 note={LoRA fine-tuning of Gemma 3 270M on the SciInstruct dataset.}
7}
APA:
ps2program. (2025).
Gemma 3 270M SciInstruct LoRA. Hugging Face Model Hub.
https://huggingface.co/ps2program/gemma3-270m-scisinstructModel Card Authors
ps2program
Model Card Contact
[Your preferred contact method or email address]
