This model (sweatSmile/Qwen3-0.6B-4bit-NEET) is a fine-tuned version of Qwen/Qwen3-0.6B specifically trained on biology multiple-choice questions for NEET exam preparation using LoRA (Low-Rank Adaptation) with 4-bit quantization. The model has been optimized to understand and generate responses for biology questions in a structured format.

Model Details

Model Type: Causal Language Model
Base Model: Qwen/Qwen3-0.6B
This Model: sweatSmile/Qwen3-0.6B-4bit-NEET
Fine-tuning Method: Supervised Fine-Tuning (SFT) with LoRA
Quantization: 4-bit (NF4)
LoRA Configuration:
- Rank (r): 8
- Alpha: 16
- Target Modules: All linear layers
- Dropout: 0.05
Language: English
License: Apache-2.0
Parameters: 0.6B (4-bit quantized base + LoRA adapters)
Max Sequence Length: 32 tokens

Training Data

Dataset: sweatSmile/neet-biology-qa
Dataset Size: 793 samples after preprocessing
Format: Multiple choice questions with Subject, Question, Options (A-D), and Answer
Subject Focus: Biology (NEET exam preparation)

Dataset Quality Evaluation

To ensure the quality of the training data, we conducted a comprehensive evaluation using random samples from the dataset with an LLM-as-judge approach. The evaluation results demonstrate the reliability of our training data:

Quality Metrics (LLM Judge Evaluation):

Primary Evaluation: 82.4% correctness score (28/34 samples correct)
Secondary Evaluation: 85.3% correctness score (29/34 samples correct)
Small Sample Validation: 100% correctness score (3/3 samples correct)
Average Model Response Latency: 4.97-7.55 seconds

Evaluation Details:

Random samples were extracted from the training dataset for independent quality assessment
Multiple evaluation runs were conducted to ensure consistency
LLM-as-judge methodology was used to verify answer correctness
All evaluation runs completed successfully with no errors
High consistency across different sample sets indicates robust dataset quality

The consistently high correctness scores (82-100%) across different random samples validate that the dataset contains accurate, well-formatted biology questions suitable for NEET preparation. This quality assessment provides confidence in the model's training foundation and expected performance on similar biology multiple-choice questions.

Training Configuration

Training Method: Supervised Fine-Tuning with TRL SFTTrainer + LoRA
Batch Size: 1 (per device) × 16 (gradient accumulation) = 16 effective batch size
Learning Rate: 3e-4 with cosine scheduler
Warmup Ratio: 0.1
Epochs: 5
Total Training Steps: ~245
Weight Decay: 0.01
Gradient Clipping: 1.0
Precision: FP16 for memory efficiency
Sequence Length: 32 tokens (tokenized with padding)

Memory Optimizations Applied

4-bit Quantization: NF4 quantization for efficient memory usage
LoRA Fine-tuning: Only fine-tune small adapter layers instead of full model
Gradient Checkpointing: Enabled for memory efficiency
Sequence Length: Set to 32 tokens with padding
Batch Size: Minimized to 1 per device with gradient accumulation
Packing: Disabled (uses padding instead)
Mixed Precision: FP16 enabled

Data Format

The model was trained on data formatted as:

Subject: biology
Question: Which of the following tissues is responsible for secondary growth in dicot stems?

Options:
A. Intercalary meristem
B. Lateral meristem
C. Apical meristem
D. Dermal tissue

Answer: B<|endoftext|>

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load the fine-tuned model directly
model = AutoModelForCausalLM.from_pretrained(
    "sweatSmile/Qwen3-0.6B-4bit-NEET",
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("sweatSmile/Qwen3-0.6B-4bit-NEET")

# Example usage
prompt = """Subject: biology
Question: Which organelle is responsible for photosynthesis?

Options:
A. Mitochondria
B. Chloroplast
C. Nucleus
D. Ribosome

Answer:"""

inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=32)
with torch.no_grad():
    outputs = model.generate(
        inputs.input_ids,
        max_new_tokens=5,
        do_sample=False,
        pad_token_id=tokenizer.eos_token_id
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Training Environment

Framework: Transformers + TRL + PEFT (LoRA)
Quantization: 4-bit NF4 with BitsAndBytesConfig
Precision: Mixed precision (FP16)
Gradient Checkpointing: Enabled for memory efficiency
Hardware Requirements: Optimized for limited VRAM (~22GB GPU)
Training Time: ~245 steps across 5 epochs

Model Performance & Quality Assurance

The model's training data quality was rigorously validated through independent evaluation:

Dataset Correctness: 82-85% accuracy confirmed through LLM-as-judge evaluation
Sample Coverage: Multiple random samples tested across different evaluation runs
Consistency: High correctness scores maintained across various sample sets
Reliability: Zero errors in evaluation pipeline, indicating robust data processing

This quality assurance process ensures that the model was trained on accurate, well-curated biology questions appropriate for NEET exam preparation.

Model Outputs

The model generates answers in the format "A", "B", "C", or "D" following the multiple-choice question structure it was trained on.

Limitations

Sequence Length: Limited to 32 tokens due to memory optimization
Domain Specific: Optimized specifically for NEET biology questions
Format Dependent: Works best with the specific question format used in training
LoRA Constraints: Only certain layers are fine-tuned through LoRA adapters
Quantization Effects: 4-bit quantization may slightly impact precision

Training Logs

Total Samples: 793
Effective Batch Size: 16 (1 × 16 gradient accumulation)
Steps per Epoch: ~49
Total Training Steps: ~245
Learning Rate: 3e-4 (higher for LoRA training)
Memory Usage: Optimized for 22GB GPU memory with 4-bit + LoRA

Files

adapter_config.json: LoRA adapter configuration
adapter_model.bin: LoRA adapter weights
config.json: Model configuration
tokenizer.json: Tokenizer configuration
training_logs/: Training logs and checkpoints
evaluation_results/: Dataset quality evaluation logs and metrics

Citation

bibtex

@misc{mcq-biology-lora-model-2024,
  title={MCQ Biology Model - LoRA Fine-tuned Qwen3-0.6B for NEET Preparation},
  author={sweatSmile},
  year={2024},
  howpublished={\\url{https://huggingface.co/sweatSmile/Qwen3-0.6B-4bit-NEET}},
  note={Fine-tuned with LoRA on NEET biology questions using 4-bit quantization with validated dataset quality}
}

Acknowledgments

Base model: Qwen/Qwen3-0.6B
Dataset: sweatSmile/neet-biology-qa
Framework: Hugging Face Transformers, TRL, and PEFT
Quantization: BitsAndBytesConfig with NF4
Fine-tuning: LoRA (Low-Rank Adaptation)
Quality Assurance: LLM-as-judge evaluation methodology