ANMOLGPT-4B-v0.4
ANMOLGPT-4B-v0.4 is the fourth iteration of the ANMOLGPT open-source language model series.
v0.4 is based on Qwen3.5-4B and was fine-tuned using the SlimOrca instruction dataset with parameter-efficient LoRA fine-tuning.
The goal of v0.4 was to improve general instruction following, knowledge, and reasoning capabilities while preserving the improvements achieved in previous ANMOLGPT versions.
An interesting result from this experiment was a significant improvement on TruthfulQA, despite TruthfulQA not being used as the v0.4 training dataset.
Model Details
| Property | Value |
|---|
| Model | ANMOLGPT-4B-v0.4 |
| Base Model | Qwen3.5-4B |
| Fine-tuning Dataset | SlimOrca |
| Fine-tuning Method | LoRA |
| Training Framework | Unsloth |
| Model Framework | Hugging Face Transformers |
| Quantization | 4-bit |
| Parameters | Approximately 4B |
| Language | English |
| Developer | Anmol Dhandhania |
Training
ANMOLGPT-4B-v0.4 was created by fine-tuning the Qwen3.5-4B base model on the SlimOrca instruction dataset.
The experiment was designed to investigate whether a high-quality instruction and reasoning dataset could improve general model performance without increasing the model's parameter count.
Training approach
- Base model: Qwen3.5-4B
- Dataset: SlimOrca
- Fine-tuning: LoRA
- Quantization: 4-bit
- Training framework: Unsloth
- Model framework: Hugging Face Transformers
The model was trained using parameter-efficient fine-tuning rather than updating all model parameters.
Evaluation
ANMOLGPT-4B-v0.4 was evaluated using the EleutherAI LM Evaluation Harness.
The evaluation suite included:
- HellaSwag
- PIQA
- ARC-Easy
- Winogrande
- TruthfulQA
- MMLU
- GSM8K
Most benchmarks were evaluated using 0-shot configurations.
GSM8K was evaluated using its 5-shot configuration.
Benchmark Results
| Benchmark | Metric | Score |
|---|
| HellaSwag | Accuracy | 55.06% |
| HellaSwag | Accuracy (normalized) | 73.40% |
| PIQA | Accuracy | 77.86% |
| PIQA | Accuracy (normalized) | 78.29% |
| ARC-Easy | Accuracy | 83.75% |
| ARC-Easy | Accuracy (normalized) | 79.21% |
| Winogrande | Accuracy | 70.64% |
| TruthfulQA MC2 | Accuracy | 54.61% |
| MMLU | Accuracy | 73.58% |
| GSM8K | Flexible Extract | 54.36% |
| GSM8K | Strict Match | 46.85% |
GSM8K
GSM8K was evaluated using 5-shot prompting.
| Metric | Score | Standard Error |
|---|
| Flexible Extract | 54.36% | ±1.37% |
| Strict Match | 46.85% | ±1.37% |
The flexible-extract metric allows the evaluation harness to extract the final numerical answer from the model's generated response.
The strict-match metric requires the generated answer to satisfy the stricter answer-format criteria.
MMLU
Overall MMLU accuracy:
73.58% ± 0.35%
Category Results
| Category | Accuracy |
|---|
| Humanities | 66.14% |
| Other | 77.44% |
| Social Sciences | 82.94% |
| STEM | 71.74% |
Selected Subject Results
| Subject | Accuracy |
|---|
| High School Government & Politics | 93.78% |
| Marketing | 92.31% |
| High School Psychology | 91.93% |
| High School Biology | 90.32% |
| High School Microeconomics | 90.76% |
| Sociology | 88.06% |
| High School World History | 86.08% |
| High School US History | 85.78% |
| Computer Security | 85.00% |
| High School European History | 84.85% |
| High School Geography | 86.87% |
| High School Computer Science | 82.00% |
| College Biology | 88.89% |
| Conceptual Physics | 83.83% |
| Astronomy | 82.89% |
Some areas remain challenging:
| Subject | Accuracy |
|---|
| Abstract Algebra | 52.00% |
| High School Mathematics | 50.37% |
| College Mathematics | 55.00% |
| College Physics | 56.86% |
| College Chemistry | 60.00% |
| Machine Learning | 62.50% |
| Elementary Mathematics | 64.55% |
These areas are potential targets for future training iterations.
Comparison with ANMOLGPT-4B-v0.3
v0.3 focused on targeted fine-tuning using TruthfulQA-oriented data.
v0.4 instead used SlimOrca to investigate improvements in general instruction following and reasoning.
| Benchmark | v0.3 | v0.4 | Change |
|---|
| HellaSwag | 54.58% | 55.06% | +0.48 |
| HellaSwag normalized | 72.80% | 73.40% | +0.60 |
| PIQA | 78.13% | 77.86% | -0.27 |
| PIQA normalized | 78.45% | 78.29% | -0.16 |
| ARC-Easy | 80.13% | 83.75% | +3.62 |
| ARC-Easy normalized | 76.52% | 79.21% | +2.69 |
| Winogrande | 70.56% | 70.64% | +0.08 |
| TruthfulQA MC2 | 45.85% | 54.61% | +8.76 |
| MMLU | 73.34% | 73.58% | +0.24 |
The results indicate that v0.4 improved several general reasoning and knowledge benchmarks while largely maintaining the performance achieved by v0.3.
The largest improvement was observed on TruthfulQA MC2:
45.85% → 54.61%
This is particularly interesting because TruthfulQA was not used as the v0.4 training dataset.
Further investigation is required to understand the source of this improvement and to rule out potential dataset overlap or contamination.
v0.3 vs v0.4 Training Strategy
v0.3
Primary focus:
Truthfulness and factuality
Training data included TruthfulQA-oriented examples.
v0.4
Primary focus:
General instruction following and reasoning
Training dataset:
SlimOrca
The objective was to determine whether broader high-quality instruction data could produce improvements across multiple benchmarks.
Intended Use
ANMOLGPT-4B-v0.4 is primarily intended for research, experimentation, and development involving:
- General-purpose text generation
- Instruction following
- Question answering
- Reasoning experiments
- Educational applications
- LLM fine-tuning research
- Local AI experimentation
- Lightweight generative AI applications
- Model compression and efficiency research
The model can also serve as a starting point for further fine-tuning.
Limitations
ANMOLGPT-4B-v0.4 is an experimental open-source model.
It should not be considered a production-ready or safety-critical system.
The model may:
- Generate factually incorrect information
- Hallucinate information
- Produce incorrect mathematical reasoning
- Fail on complex multi-step reasoning
- Generate inconsistent answers
- Produce incorrect code
- Reflect limitations present in the base model
- Reflect biases and limitations present in the training data
Benchmark performance should not be interpreted as a guarantee of real-world performance.
The model should not be used as the sole basis for medical, legal, financial, safety-critical, or other high-stakes decisions.
Evaluation Configuration
The model was evaluated using the EleutherAI LM Evaluation Harness.
Evaluation configuration:
| Benchmark | n-shot |
|---|
| HellaSwag | 0 |
| PIQA | 0 |
| ARC-Easy | 0 |
| Winogrande | 0 |
| TruthfulQA MC2 | 0 |
| MMLU | 0 |
| GSM8K | 5 |
GSM8K was evaluated using:
- Flexible extraction
- Strict matching
For reproducibility, evaluations should use the same task versions, evaluation harness configuration, model-loading configuration, and generation parameters.
Model Usage
Example using Hugging Face Transformers:
1from transformers import AutoTokenizer, AutoModelForCausalLM
2
3model_id = "anmoldhandhania93/ANMOLGPT-4B-v0.4"
4
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6
7model = AutoModelForCausalLM.from_pretrained(
8 model_id,
9 device_map="auto"
10)
11
12prompt = "Explain what machine learning is in simple terms."
13
14inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
15
16outputs = model.generate(
17 **inputs,
18 max_new_tokens=256,
19 temperature=0.7,
20 do_sample=True
21)
22
23response = tokenizer.decode(
24 outputs[0],
25 skip_special_tokens=True
26)
27
28print(response)
