Aryabhata-1.0 GGUF Models
Model Generation Details
This model was generated using
llama.cpp at commit
221c0e0c.
Quantization Beyond the IMatrix
I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.
In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the
--tensor-type option in
llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉
Layer bumping with llama.cppWhile this does increase model file size, it significantly improves precision for a given quantization level.
I'd love your feedback—have you tried this? How does it perform for you?
Click here to get info on choosing the right GGUF model format
Aryabhatta 1.0 : An exam-focused language model for JEE Math
Overview
Aryabhata 1.0 is a 7B parameter small language model for mathematics developed by Physics Wallah AI Research, optimized for high-stakes Indian competitive exams like JEE Mains. Despite its compact size, Aryabhata 1.0 achieves state-of-the-art performance on exam-centric reasoning tasks with impressive token efficiency and low inference cost.
🚧 Aryabhata 1.0 is an experimental release. We are actively seeking feedback — please contribute in the Discussion tab of this repo.
🧠 Key Features
- Architecture: 7B parameter causal decoder-based model.
- Exam-Centric Optimization: Specifically tuned for JEE-level Mathematics reasoning.
- High Accuracy:
- 86% on JEE Mains January 2025 session.
- 90.2% on JEE Mains April 2025 session.
- Token Efficiency: Operates effectively around a ~2K token window, compared to ~8K required by other reasoning models.
- Compute Efficient: Trained on a 1x2 NVIDIA H100 GPU using optimized pipeline.
🛠️ Training Details
- Training Data: ~130K problem-solution pairs curated from proprietary Physics Wallah exam datasets.
- Training Pipeline:
- Model Merging
- Rejection Sampling
- Supervised Fine-Tuning (SFT)
- Reinforcement Learning with Verifiable Rewards (RLVR)
🔀 Model Merging
We began with model merging (Weighted average) to build a strong initialization (Aryabhata 0.5) by combining diverse model capabilities:
- Qwen 2.5 Math: A robust math-centric LLM with solid symbolic math foundations.
- Ace Math: An enhanced version of Qwen 2.5 Math, fine-tuned by NVIDIA for improved accuracy in mathematics benchmarks.
- DeepSeek R1 Distill Qwen: A long-form reasoning model, fine-tuned on reasoning traces distilled from DeepSeek R1.
📚 Data Curation + Rejection Sampling
We extracted ~250K raw questions from Physics Wallah's internal database and applied aggressive filtering and cleaning:
- Removed: diagram-based, non-English, and option-heavy questions.
- Kept: questions matching the distribution of JEE Main 2019–2024.
Final curated dataset: ~130K high-quality questions.
For each question:
- Generated 4 CoTs using Aryabhata 0.5.
- Retained only those leading to correct final answers.
Resulting Dataset:
- ~100K questions
- ~350K high-quality CoTs
We used this dataset for SFT.
🎯 Reinforcement Learning with Verifiable Rewards (RLVR)
We used a custom in-house variant of Group Relative Policy Optimization (GRPO), adapted for math-specific reward functions.
- Removed KL-divergence penalty
- Removed clipping
We used RLVR on the remaining ~30K questions.
This multi-phase training strategy allows Aryabhata 1.0 to capture pedagogy-aligned reasoning patterns, making it highly effective for solving real student queries in mathematics.
📊 Performance Highlights
Evaluation Setup
All evaluations were performed with temperature = 0.0, and we report pass@1 accuracy.
Evaluation Datasets
We evaluated the model on two sets of official JEE Mains 2025 mathematics papers:
- January Session: 10 question papers containing 250 questions.
- April Session: 9 question papers containing 225 questions.
Each paper includes a mix of:
- Multiple Choice Questions (MCQs) with one correct option
- Numeric Answer Type (NAT) questions requiring precise numerical responses
Evaluation Metric
We used a composite evaluation metric to reflect real-world grading rigor and reduce false positives:
- Float Match
- Compares predicted and target answers within a tolerance (±1e-9)
- Handles rounding artifacts and small numerical errors robustly
- String Match
- Used for symbolic answers (e.g., fractions, radicals)
- Uses strict exact match — predictions must match ground truth character-for-character
- LLM-as-Judge (GPT-4o-mini)
- Used for Mathematical equivalence for ambiguous formats
🔹 Accuracy Comparison Across Models
Aryabhata has the best accuracy on JEE Main Maths, on par with frontier models
🔹 Accuracy vs Token Usage
Aryabhata is on par with frontier models in terms of accuracy vs token usage
🔧 Intended Use
Primary Use Cases:
- Competitive exam preparation (JEE Main level mathematics problems)
- Question answering and doubt-solving systems
- Educational tutoring and concept explanation
💡 How to Use
🧪 Using with 🤗 Transformers
1from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
2
3model_id = "PhysicsWallahAI/Aryabhata-1.0"
4
5tokenizer = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(model_id)
7
8
9# Define stop strings
10stop_strings = ["<|im_end|>", "<|end|>", "<im_start|>", "```python\n", "<|im_start|>", "]}}]}}]"]
11
12def strip_bad_tokens(s, stop_strings):
13 for suffix in stop_strings:
14 if s.endswith(suffix):
15 return s[:-len(suffix)]
16 return s
17
18
19# Create generation config (can also set temperature, top_p, etc.)
20generation_config = GenerationConfig(
21 max_new_tokens=4096,
22 stop_strings = stop_strings
23)
24
25query = 'Find all the values of \\sqrt[3]{1}'
26messages = [{'role': 'system', 'content': 'Think step-by-step; put only the final answer inside \\boxed{}.'},
27 {'role': 'user', 'content': query}]
28
29text = tokenizer.apply_chat_template(
30 messages,
31 tokenize=False,
32 add_generation_prompt=True
33)
34inputs = tokenizer([text], return_tensors="pt")
35outputs = model.generate(**inputs, generation_config=generation_config, tokenizer=tokenizer)
36
37print(strip_bad_tokens(tokenizer.decode(outputs[0], skip_special_tokens=True), stop_strings))
⚡ Using with vLLM
To run the model efficiently using vLLM:
1from vllm import LLM, SamplingParams
2
3# Initialize model (downloads from Hugging Face if not local)
4llm = LLM(model="PhysicsWallahAI/Aryabhata-1.0")
5
6# Define prompt and sampling configuration
7query = 'Find all the values of \\sqrt[3]{1}'
8messages = [{'role': 'system', 'content': 'Think step-by-step; put only the final answer inside \\boxed{}.'},
9 {'role': 'user', 'content': query}]
10sampling_params = SamplingParams(temperature=0.0, max_tokens=4*1024, stop=["<|im_end|>", "<|end|>", "<im_start|>", "```python\n", "<|im_start|>", "]}}]}}]"])
11
12# Run inference
13results = llm.chat(messages, sampling_params)
14
15# Print result
16print(results[0].outputs[0].text.strip())
🚀 Roadmap
Aryabhata 2.0 (Upcoming):
- Extending domain coverage to Physics and Chemistry
- Supporting JEE Advanced, NEET, and Foundation syllabus
- Further optimization for affordability and accuracy in real-time deployments
🤝 Citation
If you use this model, please cite:
1@misc{Aryabhata2025,
2 title = {Aryabhata 1.0: A compact, exam-focused language model tailored for mathematics in Indian competitive exams, especially JEE Main.},
3 author = {Physics Wallah AI Research},
4 year = {2025},
5 note = {\url{https://huggingface.co/PhysicsWallahAI/Aryabhata-1.0}},
6}
7
8<!--End Original Model Card-->
9
10---
11
12# <span id="testllm" style="color: #7F7FFF;">🚀 If you find these models useful</span>
13
14Help me test my **AI-Powered Quantum Network Monitor Assistant** with **quantum-ready security checks**:
15
16👉 [Quantum Network Monitor](https://readyforquantum.com/?assistant=open&utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme)
17
18
19The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : [Source Code Quantum Network Monitor](https://github.com/Mungert69). You will also find the code I use to quantize the models if you want to do it yourself [GGUFModelBuilder](https://github.com/Mungert69/GGUFModelBuilder)
20
21💬 **How to test**:
22 Choose an **AI assistant type**:
23 - `TurboLLM` (GPT-4.1-mini)
24 - `HugLLM` (Hugginface Open-source models)
25 - `TestLLM` (Experimental CPU-only)
26
27### **What I’m Testing**
28I’m pushing the limits of **small open-source models for AI network monitoring**, specifically:
29- **Function calling** against live network services
30- **How small can a model go** while still handling:
31 - Automated **Nmap security scans**
32 - **Quantum-readiness checks**
33 - **Network Monitoring tasks**
34
35🟡 **TestLLM** – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):
36- ✅ **Zero-configuration setup**
37- ⏳ 30s load time (slow inference but **no API costs**) . No token limited as the cost is low.
38- 🔧 **Help wanted!** If you’re into **edge-device AI**, let’s collaborate!
39
40### **Other Assistants**
41🟢 **TurboLLM** – Uses **gpt-4.1-mini** :
42- **It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
43- **Create custom cmd processors to run .net code on Quantum Network Monitor Agents**
44- **Real-time network diagnostics and monitoring**
45- **Security Audits**
46- **Penetration testing** (Nmap/Metasploit)
47
48🔵 **HugLLM** – Latest Open-source models:
49- 🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.
50
51### 💡 **Example commands you could test**:
521. `"Give me info on my websites SSL certificate"`
532. `"Check if my server is using quantum safe encyption for communication"`
543. `"Run a comprehensive security audit on my server"`
554. '"Create a cmd processor to .. (what ever you want)" Note you need to install a [Quantum Network Monitor Agent](https://readyforquantum.com/Download/?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface_repo_readme) to run the .net code on. This is a very flexible and powerful feature. Use with caution!
56
57### Final Word
58
59I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is [open source](https://github.com/Mungert69). Feel free to use whatever you find helpful.
60
61If you appreciate the work, please consider [buying me a coffee](https://www.buymeacoffee.com/mahadeva) ☕. Your support helps cover service costs and allows me to raise token limits for everyone.
62
63I'm also open to job opportunities or sponsorship.
64
65Thank you! 😊
