en-hi-brx-deepseek-v3-dense-pt: A 150M Multilingual Model Fine-Tuned for Algorithmic Reasoning
This repository contains a 150M parameter dense DeepseekV3 model pre-trained from scratch on a 3-billion-token multilingual corpus of
English, Hindi, and Bodo.
The
Bodo data was synthetically generated, can be accessed
here.
The repository includes several checkpoints, representing a curriculum-based fine-tuning process designed to teach the model complex, multi-step algorithmic reasoning.
Model Checkpoints
The models are organized in a sequential, curriculum-based structure. Each subsequent model was fine-tuned from the previous one, building upon its learned skills.
1. base
The foundation model, pre-trained on a 3-billion-token multilingual corpus. It has a foundational understanding of English, Hindi, and Bodo grammar and vocabulary but possesses limited reasoning capabilities.
- Training Corpus: C4 (English), mC4 (Hindi), and a custom Bodo corpus (Public + Synthetic).
- Parameters: ~150M
2. finetuned_ops
This model is an "ops-expert." It was fine-tuned from the base model on a large, focused dataset of five fundamental arithmetic operations (+, -, *, /, %).
- Training Strategy: This checkpoint was trained using principles from the "grokking" literature, including extended training (40 epochs) and high weight decay (0.1), to force the model to learn a generalizable "circuit" for arithmetic rather than memorizing examples.
- Capabilities: High accuracy (~95%) on 2-digit arithmetic tasks.
3. finetuned_gcd
This model is a "sequential reasoning expert." It was fine-tuned from the finetuned_ops model to solve for the Greatest Common Divisor (GCD) using the Euclidean algorithm.
- Training Strategy: Trained on a multilingual, Chain-of-Thought (CoT) dataset. The model learns to generate the intermediate
[rem(...)] steps before providing the final answer. The data was balanced by undersampling cases where GCD=1.
- Capabilities: High accuracy (~95%) in performing multi-step, sequential reasoning for 2-digit GCD problems in English, Hindi, and Bodo.
4. finetuned_bodmas
This is the final and most capable model, designed to be a "hierarchical reasoning expert." It was fine-tuned from the finetuned_gcd model.
- Training Strategy: Trained on a CoT dataset using an "explicit rewrite" format, where the model must recalculate and rewrite the entire mathematical expression after each operation, correctly following the BODMAS order of operations.
- Capabilities: Solves multi-step arithmetic expressions involving hierarchical reasoning with reasonable accuracy (~50%).
How to Use
Load a specific fine-tuned model and the tokenizer for inference.
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
3
4# Choose the desired model checkpoint
5model_name = "akshitkr/en-hi-brx-deepseek-v3-dense-pt"
6# revision can be 'finetuned_ops', 'finetuned_gcd', or 'finetuned_bodmas'
7revision_name = "finetuned_gcd"
8
9tokenizer = AutoTokenizer.from_pretrained(model_name)
10model = AutoModelForCausalLM.from_pretrained(
11 model_name,
12 revision=revision_name,
13 torch_dtype=torch.bfloat16 # Use bfloat16 for inference
14)
15
16# Example for GCD model
17prompt = "|user|what is the gcd of 9 9 and 5 4|user||reasoning|"
18
19inputs = tokenizer(prompt, return_tensors="pt")
20outputs = model.generate(**inputs, max_new_tokens=256)
21
22print(tokenizer.decode(outputs, skip_special_tokens=True))
23# Expected output:
24# |user|what is the gcd of 9 9 and 5 4|user||reasoning|{[gcd(<9 9>,<5 4>)][rem(<9 9> % <5 4>)=<4 5>]}{[gcd(<5 4>,<4 5>)][rem(<5 4> % <4 5>)=<0 9>]}{[gcd(<4 5>,<0 9>)][rem(<4 5> % <0 9>)=<0 0>]}{[gcd(<0 9>,<0 0>)]}|reasoning||assistant|<0 9>|assistant|
