IndicBERT Probing and Fine-tuning Analysis
Extension of the paper: IndicSentEval: How Effectively do Multilingual Transformer Models encode Linguistic Properties for Indic Languages? (Aravapalli et al., 2024)
This repository reproduces the probing experiments from the IndicSentEval paper on IndicBERT (by AI4Bharat) and extends the analysis with two new contributions — stronger MLP probing and end-to-end fine-tuning — to measure IndicBERT's true linguistic capacity beyond what standard probing reveals.
Repository Structure
IndicBertology/
│
├── src/
│ ├── run_indicbert.py # Probing script — reproduces paper's LogReg methodology
│ ├── finetune_indicbert.py # Fine-tuning + Frozen MLP comparison (our extension)
│ ├── classifier.py # Original repo classifier script
│ ├── ada.py # Original repo embedding extraction script
│ └── probingData/ # Dataset CSVs (6 languages × 9 tasks)
│ ├── hindi/
│ ├── marathi/
│ ├── telugu/
│ ├── malayalam/
│ ├── kannada/
│ └── urdu/
│
├── utils/
│ └── ssfAPI.py # Parses SSF (Shakti Standard Format) annotated data
│
├── guidelines/ # Project guidelines and SSF format documentation
│
├── results/
│ ├── indicbert_results.json # Probing results — all 12 layers, all tasks
│ └── finetuning_comparison.json # Frozen MLP vs Fine-tuned comparison
│ └── finetuning_comparison.csv # Same results in CSV format
│
├── requirements.txt
└── README.md
Our Three Experimental Approaches
Approach 1 — LogisticRegression Probe (Paper's Method, Reproduced)
IndicBERT (frozen) → extract all 12 layer embeddings → LogisticRegression → best layer accuracy
Answers: Which layer best encodes each linguistic property?
Approach 2 — Frozen MLP Probe (Our Extension)
IndicBERT (frozen) → CLS token from final layer → 2-layer MLP (768→256→ReLU→n_classes) → accuracy
Answers: Can a stronger non-linear classifier extract more from the same frozen representations?
Approach 3 — End-to-End Fine-tuning (Our Extension)
IndicBERT (trainable) + MLP head → trained end-to-end with differential LRs → accuracy
Answers: What is IndicBERT's true maximum linguistic capacity for each task?
Probing Tasks
| Probing Task | Category | Number of Classes |
|---|
| Sentence Length (SentLen) | Surface | 8 |
| Bigram Shift (BShift) | Syntactic | 2 |
| Tree Depth (TreeDepth) | Syntactic | 5 |
| Subject Number (SubjNum) | Semantic | 2 |
| Object Number (ObjNum) | Semantic | 2 |
| Word Content | Semantic | var |
| Gender | Semantic | 4 |
| Number | Semantic | 3 |
| Person | Semantic | 7 |
Examples of Each Class (Hindi)
Subject Number
| Type | Example |
|---|
| sg | वैश्विक रूप से यह संक्रमण उन जगहों पर अधिक आम है जहां पर रोग - प्रतिरोधकता कम है । |
| pl | सामान्य लोगों में से आधे लोगों में ये छोटे जीव नींद के दौरान प्रवेश करते हैं । |
Object Number
| Type | Example |
|---|
| sg | उस वक्त के अंगरेज शासक लार्ड डलहौजी के नाम पर इसका नाम ' डलहौजी ' रख दिया गया । |
| pl | अगस्त के बाद गर्म कपड़े साथ रखें । |
Bigram Shift
| Type | Example |
|---|
| 0 (original) | पंजपुला एक सैरगाह है । |
| 1 (shifted) | गर्मियों में ठंडी सड़क को सब लुभाती है । |
Gender
| Type | Example |
|---|
| any | पंजपुला एक सैरगाह है । |
| m | पहले चश्मे में इतना पानी होता था कि 7 धाराएँ बनती थीं । |
| f | आज चश्मे में से एक धारा निकलती है । |
Number
| Type | Example |
|---|
| sg | डलहौजी में भीड़भाड़ के स्थान पर शांत माहौल है । |
| pl | लंबी छुट्टियाँ गुजारने वाले एकांतपसंद लोग डलहौजी बड़ी संख्या में आते हैं । |
| any | सामान्य स्वेटर व शाल हर मौसम में चाहिए । |
Person
| Type | Example |
|---|
| 1 | हम पूरी तरह तैयार होकर नहीं जाते । |
| 1h | हम क्या सबसे झगड़ा करती फिरती हैं का ? |
| 2 | चना दाल को दो घंटे पहले पानी में भिगो दें । |
| 2h | अगस्त के बाद गर्म कपड़े साथ रखें । |
| 3 | पंजपुला एक सैरगाह है । |
| 3h | सुभाष चंद्र बोस कुछ दिन यहाँ आ कर रहे थे । |
| any | कभी पंजपुला से धर्मशाला की ओर एक पैदल मार्ग जाता था । |
Languages Covered
| Language | Code | Sentences | Family | Script |
|---|
| Hindi | hi | 12,292 | Indo-Aryan | Devanagari |
| Marathi | mr | 12,029 | Indo-Aryan | Devanagari |
| Telugu | te | 3,222 | Dravidian | Telugu |
| Malayalam | ml | 7,667 | Dravidian | Malayalam |
| Kannada | kn | 9,806 | Dravidian | Kannada |
| Urdu | ur | 2,363 | Indo-Aryan | Nastaliq |
Perturbations (From Original Paper)
The original paper applies 13 text perturbations to test model robustness. These are defined in the src/ folder.
| Perturbation Type | Description |
|---|
| Append Irrelevant | Append an irrelevant sentence |
| DropAll | Drop all content words |
| DropAllNouns | Drop all noun tokens |
| DropAllVerbs | Drop all verb tokens |
| DropFirst | Drop first word |
| DropLast | Drop last word |
| DropFirstLast | Drop first and last word |
| DropRandomNoun | Drop a random noun |
| DropRandomVerb | Drop a random verb |
| KeepBoth | Keep only subject and object |
| KeepOnlyNoun | Keep only nouns |
| KeepOnlyVerb | Keep only verbs |
| Shuffle | Randomly shuffle all words |
Word Content — Mid-Frequency Words per Language
| Language | Vocabulary Size |
|---|
| Hindi | 781 |
| Marathi | 626 |
| Kannada | 416 |
| Malayalam | 188 |
| Urdu | 153 |
| Telugu | 34 |
Key Results
Probing Results — Best Layer Accuracy (LogisticRegression, Paper Method)
| Task | Hindi | Marathi | Telugu | Malayalam | Kannada | Urdu |
|---|
| SentLen | 63.1% | 25.1% | — | 30.6% | — | 18.2% |
| SubjNum | 67.5% | 71.3% | 73.2% | 76.0% | 79.7% | 89.3% |
| ObjNum | 70.1% | 88.2% | 64.7% | 79.1% | 80.9% | 86.1% |
| TreeDepth | 36.3% | 22.9% | — | 21.8% | 27.9% | 20.3% |
| BShift | 71.3% | 73.2% | 75.2% | 71.5% | 72.8% | 74.0% |
| Gender | 51.4% | 41.5% | 45.9% | — | 78.6% | 46.1% |
| Number | 60.5% | 76.4% | 55.8% | — | 76.3% | 67.2% |
| Person | 66.2% | 88.2% | 68.5% | — | 90.7% | 72.8% |
Fine-tuning Comparison — Telugu (Frozen MLP vs Fine-tuned)
| Task | Frozen MLP | Fine-tuned | Δ |
|---|
| SentLen | 82.6% | 96.9% | +14.3% |
| SubjNum | 71.9% | 83.0% | +11.1% |
| ObjNum | 66.4% | 66.4% | 0.0% |
| TreeDepth | 55.8% | 59.7% | +3.9% |
| BShift | 71.8% | 78.6% | +6.8% |
| Gender | 47.9% | 82.4% | +34.5% |
| Number | 51.6% | 89.8% | +38.1% |
| Person | 69.3% | 88.2% | +18.9% |
Key Findings
1. Probing underestimates surface task capacity — SentLen accuracy in Urdu goes from 18.2% (LogReg probing) to 84.1% (fine-tuned), a +66% gain. IndicBERT has strong surface encoding capacity that probing alone cannot reveal.
2. Telugu morphology shows the largest fine-tuning gains — Number: +38.1%, Gender: +34.5%. Frozen representations fail to capture Telugu morphological features that the model can actively learn.
3. TreeDepth remains hard regardless of approach — Even with full fine-tuning, tree depth peaks at ~59.7% (Telugu), indicating a fundamental limitation in IndicBERT's syntactic depth encoding.
4. Kannada Person is already near-perfectly encoded — Frozen MLP: 89.6%, Fine-tuned: 90.8% (Δ = +1.2%). Pretraining naturally encodes Kannada person morphology near-perfectly.
5. Delta (Δ) analysis is our new contribution — Large Δ means probing underestimates model capacity. Zero Δ means frozen representations are already optimal. Negative Δ means fine-tuning overfit on small data.
Setup
1. Clone the repository
2. Install dependencies
bash
pip install -r requirements.txt
3. Download IndicBERT
The model is gated on HuggingFace. Request access at
https://huggingface.co/ai4bharat/indic-bert, then place the model files at
./indic-bert-local/indic-bert/The folder should contain:
indic-bert-local/indic-bert/
├── config.json
├── pytorch_model.bin
├── spiece.model
└── spiece.vocab
Running
Probing experiments — reproduces the paper's methodology
bash
python src/run_indicbert.py
Extracts embeddings from all 12 IndicBERT layers, trains LogisticRegression on each, reports best-layer accuracy.
Results saved to results/indicbert_results.json
Fine-tuning comparison — our extension
bash
python src/finetune_indicbert.py
Trains frozen MLP probe and fine-tunes IndicBERT end-to-end for each task.
Results saved to results/finetuning_comparison.json and results/finetuning_comparison.csv
Implementation Notes
keep_accents=True in tokenizer — critical for preserving Indic vowel matras (diacritics)- CLS token (index 0 of last hidden state) used as sentence representation
- Train / Dev / Test split: 70% / 10% / 20% with stratified sampling
- Early stopping with patience=3 on validation accuracy
- Fine-tuning uses differential learning rates:
2e-5 for IndicBERT, 1e-3 for classifier head
- Fresh IndicBERT weights loaded for each task to prevent cross-task interference
- MPS (Apple Silicon GPU) used for acceleration
Requirements
torch>=1.13.0
transformers>=4.30.0
sentencepiece>=0.1.99
numpy>=1.23.0
scikit-learn>=1.2.0
pandas>=1.5.0
About IndicBERT
IndicBERT is an ALBERT-based multilingual model created by AI4Bharat (IIT Madras) and pretrained on ~9 billion tokens across 12 Indian languages using Masked Language Modeling (MLM) and Sentence Order Prediction (SOP). It has 12 transformer layers with a hidden dimension of 768.
- Model:
ai4bharat/indic-bert
- Architecture: ALBERT-base
- Hidden size: 768
- Layers: 12
- Vocabulary: 200,000 SentencePiece tokens
Original Paper and Codebase
This repository builds on the original IndicBertology codebase:
- Paper: IndicSentEval — Aravapalli et al., 2024. arXiv: 2410.02611
- Original repo: https://github.com/aforakhilesh/IndicBertology
bibtex
@article{aravapalli2024indicsenteval,
title = {IndicSentEval: How Effectively do Multilingual Transformer
Models encode Linguistic Properties for Indic Languages?},
author = {Aravapalli, Akhilesh and others},
journal = {arXiv preprint arXiv:2410.02611},
year = {2024},
url = {
https://arxiv.org/abs/2410.02611}
}
Acknowledgements
- AI4Bharat for the IndicBERT model and IndicCorp dataset
- Authors of IndicSentEval for the probing dataset and methodology
