This is the first publicly released model fine-tuned specifically for the Tanglish technical register used by Tamil engineering students and developers (e.g., "indha gradient descent enna na...", "konjam sollunga", "intha for-loop epdi work pannuthu").

Model Details

Base model: meta-llama/Llama-3.1-8B-Instruct
Fine-tuning method: QLoRA (4-bit NF4 quantization + LoRA adapters)
LoRA config: rank=8, alpha=16, target modules: q_proj, v_proj, k_proj, o_proj
Training framework: trl.SFTTrainer with peft
Trainable parameters: 6,815,744 (0.15% of base model)
Adapter size: 26 MB
Training data: dheepakkaran/TamilTech-QA (4,415 samples)
Hardware: Single NVIDIA T4 (16 GB) on Kaggle
Training time: ~5 hours for 1 epoch
Precision: bf16 training, fp16 compute

Training data

The model was fine-tuned on TamilTech-QA, a Tanglish technical QA dataset of 4,415 samples (3,536 train / 431 val / 447 test) drawn from:

12 Tamil tech YouTube channels (92% of training data)
GPT-4o-mini-generated synthetic Tanglish pairs (8%)

Topics covered: Python, DSA, ML, ECE, web, OS, networking, databases, algorithms, debugging.

Evaluation results

Evaluated on 447 held-out test samples comparing base Llama-3.1-8B (zero-shot) vs. this fine-tuned model:

Metric	Base zero-shot	Fine-tuned (ours)	Change
Perplexity	57.05	12.40	-78% ⭐
BLEU-4	0.0012	0.0012	~
ROUGE-L	0.0206	0.0201	~
BERTScore F1	0.5245	0.5018	-4%
CSPS (Code-Switch Preservation Score)	0.7268	0.7539	+3.7%
TTR (Technical Term Retention)	0.3880	0.4193	+8%
TCF (Tamil Connector Fluency)	0.4989	0.5078	+1.8%
Answer length ratio	14.50x	14.04x	-3%

Headline result

Perplexity drops 78% (57 → 12) — the model has clearly learned the Tanglish language modeling distribution. CSPS, TTR, and TCF (novel metrics for code-switched generation, defined in the dataset card) all improve modestly, indicating better code-switching preservation and technical term retention.

Surface generation metrics (BLEU/ROUGE) show marginal change. We attribute this to:

Only 1 epoch trained (limited by T4 + Kaggle's 12-hour session cap)
Prompt format mismatch between training (ChatML) and evaluation (Q/A)
Short reference answers in the test set

Inference example

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

base = "meta-llama/Llama-3.1-8B-Instruct"
adapter = "dheepakkaran/TamilTech-QA-Llama3.1-8B-QLoRA"

tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(model, adapter)

prompt = (
    "<|im_start|>system\n"
    "You are a helpful Tanglish technical assistant.<|im_end|>\n"
    "<|im_start|>user\nindha gradient descent enna na konjam explain pannunga<|im_end|>\n"
    "<|im_start|>assistant\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Intended use

Research on code-switched (Tanglish) NLP and Indian-language low-resource modeling
Education tools for Tamil engineering students
Baseline for future Tanglish technical assistants

Limitations

The model is trained on only 1 epoch with 4,415 samples — production quality requires more data and longer training.
Output style still leans verbose / English-heavy (length ratio ~14x). The model has learned the Tanglish distribution at the language-modeling level but the autoregressive style transfer is incomplete.
Domain: skewed toward Python, ML, gadget reviews (based on YouTube source channels).
Not safety red-teamed. Don't deploy in user-facing safety-critical settings.
May hallucinate technical specifications, API signatures, or numeric facts — verify any code/specs it produces.

Bias and risks

Inherits biases from base Llama-3.1 + YouTube comment data (Indian male-dominated audience, certain stylistic patterns).
The Tamil/English ratio in outputs is heavily skewed toward English; this is a known limitation.

Training hyper-parameters

Setting	Value
Optimizer	paged_adamw_8bit
Learning rate	2e-4
LR scheduler	cosine
Warmup ratio	0.03
Per-device batch size	2
Gradient accumulation	8 (effective batch 16)
Max sequence length	512
Epochs	1
Mixed precision	bf16
LoRA rank / alpha	8 / 16
Dropout	0.05
Quantization	4-bit NF4 + double quant

Final training loss: 1.55 → 0.97 (eval loss).

Repository and code

Full reproducible pipeline (data collection, preprocessing, training, evaluation, custom metrics) available in the source code distribution. See the dataset card for citation details on the underlying TamilTech-QA corpus.

Citation

bibtex

@misc{tamiltech-qa-2026,
  title  = {TamilTech-QA: A Tanglish Technical Question-Answering Dataset and Fine-Tuned LLM},
  author = {Dheepak Karan E S},
  year   = {2026},
  url    = {https://huggingface.co/dheepakkaran/TamilTech-QA-Llama3.1-8B-QLoRA},
}

Acknowledgements

Base model courtesy of Meta AI (Llama 3.1). Training infrastructure courtesy of Kaggle (free T4 GPU). Training data sourced from public Tamil tech YouTube channels with attribution.