You can deploy this model and start earning money today!

stefanj0/
qwen2.5-3b-qlora-latex-fluentmath

Name: qwen2.5-3b-qlora-latex-fluentmath
Availability: InStock

Views

No views yet

Qwen2.5-3B QLoRA: LaTeX → FluentMath Translation

Fine-tuned model for converting LaTeX mathematical expressions to natural spoken English (FluentMath style)

This model is a QLoRA (4-bit quantized + LoRA adapters) fine-tuned version of Qwen/Qwen2.5-3B-Instruct for the specialized task of translating LaTeX mathematical notation into natural, conversational English suitable for text-to-speech systems.

Model Description

Base Model: Qwen/Qwen2.5-3B-Instruct (2.93B parameters)
Fine-tuning Method: QLoRA (4-bit quantization + LoRA adapters)
Training Data: 10,000 augmented LaTeX → FluentMath pairs from AutoMathText dataset
Task: Sequence-to-sequence translation of mathematical notation
License: Apache 2.0 (inherited from base model)

Performance Metrics

Evaluated on 500 held-out test samples:

Metric	Score
Exact Match Accuracy	42.60%
BLEU (F1)	93.39%
Edit Distance Similarity	90.56%

Comparison with 270M baseline (Gemma-3-270m):

Exact Match: +42.60 percentage points
BLEU: +5.75 percentage points
Similarity: +9.36 percentage points
VRAM: 5.86GB (less than 270M full precision!)

Training Details

Training Configuration

Learning Rate: 2e-4 (higher for LoRA)
Epochs: 7
Batch Size: 1 per device
Gradient Accumulation: 16 steps (effective batch size: 16)
Optimizer: paged_adamw_8bit (memory-efficient)
LR Scheduler: Cosine with 10% warmup
Precision: BF16
LoRA Parameters:
- Rank (r): 16
- Alpha: 32
- Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
- Dropout: 0.05

Training Results

Final Training Accuracy: 94.45% (mean token accuracy)
Final Training Loss: 0.198
Training Time: ~3.5 hours on RTX 4070 SUPER (12GB)
Peak VRAM: ~6-7GB during training

Data Augmentation Strategy

Training data includes controlled noise to improve robustness:

60% Clean: Original high-quality verbalizations
20% Tier 1: Semantic-preserving variations (whitespace, notation alternatives)
15% Tier 2: Delimiter variations (unmatched braces, extra brackets)
5% Tier 2: Structural variations (command typos, missing braces)

Usage

Installation

pip install torch transformers peft accelerate bitsandbytes

Inference

python

import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

# Load model with LoRA adapters
model = AutoPeftModelForCausalLM.from_pretrained(
    "stefanj0/qwen2.5-3b-qlora-latex-fluentmath",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("stefanj0/qwen2.5-3b-qlora-latex-fluentmath")

def translate_latex(latex: str) -> str:
    """Translate LaTeX to FluentMath."""
    user_prompt = (
        "Convert the following LaTeX mathematical expression to natural spoken English:\n\n"
        f"LaTeX: {latex}\n\n"
        "Spoken form:"
    )

    # Format as chat
    prompt = tokenizer.apply_chat_template(
        [{"role": "user", "content": user_prompt}],
        tokenize=False,
        add_generation_prompt=True
    )

    # Generate
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=256,
            pad_token_id=tokenizer.eos_token_id,
        )

    # Parse output
    generated = tokenizer.decode(outputs[0], skip_special_tokens=True)
    if "Spoken form:" in generated:
        verbalization = generated.split("Spoken form:")[-1].strip()
    else:
        verbalization = generated.strip()

    # Remove role prefix if present
    if verbalization.startswith("assistant\n"):
        verbalization = verbalization[10:].strip()

    return verbalization

# Example usage
latex = r"\int_a^b f'(x)\,dx = f(b) - f(a)"
translation = translate_latex(latex)
print(translation)
# Output: "the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh"

Example Translations

LaTeX	FluentMath Output
`\frac{a}{b}`	a over b
`x^2 + y^2 = r^2`	x squared plus y squared equals r squared
`\int_a^b f'(x)\,dx = f(b) - f(a)`	the integral from eigh to b of, f prime, of x, d x, equals, f of b minus f of eigh
`P(A\|B) = \frac{P(B\|A)P(A)}{P(B)}`	P, the quantity Eigh divides B, equals, the fraction with numerator, P, the quantity B divides Eigh, P of Eigh, and denominator P of B
`\lim_{n\to\infty} \sum_{k=1}^{n} \frac{1}{n}`	the limit as n approaches infinity, of, sum from k equals 1 to n of, 1 over n
`A\vec{v} = \lambda\vec{v}`	Eigh, bold v, equals, lambda bold v
`i\hbar\frac{\partial}{\partial t}\Psi = \hat{H}\Psi`	i h bar, the fraction with numerator, partial derivative, and denominator partial derivative t, Psi, equals, H hat, Psi

Use Cases

Text-to-Speech Systems: Convert mathematical papers to audio
Accessibility: Help visually impaired users access mathematical content
Education: Generate natural language descriptions of mathematical expressions
Data Augmentation: Create training data for math-aware language models
Documentation: Auto-generate spoken descriptions for LaTeX in technical docs

Limitations

Letter Pronunciation: Capital 'A' rendered as "Eigh" (from speech synthesis rules)
Vector Notation: Vectors shown as "bold v" instead of "v with right arrow"
Custom Macros: Does not support \newcommand or custom LaTeX macros
Long Expressions: May truncate expressions longer than 256 tokens
Domain Coverage: Trained primarily on academic mathematics; may struggle with domain-specific notation

Technical Details

Why QLoRA?

QLoRA (Quantized LoRA) enables training large models on consumer GPUs:

4-bit Quantization: 75% memory reduction (stores weights in 4 bits vs 16)
LoRA Adapters: Train only ~1% of parameters (64M trainable out of 2.93B total)
Quality: Achieves 95-98% of full fine-tuning performance
Efficiency: Trains 11x larger model (3B vs 270M) using LESS memory (5.86GB vs 6-7GB)

Architecture

Base: Qwen2.5-3B-Instruct (decoder-only transformer)
Quantization: 4-bit NormalFloat (NF4) with double quantization
LoRA Rank: 16 (adapters inject low-rank updates into attention and MLP layers)
Target Modules: All attention projections (Q, K, V, O) and MLP gates/projections

Training Pipeline

Data Extraction: LaTeX expressions extracted from AutoMathText dataset
Verbalization: Converted to speech using mathwords library (Rust/PyO3)
Evaluation: Quality-checked with Qwen3-0.6B evaluator (93% accuracy)
Correction: Post-processed to fix common failure patterns
Augmentation: Noise added for robustness (10K samples total)
Fine-tuning: 7 epochs with QLoRA on RTX 4070 SUPER (12GB)

Citation

If you use this model, please cite:

bibtex

@misc{qwen25-3b-qlora-latex-fluentmath,
  author = {Stefan J.},
  title = {Qwen2.5-3B QLoRA: LaTeX to FluentMath Translation},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/stefanj0/qwen2.5-3b-qlora-latex-fluentmath}
}

Also cite the base model:

bibtex

@article{qwen2.5,
  title={Qwen2.5: A Party of Foundation Models},
  author={Qwen Team},
  year={2024},
  journal={arXiv preprint arXiv:2412.xxxxx}
}

Framework Versions

PEFT: 0.18.0
TRL: 0.26.2
Transformers: 4.57.3
PyTorch: 2.9.1
Datasets: 4.4.2
Tokenizers: 0.22.1

Acknowledgments

Base Model: Qwen/Qwen2.5-3B-Instruct by Alibaba Cloud
Training Framework: Hugging Face Transformers
QLoRA Implementation: PEFT + bitsandbytes
Dataset: AutoMathText by Math-AI
Verbalization: mathwords library (Rust/PyO3 wrapper for MathCAT)

Model Card Authors

Stefan J. (stefanj0)

Model Card Contact

For questions or issues, please open an issue on the model's discussion page.