This repository contains a LoRA (Low-Rank Adaptation) fine-tuned version of Qwen/Qwen3-1.7B. This model is specialized for generating Python code to solve linear algebra problems described in natural language.

Model Details

Model Description

This is a specialized code-generation model. It takes a textual description of a linear algebra problem and outputs a Python script, typically using the numpy library, that solves it.

Developed by: premjatin
Model type: LoRA fine-tune
Language(s): English, Python
License: MIT
Finetuned from model: Qwen/Qwen3-1.7B

How to Get Started with the Model

Use the code below to get started with the model. This example uses 4-bit quantization for efficient inference on consumer hardware.

python

# First, install necessary libraries
# !pip install transformers torch peft accelerate bitsandbytes

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

BASE_MODEL_ID = "Qwen/Qwen3-1.7B"
ADAPTER_MODEL_ID = "premjatin/qwen-linear-algebra-coder"


quantization_config = BitsAndBytesConfig(load_in_4bit=True)

# Load the base model and tokenizer
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
    quantization_config=quantization_config,
)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

# Load the LoRA adapter
model = PeftModel.from_pretrained(base_model, ADAPTER_MODEL_ID)

# --- Define a problem ---
problem_description = """
Find the eigenvalues and eigenvectors of the following 3x3 matrix:
[[4, 0, 1],
 [-2, 1, 0],
 [-2, 0, 1]]
"""

# --- Create a structured prompt ---
prompt = f"""### INSTRUCTION:
You are an AI assistant that generates Python code to solve linear algebra problems.

### PROBLEM:
{problem_description}

### PYTHON SOLUTION:

--- Generate the solution ---

inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=256, do_sample=False, )

response_text = tokenizer.decode(outputs[0], skip_special_tokens=True) print(response_text)


Training Details

Training Data

The model was fine-tuned on a custom dataset of approximately 10,000 problem-and-solution pairs related to linear algebra. The data was structured to teach the model how to convert a word problem into a Python script.
Training Hyperparameters
The model was trained using the PEFT library with the following LoRA configuration:

{
  "r": 16,
  "lora_alpha": 32,
  "lora_dropout": 0.05,
  "bias": "none",
  "task_type": "CAUSAL_LM",
  "target_modules": [
    "down_proj", "k_proj", "v_proj",
    "gate_proj", "up_proj", "q_proj", "o_proj"
  ]
}

Bias, Risks, and Limitations

Code Correctness: The generated code is not guaranteed to be 100% correct or optimal. Always review and test the code before using it in a production environment.

Scope: The model's capabilities are limited to the types of problems present in its training data. It may not perform well on highly complex or novel problems.

Library Bias: The model was primarily trained on solutions using numpy and will heavily favor it in its responses.