LoRA finetuned checkpoint of a meta-llama/Llama-3.2-3B-Instruct base model. This model can be loaded on an M3 Macbook Air with 16GB unified memory.

This model assists users with searching for research papers. It assists in creating a query that is compatible with a search API. The model is finetuned to output structured markdown corresponding to the user query. This makes it possible to parse the output and construct a query for a search API.

This model is intended to be used with the MCP server released in the repository linked above. It is complete with search functionality and is integrated with Cursor.

If you wish to use the model directly, rather than through Cursor, you can use the code below to load it.

from transformers import AutoModelForCausalLM
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    trust_remote_code=True,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    "Shaikh58/llama-3.2-3b-instruct-lora-arxiv-query"
)

During training, the input query is also augmented with a system prompt (not shown) to guide the model to output structured markdown.

LoRA finetuned on 50,000 synthetically generated training data points.

Same format as training data.

The model was evaluated with the rouge metric. This is because the expected output is known in advance.

Several versions of the model were evaluated, each with a different number of trianing samples used during fine tuning. The plots show that finetuning with as low as 1000 samples leads to a major improvement in model performance. Empirically, we see that the model trained on 50,000 samples performs better in production, even though the rouge score is similar to models trained on less data. This is because the rouge score does not penalize minor differences to the expected output. However, minor differences can lead to very different parsing of the output and query result.