You can deploy this model and start earning money today!

Johonson/
adasparse-1B

peft

llama

sparse-retrieval

information-retrieval

msmarco

meta-llama/Llama-3.2-1B

adapter

llama3.2

Views

No views yet

AdaSparse-1B (LLaMA-3.2-1B, MS MARCO)

AdaSparse sparse retriever built on meta-llama/Llama-3.2-1B, trained on MS MARCO with contrastive + knowledge-distillation loss, an adaptive top-k pruning and a learned per-term threshold. This repository contains the LoRA adapter (including the learned q_thres/d_thres thresholding modules), the tokenizer, and the retriever config.

Usage

Requires the AdaSparse codebase:

python

import torch
from transformers import AutoTokenizer
from scaling_retriever.modeling.llm_encoder import LlamaBiSparse

model = LlamaBiSparse.load_from_lora("Johonson/adasparse-1B")
tokenizer = AutoTokenizer.from_pretrained("Johonson/adasparse-1B")

queries = ["What is the capital of France?"]
passages = ["Paris is the capital of France."]

tokenized_queries = tokenizer(queries, max_length=192, truncation=True,
                              padding="longest", return_tensors="pt")
tokenized_passages = tokenizer(passages, max_length=192, truncation=True,
                               padding="longest", return_tensors="pt")

query_embeds = model.query_encode(**tokenized_queries)
doc_embeds = model.doc_encode(**tokenized_passages)
scores = torch.matmul(query_embeds, doc_embeds.T)

Note: the base model meta-llama/Llama-3.2-1B is gated — request access on its model page first.