Qodo-Embed-1 is a state-of-the-art code embedding model designed for retrieval tasks in the software development domain.
It is offered in two sizes: lite (1.5B) and medium (7B). The model is optimized for natural language-to-code and code-to-code retrieval, making it highly effective for applications such as code search, retrieval-augmented generation (RAG), and contextual understanding of programming languages.
This model outperforms all previous open-source models in the COIR and MTEB leaderboards, achieving best-in-class performance with a significantly smaller size compared to competing models.
Languages Supported:
Python
C++
C#
Go
Java
Javascript
PHP
Ruby
Typescript
Model Information
Model Size: 7B
Embedding Dimension: 3584
Max Input Tokens: 32k
Requirements
transformers>=4.39.2
flash_attn>=2.5.6
Usage
Sentence Transformers
python
1from sentence_transformers import SentenceTransformer
23# Download from the 🤗 Hub4model = SentenceTransformer("Qodo/Qodo-Embed-1-7B")5# Run inference6sentences =[7'accumulator = sum(item.value for item in collection)',8'result = reduce(lambda acc, curr: acc + curr.amount, data, 0)',9'matrix = [[i*j for j in range(n)] for i in range(n)]'10]11embeddings = model.encode(sentences)12print(embeddings.shape)13# [3, 1536]1415# Get the similarity scores for the embeddings16similarities = model.similarity(embeddings, embeddings)17print(similarities.shape)18# [3, 3]
Transformers
python
1import torch
2import torch.nn.functional as F
34from torch import Tensor
5from transformers import AutoTokenizer, AutoModel
678deflast_token_pool(last_hidden_states: Tensor,9 attention_mask: Tensor)-> Tensor:10 left_padding =(attention_mask[:,-1].sum()== attention_mask.shape[0])11if left_padding:12return last_hidden_states[:,-1]13else:14 sequence_lengths = attention_mask.sum(dim=1)-115 batch_size = last_hidden_states.shape[0]16return last_hidden_states[torch.arange(batch_size, device=last_hidden_states.device), sequence_lengths]171819# Each query must come with a one-sentence instruction that describes the task20queries =[21'how to handle memory efficient data streaming',22'implement binary tree traversal'23]2425documents =[26"""def process_in_chunks():
27 buffer = deque(maxlen=1000)
28 for record in source_iterator:
29 buffer.append(transform(record))
30 if len(buffer) >= 1000:
31 yield from buffer
32 buffer.clear()""",3334"""class LazyLoader:
35 def __init__(self, source):
36 self.generator = iter(source)
37 self._cache = []
3839 def next_batch(self, size=100):
40 while len(self._cache) < size:
41 try:
42 self._cache.append(next(self.generator))
43 except StopIteration:
44 break
45 return self._cache.pop(0) if self._cache else None""",4647"""def dfs_recursive(root):
48 if not root:
49 return []
50 stack = []
51 stack.extend(dfs_recursive(root.right))
52 stack.append(root.val)
53 stack.extend(dfs_recursive(root.left))
54 return stack"""55]56input_texts = queries + documents
5758tokenizer = AutoTokenizer.from_pretrained('Qodo/Qodo-Embed-1-7B', trust_remote_code=True)59model = AutoModel.from_pretrained('Qodo/Qodo-Embed-1-7B', trust_remote_code=True)6061max_length =81926263# Tokenize the input texts64batch_dict = tokenizer(input_texts, max_length=max_length, padding=True, truncation=True, return_tensors='pt')65outputs = model(**batch_dict)66embeddings = last_token_pool(outputs.last_hidden_state, batch_dict['attention_mask'])6768# normalize embeddings69embeddings = F.normalize(embeddings, p=2, dim=1)70scores =(embeddings[:2] @ embeddings[2:].T)*10071print(scores.tolist())
