For retrieval task, We sampled 10 hard negative passages/answers from top50-top200 related passages/answers for each query.
For classification/clustering tasks, we sampled 5 hard negative samples from other classes/cluster for each sample.
For classification/clustering tasks, we also used the category names of each class and cluster as positive and negative samples.

Data synthesis by LLM (ZPoint-72B)

For retrieval tasks, we used LLM to rewrite each query, generating five different rewritten results.
For retrieval tasks, we also generated five new queries for some documents by LLM.
For non-retrieval tasks, we used LLM to rewrite the queries, generating five rewritten results for each query.
Finally, total amount of synthesized data is about 30 million.

Collect more data for retrieval-type tasks

miracl/miracl
FreedomIntelligence/Huatuo26M-Lite
PaddlePaddle/dureader_robust C-MTEB test filtered
THUIR/T2Ranking C-MTEB test filtered
Shitao/bge-reranker-data
Shitao/MLDR
...

We constructed a dataset of approximately 100 million training samples through collection, machine translation, and LLM synthesis. This dataset includes data from various fields such as healthcare, law, electricity, automotive, and 3C (Consumer Electronics).

Training loss

Multi-Task loss like Piccolo
Matryoshka Representation Learning

Example

python

from sentence_transformers import SentenceTransformer
sentences1 = ["这个产品真垃圾"]
sentences2 = ["我太喜欢这个产品了"]
model = SentenceTransformer('iampanda/zpoint_large_embedding_zh')
embeddings_1 = model.encode(sentences1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)