This model is built on top of the
intfloat/multilingual-e5-base checkpoint
and has been fine-tuned to specialize in Japanese medical document retrieval.
It leverages crawled Japanese medical web documents and LLM-based query generation and distilation of a strong re-ranker to achieve domain specialization.
This model is designed for Japanese medical document search. It was fine-tuned using 750,000 Japanese medical web documents.
The overall algorithm is based on the work presented in the paper (NOTE: The authors of this model are different from those of this paper):
-
LLM-Based Query Generation:
A large language model is used to generate queries from a set of 50,000 source documents.
- Similar documents in the source set are removed to ensure diversity.
- Query generation is performed using tokyotech-llm/Llama-3.1-Swallow-8B-Instruct-v0.1 with three examples provided for few-shot learning.
- Generated queries are further filtered by using the LLM to check for the inclusion of relevant medical or health-related knowledge; queries failing this check are removed.
-
Candidate Query Validation & Re-ranking:
- The generated queries are used to search the Japanese medical documents using intfloat/multilingual-e5-base.
Only queries in which the original source document appears within the top 100 results are retained.
- A re-ranking step is performed using the cl-nagoya/ruri-reranker-large model.
- Only queries where the original document is ranked at the top are kept.
- The top result is treated as a positive example.
- For candidates ranked between 1 and 100, a min-max scaling is applied. Documents scoring above a threshold (defined as Top 1 score * α) are removed, as they might already be relevant.
- The top 20 of the remaining documents are then used as negative examples.
-
Training Loss:
The model is trained using a combination of:
- InfoNCE Loss (DPR-style): Encouraging embeddings of queries and positive documents to be similar, and those and negative documents to be dissimilar.
- KL Divergence Loss: Minimizing the difference between the re-ranking scores and the model’s predicted scores.
