Is this your dataset? Claim it with the Hugging Face account that owns it.

stai-tuebingen/
faiss-smollm

Name: faiss-smollm
Creator: stai-tuebingen
License: https://spdx.org/licenses/ODC-By-1.0.html

Views

No views yet

FAISS-Based Novelty Detection for SmolLM and SmolLM2

This tutorial demonstrates how to the measure novelty of text queries with respect to the provided SmolLM and SmolLM2 pretraining corpora, with optional ColBERTv2 re-ranking for improved precision.

Overview

The pipeline consists of four main steps:

Generate Embeddings - Encode your queries using a sentence transformer FAISS Search - Retrieve top-K most similar documents from the pretraining corpus Combine Results -… See the full description on the dataset page: https://huggingface.co/datasets/stai-tuebingen/faiss-smollm.