We project queries into PCA-reduced embedding space for semantic search — 50 dimensions, 97% reduction, 8% better precision.
Eigen-Projected Semantic Search with Dimensionality Reduction
The Problem
Standard semantic search in 1536 dimensions suffers from the concentration of distances phenomenon. High-dimensional similarity search is both slow and less discriminative than search in a reduced semantic subspace.
