Domain-specific contrastive embedding model for Amazon Fashion product search, trained on 1.25M (review, product metadata) pairs from the Amazon Reviews 2023 Fashion dataset.

Approach

Based on BLaIR (Hou et al., 2024) — SimCSE-style contrastive learning with infoNCE loss and in-batch negatives, plus auxiliary MLM objective.

Base model: roberta-base (125M params)
Training data: 1.25M filtered Fashion review-product pairs (rating >= 4, text length >= 50)
Loss: infoNCE (temp=0.05) + MLM aux (weight=0.1)
Pooling: [CLS] token through MLP, L2-normalized
Output: 768-d embeddings

Usage

python

import torch
from transformers import RobertaTokenizer
from models import RobertaForCL

# Load
tokenizer = RobertaTokenizer.from_pretrained("benzbe/blair-fashion-base")
model = RobertaForCL.from_pretrained(
    "benzbe/blair-fashion-base",
    temp=0.05, pooler_type="cls", do_mlm=False
)
model.eval()

# Encode texts
texts = ["comfortable running shoes", "leather handbag", "summer floral dress"]
inputs = tokenizer(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")

with torch.no_grad():
    embeddings = model.sentence_embed(inputs["input_ids"], inputs["attention_mask"])
    # Shape: (3, 768), L2-normalized

# Cosine similarity
sim = embeddings @ embeddings.T
print(sim)

Performance

On a 10-query ground-truth test against 100 random Fashion products:

90% correct@1 (9/10 exact match at top result)
100% correct@5 (all correct items within top 5)

Training Details

Parameter	Value
Epochs	5+ (stopped at ~89K steps)
Batch size	64 × 4 accumulation = 256 effective
Learning rate	5e-5 (linear warmup 10%)
Max seq length	128
Best val loss	0.8708
GPU	RTX 4090 (24GB)