COIP — Contrastive Olfaction-Image Pretraining
Joint embeddings of
smell and sight, trained on
New York Smells: 7,000 paired
image / electronic-nose recordings captured in the wild across New York City with a
Cyranose 320 (32 conducting-polymer sensors).
Two towers project into a shared 128-d L2-normalised space, so a dot product between an
olfaction embedding and an image embedding is a cosine similarity. Given a smell, you can
retrieve the image of what produced it; given the embedding alone, a linear probe can
name the scene, object, or material.
Usage
1from nys.models import from_pretrained
2
3model = from_pretrained("transformer")
4
5image_emb, smell_emb = model(image, raw_smell) # both (B, 128), L2-normalised
6similarity = smell_emb @ image_emb.t()
raw_smell is (B, 28, 32) — concat(baseline_raw[:14], sample_raw[:14]), z-scored per
sensor with training-split statistics. The smellprint model instead takes the (B, 32)
hand-crafted feature. nys.data.SmellDataset assembles either from the released dataset.
Models
| Encoder | Input | File | Architecture |
|---|
transformer | raw (28, 32) | transformer/model.pth | CLS + 6-layer pre-norm transformer |
cnn | raw (28, 32) | cnn/model.pth | 4 conv blocks over the sensor matrix |
mlp | raw (28, 32) | mlp/model.pth | 3 × Linear/ReLU/LayerNorm |
smellprint | (32,) | smellprint/model.pth | MLP over the smellprint feature |
Each is ~96 MB, trained on the object-level split with the image tower finetuned from
ImageNet initialisation, and carries its training configuration under legacy_opt.
Cross-modal retrieval
Smell query against a gallery of image embeddings, N = 933.
| Smell encoder | Mean rank ↓ | Median rank ↓ | R@5 ↑ | R@10 ↑ | R@20 ↑ |
|---|
| Chance | 467 | 467 | 0.54 | 1.07 | 2.14 |
| MLP (smellprint) | 362.6 | 324 | 1.61 | 2.79 | 5.68 |
| MLP (raw) | 134.4 | 48 | 13.50 | 22.19 | 33.33 |
| CNN (raw) | 125.0 | 37 | 17.90 | 26.69 | 38.05 |
| Transformer (raw) | 106.1 | 30 | 17.36 | 28.72 | 41.80 |
Training on the raw olfactory signal substantially outperforms the hand-crafted
smellprint on every metric, and the transformer is the strongest of the raw encoders.
Recognising scenes, objects and materials from smell
A linear probe on the frozen olfaction embedding, on the object-level test split.
Random-weight encoders are shown for reference.
| Encoder | Init | Scene (8) | Material (51) | Object (49) |
|---|
| — | chance | 12.5 | 2.0 | 2.0 |
| MLP (smellprint) | random | 29.5 | 2.8 | 3.7 |
| MLP (smellprint) | COIP | 32.7 | 5.0 | 4.8 |
| MLP (raw) | random | 90.6 | 10.8 | 17.1 |
| MLP (raw) | COIP | 94.2 | 10.4 | 17.7 |
| CNN (raw) | random | 71.2 | 9.1 | 12.6 |
| CNN (raw) | COIP | 93.2 | 10.1 | 18.4 |
| Transformer (raw) | random | 72.2 | 8.5 | 10.9 |
| Transformer (raw) | COIP | 87.4 | 14.5 | 16.1 |
Scene is top-1; material and object are balanced accuracy (mean per-class
recall). Those two tasks have 51 and 49 heavily skewed classes — the largest
material class alone is 13.7% of the test set — so unbalanced top-1 there
largely measures the class prior. Scene's 8 classes are roughly even, where the
two metrics coincide.
Scene labels are per-session; object and material labels are GPT-4o annotations
derived from the paired image. All three tasks are well above chance from smell alone.
Scope
60 recording sessions in New York City over roughly three months. Electronic-nose
response varies with temperature, humidity and sensor age, so behaviour on other
hardware, other climates, or odorants unlike those in the dataset is untested.
Citation
1@article{ozguroglu2025smell,
2 title={New York Smells: A Large Multimodal Dataset for Olfaction},
3 author={Ozguroglu, Ege and Liang, Junbang and Liu, Ruoshi and Chiquier, Mia and DeTienne, Michael and Qian, Wesley Wei and Horowitz, Alexandra and Owens, Andrew and Vondrick, Carl},
4 journal={arXiv preprint arXiv:2511.20544},
5 year={2025}
6}
License
CC BY 4.0, matching the dataset.
