Reduced SigLIP for Person Visual Descriptions

---

Model Details for Reduced version

• Base model: google/siglip2-base-patch16-224
• Reduced dimension: 64
• Architecture modifications: Added two linear layers (for text and image) to project embeddings from the original dimension down to reduced_dim.

---

Intended Uses & Limitations

Example Applications

• Person retrieval based on textual or visual descriptions of the person
• Person re-identification
• Embedding extraction for retrieval systems

Limitations and Bias

• May inherit biases from the base SigLIP model and training data
• Not suitable for tasks requiring detailed fine-grained recognition without further training
• Trained on surveillance data; suitable for tasks where a substantial portion of the person is visible

---

Training

Loss function

• Type: Soft contrastive loss with label smoothing; optional image-to-image contrastive loss for same-identity image pairs.
• Description: The model is trained to align text and image embeddings using a modified contrastive objective. Instead of relying on hard one-hot targets, label smoothing allocates a small probability mass to all other samples in the batch. All embeddings are normalized prior to similarity computation, and the loss is applied symmetrically in both the image-to-text and text-to-image directions. To encourage re-identification and emphasize clothing features rather than pose or background, an additional image-to-image contrastive loss can be incorporated.

Evaluation Metrics

• Truncated Cumulative Matching Characteristic (CMC) AUC
  • Measures the fraction of queries where the correct match appears within the top K ranks (e.g., top-10). Unlike MRR or strict top-1 accuracy, this metric rewards consistent retrieval of all relevant matches near the top ranks, rather than a few perfect hits with others ranked very low.

Datasets

• Sources: CUHK-PEDES, ICFG-PEDES, IIITD, ITCPR, PRW-TBPS, PETA, (part of SYNTH and RSTPReid for testing)
• Processing: Text descriptions were processed with the Mistral LLM to remove ambiguous information about pose or context, leaving only clear visual characteristics in a structured format, where key features are separated with comma. During train/val/test the original splits are respected as much as possible.

• CUHK-PEDES: test: 1000 identities following the original split; validation: 150 identities sampled from the original validation split; train: all remaining identities
• ICFG-PEDES: test: 946 identities following the original split; validation: 150 identities sampled from the original training split; train: all remaining identities
• IIITD: test: 2500 identities following the original split; validation: 150 identities sampled from the original validation split; train: all remaining identities
• ITCPR: Originally designed for zero-shot evaluation, so a new split is created:test: 1000 randomly selected identities; validation: 150 randomly selected identities; train: all remaining identities
• PRW: test: 450 identities following the original split; validation: 150 identities sampled from the original training split; train: all remaining identities
• PETA: The original random image-level split can cause identity leakage. Therefore, the identity-based split from PETA-ZS is adopted: test: 1706 identities following the original split; validation: 150 identities sampled from the original validation split; train: all remaining identities
• RSTPReid: Only the 200 test images are used for evaluation due to annotation errors in the rest of the dataset.
• SYNTH: 1000 random images are used for the test set because the generated captions are noisy and sometimes inaccurate.

Training Setup

• Frozen pretrained base model initially; only projection layers were trained.
• Fine-tuning: later trained the model head and then the whole model.
• Optimizer: AdamW
• Learning rate: 1e-4 for warm-up, 5e-6 for the rest
• Epochs: 4 epochs for projection layer warm-up, 4 epochs for model head fine-tuning, and 50 epochs for full model fine-tuning
• Visual Augmentations: random operations including small rotations, scale, hue variations, horizontal flip, and color jitter
• Text Augmentations: random subsets of key features are selected and removed from the comma-separated description strings to create augmented training samples
• Training Codes: codes are available on [github] (https://github.com/MarketaJu/ReducedSiglipForPersonDescription)
• Package Versions: torch==2.9.1, transformers==4.57.3, pillow==12.0.0, torchvision==0.24.1

---

Results

Evaluation of the Model per Dataset

• For each text query, the goal is to retrieve the correct identity, not the exact corresponding image.
• During evaluation, scores are computed over all images belonging to the same identity, and the maximum score is taken to represent that identity.
• These identity-level scores are then ranked, and the following retrieval metrics are calculated:
  • Top-k: standard top-1, top-5, top-10 accuracy
  • MRR: Mean Reciprocal Rank
  • CMC AUC: Truncated CMC AUC evaluated up to rank 20

Cross-Model Comparison

---

Usage