This is a sentence-transformers model finetuned from sentence-transformers/clip-ViT-B-32 on the skin-positive-negative-pairs dataset. It maps sentences & paragraphs to a None-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/clip-ViT-B-32
Maximum Sequence Length: 77 tokens
Output Dimensionality: None dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- skin-positive-negative-pairs

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): CLIPModel()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("TatvaRA/clip-skin-embeddings")
# Run inference
sentences = [
    'The image depicts the abdominal region with a reticulated, or net-like, erythematous pattern on the skin. The area shows a faint, reddish-brown discoloration with a lacy appearance, indicative of erythema ab igne. The borders of the discoloration are irregular but well-demarcated. No blistering or open lesions are visible, and the skin surface appears intact without signs of scaling or crusting.',
    'The image depicts multiple small, soft, flesh-colored to slightly hyperpigmented skin lesions located on the eyelid and surrounding skin. These lesions are pedunculated with a smooth surface, typical of skin tags (acrochordons). Some are more prominent and raised, while others are flatter against the skin surface. The surrounding skin appears slightly wrinkled, consistent with normal aging changes.',
    'The image shows an elderly person with erythematous patches and papules prominently located on the central face, including the cheeks, nose, and forehead. The skin findings display a symmetrical distribution. The nose is slightly bulbous, suggesting possible rhinophyma changes. The lesions do not have well-defined borders and there is no visible scaling or pustular formations. The background skin appears sun-damaged with fine wrinkling, but without significant telangiectasia visible in this image.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3884, 0.3253],
#         [0.3884, 1.0000, 0.5531],
#         [0.3253, 0.5531, 1.0000]])

Evaluation

Metrics

Triplet

Datasets: skin-dataset-train and skin-dataset-valid
Evaluated with TripletEvaluator

Metric	skin-dataset-train	skin-dataset-valid
cosine_accuracy	1.0	1.0

Training Details

Training Dataset

skin-positive-negative-pairs

Dataset: skin-positive-negative-pairs at a6e13e3
Size: 105 training samples
Columns: anchor, positive, and negative
Approximate statistics based on the first 105 samples:
anchor positive negative
type PIL.PngImagePlugin.PngImageFile string string
details
min: 46 tokens
mean: 75.37 tokens
max: 77 tokens
min: 46 tokens
mean: 75.55 tokens
max: 77 tokens

Samples:

anchor	positive	negative
`<PIL.PngImagePlugin.PngImageFile image mode=RGB size=1022x767 at 0x7F68FEB0AD40>`	The image depicts a melanocytic nevus located on the skin. It is characterized by a well-defined, round brown lesion centrally located on a lighter skin background. The pigmentation appears uniform with a slightly darker speckled pattern distributed evenly throughout the nevus. The borders are smooth and clearly demarcated from the surrounding skin. There is no apparent erythema, ulceration, or significant secondary changes. Sparse hair is visible around the lesion, and the overall lesion is flat, without palpable elevation.	`The image shows the upper lip, featuring a cluster of small, grouped vesicles on an erythematous base. The vesicles are clear, filled with fluid, and some are beginning to coalesce. There is minimal surrounding erythema, and the lesions are located at the vermillion border of the lip. The borders of the vesicles are well-defined, and there are no signs of crusting at this stage.`
`<PIL.PngImagePlugin.PngImageFile image mode=RGB size=294x222 at 0x7F68FEB0A0E0>`	`The image shows a well-defined, erythematous plaque on the skin with a rough, scaly surface. The lesion is located on a hand and exhibits significant inflammation and lichenification. There is evidence of excoriation and possible oozing, suggesting chronic irritation and scratching. The borders of the lesion are distinct, and the surrounding skin appears dry.`	`The image displays multiple open comedones, appearing as black or dark brown plugs, located on the periocular region of the face. These lesions have a characteristic distribution, clustering under the eye and on the upper cheek area. The surrounding skin exhibits signs of chronic sun damage, including fine wrinkles and a mildly rough texture. The borders of the comedones are well-defined, and there is no evidence of erythema or secondary infection around the lesions.`
`<PIL.PngImagePlugin.PngImageFile image mode=RGB size=600x450 at 0x7F68FEB0B160>`	`The image shows a close-up view of the skin with an irregularly shaped lesion displaying a mixture of colors, including pink, brown, and some erythematous areas. The lesion has uneven borders and appears slightly raised with scaly, crusted surfaces. There are areas of keratinization, with the lesion being distributed in a somewhat asymmetrical pattern. Surrounding skin appears less affected, with some visible telangiectasia.`	The image shows the back of an individual with multiple vesicular lesions. These lesions are scattered across the skin surface and vary in size. They appear as small, red, blister-like eruptions that are slightly raised. Some lesions exhibit a clear vesicular head, while others are crusted over, indicating different stages of development. The distribution is widespread across the back, with a noticeable absence of clustering. The lesions have well-defined borders, and the surrounding skin appears otherwise normal without significant erythema or swelling.

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Evaluation Dataset

skin-positive-negative-pairs

Dataset: skin-positive-negative-pairs at a6e13e3
Size: 22 evaluation samples
Columns: anchor, positive, and negative
Approximate statistics based on the first 22 samples:
anchor positive negative
type PIL.PngImagePlugin.PngImageFile string string
details
min: 74 tokens
mean: 76.59 tokens
max: 77 tokens
min: 60 tokens
mean: 75.59 tokens
max: 77 tokens

Samples:

anchor	positive	negative
`<PIL.PngImagePlugin.PngImageFile image mode=RGB size=294x222 at 0x7F68FEB0BE80>`	The image shows an area of skin with multiple open comedones, commonly known as blackheads, scattered across the affected area. These comedones are predominantly located on the cheek and temple of the face. The skin appears weathered and sallow, with some areas exhibiting signs of solar damage, including wrinkling and yellowish discoloration. The borders of the lesions are irregular, and there are multiple large, open comedones. There are secondary changes, such as mild erythema and induration in some areas, suggesting chronic sun exposure.	`The image displays toenails with signs of fungal infection. The nails appear thickened and discolored, predominantly yellowish-brown. There is noticeable onycholysis, especially in the larger toenail, with crumbling and irregular borders. The surface of the nails looks rough and uneven, and there is subungual debris. The surrounding skin appears intact without visible erythema or scaling.`
`<PIL.PngImagePlugin.PngImageFile image mode=RGB size=294x222 at 0x7F68FEB0BC70>`	`The image shows a circular, red lesion on the skin, located on the cheek. It features a central elevated red spot with radiating capillaries, resembling a spider's web. The surrounding skin appears slightly erythematous. The lesion has well-defined borders and is isolated with no other lesions visible in the immediate vicinity.`	`The image shows an area of skin with multiple erythematous, excoriated papules and plaques, some of which are coalescing. The lesions are distributed on the flexural aspect of the extremity. The skin appears dry and lichenified with some areas showing slight scaling. Borders of the individual lesions are irregular, and there is no clear demarcation from the surrounding skin.`
`<PIL.PngImagePlugin.PngImageFile image mode=RGB size=472x720 at 0x7F68FEB0BBE0>`	The image shows an elderly person with erythematous patches and papules prominently located on the central face, including the cheeks, nose, and forehead. The skin findings display a symmetrical distribution. The nose is slightly bulbous, suggesting possible rhinophyma changes. The lesions do not have well-defined borders and there is no visible scaling or pustular formations. The background skin appears sun-damaged with fine wrinkling, but without significant telangiectasia visible in this image.	`The image shows a round, erythematous (red) lesion with a well-defined, slightly raised border. The center of the lesion appears to be clearer with minor scaling, typical of a ringworm (tinea corporis) infection. The lesion is located on skin likely from a limb or torso. The margins are distinct with some subtle scaling toward the periphery.`

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
learning_rate: 0.0001
num_train_epochs: 7

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 0.0001
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 7
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.0
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	skin-dataset-train_cosine_accuracy	skin-dataset-valid_cosine_accuracy
-1	-1	-	-	0.9619	1.0
0.1429	1	2.7596	-	-	-
0.2857	2	2.5414	-	-	-
0.4286	3	2.6204	-	-	-
0.5714	4	2.4667	-	-	-
0.7143	5	2.4014	-	-	-
0.8571	6	2.5579	-	-	-
1.0	7	2.0899	2.2113	-	-
1.1429	8	2.1459	-	-	-
1.2857	9	2.0529	-	-	-
1.4286	10	1.8848	-	-	-
1.5714	11	2.0387	-	-	-
1.7143	12	2.0944	-	-	-
1.8571	13	2.1068	-	-	-
2.0	14	1.6342	2.1221	-	-
0.1429	1	1.7895	-	-	-
0.2857	2	1.6297	-	-	-
0.4286	3	1.9204	-	-	-
0.5714	4	1.9449	-	-	-
0.7143	5	1.8608	-	-	-
0.8571	6	2.186	-	-	-
1.0	7	1.522	1.8561	-	-
1.1429	8	1.5326	-	-	-
1.2857	9	1.3086	-	-	-
1.4286	10	1.1582	-	-	-
1.5714	11	1.4275	-	-	-
1.7143	12	1.3807	-	-	-
1.8571	13	1.4184	-	-	-
2.0	14	0.909	1.7285	-	-
2.1429	15	1.2727	-	-	-
2.2857	16	1.3549	-	-	-
2.4286	17	1.0448	-	-	-
2.5714	18	1.0773	-	-	-
2.7143	19	1.1036	-	-	-
2.8571	20	0.9444	-	-	-
3.0	21	0.5148	1.6311	-	-
3.1429	22	0.8764	-	-	-
3.2857	23	0.809	-	-	-
3.4286	24	0.8724	-	-	-
3.5714	25	0.8276	-	-	-
3.7143	26	0.8923	-	-	-
3.8571	27	0.797	-	-	-
4.0	28	0.5056	1.5989	-	-
4.1429	29	0.7625	-	-	-
4.2857	30	0.6356	-	-	-
4.4286	31	0.8352	-	-	-
4.5714	32	0.6936	-	-	-
4.7143	33	0.6338	-	-	-
4.8571	34	0.5392	-	-	-
5.0	35	0.3826	1.5942	-	-
5.1429	36	0.6252	-	-	-
5.2857	37	0.5969	-	-	-
5.4286	38	0.4948	-	-	-
5.5714	39	0.7617	-	-	-
5.7143	40	0.6775	-	-	-
5.8571	41	0.6289	-	-	-
6.0	42	0.2195	1.5698	-	-
6.1429	43	0.5786	-	-	-
6.2857	44	0.6049	-	-	-
6.4286	45	0.5162	-	-	-
6.5714	46	0.5841	-	-	-
6.7143	47	0.6579	-	-	-
6.8571	48	0.6702	-	-	-
7.0	49	0.2975	1.5598	-	-
-1	-1	-	-	1.0	1.0

Framework Versions

Python: 3.10.12
Sentence Transformers: 5.0.0
Transformers: 4.53.2
PyTorch: 2.7.1+cu126
Accelerate: 1.9.0
Datasets: 4.0.0
Tokenizers: 0.21.2

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

	anchor	positive	negative
type	PIL.PngImagePlugin.PngImageFile	string	string
details		min: 46 tokens mean: 75.37 tokens max: 77 tokens	min: 46 tokens mean: 75.55 tokens max: 77 tokens

TatvaRA/clip-skin-embeddings-finetuned

SentenceTransformer based on sentence-transformers/clip-ViT-B-32

Model Details

Model Description

Model Sources

Full Model Architecture

Usage

Direct Usage (Sentence Transformers)

Evaluation

Metrics

Triplet

Training Details

Training Dataset

skin-positive-negative-pairs

Evaluation Dataset

skin-positive-negative-pairs

Training Hyperparameters

Non-Default Hyperparameters

All Hyperparameters

Training Logs

Framework Versions

Citation

BibTeX

Sentence Transformers

MultipleNegativesRankingLoss

TatvaRA/
clip-skin-embeddings-finetuned