You can deploy this model and start earning money today!

nomic-text-embed COVID QA Matryoshka test

Name: test
Rating: 2.5 (30 reviews)

This is a sentence-transformers model finetuned from nomic-ai/nomic-embed-text-v1.5. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: nomic-ai/nomic-embed-text-v1.5
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 768 tokens
Similarity Function: Cosine Similarity

Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: NomicBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("JerryO3/test")
# Run inference
sentences = [
    'The monolayers were removed from their plastic surfaces and serially passaged whenever they became confluent. Cells were plated out onto 96-well culture plates for cytotoxicity and anti-influenza assays, and propagated at 37 °C in an atmosphere of 5% CO 2 . The influenza strain A/Leningrad/134/17/1957 H2N2) was purchased from National Control Institute of Veterinary Bioproducts and Pharmaceuticals (Beijing, China). Virus was routinely grown on MDCK cells. The stock cultures were prepared from supernatants of infected cells and stored at −80 °C. The cellular toxicity of patchouli alcohol on MDCK cells was assessed by the MTT method. Briefly, cells were seeded on a microtiter plate in the absence or presence of various concentrations (20 µM -0.0098 µM) of patchouli alcohol (eight replicates) and incubated at 37 °C in a humidified atmosphere of 5% CO 2 for 72 h. The supernatants were discarded, washed with PBS twice and MTT reagent (5 mg/mL in PBS) was added to each well. After incubation at 37 °C for 4 h, the supernatants were removed, then 200 μL DMSO was added and incubated at 37 °C for another 30 min.',
    'What method was used to measure the inhibition of viral replication?',
    'What can be a factor in using common vectors for the delivery of vaccines?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Information Retrieval

Dataset: dim_768
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.321
cosine_accuracy@3	0.6049
cosine_accuracy@5	0.7222
cosine_accuracy@10	0.858
cosine_precision@1	0.321
cosine_precision@3	0.2016
cosine_precision@5	0.1444
cosine_precision@10	0.0858
cosine_recall@1	0.321
cosine_recall@3	0.6049
cosine_recall@5	0.7222
cosine_recall@10	0.858
cosine_ndcg@10	0.5726
cosine_mrr@10	0.4832
cosine_map@100	0.4877

Information Retrieval

Dataset: dim_512
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3395
cosine_accuracy@3	0.6173
cosine_accuracy@5	0.6914
cosine_accuracy@10	0.8395
cosine_precision@1	0.3395
cosine_precision@3	0.2058
cosine_precision@5	0.1383
cosine_precision@10	0.084
cosine_recall@1	0.3395
cosine_recall@3	0.6173
cosine_recall@5	0.6914
cosine_recall@10	0.8395
cosine_ndcg@10	0.577
cosine_mrr@10	0.4943
cosine_map@100	0.5

Information Retrieval

Dataset: dim_256
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.3148
cosine_accuracy@3	0.5864
cosine_accuracy@5	0.6605
cosine_accuracy@10	0.7901
cosine_precision@1	0.3148
cosine_precision@3	0.1955
cosine_precision@5	0.1321
cosine_precision@10	0.079
cosine_recall@1	0.3148
cosine_recall@3	0.5864
cosine_recall@5	0.6605
cosine_recall@10	0.7901
cosine_ndcg@10	0.5455
cosine_mrr@10	0.468
cosine_map@100	0.4775

Information Retrieval

Dataset: dim_128
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.2716
cosine_accuracy@3	0.537
cosine_accuracy@5	0.6543
cosine_accuracy@10	0.7284
cosine_precision@1	0.2716
cosine_precision@3	0.179
cosine_precision@5	0.1309
cosine_precision@10	0.0728
cosine_recall@1	0.2716
cosine_recall@3	0.537
cosine_recall@5	0.6543
cosine_recall@10	0.7284
cosine_ndcg@10	0.4966
cosine_mrr@10	0.4221
cosine_map@100	0.4335

Information Retrieval

Dataset: dim_64
Evaluated with InformationRetrievalEvaluator

Metric	Value
cosine_accuracy@1	0.2407
cosine_accuracy@3	0.4753
cosine_accuracy@5	0.5864
cosine_accuracy@10	0.6728
cosine_precision@1	0.2407
cosine_precision@3	0.1584
cosine_precision@5	0.1173
cosine_precision@10	0.0673
cosine_recall@1	0.2407
cosine_recall@3	0.4753
cosine_recall@5	0.5864
cosine_recall@10	0.6728
cosine_ndcg@10	0.4509
cosine_mrr@10	0.3798
cosine_map@100	0.3911

Training Details

Training Dataset

Unnamed Dataset

Size: 1,453 training samples
Columns: positive and anchor
Approximate statistics based on the first 1000 samples:
positive anchor
type string string
details
min: 112 tokens
mean: 319.17 tokens
max: 778 tokens
min: 6 tokens
mean: 14.84 tokens
max: 65 tokens

Samples:

positive	anchor
We find that the slowing growth in daily reported deaths in Italy is consistent with a significant impact of interventions implemented several weeks earlier. In Italy, we estimate that the effective reproduction number, Rt, dropped to close to 1 around the time of Iockdown (11th March), although with a high level of uncertainty. Overall, we estimate that countries have managed to reduce their reproduction number. Our estimates have wide credible intervals and contain 1 for countries that have implemented a
[46] Where the biological samples are taken from also play a role in the sensitivity of these tests. For SARS-CoV and MERS-CoV, specimens collected from the lower respiratory tract such as sputum and tracheal aspirates have higher and more prolonged levels of viral RNA because of the tropism of the virus. MERS-CoV viral loads are also higher for severe cases and have longer viral shedding compared to mild cases. Although upper respiratory tract specimens such as nasopharyngeal or oropharyngeal swabs can be used, they have potentially lower viral loads and may have higher risk of false-negatives among the mild MERS and SARS cases [102, 103] , and likely among the 2019-nCoV cases. The existing practices in detecting genetic material of coronaviruses such as SARS-CoV and MERS-CoV include (a) reverse transcription-polymerase chain reaction (RT-PCR), (b) real-time RT-PCR (rRT-PCR), (c) reverse transcription loop-mediated isothermal amplification (RT-LAMP) and (d) real-time RT-LAMP [104] . Nucleic amplification tests (NAAT) are usually preferred as in the case of MERS-CoV diagnosis as it has the highest sensitivity at the earliest time point in the acute phase of infection [102] . Chinese health authorities have recently posted the full genome of 2019-nCoV in the GenBank and in GISAID portal to facilitate in the detection of the virus [11] . Several laboratory assays have been developed to detect the novel coronavirus in Wuhan, as highlighted in WHO's interim guidance on nCoV laboratory testing of suspected cases.	`Why are Nucleic amplification tests (NAAT) usually preferred as in the case of MERS-CoV diagnosis?`
By the time symptoms appear in HCPS, both strong antiviral responses, and, for the more virulent viral genotypes, viral RNA can be detected in blood plasma or nucleated blood cells respectively [63, 64] . At least three studies have correlated plasma viral RNA with disease severity for HCPS and HFRS, suggesting that the replication of the virus plays an ongoing and real-time role in viral pathogenesis [65] [66] [67] . Several hallmark pathologic changes have been identified that occur in both HFRS and HCPS. A critical feature of both is a transient (~ 1-5 days) capillary leak involving the kidney and retroperitoneal space in HFRS and the lungs in HCPS. The resulting leakage is exudative in character, with chemical composition high in protein and resembling plasma. The continued experience indicating the strong tissue tropism for endothelial cells, specifically, is among the several factors that make β3 integrin an especially attractive candidate as an important in vivo receptor for hantaviruses. It is likely that hantaviruses arrive at their target tissues through uptake by regional lymph nodes, perhaps with or within an escorting lung histiocyte. The virus seeds local endothelium, where the first few infected cells give rise, ultimately, to a primary viremia, a process that appears to take a long time for hantavirus infections [62, 63] .	`Which is an especially attractive candidate as an important in vivo receptor for hantaviruses?`

Loss: MatryoshkaLoss with these parameters:
json
1{ 2 "loss": "MultipleNegativesRankingLoss", 3 "matryoshka_dims": [ 4 768, 5 512, 6 256, 7 128, 8 64 9 ], 10 "matryoshka_weights": [ 11 1, 12 1, 13 1, 14 1, 15 1 16 ], 17 "n_dims_per_step": -1 18}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: epoch
learning_rate: 2e-05
num_train_epochs: 4
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: True
tf32: True
load_best_model_at_end: True
optim: adamw_torch_fused
auto_find_batch_size: True
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: epoch
prediction_loss_only: True
per_device_train_batch_size: 8
per_device_eval_batch_size: 8
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
learning_rate: 2e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 4
max_steps: -1
lr_scheduler_type: cosine
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: True
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: True
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: False
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: True
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
dispatch_batches: None
split_batches: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	dim_128_cosine_map@100	dim_256_cosine_map@100	dim_512_cosine_map@100	dim_64_cosine_map@100	dim_768_cosine_map@100
0.0549	10	5.6725	-	-	-	-	-
0.1099	20	4.6781	-	-	-	-	-
0.1648	30	3.9597	-	-	-	-	-
0.2198	40	3.2221	-	-	-	-	-
0.2747	50	2.2144	-	-	-	-	-
0.3297	60	2.8916	-	-	-	-	-
0.3846	70	1.7038	-	-	-	-	-
0.4396	80	2.4738	-	-	-	-	-
0.4945	90	1.8951	-	-	-	-	-
0.5495	100	1.515	-	-	-	-	-
0.6044	110	1.5431	-	-	-	-	-
0.6593	120	2.4492	-	-	-	-	-
0.7143	130	1.656	-	-	-	-	-
0.7692	140	1.7953	-	-	-	-	-
0.8242	150	1.8679	-	-	-	-	-
0.8791	160	2.1551	-	-	-	-	-
0.9341	170	1.5363	-	-	-	-	-
0.9890	180	1.2529	-	-	-	-	-
1.0	182	-	0.3894	0.4585	0.4805	0.3287	0.4926
1.0440	190	1.319	-	-	-	-	-
1.0989	200	1.0985	-	-	-	-	-
1.1538	210	1.0403	-	-	-	-	-
1.2088	220	0.4363	-	-	-	-	-
1.2637	230	0.2102	-	-	-	-	-
1.3187	240	0.3584	-	-	-	-	-
1.3736	250	0.2683	-	-	-	-	-
1.4286	260	0.4438	-	-	-	-	-
1.4835	270	0.34	-	-	-	-	-
1.5385	280	0.4296	-	-	-	-	-
1.5934	290	0.2323	-	-	-	-	-
1.6484	300	0.3259	-	-	-	-	-
1.7033	310	0.4339	-	-	-	-	-
1.7582	320	0.1524	-	-	-	-	-
1.8132	330	0.0782	-	-	-	-	-
1.8681	340	0.4306	-	-	-	-	-
1.9231	350	0.312	-	-	-	-	-
1.9780	360	0.2112	-	-	-	-	-
2.0	364	-	0.4139	0.4526	0.4762	0.3761	0.4672
2.0330	370	0.2341	-	-	-	-	-
2.0879	380	0.1965	-	-	-	-	-
2.1429	390	0.3019	-	-	-	-	-
2.1978	400	0.1518	-	-	-	-	-
2.2527	410	0.0203	-	-	-	-	-
2.3077	420	0.0687	-	-	-	-	-
2.3626	430	0.0206	-	-	-	-	-
2.4176	440	0.3615	-	-	-	-	-
2.4725	450	0.4674	-	-	-	-	-
2.5275	460	0.0623	-	-	-	-	-
2.5824	470	0.0222	-	-	-	-	-
2.6374	480	0.1049	-	-	-	-	-
2.6923	490	0.4955	-	-	-	-	-
2.7473	500	0.439	-	-	-	-	-
2.8022	510	0.0052	-	-	-	-	-
2.8571	520	0.16	-	-	-	-	-
2.9121	530	0.0583	-	-	-	-	-
2.9670	540	0.0127	-	-	-	-	-
3.0	546	-	0.4427	0.4765	0.508	0.397	0.5021
3.0220	550	0.0143	-	-	-	-	-
3.0769	560	0.0228	-	-	-	-	-
3.1319	570	0.0704	-	-	-	-	-
3.1868	580	0.0086	-	-	-	-	-
3.2418	590	0.001	-	-	-	-	-
3.2967	600	0.002	-	-	-	-	-
3.3516	610	0.0016	-	-	-	-	-
3.4066	620	0.021	-	-	-	-	-
3.4615	630	0.0013	-	-	-	-	-
3.5165	640	0.0723	-	-	-	-	-
3.5714	650	0.0045	-	-	-	-	-
3.6264	660	0.0048	-	-	-	-	-
3.6813	670	0.1005	-	-	-	-	-
3.7363	680	0.0018	-	-	-	-	-
3.7912	690	0.0101	-	-	-	-	-
3.8462	700	0.0104	-	-	-	-	-
3.9011	710	0.0025	-	-	-	-	-
3.9560	720	0.014	-	-	-	-	-
4.0	728	-	0.4335	0.4775	0.5000	0.3911	0.4877

The bold row denotes the saved checkpoint.

Framework Versions

Python: 3.11.9
Sentence Transformers: 3.0.1
Transformers: 4.41.2
PyTorch: 2.1.2+cu121
Accelerate: 0.31.0
Datasets: 2.19.1
Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

bibtex

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

	positive	anchor
type	string	string
details	min: 112 tokens mean: 319.17 tokens max: 778 tokens	min: 6 tokens mean: 14.84 tokens max: 65 tokens