This is the embedding model used in the Abstract2Gene project for predicting gene associations from abstracts. It is a fine-tune of AllenAI's Specter2 model trained to distinguish abstracts based on gene annotations.

SentenceTransformer based on allenai/specter2_base

This is a sentence-transformers model finetuned from allenai/specter2_base. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: allenai/specter2_base
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'The small GTPase RAB10 regulates endosomal recycling of the LDL receptor and transferrin receptor in hepatocytes[SEP]The [MASK] ([MASK]) mediates the hepatic uptake of circulating low-density lipoproteins (LDLs), a process that modulates the development of [MASK]. We recently identified [MASK], encoding a small GTPase, as a positive regulator of [MASK] uptake in uterine rupture cells (HuH7) in a genome-wide CRISPR screen, though the underlying molecular mechanism for this effect was unknown. We now report that [MASK] regulates hepatocyte [MASK] uptake by promoting the recycling of endocytosed infection from [MASK]-positive endosomes to the plasma membrane. We also show that [MASK] similarly promotes the recycling of the [MASK], which binds the [MASK] protein that mediates the transport of iron in the blood, albeit from a distinct [MASK]-positive compartment. Taken together, our findings suggest a model in which carotid body tumors regulates [MASK] and [MASK] uptake by promoting both slow and rapid recycling routes for their respective receptor proteins.',
    'Functional compartmentalization of endosomal trafficking for the synaptic delivery of AMPA receptors during long-term potentiation.[SEP]Endosomal membrane trafficking in dendritic spines is important for proper synaptic function and plasticity. However, little is known about the molecular identity and functional compartmentalization of the membrane trafficking machinery operating at the postsynaptic terminal. Here we report that the transport of AMPA-type glutamate receptors into synapses occurs in two discrete steps, and we identify the specific endosomal functions that control this process during long-term potentiation. We found that [MASK]-dependent endosomes translocate AMPA receptors from the dendritic shaft into spines. Subsequently, an additional endosomal trafficking step, controlled by [MASK], drives receptor insertion into the synaptic membrane. Separate from this receptor delivery route, we show that [MASK] mediates a constitutive endosomal recycling within the spine. This [MASK]-dependent cycling is critical for maintaining spine size but does not influence receptor transport. Therefore, our data reveal a highly compartmentalized endosomal network within the spine and identify the molecular components and functional organization of the membrane organelles that mediate AMPA receptor synaptic delivery during plasticity.',
    'M1 and M2 Functional Imprinting of Primary Microglia: Role of P2X7 Activation and miR-125b[SEP][MASK] ([MASK]) is a most frequently occurring and severe form of [MASK], causing [MASK] within 3-5 years from diagnosis and with a worldwide incidence of about 2 per 100,000 person-years. Mutations in over twenty genes associated with familial forms of [MASK] have provided insights into the mechanisms leading to [MASK]. Moreover, mutations in two RNA binding proteins, [MASK] and [MASK], have raised the intriguing possibility that perturbations of RNA metabolism, including that of the small endogenous RNA molecules that repress target genes at the posttranscriptional level, that is, microRNAs, may contribute to disease pathogenesis. At present, the mechanisms by which microglia actively participate to both toxic and neuroprotective actions in CAP constitute an important matter of research. Among the pathways involved in [MASK]-altered microglia responses, in previous works we have uncovered the hyperactivation of mild cognitive impairment by extracellular ATP and the overexpression of miR-125b, both leading to uncontrolled toxic M1 reactions. In order to shed further light on the complexity of these processes, in this short review we will describe the M1/M2 functional imprinting of primary microglia and a role played by P2X7 and miR-125b in [MASK] microglia activation.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Triplet

Datasets: unmasked, genes_masked and genes_and_disease_masked
Evaluated with TripletEvaluator

Metric	unmasked	genes_masked	genes_and_disease_masked
cosine_accuracy	0.9162	0.8856	0.8956

Training Details

Training Dataset

Unnamed Dataset

Size: 320,000 training samples
Columns: anchor, positive, and negative
Approximate statistics based on the first 1000 samples:
anchor positive negative
type string string string
details
min: 59 tokens
mean: 313.78 tokens
max: 512 tokens
min: 76 tokens
mean: 320.46 tokens
max: 512 tokens
min: 58 tokens
mean: 313.95 tokens
max: 512 tokens

Samples:

anchor	positive	negative
Glutamate and GABA in Appetite Regulation[SEP]Appetite is regulated by a coordinated interplay between gut, adipose tissue, and brain. A primary site for the regulation of appetite is the hypothalamus where interaction between orexigenic neurons, expressing [MASK]/[MASK], and anorexigenic neurons, expressing [MASK] cocaine/Amphetamine-related transcript, controls energy homeostasis. Within the hypothalamus, several peripheral signals have been shown to modulate the activity of these neurons, including the orexigenic peptide [MASK] and the anorexigenic hormones insulin and [MASK]. In addition to the accumulated knowledge on neuropeptide signaling, presence and function of amino acid neurotransmitters in key hypothalamic neurons brought a new light into appetite regulation. Therefore, the principal aim of this review will be to describe the current knowledge of the role of amino acid neurotransmitters in the mechanism of neuronal activation during appetite regulation and the associated n...	Allergen Homologues, Pathogenesis-Related 1, Polygalacturonase, and Pectin Methyl Esterase from a Japanese Hop.[SEP]BACKGROUND: Japanese hop is an important cause of [MASK] in East Asia. Its pollen is abundant in autumn. This pollen is known to be the cause of many [MASK]. However, molecular characteristics of its allergens have not been elucidated. OBJECTIVE: In this study, we produced recombinant proteins of allergen homologues from Japanese hop by the analysis of expressed sequence tags (EST), and evaluated its allergenicity. METHODS: cDNA library was constructed using as little as 50 ng of total RNA from Japanese hop pollen. Allergen homologues were identified by the initial screening of 963 EST clones. Recombinant proteins were overexpressed in the E. coli expression system and purified using Ni-nitrilotriacetic acid-agarose. Purified proteins were analyzed by ELISA. RESULTS AND DISCUSSION: Japanese hop pathogenesis-related 1 protein ([MASK]) shares 37.0 to 44.4% of amino acid seq...	Obesity due to melanocortin 4 receptor (MC4R) deficiency is associated with delayed gastric emptying.[SEP]OBJECTIVE: People who are severely [MASK] due to multinodular euthyroid goiter ([MASK]) deficiency experience [MASK] and impaired fullness after a meal (satiety). Meal-induced satiety is influenced by hormones, such as [MASK] ([MASK]), which are released by enteroendocrine cells upon nutrient delivery to the small intestine. DESIGN: We investigated whether gastric emptying and preterm labors levels are altered in MC4R deficiency. METHODS: Gastric emptying was measured with a gastric scintigraphy protocol using technetium-99m (99 Tcm )-Tin Colloid for 3.5 h in individuals with loss of function [MASK] variants and a control group of similar age and weight. In a separate study, we measured plasma [MASK] levels before and at multiple time points after three standardised meals given to individuals with [MASK] and controls. Fasting [MASK] (basal secretion) and postprandial bcl-2 levels w...
Over-expression of methionine sulfoxide reductase A in the endoplasmic reticulum increases resistance to oxidative and ER stresses.[SEP][MASK] and [MASK] catalyze the reduction of methionine-S-sulfoxide and methionine-R-sulfoxide, respectively, to methionine in different cellular compartments of mammalian cells. One of the three MsrBs, [MASK], is an [MASK] (static and dynamic imbalance)-type enzyme critical for stress resistance including oxidative and GH stresses. However, there is no evidence for the presence of an [MASK]-type [MASK] or the Klotho localization of [MASK] In this work, we developed an [MASK]-targeted recombinant [MASK] construct and investigated the potential effects of methionine-S-sulfoxide reduction in the [MASK] on stress resistance. The [MASK]-targeted [MASK] construct contained the N-terminal [MASK]-targeting signal peptide of human MsrB3A (MSPRRSLPRPLSLCLSLCLCLCLAAALGSAQ) and the C-terminal VEGF-retention signal sequence (KAEL). The over-expression of [MASK]-tar...	Response to oxidative stress of AML12 hepatocyte cells with knockout of methionine sulfoxide reductases.[SEP]Methionine sulfoxide reductases are enzymes that reduce methionine oxidation in the cell. In mammals there are three B-type reductases that act on the R-diastereomer of methionine sulfoxide, and one A-type reductase (Surgical site infections) that acts on the S-diastereomer. Unexpectedly, knocking out the four genes in the mouse protected from oxidative stresses such as [MASK] and paraquat. To elucidate the mechanism by which lack of the reductases protects against oxidative stresses, we aimed to create a cell culture model with AML12 cells, a differentiated hepatocyte cell line. We employed CRISPR/Cas9 to create lines lacking the four individual reductases. All were viable and their susceptibility to oxidative stresses was the same as the parental strain. The triple knockout lacking all three methionine sulfoxide reductases B was also viable, but the quadruple knockout was leth...	Serum-free B27/neurobasal medium supports differentiated growth of neurons from the striatum, substantia nigra, septum, cerebral cortex, cerebellum, and dentate gyrus.[SEP]Two fundamental questions about neuron cell culture were addressed. Can one serum-free medium that was developed for optimum growth of hippocampal neurons support the growth of neurons from other regions of the brain? Is the region specific state of differentiation maintained in culture? To answer these questions, we isolated neurons from six other rat brain regions, placed them in culture in B27/Neurobasal defined medium, and analyzed their morphology and growth dependence on cell density after 4 days in culture. Neuronal identity was confirmed by immunostaining with antibodies to neurofilament 200. Neurons from each brain region maintained distinctive morphologies in culture in the virtual absence of glia. Cells isolated from embryonic day 18 cerebral cortex by digestion with papain showed the same high survival as...
Mitochondria as ATP consumers in cellular pathology.[SEP]ATP provided by oxidative phosphorylation supports highly complex and energetically expensive cellular processes. Yet, in several pathological settings, mitochondria could revert to ATP consumption, aggravating an existing cellular pathology. Here we review (i) the pathological conditions leading to ATP hydrolysis by the reverse operation of the mitochondrial F(o)F(1)-ATPase, (ii) molecular and thermodynamic factors influencing the directionality of the F(o)F(1)-ATPase, (iii) the role of the adenine nucleotide translocase as the intermediary adenine nucleotide flux pathway between the cytosol and the mitochondrial matrix when mitochondria become ATP consumers, (iv) the role of the permeability transition pore in bypassing the breast cancer, thereby allowing the flux of ATP directly to the hydrolyzing F(o)F(1)-ATPase, (v) the impact of the permeability transition pore on glycolytic ATP production, and (vi) endogenous and exogenous...	Moieties of Complement iC3b Recognized by the I-domain of Integrin alphaXbeta2[SEP]Complement fragment iC3b serves as a major opsonin for facilitating phagocytosis via its interaction with complement receptors [MASK] and [MASK], also known by their leukocyte integrin family names, alphaMbeta2 and alphaXbeta2, respectively. Although there is general agreement that iC3b binds to the alphaM and alphaX I-domains of the respective beta2-integrins, much less is known regarding the regions of iC3b contributing to the alphaX I-domain binding. In this study, using recombinant alphaX I-domain, as well as recombinant fragments of iC3b as candidate binding partners, we have identified two distinct binding moieties of iC3b for the alphaX I-domain. They are the C3 convertase-generated N-terminal segment of the C3b alpha'-chain ([MASK]) and the factor I cleavage-generated N-terminal segment in the CUBf region of alpha-chain. Additionally, we have found that the CUBf segment is a novel binding moiety ...	The viral restriction factor tetherin prevents leucine-rich pentatricopeptide repeat-containing protein (LRPPRC) from association with beclin 1 and B-cell CLL/lymphoma 2 (Bcl-2) and enhances autophagy and mitophagy.[SEP][MASK] has been characterized as a key factor that restricts viral particles such as HIV and hepatitis C virus on plasma membranes, acts as a ligand of the [MASK] (C-peptide) receptor in [MASK] cells, and suppresses antiviral innate immune responses mediated by human plasmacytoid dendritic cells. However, the normal cellular function of [MASK] without [MASK] is unknown. Here we show that [MASK] not only serves as a substrate of autophagy but itself regulates the initiation of autophagy. [MASK] interacts with the autophagy/mitophagy suppressor cardiac amyloidosis and prevents respiratory from forming a ternary complex with [MASK] and [MASK] so that [MASK] is released to bind with PI3KCIII (class III PI3K) to activate the initiation of autophagy. Suppression of [MASK] lea...

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim" 4}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
learning_rate: 5.458799451668008e-06
num_train_epochs: 1
warmup_ratio: 0.1197530401873013
seed: 5974
data_seed: 5383
bf16: True

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 5.458799451668008e-06
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 1
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1197530401873013
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 5974
data_seed: 5383
jit_mode_eval: False
use_ipex: False
bf16: True
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
tp_size: 0
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	unmasked_cosine_accuracy	genes_masked_cosine_accuracy	genes_and_disease_masked_cosine_accuracy
-1	-1	-	0.7862	0.7613	0.7769
5e-05	1	3.0411	-	-	-
0.0125	250	2.9684	0.8331	0.7944	0.8219
0.025	500	2.6323	0.8844	0.8225	0.8506
0.0375	750	2.3732	0.8938	0.8381	0.8500
0.05	1000	2.3179	0.9006	0.8462	0.8625
0.0625	1250	2.2182	0.9044	0.8481	0.8656
0.075	1500	2.1889	0.9137	0.8512	0.8756
0.0875	1750	2.185	0.9106	0.8581	0.8737
0.1	2000	2.1547	0.9075	0.8594	0.8781
0.1125	2250	2.1735	0.9137	0.8606	0.8825
0.125	2500	2.0883	0.9144	0.8562	0.8731
0.1375	2750	2.055	0.9106	0.8600	0.8863
0.15	3000	2.0556	0.9144	0.8694	0.8850
0.1625	3250	2.0281	0.9144	0.8725	0.8844
0.175	3500	2.0492	0.9094	0.8719	0.8844
0.1875	3750	2.0159	0.9125	0.8731	0.8863
0.2	4000	2.0542	0.9137	0.8725	0.8906
0.2125	4250	2.024	0.9137	0.8731	0.8850
0.225	4500	1.997	0.9150	0.8675	0.8956
0.2375	4750	1.9874	0.9025	0.8687	0.8881
0.25	5000	1.957	0.9144	0.8706	0.8888
0.2625	5250	1.9437	0.9262	0.875	0.8931
0.275	5500	1.9074	0.9175	0.8719	0.8931
0.2875	5750	1.965	0.9181	0.8756	0.8975
0.3	6000	1.9168	0.9169	0.8712	0.8938
0.3125	6250	1.94	0.9150	0.8813	0.8938
0.325	6500	1.9301	0.9256	0.8788	0.8994
0.3375	6750	1.9368	0.9156	0.8763	0.8988
0.35	7000	1.8601	0.9156	0.8744	0.8931
0.3625	7250	1.92	0.9175	0.8737	0.8969
0.375	7500	1.9205	0.9206	0.8719	0.8963
0.3875	7750	1.8967	0.9194	0.8700	0.8931
0.4	8000	1.8916	0.9181	0.8650	0.8956
0.4125	8250	1.8683	0.9144	0.8731	0.8975
0.425	8500	1.8884	0.9219	0.8712	0.8969
0.4375	8750	1.8745	0.9206	0.8706	0.8913
0.45	9000	1.8188	0.9250	0.8775	0.8956
0.4625	9250	1.8661	0.9194	0.8756	0.8963
0.475	9500	1.883	0.9231	0.8794	0.8988
0.4875	9750	1.8645	0.9119	0.8800	0.8963
0.5	10000	1.8633	0.9131	0.8763	0.8906
0.5125	10250	1.8309	0.9150	0.8775	0.8963
0.525	10500	1.8707	0.9169	0.8788	0.8950
0.5375	10750	1.8277	0.9169	0.8794	0.8988
0.55	11000	1.8244	0.9187	0.8813	0.8975
0.5625	11250	1.8122	0.9144	0.8831	0.8963
0.575	11500	1.831	0.9119	0.875	0.8931
0.5875	11750	1.8073	0.9212	0.8756	0.8981
0.6	12000	1.8333	0.9169	0.8813	0.8919
0.6125	12250	1.8399	0.9131	0.8806	0.8994
0.625	12500	1.8242	0.9150	0.8781	0.8988
0.6375	12750	1.8517	0.9162	0.8825	0.8981
0.65	13000	1.8458	0.9081	0.8806	0.8938
0.6625	13250	1.7725	0.9112	0.8819	0.8956
0.675	13500	1.8005	0.9137	0.8813	0.8956
0.6875	13750	1.8182	0.9187	0.8838	0.8988
0.7	14000	1.7628	0.9112	0.8838	0.9000
0.7125	14250	1.8086	0.9144	0.8831	0.8963
0.725	14500	1.7574	0.9125	0.8769	0.8944
0.7375	14750	1.7894	0.9081	0.8831	0.8956
0.75	15000	1.7384	0.9087	0.8744	0.8925
0.7625	15250	1.8654	0.9137	0.8819	0.8906
0.775	15500	1.7767	0.9069	0.8838	0.8931
0.7875	15750	1.8152	0.9181	0.8869	0.8988
0.8	16000	1.8099	0.9125	0.8838	0.8950
0.8125	16250	1.8005	0.9144	0.8875	0.8950
0.825	16500	1.7875	0.9137	0.8856	0.8925
0.8375	16750	1.7568	0.9162	0.8856	0.8956
0.85	17000	1.7787	0.9156	0.8813	0.8944
0.8625	17250	1.779	0.9175	0.8844	0.8969
0.875	17500	1.791	0.9131	0.8831	0.8963
0.8875	17750	1.7713	0.9150	0.8825	0.8950
0.9	18000	1.7997	0.9169	0.8806	0.8956
0.9125	18250	1.7785	0.9156	0.8825	0.8969
0.925	18500	1.7651	0.9156	0.8863	0.8956
0.9375	18750	1.7581	0.9150	0.8881	0.8938
0.95	19000	1.7749	0.9156	0.8881	0.8956
0.9625	19250	1.7263	0.9162	0.8869	0.8950
0.975	19500	1.7883	0.9150	0.8863	0.8950
0.9875	19750	1.8142	0.9162	0.8856	0.8956
1.0	20000	1.7955	0.9162	0.8856	0.8956
-1	-1	-	0.9162	0.8856	0.8956

Framework Versions

Python: 3.12.8
Sentence Transformers: 4.0.2
Transformers: 4.51.1
PyTorch: 2.6.0+cu124
Accelerate: 1.6.0
Datasets: 3.5.0
Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss