This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1 on the ai_alignment dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: sentence-transformers/all-distilroberta-v1
Maximum Sequence Length: 512 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- ai_alignment

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

python

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pfrenee/distilroberta_ai_alignment")
# Run inference
queries = [
    "Data engineering, ETL workflows, cloud-based data solutions",
]
documents = [
    "Qualifications and Skills Education: Bachelor's degree in Computer Science or a related field. Experience: 5+ years in Software Engineering with a focus on Data Engineering. Technical Proficiency: Expertise in Python; familiarity with JavaScript and Java is beneficial. Proficient in SQL (Postgres, Presto/Trino dialects), ETL workflows, and workflow orchestration systems (e.g. Airflow, Prefect). Knowledge of modern data file formats (e.g. Parquet, Avro, ORC) and Python data tools (e.g. pandas, Dask, Ray). Cloud and Data Solutions: Experience in building cloud-based Data Warehouse/Data Lake solutions (AWS Athena, Redshift, Snowflake) and familiarity with AWS cloud services and infrastructure-as-code tools (CDK, Terraform). Communication Skills: Excellent communication and presentation skills, fluent in English. Work Authorization: Must be authorized to work in the US. \nWork Schedule Hybrid work schedule: Minimum 3 days per week in the San Francisco office (M/W/Th), with the option to work remotely 2 days per week. \nSalary Range: $165,000-$206,000 base depending on experience \nBonus: Up to 20% annual performance bonus \nGenerous benefits package: Fully paid healthcare, monthly reimbursements for gym, commuting, cell phone & home wifi.",
    "Experience with LLMs and PyTorch: Extensive experience with large language models and proficiency in PyTorch.Expertise in Parallel Training and GPU Cluster Management: Strong background in parallel training methods and managing large-scale training jobs on GPU clusters.Analytical and Problem-Solving Skills: Ability to address complex challenges in model training and optimization.Leadership and Mentorship Capabilities: Proven leadership in guiding projects and mentoring team members.Communication and Collaboration Skills: Effective communication skills for conveying technical concepts and collaborating with cross-functional teams.Innovation and Continuous Learning: Passion for staying updated with the latest trends in AI and machine learning.\n\nWhat We Offer\n\nMarket competitive and pay equity-focused compensation structure100% paid health insurance for employees with 90% coverage for dependentsAnnual lifestyle wallet for personal wellness, learning and development, and more!Lifetime maximum benefit for family forming and fertility benefitsDedicated mental health support for employees and eligible dependentsGenerous time away including company holidays, paid time off, sick time, parental leave, and more!Lively office environment with catered meals, fully stocked kitchens, and geo-specific commuter benefits\n\nBase pay for the successful applicant will depend on a variety of job-related factors, which may include education, training, experience, location, business needs, or market demands. The expected salary range for this role is based on the location where the work will be performed and is aligned to one of 3 compensation zones. This role is also eligible to participate in a Robinhood bonus plan and Robinhood’s equity plan. For other locations not listed, compensation can be discussed with your recruiter during the interview process.\n\nZone 1 (Menlo Park, CA; New York, NY; Bellevue, WA; Washington, DC)\n\n$187,000—$220,000 USD\n\nZone 2 (Denver, CO; Westlake, TX; Chicago, IL)\n\n$165,000—$194,000 USD\n\nZone 3 (Lake Mary, FL)\n\n$146,000—$172,000 USD\n\nClick Here To Learn More About Robinhood’s Benefits.\n\nWe’re looking for more growth-minded and collaborative people to be a part of our journey in democratizing finance for all. If you’re ready to give 100% in helping us achieve our mission—we’d love to have you apply even if you feel unsure about whether you meet every single requirement in this posting. At Robinhood, we're looking for people invigorated by our mission, values, and drive to change the world, not just those who simply check off all the boxes.\n\nRobinhood embraces a diversity of backgrounds and experiences and provides equal opportunity for all applicants and employees. We are dedicated to building a company that represents a variety of backgrounds, perspectives, and skills. We believe that the more inclusive we are, the better our work (and work environment) will be for everyone. Additionally, Robinhood provides reasonable accommodations for candidates on request and respects applicants' privacy rights. To review Robinhood's Privacy Policy please review the specific policy applicable to your country.",
    "experience with Transformers\nNeed to be 8+ year's of work experience. \nWe need a Data Scientist with demonstrated expertise in training and evaluating transformers such as BERT and its derivatives.\nRequired: Proficiency with Python, pyTorch, Linux, Docker, Kubernetes, Jupyter. Expertise in Deep Learning, Transformers, Natural Language Processing, Large Language Models\nPreferred: Experience with genomics data, molecular genetics. Distributed computing tools like Ray, Dask, Spark",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4493, 0.0204, 0.0266]])

Evaluation

Metrics

Triplet

Datasets: ai-job-validation and ai-job-test
Evaluated with TripletEvaluator

Metric	ai-job-validation	ai-job-test
cosine_accuracy	0.9802	0.9709

Training Details

Training Dataset

ai_alignment

Dataset: ai_alignment at bb2b8ee
Size: 814 training samples
Columns: query, job_description_pos, and job_description_neg

Approximate statistics based on the first 814 samples:

	query	job_description_pos	job_description_neg
type	string	string	string
details	min: 8 tokens mean: 14.97 tokens max: 41 tokens	min: 7 tokens mean: 349.01 tokens max: 512 tokens	min: 7 tokens mean: 347.16 tokens max: 512 tokens

Samples:

query	job_description_pos	job_description_neg
`Python design patterns, Snowflake data warehousing, AWS data pipeline optimization`	Requirements:[object Object]- Good communication; and problem-solving abilities- Ability to work as an individual contributor; collaborating with Global team- Strong experience with Data Warehousing- OLTP, OLAP, Dimension, Facts, Data Modeling- Expertise implementing Python design patterns (Creational, Structural and Behavioral Patterns)- Expertise in Python building data application including reading, transforming; writing data sets- Strong experience in using boto3, pandas, numpy, pyarrow, Requests, Fast API, Asyncio, Aiohttp, PyTest, OAuth 2.0, multithreading, multiprocessing, snowflake python connector; Snowpark- Experience in Python building data APIs (Web/REST APIs)- Experience with Snowflake including SQL, Pipes, Stream, Tasks, Time Travel, Data Sharing, Query Optimization- Experience with Scripting language in Snowflake including SQL Stored Procs, Java Script Stored Procedures; Python UDFs- Understanding of Snowflake Internals; experience in integration with Reporting; UI applications- Stron...	QUALIFICATIONS Required Certifications DoD IAT Level III Certification (Must obtain within 180 days of hire). Education, Background, and Years of Experience 3-5 years of Data Analyst experience. ADDITIONAL SKILLS & QUALIFICATIONS Required Skills At least 3 years of hands-on experience with query languages, such as SQL and Kusto to facilitate robust reporting capabilities. Preferred Skills Understanding of Microsoft Power Platform. Power BI authoring, in combination with designing and integrating with data sources. Tier III, Senior Level Experience with Kusto Query Language (KQL). Tier III, Senior Level Experience with Structured Query Language (SQL). WORKING CONDITIONS Environmental Conditions Contractor site with 0%-10% travel possible. Possible off-hours work to support releases and outages. General office environment. Work is generally sedentary in nature but may require standing and walking for up to 10% of the time. The working environment is generally favorable. Lighting and temp...
`Data Science in Marketing, Customer LTV Modeling, Experimentation Frameworks`	experience. You are comfortable with a range of statistical and ML techniques with the ability to apply them to deliver measurable business impact at Turo.[object Object][object Object]You’re someone who constantly thinks about how data can support Turo’s work across domains, actively utilizing it to work-through challenges and unlock new opportunities. You’re proficient in translating unstructured problems into tangible mathematical frameworks, and are able to bring others with you on that journey. You’re someone who enjoys working with business stakeholders to drive experimentation and foster a data-centric culture. You’re able to recognize the right tools for each problem and design solutions that scale the impact of your work. You have a passion for contributing to a best in class product and take ownership of your work from inception to implementation and beyond.[object Object][object Object]What You Will Do[object Object][object Object]Turo’s marketplace has enjoyed continued growth as a business, which has in part been achieved through significant Marketing inv...	requirements.Prepares and presents results of analysis along with improvements and/or recommendations to the business at all levels of management.Coordinates with global sourcing team and peers to aggregate data align reporting.Maintain data integrity of databases and make changes as required to enhance accuracy, usefulness and access.Acts as a Subject Matter Expert (SME) for key systems/processes in subject teams and day-to-day functions.Develops scenario planning tools/models (exit/maintain/grow). Prepares forecasts and analyzes trends in general business conditions.Request for Proposal (RFP) activities – inviting suppliers to participate in RFP, loading RFP into Sourcing tool, collecting RFP responses, conducting qualitative and quantitative analyses.Assists Sourcing Leads in maintaining pipeline, reports on savings targets.[object Object]Qualifications:Bachelors Degree is required.Minimum of 4 years of relevant procurement analyst experience.Advanced Excel skills are required.C.P.M., C.P.S.M., o...
`education workforce data analysis R Tableau`	experience as an SME in complex enterprise-level projects, 5+ years of experience analyzing info and statistical data to prepare reports and studies for professional use, and experience working with education and workforce data.[object Object]If you’re interested, I'll gladly provide more details about the role and further discuss your qualifications.[object Object]Thanks,Stephen M HrutkaPrincipal Consultantwww.hruckus.com[object Object]Executive Summary: HRUCKUS is looking to hire a Data Analyst resource to provide data analysis and management support. The Data Analyst must have at least 10 years of overall experience.[object Object]Position Description: The role of the Data Analyst is to provide data analysis support for the Office of Education Through Employment Pathways, which is located within the Office of the Deputy Mayor for Education. This is a highly skilled position requiring familiarity with educational data and policies.[object Object]The position will require the resources to produce data analysis, focusing on education and workforce-relate...	Experience of Delta Lake, DWH, Data Integration, Cloud, Design and Data Modelling.• Proficient in developing programs in Python and SQL• Experience with Data warehouse Dimensional data modeling.• Working with event based/streaming technologies to ingest and process data.• Working with structured, semi structured and unstructured data.• Optimize Databricks jobs for performance and scalability to handle big data workloads. • Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks. • Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.• Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process.• Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards.[object Object]Qualifications:[object Object]• 5+ years Python coding experience.• 5+ years - SQL...

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false 5}

Evaluation Dataset

ai_alignment

Dataset: ai_alignment at bb2b8ee
Size: 101 evaluation samples
Columns: query, job_description_pos, and job_description_neg

Approximate statistics based on the first 101 samples:

	query	job_description_pos	job_description_neg
type	string	string	string
details	min: 10 tokens mean: 14.79 tokens max: 23 tokens	min: 61 tokens mean: 366.96 tokens max: 512 tokens	min: 27 tokens mean: 372.63 tokens max: 512 tokens

Samples:

query	job_description_pos	job_description_neg
`Statistical programming SAS, clinical development, AAV gene therapy`	QUALIFICATIONS: [object Object][object Object]Education:[object Object][object Object]12 years of related experience with a Bachelor’s degree; or 8 years and a Master’s degree; or a PhD with 5 years experience; or equivalent experience[object Object][object Object]Experience:[object Object][object Object]Work experience in biotech/pharmaceutical industry or medical research for a minimum of 8 years (or 4 years for a PhD with relevant training)Experience in clinical developmentExperience in ophthalmology and/or biologic/gene therapy a plus[object Object][object Object]Skills:[object Object][object Object]Strong SAS programming skills required with proficiency in SAS/BASE, SAS Macros, SAS/Stat and ODS (proficiency in SAS/SQL, SAS/GRAPH or SAS/ACCESS is a plus)Proficiency in R programming a plusProficiency in Microsoft Office Apps, such as WORD, EXCEL, and PowerPoint (familiar with the “Chart” features in EXCEL/PowerPoint a plus)Good understanding of standards specific to clinical trials such as CDISC, SDTM, and ADaM, MedDRA, WHODRUGExperience with all clinical phases (I, II, III, and IV) is desirableExperience with BLA/IND submissions is strongly desir...	requirements may change at any time.[object Object][object Object]Qualifications[object Object][object Object] Qualification:[object Object]• BS degree in Computer Science, Computer Engineering or other relevant majors.[object Object]• Excellent programming, debugging, and optimization skills in general purpose programming languages[object Object]• Ability to think critically and to formulate solutions to problems in a clear and concise way.[object Object][object Object]Preferred Qualifications:[object Object]• Experience with one or more general purpose programming languages including but not limited to: Go, C/C++, Python.[object Object]• Good understanding in one of the following domains: ad fraud detection, risk control, quality control, adversarial engineering, and online advertising systems.[object Object]• Good knowledge in one of the following areas: machine learning, deep learning, backend, large-scale systems, data science, full-stack.[object Object][object Object]TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workpla...
`ETL pipeline design, bulk data solutions, classified environments`	Skills & Experience:Must hold a TS/SCI Full Scope Polygraph clearance, and have experience working in classified environments.Professional experience with Python and a JVM language (e.g., Scala) 4+ years of experience designing and maintaining ETL pipelines Experience using Apache SparkExperience with SQL (e.g., Postgres) and NoSQL (e.g., Cassandra, ElasticSearch, etc.)databases Experience working on a cloud platform like GCP, AWS, or Azure Experience working collaboratively with git [object Object]Desired Skills & Experience:Understanding of Docker/Kubernetes Understanding of or interest in knowledge graphsExperienced in supporting and working with internal teams and customers in a dynamic environment Passionate about open source development and innovative technology[object Object]Benefits: Limitless growth and learning opportunitiesA collaborative and positive culture - your team will be as smart and driven as youA strong commitment to diversity, equity & inclusionExceedingly generous vacation leave, parental l...	experience with all aspects of the software development lifecycle, from design to deployment. Demonstrate understanding of the full life data lifecycle and the role that high-quality data plays across applications, machine learning, business analytics, and reporting. Lead and take ownership of assigned technical projects in a fast-paced environment. [object Object]What you need to succeed (minimum qualifications)3-5+ years of experienceFamiliar with best practices for data ingestion and data designDevelop initial queries for profiling data, validating analysis, testing assumptions, driving data quality assessment specifications, and define a path to deploymentIdentify necessary business rules for extracting data along with functional or technical risks related to data sources (e.g. data latency, frequency, etc.)Knowledge of working with queries/applications, including performance tuning, utilizing indexes, and materialized views to improve query performanceContinuously improve quality, efficiency, a...
`Provider data analysis, healthcare compliance, business process improvement`	requirements of health plan as it pertains to contracting, benefits, prior authorizations, fee schedules, and other business requirements.[object Object]•Analyze and interpret data to determine appropriate configuration changes.• Accurately interprets specific state and/or federal benefits, contracts as well as additional business requirements and converting these terms to configuration parameters.• Oversees coding, updating, and maintaining benefit plans, provider contracts, fee schedules and various system tables through the user interface.• Applies previous experience and knowledge to research and resolve claim/encounter issues, pended claims and update system(s) as necessary.• Works with fluctuating volumes of work and can prioritize work to meet deadlines and needs of user community.• Provides analytical, problem-solving foundation including definition and documentation, specifications.• Recognizes, identifies and documents changes to existing business processes and identifies new opportunities...	experience.Required Skills: ADF pipelines, SQL, Kusto, Power BI, Cosmos (Scope Scripts). Power Bi, ADX (Kusto), ADF, ADO, Python/C#.Good to have – Azure anomaly Alerting, App Insights, Azure Functions, Azure FabricQualifications for the role 5+ years experience building and optimizing ‘big data’ data pipelines, architectures and data sets. Specific experience working with COSMOS and Scope is required for this role. Experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases is a plus. Experience with investigating and on-boarding new data sources in a big-data environment, including forming relationships with data engineers cross-functionally to permission, mine and reformat new data sets. Strong analytic skills related to working with unstructured data sets. A successful history of manipulating, processing and extracting value from large disconnected datasets.

Loss: MultipleNegativesRankingLoss with these parameters:
json
1{ 2 "scale": 20.0, 3 "similarity_fct": "cos_sim", 4 "gather_across_devices": false 5}

Training Hyperparameters

Non-Default Hyperparameters

eval_strategy: steps
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
learning_rate: 1e-05
num_train_epochs: 6
warmup_ratio: 0.1
batch_sampler: no_duplicates

All Hyperparameters

Click to expand

overwrite_output_dir: False
do_predict: False
eval_strategy: steps
prediction_loss_only: True
per_device_train_batch_size: 16
per_device_eval_batch_size: 16
per_gpu_train_batch_size: None
per_gpu_eval_batch_size: None
gradient_accumulation_steps: 1
eval_accumulation_steps: None
torch_empty_cache_steps: None
learning_rate: 1e-05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
max_grad_norm: 1.0
num_train_epochs: 6
max_steps: -1
lr_scheduler_type: linear
lr_scheduler_kwargs: {}
warmup_ratio: 0.1
warmup_steps: 0
log_level: passive
log_level_replica: warning
log_on_each_node: True
logging_nan_inf_filter: True
save_safetensors: True
save_on_each_node: False
save_only_model: False
restore_callback_states_from_checkpoint: False
no_cuda: False
use_cpu: False
use_mps_device: False
seed: 42
data_seed: None
jit_mode_eval: False
use_ipex: False
bf16: False
fp16: False
fp16_opt_level: O1
half_precision_backend: auto
bf16_full_eval: False
fp16_full_eval: False
tf32: None
local_rank: 0
ddp_backend: None
tpu_num_cores: None
tpu_metrics_debug: False
debug: []
dataloader_drop_last: False
dataloader_num_workers: 0
dataloader_prefetch_factor: None
past_index: -1
disable_tqdm: False
remove_unused_columns: True
label_names: None
load_best_model_at_end: False
ignore_data_skip: False
fsdp: []
fsdp_min_num_params: 0
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
fsdp_transformer_layer_cls_to_wrap: None
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
deepspeed: None
label_smoothing_factor: 0.0
optim: adamw_torch_fused
optim_args: None
adafactor: False
group_by_length: False
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
dataloader_pin_memory: True
dataloader_persistent_workers: False
skip_memory_metrics: True
use_legacy_prediction_loop: False
push_to_hub: False
resume_from_checkpoint: None
hub_model_id: None
hub_strategy: every_save
hub_private_repo: None
hub_always_push: False
hub_revision: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
include_inputs_for_metrics: False
include_for_metrics: []
eval_do_concat_batches: True
fp16_backend: auto
push_to_hub_model_id: None
push_to_hub_organization: None
mp_parameters:
auto_find_batch_size: False
full_determinism: False
torchdynamo: None
ray_scope: last
ddp_timeout: 1800
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
include_tokens_per_second: False
include_num_input_tokens_seen: False
neftune_noise_alpha: None
optim_target_modules: None
batch_eval_metrics: False
eval_on_start: False
use_liger_kernel: False
liger_kernel_config: None
eval_use_gather_object: False
average_tokens_across_devices: False
prompts: None
batch_sampler: no_duplicates
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	ai-job-validation_cosine_accuracy	ai-job-test_cosine_accuracy
-1	-1	-	-	0.8614	-
1.9608	100	0.848	0.3421	0.9802	-
3.9216	200	0.3142	0.3138	0.9802	-
5.8824	300	0.1828	0.3009	0.9802	-
-1	-1	-	-	0.9802	0.9709

Framework Versions

Python: 3.12.11
Sentence Transformers: 5.1.0
Transformers: 4.55.4
PyTorch: 2.8.0
Accelerate: 1.10.1
Datasets: 4.0.0
Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

bibtex

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

bibtex

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}