pfrenee/distilroberta_ai_alignment

SentenceTransformer based on sentence-transformers/all-distilroberta-v1

This is a sentence-transformers model finetuned from sentence-transformers/all-distilroberta-v1 on the ai_alignment dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-distilroberta-v1
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • ai_alignment

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'RobertaModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("pfrenee/distilroberta_ai_alignment")
# Run inference
queries = [
    "Data engineering, ETL workflows, cloud-based data solutions",
]
documents = [
    "Qualifications and Skills Education: Bachelor's degree in Computer Science or a related field. Experience: 5+ years in Software Engineering with a focus on Data Engineering. Technical Proficiency: Expertise in Python; familiarity with JavaScript and Java is beneficial. Proficient in SQL (Postgres, Presto/Trino dialects), ETL workflows, and workflow orchestration systems (e.g. Airflow, Prefect). Knowledge of modern data file formats (e.g. Parquet, Avro, ORC) and Python data tools (e.g. pandas, Dask, Ray). Cloud and Data Solutions: Experience in building cloud-based Data Warehouse/Data Lake solutions (AWS Athena, Redshift, Snowflake) and familiarity with AWS cloud services and infrastructure-as-code tools (CDK, Terraform). Communication Skills: Excellent communication and presentation skills, fluent in English. Work Authorization: Must be authorized to work in the US. \nWork Schedule Hybrid work schedule: Minimum 3 days per week in the San Francisco office (M/W/Th), with the option to work remotely 2 days per week. \nSalary Range: $165,000-$206,000 base depending on experience \nBonus: Up to 20% annual performance bonus \nGenerous benefits package: Fully paid healthcare, monthly reimbursements for gym, commuting, cell phone & home wifi.",
    "Experience with LLMs and PyTorch: Extensive experience with large language models and proficiency in PyTorch.Expertise in Parallel Training and GPU Cluster Management: Strong background in parallel training methods and managing large-scale training jobs on GPU clusters.Analytical and Problem-Solving Skills: Ability to address complex challenges in model training and optimization.Leadership and Mentorship Capabilities: Proven leadership in guiding projects and mentoring team members.Communication and Collaboration Skills: Effective communication skills for conveying technical concepts and collaborating with cross-functional teams.Innovation and Continuous Learning: Passion for staying updated with the latest trends in AI and machine learning.\n\nWhat We Offer\n\nMarket competitive and pay equity-focused compensation structure100% paid health insurance for employees with 90% coverage for dependentsAnnual lifestyle wallet for personal wellness, learning and development, and more!Lifetime maximum benefit for family forming and fertility benefitsDedicated mental health support for employees and eligible dependentsGenerous time away including company holidays, paid time off, sick time, parental leave, and more!Lively office environment with catered meals, fully stocked kitchens, and geo-specific commuter benefits\n\nBase pay for the successful applicant will depend on a variety of job-related factors, which may include education, training, experience, location, business needs, or market demands. The expected salary range for this role is based on the location where the work will be performed and is aligned to one of 3 compensation zones. This role is also eligible to participate in a Robinhood bonus plan and Robinhood’s equity plan. For other locations not listed, compensation can be discussed with your recruiter during the interview process.\n\nZone 1 (Menlo Park, CA; New York, NY; Bellevue, WA; Washington, DC)\n\n$187,000—$220,000 USD\n\nZone 2 (Denver, CO; Westlake, TX; Chicago, IL)\n\n$165,000—$194,000 USD\n\nZone 3 (Lake Mary, FL)\n\n$146,000—$172,000 USD\n\nClick Here To Learn More About Robinhood’s Benefits.\n\nWe’re looking for more growth-minded and collaborative people to be a part of our journey in democratizing finance for all. If you’re ready to give 100% in helping us achieve our mission—we’d love to have you apply even if you feel unsure about whether you meet every single requirement in this posting. At Robinhood, we're looking for people invigorated by our mission, values, and drive to change the world, not just those who simply check off all the boxes.\n\nRobinhood embraces a diversity of backgrounds and experiences and provides equal opportunity for all applicants and employees. We are dedicated to building a company that represents a variety of backgrounds, perspectives, and skills. We believe that the more inclusive we are, the better our work (and work environment) will be for everyone. Additionally, Robinhood provides reasonable accommodations for candidates on request and respects applicants' privacy rights. To review Robinhood's Privacy Policy please review the specific policy applicable to your country.",
    "experience with Transformers\nNeed to be 8+ year's of work experience. \nWe need a Data Scientist with demonstrated expertise in training and evaluating transformers such as BERT and its derivatives.\nRequired: Proficiency with Python, pyTorch, Linux, Docker, Kubernetes, Jupyter. Expertise in Deep Learning, Transformers, Natural Language Processing, Large Language Models\nPreferred: Experience with genomics data, molecular genetics. Distributed computing tools like Ray, Dask, Spark",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.4493, 0.0204, 0.0266]])

Evaluation

Metrics

Triplet

• Datasets: ai-job-validation and ai-job-test
• Evaluated with TripletEvaluator

Metric	ai-job-validation	ai-job-test
cosine_accuracy	0.9802	0.9709

Training Details

Training Dataset

ai_alignment

• Dataset: ai_alignment at bb2b8ee
• Size: 814 training samples
• Columns: query, job_description_pos, and job_description_neg
• Approximate statistics based on the first 814 samples:

  	query	job_description_pos	job_description_neg
  type	string	string	string
  details	min: 8 tokens mean: 14.97 tokens max: 41 tokens	min: 7 tokens mean: 349.01 tokens max: 512 tokens	min: 7 tokens mean: 347.16 tokens max: 512 tokens

• Samples:

  query	job_description_pos	job_description_neg
  Python design patterns, Snowflake data warehousing, AWS data pipeline optimization	Requirements: - Good communication; and problem-solving abilities- Ability to work as an individual contributor; collaborating with Global team- Strong experience with Data Warehousing- OLTP, OLAP, Dimension, Facts, Data Modeling- Expertise implementing Python design patterns (Creational, Structural and Behavioral Patterns)- Expertise in Python building data application including reading, transforming; writing data sets- Strong experience in using boto3, pandas, numpy, pyarrow, Requests, Fast API, Asyncio, Aiohttp, PyTest, OAuth 2.0, multithreading, multiprocessing, snowflake python connector; Snowpark- Experience in Python building data APIs (Web/REST APIs)- Experience with Snowflake including SQL, Pipes, Stream, Tasks, Time Travel, Data Sharing, Query Optimization- Experience with Scripting language in Snowflake including SQL Stored Procs, Java Script Stored Procedures; Python UDFs- Understanding of Snowflake Internals; experience in integration with Reporting; UI applications- Stron...	QUALIFICATIONS Required Certifications DoD IAT Level III Certification (Must obtain within 180 days of hire). Education, Background, and Years of Experience 3-5 years of Data Analyst experience. ADDITIONAL SKILLS & QUALIFICATIONS Required Skills At least 3 years of hands-on experience with query languages, such as SQL and Kusto to facilitate robust reporting capabilities. Preferred Skills Understanding of Microsoft Power Platform. Power BI authoring, in combination with designing and integrating with data sources. Tier III, Senior Level Experience with Kusto Query Language (KQL). Tier III, Senior Level Experience with Structured Query Language (SQL). WORKING CONDITIONS Environmental Conditions Contractor site with 0%-10% travel possible. Possible off-hours work to support releases and outages. General office environment. Work is generally sedentary in nature but may require standing and walking for up to 10% of the time. The working environment is generally favorable. Lighting and temp...
  Data Science in Marketing, Customer LTV Modeling, Experimentation Frameworks	experience. You are comfortable with a range of statistical and ML techniques with the ability to apply them to deliver measurable business impact at Turo. You’re someone who constantly thinks about how data can support Turo’s work across domains, actively utilizing it to work-through challenges and unlock new opportunities. You’re proficient in translating unstructured problems into tangible mathematical frameworks, and are able to bring others with you on that journey. You’re someone who enjoys working with business stakeholders to drive experimentation and foster a data-centric culture. You’re able to recognize the right tools for each problem and design solutions that scale the impact of your work. You have a passion for contributing to a best in class product and take ownership of your work from inception to implementation and beyond. What You Will Do Turo’s marketplace has enjoyed continued growth as a business, which has in part been achieved through significant Marketing inv...	requirements.Prepares and presents results of analysis along with improvements and/or recommendations to the business at all levels of management.Coordinates with global sourcing team and peers to aggregate data align reporting.Maintain data integrity of databases and make changes as required to enhance accuracy, usefulness and access.Acts as a Subject Matter Expert (SME) for key systems/processes in subject teams and day-to-day functions.Develops scenario planning tools/models (exit/maintain/grow). Prepares forecasts and analyzes trends in general business conditions.Request for Proposal (RFP) activities – inviting suppliers to participate in RFP, loading RFP into Sourcing tool, collecting RFP responses, conducting qualitative and quantitative analyses.Assists Sourcing Leads in maintaining pipeline, reports on savings targets. Qualifications:Bachelors Degree is required.Minimum of 4 years of relevant procurement analyst experience.Advanced Excel skills are required.C.P.M., C.P.S.M., o...
  education workforce data analysis R Tableau	experience as an SME in complex enterprise-level projects, 5+ years of experience analyzing info and statistical data to prepare reports and studies for professional use, and experience working with education and workforce data. If you’re interested, I'll gladly provide more details about the role and further discuss your qualifications. Thanks,Stephen M HrutkaPrincipal Consultantwww.hruckus.com Executive Summary: HRUCKUS is looking to hire a Data Analyst resource to provide data analysis and management support. The Data Analyst must have at least 10 years of overall experience. Position Description: The role of the Data Analyst is to provide data analysis support for the Office of Education Through Employment Pathways, which is located within the Office of the Deputy Mayor for Education. This is a highly skilled position requiring familiarity with educational data and policies. The position will require the resources to produce data analysis, focusing on education and workforce-relate...	Experience of Delta Lake, DWH, Data Integration, Cloud, Design and Data Modelling.• Proficient in developing programs in Python and SQL• Experience with Data warehouse Dimensional data modeling.• Working with event based/streaming technologies to ingest and process data.• Working with structured, semi structured and unstructured data.• Optimize Databricks jobs for performance and scalability to handle big data workloads. • Monitor and troubleshoot Databricks jobs, identify and resolve issues or bottlenecks. • Implement best practices for data management, security, and governance within the Databricks environment. Experience designing and developing Enterprise Data Warehouse solutions.• Proficient writing SQL queries and programming including stored procedures and reverse engineering existing process.• Perform code reviews to ensure fit to requirements, optimal execution patterns and adherence to established standards. Qualifications: • 5+ years Python coding experience.• 5+ years - SQL...

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

ai_alignment

• Dataset: ai_alignment at bb2b8ee
• Size: 101 evaluation samples
• Columns: query, job_description_pos, and job_description_neg
• Approximate statistics based on the first 101 samples:

  	query	job_description_pos	job_description_neg
  type	string	string	string
  details	min: 10 tokens mean: 14.79 tokens max: 23 tokens	min: 61 tokens mean: 366.96 tokens max: 512 tokens	min: 27 tokens mean: 372.63 tokens max: 512 tokens

• Samples:

  query	job_description_pos	job_description_neg
  Statistical programming SAS, clinical development, AAV gene therapy	QUALIFICATIONS: Education: 12 years of related experience with a Bachelor’s degree; or 8 years and a Master’s degree; or a PhD with 5 years experience; or equivalent experience Experience: Work experience in biotech/pharmaceutical industry or medical research for a minimum of 8 years (or 4 years for a PhD with relevant training)Experience in clinical developmentExperience in ophthalmology and/or biologic/gene therapy a plus Skills: Strong SAS programming skills required with proficiency in SAS/BASE, SAS Macros, SAS/Stat and ODS (proficiency in SAS/SQL, SAS/GRAPH or SAS/ACCESS is a plus)Proficiency in R programming a plusProficiency in Microsoft Office Apps, such as WORD, EXCEL, and PowerPoint (familiar with the “Chart” features in EXCEL/PowerPoint a plus)Good understanding of standards specific to clinical trials such as CDISC, SDTM, and ADaM, MedDRA, WHODRUGExperience with all clinical phases (I, II, III, and IV) is desirableExperience with BLA/IND submissions is strongly desir...	requirements may change at any time. Qualifications Qualification: • BS degree in Computer Science, Computer Engineering or other relevant majors. • Excellent programming, debugging, and optimization skills in general purpose programming languages • Ability to think critically and to formulate solutions to problems in a clear and concise way. Preferred Qualifications: • Experience with one or more general purpose programming languages including but not limited to: Go, C/C++, Python. • Good understanding in one of the following domains: ad fraud detection, risk control, quality control, adversarial engineering, and online advertising systems. • Good knowledge in one of the following areas: machine learning, deep learning, backend, large-scale systems, data science, full-stack. TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workpla...
  ETL pipeline design, bulk data solutions, classified environments	Skills & Experience:Must hold a TS/SCI Full Scope Polygraph clearance, and have experience working in classified environments.Professional experience with Python and a JVM language (e.g., Scala) 4+ years of experience designing and maintaining ETL pipelines Experience using Apache SparkExperience with SQL (e.g., Postgres) and NoSQL (e.g., Cassandra, ElasticSearch, etc.)databases Experience working on a cloud platform like GCP, AWS, or Azure Experience working collaboratively with git Desired Skills & Experience:Understanding of Docker/Kubernetes Understanding of or interest in knowledge graphsExperienced in supporting and working with internal teams and customers in a dynamic environment Passionate about open source development and innovative technology Benefits: Limitless growth and learning opportunitiesA collaborative and positive culture - your team will be as smart and driven as youA strong commitment to diversity, equity & inclusionExceedingly generous vacation leave, parental l...	experience with all aspects of the software development lifecycle, from design to deployment. Demonstrate understanding of the full life data lifecycle and the role that high-quality data plays across applications, machine learning, business analytics, and reporting. Lead and take ownership of assigned technical projects in a fast-paced environment. What you need to succeed (minimum qualifications)3-5+ years of experienceFamiliar with best practices for data ingestion and data designDevelop initial queries for profiling data, validating analysis, testing assumptions, driving data quality assessment specifications, and define a path to deploymentIdentify necessary business rules for extracting data along with functional or technical risks related to data sources (e.g. data latency, frequency, etc.)Knowledge of working with queries/applications, including performance tuning, utilizing indexes, and materialized views to improve query performanceContinuously improve quality, efficiency, a...
  Provider data analysis, healthcare compliance, business process improvement	requirements of health plan as it pertains to contracting, benefits, prior authorizations, fee schedules, and other business requirements. •Analyze and interpret data to determine appropriate configuration changes.• Accurately interprets specific state and/or federal benefits, contracts as well as additional business requirements and converting these terms to configuration parameters.• Oversees coding, updating, and maintaining benefit plans, provider contracts, fee schedules and various system tables through the user interface.• Applies previous experience and knowledge to research and resolve claim/encounter issues, pended claims and update system(s) as necessary.• Works with fluctuating volumes of work and can prioritize work to meet deadlines and needs of user community.• Provides analytical, problem-solving foundation including definition and documentation, specifications.• Recognizes, identifies and documents changes to existing business processes and identifies new opportunities...	experience.Required Skills: ADF pipelines, SQL, Kusto, Power BI, Cosmos (Scope Scripts). Power Bi, ADX (Kusto), ADF, ADO, Python/C#.Good to have – Azure anomaly Alerting, App Insights, Azure Functions, Azure FabricQualifications for the role 5+ years experience building and optimizing ‘big data’ data pipelines, architectures and data sets. Specific experience working with COSMOS and Scope is required for this role. Experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases is a plus. Experience with investigating and on-boarding new data sources in a big-data environment, including forming relationships with data engineers cross-functionally to permission, mine and reformat new data sets. Strong analytic skills related to working with unstructured data sets. A successful history of manipulating, processing and extracting value from large disconnected datasets.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 1e-05
• num_train_epochs: 6
• warmup_ratio: 0.1
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 6
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	ai-job-validation_cosine_accuracy	ai-job-test_cosine_accuracy
-1	-1	-	-	0.8614	-
1.9608	100	0.848	0.3421	0.9802	-
3.9216	200	0.3142	0.3138	0.9802	-
5.8824	300	0.1828	0.3009	0.9802	-
-1	-1	-	-	0.9802	0.9709

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.1.0
• Transformers: 4.55.4
• PyTorch: 2.8.0
• Accelerate: 1.10.1
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}