encoder/README.md

metadata

tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:156
  - loss:MultipleNegativesRankingLoss
base_model: sentence-transformers/all-MiniLM-L6-v2
widget:
  - source_sentence: >-
      Adversaries may directly interact with the native OS application
      programming interface (API) to access system functions. Native APIs
      provide a controlled means of calling low-level OS services within the
      kernel, such as those involving hardware/devices, memory, and processes.
      (Citation: The MITRE Corporation May 2017) These native APIs are leveraged
      by the OS during system boot (when other system components are not yet
      initialized) as well as carrying out tasks and requests during routine
      operations.  Functionality provided by native APIs are often also exposed
      to user-mode applications via interfaces and libraries. For example,
      functions such as memcpy and direct operations on memory registers can be
      used to modify user and system memory space.
    sentences:
      - >-
        Screen Capture — Adversaries may attempt to perform screen capture of
        devices in the control system environment. Screenshots may be taken of
        workstations, HMIs, or other devices that display environment-relevant
        process, device, reporting, alarm, or related data. These device
        displays may reveal information regarding the ICS process, layout,
        control, and related schematics. In particular, an HMI can provide a lot
        of important industrial process information. (Citation: ICS-CERT October
        2017) Analysis of screen captures may provide the adversary with an
        understanding of intended operations and interactions between critical
        devices.
      - >-
        Native API — Adversaries may directly interact with the native OS
        application programming interface (API) to access system functions.
        Native APIs provide a controlled means of calling low-level OS services
        within the kernel, such as those involving hardware/devices, memory, and
        processes. (Citation: The MITRE Corporation May 2017) These native APIs
        are leveraged by the OS during system boot (when other system components
        are not yet initialized) as well as carrying out tasks and requests
        during routine operations.  Functionality provided by native APIs are
        often also exposed to user-mode applications via interfaces and
        libraries. For example, functions such as memcpy and direct operations
        on memory registers can be used to modify user and system memory space.
      - >-
        Unauthorized Command Message — Adversaries may send unauthorized command
        messages to instruct control system assets to perform actions outside of
        their intended functionality, or without the logical preconditions to
        trigger their expected function. Command messages are used in ICS
        networks to give direct instructions to control systems devices. If an
        adversary can send an unauthorized command message to a control system,
        then it can instruct the control systems device to perform an action
        outside the normal bounds of the device's actions. An adversary could
        potentially instruct a control systems device to perform an action that
        will cause an [Impact](https://attack.mitre.org/tactics/TA0105).
        (Citation: Bonnie Zhu, Anthony Joseph, Shankar Sastry 2011)   


        In the Maroochy Attack, the adversary used a dedicated analog two-way
        radio system to send false data and instructions to pumping stations and
        the central computer. (Citation: Marshall Abrams July 2008)  


        In the Dallas Siren incident, adversaries were able to send command
        messages to activate tornado alarm systems across the city without an
        impending tornado or other disaster. (Citation: Zack Whittaker April
        2017) (Citation: Benjamin Freed March 2019)
  - source_sentence: >-
      Adversaries may gather information about the physical process state. This
      information may be used to gain more information about the process itself
      or used as a trigger for malicious actions. The sources of process state
      information may vary such as, OPC tags, historian data, specific PLC block
      information, or network traffic.
    sentences:
      - >-
        Data from Information Repositories — Adversaries may target and collect
        data from information repositories. This can include sensitive data such
        as specifications, schematics, or diagrams of control system layouts,
        devices, and processes. Examples of information repositories include
        reference databases or local machines in the process environment, as
        well as workstations and databases in the corporate network that might
        contain information about the ICS.  (Citation: Cybersecurity &
        Infrastructure Security Agency March 2018)  Information collected from
        these systems may provide the adversary with a better understanding of
        the operational environment, vendors used, processes, or procedures of
        the ICS.  In a campaign between 2011 and 2013 against ONG organizations,
        Chinese state-sponsored actors searched document repositories for
        specific information such as, system manuals, remote terminal unit (RTU)
        sites, personnel lists, documents that included the string SCAD*, user
        credentials, and remote dial-up access information. (Citation: CISA
        AA21-201A Pipeline Intrusion July 2021)
      - >-
        Monitor Process State — Adversaries may gather information about the
        physical process state. This information may be used to gain more
        information about the process itself or used as a trigger for malicious
        actions. The sources of process state information may vary such as, OPC
        tags, historian data, specific PLC block information, or network
        traffic.
      - >-
        Remote System Discovery — Adversaries may attempt to get a listing of
        other systems by IP address, hostname, or other logical identifier on a
        network that may be used for subsequent Lateral Movement or Discovery
        techniques. Functionality could exist within adversary tools to enable
        this, but utilities available on the operating system or vendor software
        could also be used. (Citation: Enterprise ATT&CK January 2018)
  - source_sentence: >-
      Adversaries may cause a denial of view in attempt to disrupt and prevent
      operator oversight on the status of an ICS environment. This may manifest
      itself as a temporary communication failure between a device and its
      control source, where the interface recovers and becomes available once
      the interference ceases. (Citation: Corero) (Citation: Michael J. Assante
      and Robert M. Lee) (Citation: Tyson Macaulay)   An adversary may attempt
      to deny operator visibility by preventing them from receiving status and
      reporting messages. Denying this view may temporarily block and prevent
      operators from noticing a change in state or anomalous behavior. The
      environment's data and processes may still be operational, but functioning
      in an unintended or adversarial manner.  In the Maroochy attack, the
      adversary was able to temporarily shut an investigator out of the network,
      preventing them from viewing the state of the system.
    sentences:
      - >-
        Module Firmware — Adversaries may install malicious or vulnerable
        firmware onto modular hardware devices. Control system devices often
        contain modular hardware devices. These devices may have their own set
        of firmware that is separate from the firmware of the main control
        system equipment.   


        This technique is similar to [System
        Firmware](https://attack.mitre.org/techniques/T0857), but is conducted
        on other system components that may not have the same capabilities or
        level of integrity checking. Although it results in a device re-image,
        malicious device firmware may provide persistent access to remaining
        devices. (Citation: Daniel Peck,  Dale Peterson January 2009)  


        An easy point of access for an adversary is the Ethernet card, which may
        have its own CPU, RAM, and operating system. The adversary may attack
        and likely exploit the computer on an Ethernet card. Exploitation of the
        Ethernet card computer may enable the adversary to accomplish additional
        attacks, such as the following: (Citation: Daniel Peck,  Dale Peterson
        January 2009)  


        * Delayed Attack - The adversary may stage an attack in advance and
        choose when to launch it, such as at a particularly damaging time.  

        * Brick the Ethernet Card - Malicious firmware may be programmed to
        result in an Ethernet card failure, requiring a factory return.  

        * Random Attack or Failure - The adversary may load malicious firmware
        onto multiple field devices. Execution of an attack and the time it
        occurs is generated by a pseudo-random number generator.   

        * A Field Device Worm - The adversary may choose to identify all field
        devices of the same model, with the end goal of performing a device-wide
        compromise.  

        * Attack Other Cards on the Field Device - Although it is not the most
        important module in a field device, the Ethernet card is most accessible
        to the adversary and malware. Compromise of the Ethernet card may
        provide a more direct route to compromising other modules, such as the
        CPU module.
      - >-
        Denial of View — Adversaries may cause a denial of view in attempt to
        disrupt and prevent operator oversight on the status of an ICS
        environment. This may manifest itself as a temporary communication
        failure between a device and its control source, where the interface
        recovers and becomes available once the interference ceases. (Citation:
        Corero) (Citation: Michael J. Assante and Robert M. Lee) (Citation:
        Tyson Macaulay)   An adversary may attempt to deny operator visibility
        by preventing them from receiving status and reporting messages. Denying
        this view may temporarily block and prevent operators from noticing a
        change in state or anomalous behavior. The environment's data and
        processes may still be operational, but functioning in an unintended or
        adversarial manner.  In the Maroochy attack, the adversary was able to
        temporarily shut an investigator out of the network, preventing them
        from viewing the state of the system.
      - >-
        Module Firmware — Adversaries may install malicious or vulnerable
        firmware onto modular hardware devices. Control system devices often
        contain modular hardware devices. These devices may have their own set
        of firmware that is separate from the firmware of the main control
        system equipment.   


        This technique is similar to [System
        Firmware](https://attack.mitre.org/techniques/T0857), but is conducted
        on other system components that may not have the same capabilities or
        level of integrity checking. Although it results in a device re-image,
        malicious device firmware may provide persistent access to remaining
        devices. (Citation: Daniel Peck,  Dale Peterson January 2009)  


        An easy point of access for an adversary is the Ethernet card, which may
        have its own CPU, RAM, and operating system. The adversary may attack
        and likely exploit the computer on an Ethernet card. Exploitation of the
        Ethernet card computer may enable the adversary to accomplish additional
        attacks, such as the following: (Citation: Daniel Peck,  Dale Peterson
        January 2009)  


        * Delayed Attack - The adversary may stage an attack in advance and
        choose when to launch it, such as at a particularly damaging time.  

        * Brick the Ethernet Card - Malicious firmware may be programmed to
        result in an Ethernet card failure, requiring a factory return.  

        * Random Attack or Failure - The adversary may load malicious firmware
        onto multiple field devices. Execution of an attack and the time it
        occurs is generated by a pseudo-random number generator.   

        * A Field Device Worm - The adversary may choose to identify all field
        devices of the same model, with the end goal of performing a device-wide
        compromise.  

        * Attack Other Cards on the Field Device - Although it is not the most
        important module in a field device, the Ethernet card is most accessible
        to the adversary and malware. Compromise of the Ethernet card may
        provide a more direct route to compromising other modules, such as the
        CPU module.
  - source_sentence: Technique Spoof Reporting Message in tactic evasion
    sentences:
      - >-
        Hooking — Adversaries may hook into application programming interface
        (API) functions used by processes to redirect calls for execution and
        privilege escalation means. Windows processes often leverage these API
        functions to perform tasks that require reusable system resources.
        Windows API functions are typically stored in dynamic-link libraries
        (DLLs) as exported functions. (Citation: Enterprise ATT&CK)  One type of
        hooking seen in ICS involves redirecting calls to these functions via
        import address table (IAT) hooking. IAT hooking uses modifications to a
        processs IAT, where pointers to imported API functions are stored.
        (Citation: Nicolas Falliere, Liam O Murchu, Eric Chien February 2011)
      - >-
        Block Reporting Message — Adversaries may block or prevent a reporting
        message from reaching its intended target. In control systems, reporting
        messages contain telemetry data (e.g., I/O values) pertaining to the
        current state of equipment and the industrial process. By blocking these
        reporting messages, an adversary can potentially hide their actions from
        an operator.  Blocking reporting messages in control systems that manage
        physical processes may contribute to system impact, causing inhibition
        of a response function. A control system may not be able to respond in a
        proper or timely manner to an event, such as a dangerous fault, if its
        corresponding reporting message is blocked. (Citation: Bonnie Zhu,
        Anthony Joseph, Shankar Sastry 2011)  (Citation: Electricity Information
        Sharing and Analysis Center; SANS Industrial Control Systems March 2016)
      - >-
        Spoof Reporting Message — Adversaries may spoof reporting messages in
        control system environments for evasion and to impair process control.
        In control systems, reporting messages contain telemetry data (e.g., I/O
        values) pertaining to the current state of equipment and the industrial
        process. Reporting messages are important for monitoring the normal
        operation of a system or identifying important events such as deviations
        from expected values.  If an adversary has the ability to Spoof
        Reporting Messages, they can impact the control system in many ways. The
        adversary can Spoof Reporting Messages that state that the process is
        operating normally, as a form of evasion. The adversary could also Spoof
        Reporting Messages to make the defenders and operators think that other
        errors are occurring in order to distract them from the actual source of
        a problem. (Citation: Bonnie Zhu, Anthony Joseph, Shankar Sastry 2011)  
        In the Maroochy Attack, the adversary used a dedicated analog two-way
        radio system to send false data and instructions to pumping stations and
        the central computer. (Citation: Marshall Abrams July 2008)
  - source_sentence: Technique Loss of Productivity and Revenue in tactic impact
    sentences:
      - >-
        Loss of Productivity and Revenue — Adversaries may cause loss of
        productivity and revenue through disruption and even damage to the
        availability and integrity of control system operations, devices, and
        related processes. This technique may manifest as a direct effect of an
        ICS-targeting attack or tangentially, due to an IT-targeting attack
        against non-segregated environments.   In cases where these operations
        or services are brought to a halt, the loss of productivity may
        eventually present an impact for the end-users or consumers of products
        and services. The disrupted supply-chain may result in supply shortages
        and increased prices, among other consequences.   A ransomware attack on
        an Australian beverage company resulted in the shutdown of some
        manufacturing sites, including precautionary halts to protect key
        systems. (Citation: Paganini, Pierluigi June 2020) The company announced
        the potential for temporary shortages of their products following the
        attack. (Citation: Paganini, Pierluigi June 2020) (Citation: Lion
        Corporation June 2020)  In the 2021 Colonial Pipeline ransomware
        incident, the pipeline was unable to transport approximately 2.5 million
        barrels of fuel per day to the East Coast.  (Citation: Colonial Pipeline
        Company May 2021)
      - >-
        Modify Parameter — Adversaries may modify parameters used to instruct
        industrial control system devices. These devices operate via programs
        that dictate how and when to perform actions based on such parameters.
        Such parameters can determine the extent to which an action is performed
        and may specify additional options. For example, a program on a control
        system device dictating motor processes may take a parameter defining
        the total number of seconds to run that motor.      


        An adversary can potentially modify these parameters to produce an
        outcome outside of what was intended by the operators. By modifying
        system and process critical parameters, the adversary may cause
        [Impact](https://attack.mitre.org/tactics/TA0105) to equipment and/or
        control processes. Modified parameters may be turned into dangerous,
        out-of-bounds, or unexpected values from typical operations. For
        example, specifying that a process run for more or less time than it
        should, or dictating an unusually high, low, or invalid value as a
        parameter.   


        In the Maroochy Attack, Vitek Boden gained remote computer access to the
        control system and altered data so that whatever function should have
        occurred at affected pumping stations did not occur or occurred in a
        different way. The software program installed in the laptop was one
        developed by Hunter Watertech for its use in changing configurations in
        the PDS computers. This ultimately led to 800,000 liters of raw sewage
        being spilled out into the community. (Citation: Marshall Abrams July
        2008)  


        In the Oldsmar water treatment attack, adversaries raised the sodium
        hydroxide setpoint value from 100 part-per-million (ppm) to 11,100 ppm,
        far beyond normal operating levels. (Citation: Pinellas County Sheriffs
        Office February 2021)
      - >-
        Drive-by Compromise — Adversaries may gain access to a system during a
        drive-by compromise, when a user visits a website as part of a regular
        browsing session.With this technique, the user's web browser is targeted
        and exploited simply by visiting the compromised website.   The
        adversary may target a specific community, such as trusted third party
        suppliers or other industry specific groups, which often visit the
        target website. This kind of targeted attack relies on a common
        interest, and is known as a strategic web compromise or watering hole
        attack.   The National Cyber Awareness System (NCAS) has issued a
        Technical Alert (TA) regarding Russian government cyber activity
        targeting critical infrastructure sectors. (Citation: Cybersecurity &
        Infrastructure Security Agency March 2018) Analysis by DHS and FBI has
        noted two distinct categories of victims in the Dragonfly campaign on
        the Western energy sector: staging and intended targets. The adversary
        targeted the less secure networks of staging targets, including trusted
        third-party suppliers and related peripheral organizations. Initial
        access to the intended targets used watering hole attacks to target
        process control, ICS, and critical infrastructure related trade
        publications and informational websites.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - pearson_cosine
  - spearman_cosine
model-index:
  - name: SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2
    results:
      - task:
          type: semantic-similarity
          name: Semantic Similarity
        dataset:
          name: asg validation
          type: asg-validation
        metrics:
          - type: pearson_cosine
            value: 0.8016177378461883
            name: Pearson Cosine
          - type: spearman_cosine
            value: 0.8260664424048982
            name: Spearman Cosine

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Technique Loss of Productivity and Revenue in tactic impact',
    'Loss of Productivity and Revenue — Adversaries may cause loss of productivity and revenue through disruption and even damage to the availability and integrity of control system operations, devices, and related processes. This technique may manifest as a direct effect of an ICS-targeting attack or tangentially, due to an IT-targeting attack against non-segregated environments.   In cases where these operations or services are brought to a halt, the loss of productivity may eventually present an impact for the end-users or consumers of products and services. The disrupted supply-chain may result in supply shortages and increased prices, among other consequences.   A ransomware attack on an Australian beverage company resulted in the shutdown of some manufacturing sites, including precautionary halts to protect key systems. (Citation: Paganini, Pierluigi June 2020) The company announced the potential for temporary shortages of their products following the attack. (Citation: Paganini, Pierluigi June 2020) (Citation: Lion Corporation June 2020)  In the 2021 Colonial Pipeline ransomware incident, the pipeline was unable to transport approximately 2.5 million barrels of fuel per day to the East Coast.  (Citation: Colonial Pipeline Company May 2021)',
    "Drive-by Compromise — Adversaries may gain access to a system during a drive-by compromise, when a user visits a website as part of a regular browsing session.With this technique, the user's web browser is targeted and exploited simply by visiting the compromised website.   The adversary may target a specific community, such as trusted third party suppliers or other industry specific groups, which often visit the target website. This kind of targeted attack relies on a common interest, and is known as a strategic web compromise or watering hole attack.   The National Cyber Awareness System (NCAS) has issued a Technical Alert (TA) regarding Russian government cyber activity targeting critical infrastructure sectors. (Citation: Cybersecurity & Infrastructure Security Agency March 2018) Analysis by DHS and FBI has noted two distinct categories of victims in the Dragonfly campaign on the Western energy sector: staging and intended targets. The adversary targeted the less secure networks of staging targets, including trusted third-party suppliers and related peripheral organizations. Initial access to the intended targets used watering hole attacks to target process control, ICS, and critical infrastructure related trade publications and informational websites.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.3369, 0.1578],
#         [0.3369, 1.0000, 0.3874],
#         [0.1578, 0.3874, 1.0000]])

Evaluation

Metrics

Semantic Similarity

• Dataset: asg-validation
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.8016
spearman_cosine	0.8261

Training Details

Training Dataset

Unnamed Dataset

• Size: 156 training samples
• Columns: sentence_0 and sentence_1
• Approximate statistics based on the first 100 samples:

  	sentence_0	sentence_1
  type	string	string
  modality	text	text
  details	min: 8 tokens mean: 89.62 tokens max: 256 tokens	min: 55 tokens mean: 180.53 tokens max: 256 tokens

• Samples:

  sentence_0	sentence_1
  Adversaries may target devices that are transient across ICS networks and external networks. Normally, transient assets are brought into an environment by authorized personnel and do not remain in that environment on a permanent basis. (Citation: North American Electric Reliability Corporation June 2021) Transient assets are commonly needed to support management functions and may be more common in systems where a remotely managed asset is not feasible, external connections for remote access do not exist, or 3rd party contractor/vendor access is required. Adversaries may take advantage of transient assets in different ways. For instance, adversaries may target a transient asset when it is connected to an external network and then leverage its trusted access in another environment to launch an attack. They may also take advantage of installed applications and libraries that are used by legitimate end-users to interact with control system devices. Transient assets, in some cases, may no...	Transient Cyber Asset — Adversaries may target devices that are transient across ICS networks and external networks. Normally, transient assets are brought into an environment by authorized personnel and do not remain in that environment on a permanent basis. (Citation: North American Electric Reliability Corporation June 2021) Transient assets are commonly needed to support management functions and may be more common in systems where a remotely managed asset is not feasible, external connections for remote access do not exist, or 3rd party contractor/vendor access is required. Adversaries may take advantage of transient assets in different ways. For instance, adversaries may target a transient asset when it is connected to an external network and then leverage its trusted access in another environment to launch an attack. They may also take advantage of installed applications and libraries that are used by legitimate end-users to interact with control system devices. Transient asset...
  Technique Wireless Compromise in tactic initial access	Wireless Compromise — Adversaries may perform wireless compromise as a method of gaining communications and unauthorized access to a wireless network. Access to a wireless network may be gained through the compromise of a wireless device. (Citation: Alexander Bolshev, Gleb Cherbov July 2014) (Citation: Alexander Bolshev March 2014) Adversaries may also utilize radios and other wireless communication devices on the same frequency as the wireless network. Wireless compromise can be done as an initial access vector from a remote distance. A joint case study on the Maroochy Shire Water Services event examined the attack from a cyber security perspective. (Citation: Marshall Abrams July 2008) The adversary disrupted Maroochy Shire's radio-controlled sewage system by driving around with stolen radio equipment and issuing commands with them. Boden used a two-way radio to communicate with and set the frequencies of Maroochy Shire's repeater stations. A Polish student used a modified TV r...
  Adversaries may use masquerading to disguise a malicious application or executable as another file, to avoid operator and engineer suspicion. Possible disguises of these masquerading files can include commonly found programs, expected vendor executables and configuration files, and other commonplace application and naming conventions. By impersonating expected and vendor-relevant files and applications, operators and engineers may not notice the presence of the underlying malicious content and possibly end up running those masquerading as legitimate functions. Applications and other files commonly found on Windows systems or in engineering workstations have been impersonated before. This can be as simple as renaming a file to effectively disguise it in the ICS environment.	Masquerading — Adversaries may use masquerading to disguise a malicious application or executable as another file, to avoid operator and engineer suspicion. Possible disguises of these masquerading files can include commonly found programs, expected vendor executables and configuration files, and other commonplace application and naming conventions. By impersonating expected and vendor-relevant files and applications, operators and engineers may not notice the presence of the underlying malicious content and possibly end up running those masquerading as legitimate functions. Applications and other files commonly found on Windows systems or in engineering workstations have been impersonated before. This can be as simple as renaming a file to effectively disguise it in the ICS environment.

• Loss: MultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• num_train_epochs: 10
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 10
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	asg-validation_spearman_cosine
1.0	10	0.8261

Training Time

• Training: 3.1 minutes
• Evaluation: 52.6 seconds
• Total: 4.0 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.6.0
• Transformers: 4.57.6
• PyTorch: 2.11.0+cpu
• Accelerate: 1.14.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}