ThienLe/Qwen3-SecRerank

CrossEncoder

This is a Cross Encoder model trained on the json dataset using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

• Model Type: Cross Encoder
• Maximum Sequence Length: 4096 tokens
• Number of Output Labels: 1 label
• Training Dataset:
  • json

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Cross Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Cross Encoders on Hugging Face

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("ThienLe/Qwen3-SecRerank")
# Get scores for pairs of texts
pairs = [
    ['<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: Security Account Manager\nAdversaries may attempt to extract credential material from the Security Account Manager (SAM) database either through in-memory techniques or through the Windows Registry where the SAM database is stored. The SAM is a database file that contains local accounts for the host, typically those found with the net user command. Enumerating the SAM database requires SYSTEM level access. A number of tools can be used to retrieve the SAM file through in-memory techniques: Alternatively, the SAM can be extracted from the Registry with Reg: Creddump7 can then be used to process the SAM database locally to retrieve hashes.[1] Notes:\n', "<Document>: APT29\nAPT29 is threat group that has been attributed to Russia's Foreign Intelligence Service (SVR).[1][2] They have operated since at least 2008, often targeting government networks in Europe and NATO member countries, research institutes, and think tanks. APT29 reportedly compromised the Democratic National Committee starting in the summer of 2015.[3][4][5][6] In April 2021, the US and UK governments attributed the SolarWinds Compromise to the SVR; public statements included citations to APT29, Cozy Bear, and The Dukes.[7][8] Industry reporting also referred to the actors involved in this campaign as UNC2452, NOBELIUM, StellarParticle, Dark Halo, and SolarStorm.[9][10][11][12][13][14]\nAPT29 has used the reg save command to save registry hives.[4]<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"],
    ['<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: Why don\'t we use MAC address instead of IP address?\n\nI can use the system function in PHP to get the MAC address of site visitors (probably most of you know). Why do we use IP addresss to check whether someone is stealing a cookie or not?\nDoes the system function have more overhead, or is it still insecure when we don\'t send any parameter to the function?\nI know there are some situations in which users change their MAC address, but it happens less than IP address.\nCould you shed some light on it?\n', "<Document>: The reason for that is very simple: You won't get the MAC address of your website visitor over the Internet, because they are lost when the packets are routed. You can only get the MAC addresses from your subnet (through, for example, ARP).<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"],
    ['<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: Native API\nAdversaries may interact with the native OS application programming interface (API) to execute behaviors. Native APIs provide a controlled means of calling low-level OS services within the kernel, such as those involving hardware/devices, memory, and processes.[1][2] These native APIs are leveraged by the OS during system boot (when other system components are not yet initialized) as well as carrying out tasks and requests during routine operations. Adversaries may abuse these OS API functions as a means of executing behaviors. Similar to Command and Scripting Interpreter, the native API and its hierarchy of interfaces provide mechanisms to interact with and utilize various components of a victimized system. Native API functions (such as NtCreateProcess) may be directed invoked via system calls / syscalls, but these features are also often exposed to user-mode applications via interfaces and libraries.[3][4][5] For example, functions such as the Windows API CreateProcess() or GNU fork() will allow programs and scripts to start other processes.[6][7] This may allow API callers to execute a binary, run a CLI command, load modules, etc. as thousands of similar API functions exist for various system operations.[8][9][10] Higher level software frameworks, such as Microsoft .NET and macOS Cocoa, are also available to interact with native APIs. These frameworks typically provide language wrappers/abstractions to API functionalities and are designed for ease-of-use/portability of code.[11][12][13][14] Adversaries may use assembly to directly or in-directly invoke syscalls in an attempt to subvert defensive sensors and detection signatures such as user mode API-hooks.[15] Adversaries may also attempt to tamper with sensors and defensive tools associated with API monitoring, such as unhooking monitored functions via Disable or Modify Tools.\n', '<Document>: Kapeka\nKapeka is a backdoor written in C++ used against victims in Eastern Europe since at least mid-2022. Kapeka has technical overlaps with Exaramel for Windows and Prestige malware variants, both of which are linked to Sandworm Team. Kapeka may have been used in advance of Prestige deployment in late 2022.[1][2]\nKapeka utilizes WinAPI calls to gather victim system information.[124]<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'],
    ['<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: Why aren\'t infinite-depth wildcard certificates allowed?\n\nAs far as I can tell, an SSL certificate for *.example.com is good for foo.example.com and bar.example.com, but not foo.bar.example.com.\nWildcards certificates cannot have *.*.example.com as their subject. I guess this is due to the fact that certificates like example.* aren\'t allowed -- allowing characters before the wildcard can lead to a malicious user matching their certificate with the wrong domain.\nHowever, I don\'t see any problem with allowing certificates of the *.example.com variety to apply to all subdomains, including sub-subdomains to an infinite depth. I don\'t see any use case where the subdomains of a site are "trusted" but the sub-subdomains are not.\nThis probably causes many problems. As far as I can tell, there\'s no way to cleanly get certificates for all sub-subdomains; you either become a CA, or you buy certificates for each subdomain.\nWhat\'s the reasoning, if any, behind restricting *.example.com to single-depth subdomains only?\nBonus question: Similarly, is there a reason behind the blanket ban on characters before a wildcard? After all, if you allow only dots and asterisks before a wildcard, there\'s no way that the site from a different domain can be spoofed.\n', '<Document>: Technically, usage of wildcards is defined in RFC 2818, which does allow names like "*.*.example.com" or "foo.*.bar.*.example.com" or even "*.*.*". However, between theory and practice, there can be, let\'s say, practical differences (theory and practice match perfectly only in theory, not in practice). Web browsers have implemented stricter rules, because:\n\nImplementing multi-level wildcard matching takes a good five minutes more than implementing matching of names with a single wildcard.\nBrowser vendors did not trust existing CA for never issuing an "*.*.com" certificate.\nDevelopers are human beings, thus very good at not seeing what they cannot imagine, so multi-wildcard names were not implemented by people who did not realize that they were possible.\n\nSo Web browsers will apply restrictions, which RFC 6125 tries to at least document. Most RFC are pragmatist: if reality does not match specification, amend the specification, not reality. Note that browsers will also enforce extra rules, like forbidding "*.co.uk" (not all browsers use the same rules, though, and they are not documented either).\nProfessional CA also enter the dance with their own constraints, such as identity checking tests before issuing certificates, or simply unwillingness to issue too broad wildcard certificates: the core business of a professional CA is to sell many certificates, and wildcard certificates don\'t help for that. Quite the opposite, in fact. People want to buy wildcard certificates precisely to avoid buying many individual certificates.\n\nAnother theory which failed to make it into practice is Name Constraints. With this X.509 extension, it would be possible for a CA to issue a certificate to a sub-CA, restricting that sub-CA so that it may issue server certificates only in a given domain tree. A Name Constraints extension with an "explicit subtree" of value ".example.com" would allow www.example.com and foo.bar.example.com. In that model, the owner of the example.com domain would run his own CA, restricted by its über-CA to only names in the example.com subtree. That would be fine and dandy.\nUnfortunately, anything you do with X.509 certificates is completely worthless if deployed implementations (i.e. Web browsers) don\'t support it, and existing browsers don\'t support name constraints. They don\'t, because there is no certificate with name constraints to process (so that would be useless code), and there is no such certificate because Web browsers would not be able to process them anyway. To bootstrap things, someone must start the cycle, but browser vendors wait after professional CA, and professional CA are unwilling to support name constraints, for the same reasons as previously (which all come down to money, in the long run).<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n'],
    ['<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: Are Kismet (on OpenWRT) and Snort IDS (on a linux server) compatitble?\n\nI\'m trying to develop an IDS/IPS system project to include these elements:\nA router running OpenWRT running Kismet drone (Attitude Adjustment 12.09rc1)\nA Linux server (Running Kismet server + client)\nI have successfully installed Kismet drone/server on their respective platforms.\nBut I have heard that Snort is rather more well-implemented for an IDPS system.\n\nIs there a way to pass packets captured by Kismet to Snort IDS? I have looked on the internet but I have only found outdated and incomplete answers.\nWould it be a good idea to develop this system by just using Kismet as an IDS, without using Snort at all?\nAny other ideas and suggests are mostly welcome, thank you.\n', "<Document>: I think libpcap and/or tcpdump are what you're looking for. Kismet is a wireless analyzer that will display the 802.11 metadata. There was some activity awhile ago to add 802.11 intrusion detection capabilities (snort-wireless). You might still find useful remnanats of that.\nKismet pulls the 802.11 frame data and analyzes that, then examines some of the TCP/IP data (when it's unencrypted or decrypted) for additional network activity reporting. Kismet can also log the captured data, which you could then feed into an IDS (or any other process) using tcpreplay.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    '<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: Security Account Manager\nAdversaries may attempt to extract credential material from the Security Account Manager (SAM) database either through in-memory techniques or through the Windows Registry where the SAM database is stored. The SAM is a database file that contains local accounts for the host, typically those found with the net user command. Enumerating the SAM database requires SYSTEM level access. A number of tools can be used to retrieve the SAM file through in-memory techniques: Alternatively, the SAM can be extracted from the Registry with Reg: Creddump7 can then be used to process the SAM database locally to retrieve hashes.[1] Notes:\n',
    [
        "<Document>: APT29\nAPT29 is threat group that has been attributed to Russia's Foreign Intelligence Service (SVR).[1][2] They have operated since at least 2008, often targeting government networks in Europe and NATO member countries, research institutes, and think tanks. APT29 reportedly compromised the Democratic National Committee starting in the summer of 2015.[3][4][5][6] In April 2021, the US and UK governments attributed the SolarWinds Compromise to the SVR; public statements included citations to APT29, Cozy Bear, and The Dukes.[7][8] Industry reporting also referred to the actors involved in this campaign as UNC2452, NOBELIUM, StellarParticle, Dark Halo, and SolarStorm.[9][10][11][12][13][14]\nAPT29 has used the reg save command to save registry hives.[4]<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
        "<Document>: The reason for that is very simple: You won't get the MAC address of your website visitor over the Internet, because they are lost when the packets are routed. You can only get the MAC addresses from your subnet (through, for example, ARP).<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
        '<Document>: Kapeka\nKapeka is a backdoor written in C++ used against victims in Eastern Europe since at least mid-2022. Kapeka has technical overlaps with Exaramel for Windows and Prestige malware variants, both of which are linked to Sandworm Team. Kapeka may have been used in advance of Prestige deployment in late 2022.[1][2]\nKapeka utilizes WinAPI calls to gather victim system information.[124]<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n',
        '<Document>: Technically, usage of wildcards is defined in RFC 2818, which does allow names like "*.*.example.com" or "foo.*.bar.*.example.com" or even "*.*.*". However, between theory and practice, there can be, let\'s say, practical differences (theory and practice match perfectly only in theory, not in practice). Web browsers have implemented stricter rules, because:\n\nImplementing multi-level wildcard matching takes a good five minutes more than implementing matching of names with a single wildcard.\nBrowser vendors did not trust existing CA for never issuing an "*.*.com" certificate.\nDevelopers are human beings, thus very good at not seeing what they cannot imagine, so multi-wildcard names were not implemented by people who did not realize that they were possible.\n\nSo Web browsers will apply restrictions, which RFC 6125 tries to at least document. Most RFC are pragmatist: if reality does not match specification, amend the specification, not reality. Note that browsers will also enforce extra rules, like forbidding "*.co.uk" (not all browsers use the same rules, though, and they are not documented either).\nProfessional CA also enter the dance with their own constraints, such as identity checking tests before issuing certificates, or simply unwillingness to issue too broad wildcard certificates: the core business of a professional CA is to sell many certificates, and wildcard certificates don\'t help for that. Quite the opposite, in fact. People want to buy wildcard certificates precisely to avoid buying many individual certificates.\n\nAnother theory which failed to make it into practice is Name Constraints. With this X.509 extension, it would be possible for a CA to issue a certificate to a sub-CA, restricting that sub-CA so that it may issue server certificates only in a given domain tree. A Name Constraints extension with an "explicit subtree" of value ".example.com" would allow www.example.com and foo.bar.example.com. In that model, the owner of the example.com domain would run his own CA, restricted by its über-CA to only names in the example.com subtree. That would be fine and dandy.\nUnfortunately, anything you do with X.509 certificates is completely worthless if deployed implementations (i.e. Web browsers) don\'t support it, and existing browsers don\'t support name constraints. They don\'t, because there is no certificate with name constraints to process (so that would be useless code), and there is no such certificate because Web browsers would not be able to process them anyway. To bootstrap things, someone must start the cycle, but browser vendors wait after professional CA, and professional CA are unwilling to support name constraints, for the same reasons as previously (which all come down to money, in the long run).<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n',
        "<Document>: I think libpcap and/or tcpdump are what you're looking for. Kismet is a wireless analyzer that will display the 802.11 metadata. There was some activity awhile ago to add 802.11 intrusion detection capabilities (snort-wireless). You might still find useful remnanats of that.\nKismet pulls the 802.11 frame data and analyzes that, then examines some of the TCP/IP data (when it's unencrypted or decrypted) for additional network activity reporting. Kismet can also log the captured data, which you could then feed into an IDS (or any other process) using tcpreplay.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n",
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Training Details

Training Dataset

json

• Dataset: json
• Size: 49,346 training samples
• Columns: sentence1 and sentence2
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2
  type	string	string
  details	min: 414 characters mean: 1301.6 characters max: 19975 characters	min: 84 characters mean: 1069.28 characters max: 9940 characters

• Samples:

  sentence1	sentence2
  <|im_start|>system Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|> <|im_start|>user : Given a web search query, retrieve relevant passages that answer the query : Is it possible to send data from an open-source program but make it impossible for a user with source code to do the same? If I want to store a global scoreboard for a game running locally on the user's computer and I want to make sure that all the requests coming to the server are really generated by the game and not spoofed by the users, is there anything that can be done to prevent cheating? The program is open-source, so no obfuscation can be used. The problem I have with coming up with a solution is that whatever I choose to implement in code must necessarily include all the components necessary for creating a spoofed program that can send any data user wants, especially any encryption keys and hash...	: In a word, no. The best option is to move all the actual game logic to the server, and have the client be a thin client that just displays state and sends input. That wouldn't prevent various types of cheating (such as game automation tools) but it's the only way to comprehensively avoid the user sending fraudulent game results. It's a popular approach in multi-player, too, as it avoids telling the user anything they don't need to know. However, it's considerably more expensive in server hardware. The next option is to make it possible to validate high scores. One option would be to record, for every game, the RNG seed value followed by every input that the user provides combined with a timestamp of some sort (for turn-based games this is just an action count; for real-time games it would probably be a game engine frame/tick count). Transmitting all of that back to the server would be a lot, but - combined with the game version and level or whatever - allow the server to ef...
  <|im_start|>system Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|> <|im_start|>user : Given a web search query, retrieve relevant passages that answer the query : Encrypted/Encoded File Adversaries may encrypt or encode files to obfuscate strings, bytes, and other specific patterns to impede detection. Encrypting and/or encoding file content aims to conceal malicious artifacts within a file used in an intrusion. Many other techniques, such as Software Packing, Steganography, and Embedded Payloads, share this same broad objective. Encrypting and/or encoding files could lead to a lapse in detection of static signatures, only for this malicious content to be revealed (i.e., Deobfuscate/Decode Files or Information) at the time of execution/use. This type of file obfuscation can be applied to many file artifacts present on victim hosts, such as malware log/configuration...	: Dark Caracal Dark Caracal is threat group that has been attributed to the Lebanese General Directorate of General Security (GDGS) and has operated since at least 2012. [1] Dark Caracal has obfuscated strings in Bandook by base64 encoding, and then encrypting them.[58]<|im_end|> <|im_start|>assistant
  <|im_start|>system Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|> <|im_start|>user : Given a web search query, retrieve relevant passages that answer the query : OS Credential Dumping Adversaries may attempt to dump credentials to obtain account login and credential material, normally in the form of a hash or a clear text password. Credentials can be obtained from OS caches, memory, or structures.[1] Credentials can then be used to perform Lateral Movement and access restricted information. Several of the tools mentioned in associated sub-techniques may be used by both adversaries and professional security testers. Additional custom tools likely exist as well.	: Ember Bear Ember Bear is a Russian state-sponsored cyber espionage group that has been active since at least 2020, linked to Russia's General Staff Main Intelligence Directorate (GRU) 161st Specialist Training Center (Unit 29155).[1] Ember Bear has primarily focused operations against Ukrainian government and telecommunication entities, but has also operated against critical infrastructure entities in Europe and the Americas.[2] Ember Bear conducted the WhisperGate destructive wiper attacks against Ukraine in early 2022.[3][4][1] There is some confusion as to whether Ember Bear overlaps with another Russian-linked entity referred to as Saint Bear. At present available evidence strongly suggests these are distinct activities with different behavioral profiles.[2][5] Ember Bear gathers credential material from target systems, such as SSH keys, to facilitate access to victim environments.[12]<|im_end|> <|im_start|>assistant

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 10.0,
      "num_negatives": 4,
      "activation_fn": "torch.nn.modules.activation.Sigmoid",
      "mini_batch_size": 32
  }
  

Evaluation Dataset

json

• Dataset: json
• Size: 12,337 evaluation samples
• Columns: sentence1 and sentence2
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2
  type	string	string
  details	min: 352 characters mean: 1300.22 characters max: 10030 characters	min: 85 characters mean: 1080.48 characters max: 9526 characters

• Samples:

  sentence1	sentence2
  <|im_start|>system Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|> <|im_start|>user : Given a web search query, retrieve relevant passages that answer the query : Security Account Manager Adversaries may attempt to extract credential material from the Security Account Manager (SAM) database either through in-memory techniques or through the Windows Registry where the SAM database is stored. The SAM is a database file that contains local accounts for the host, typically those found with the net user command. Enumerating the SAM database requires SYSTEM level access. A number of tools can be used to retrieve the SAM file through in-memory techniques: Alternatively, the SAM can be extracted from the Registry with Reg: Creddump7 can then be used to process the SAM database locally to retrieve hashes.[1] Notes:	: APT29 APT29 is threat group that has been attributed to Russia's Foreign Intelligence Service (SVR).[1][2] They have operated since at least 2008, often targeting government networks in Europe and NATO member countries, research institutes, and think tanks. APT29 reportedly compromised the Democratic National Committee starting in the summer of 2015.[3][4][5][6] In April 2021, the US and UK governments attributed the SolarWinds Compromise to the SVR; public statements included citations to APT29, Cozy Bear, and The Dukes.[7][8] Industry reporting also referred to the actors involved in this campaign as UNC2452, NOBELIUM, StellarParticle, Dark Halo, and SolarStorm.[9][10][11][12][13][14] APT29 has used the reg save command to save registry hives.[4]<|im_end|> <|im_start|>assistant
  <|im_start|>system Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|> <|im_start|>user : Given a web search query, retrieve relevant passages that answer the query : Why don't we use MAC address instead of IP address? I can use the system function in PHP to get the MAC address of site visitors (probably most of you know). Why do we use IP addresss to check whether someone is stealing a cookie or not? Does the system function have more overhead, or is it still insecure when we don't send any parameter to the function? I know there are some situations in which users change their MAC address, but it happens less than IP address. Could you shed some light on it?	: The reason for that is very simple: You won't get the MAC address of your website visitor over the Internet, because they are lost when the packets are routed. You can only get the MAC addresses from your subnet (through, for example, ARP).<|im_end|> <|im_start|>assistant
  <|im_start|>system Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no".<|im_end|> <|im_start|>user : Given a web search query, retrieve relevant passages that answer the query : Native API Adversaries may interact with the native OS application programming interface (API) to execute behaviors. Native APIs provide a controlled means of calling low-level OS services within the kernel, such as those involving hardware/devices, memory, and processes.[1][2] These native APIs are leveraged by the OS during system boot (when other system components are not yet initialized) as well as carrying out tasks and requests during routine operations. Adversaries may abuse these OS API functions as a means of executing behaviors. Similar to Command and Scripting Interpreter, the native API and its hierarchy of interfaces provide mechanisms to interact with and utilize various components of a vict...	: Kapeka Kapeka is a backdoor written in C++ used against victims in Eastern Europe since at least mid-2022. Kapeka has technical overlaps with Exaramel for Windows and Prestige malware variants, both of which are linked to Sandworm Team. Kapeka may have been used in advance of Prestige deployment in late 2022.[1][2] Kapeka utilizes WinAPI calls to gather victim system information.[124]<|im_end|> <|im_start|>assistant

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 10.0,
      "num_negatives": 4,
      "activation_fn": "torch.nn.modules.activation.Sigmoid",
      "mini_batch_size": 32
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 1
• warmup_steps: 200
• optim: adamw_8bit
• gradient_accumulation_steps: 2
• bf16: True
• gradient_checkpointing: True
• eval_strategy: steps
• dataloader_num_workers: 4
• dataloader_pin_memory: False

All Hyperparameters

Click to expand

• per_device_train_batch_size: 8
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 200
• optim: adamw_8bit
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 2
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: True
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: steps
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 4
• dataloader_pin_memory: False
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss
0.2594	800	0.0868	0.1018
0.3890	1200	0.0697	0.0742
0.5187	1600	0.0502	0.0609
0.6484	2000	0.0468	0.0405
0.7781	2400	0.0361	0.0316
0.9078	2800	0.0257	0.0303

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.3
• Transformers: 5.2.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.5.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}