| --- |
| tags: |
| - ColBERT |
| - PyLate |
| - sentence-transformers |
| - sentence-similarity |
| - feature-extraction |
| - generated_from_trainer |
| - dataset_size:1188486 |
| - loss:Contrastive |
| base_model: jhu-clsp/ettin-encoder-17m |
| datasets: |
| - benjamintli/code-retrieval-combined-v2-llm-negatives |
| pipeline_tag: sentence-similarity |
| library_name: PyLate |
| --- |
| |
| # PyLate model based on jhu-clsp/ettin-encoder-17m |
|
|
| This is a [PyLate](https://github.com/lightonai/pylate) model finetuned from [jhu-clsp/ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) on the [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives) dataset. It maps sentences & paragraphs to sequences of 128-dimensional dense vectors and can be used for semantic textual similarity using the MaxSim operator. |
|
|
| ## Model Details |
|
|
| ### Model Description |
| - **Model Type:** PyLate model |
| - **Base model:** [jhu-clsp/ettin-encoder-17m](https://huggingface.co/jhu-clsp/ettin-encoder-17m) <!-- at revision 987607455c61e7a5bbc85f7758e0512ea6d0ae4c --> |
| - **Document Length:** 180 tokens |
| - **Query Length:** 32 tokens |
| - **Output Dimensionality:** 128 tokens |
| - **Similarity Function:** MaxSim |
| - **Training Dataset:** |
| - [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives) |
| <!-- - **Language:** Unknown --> |
| <!-- - **License:** Unknown --> |
|
|
| ### Model Sources |
|
|
| - **Documentation:** [PyLate Documentation](https://lightonai.github.io/pylate/) |
| - **Repository:** [PyLate on GitHub](https://github.com/lightonai/pylate) |
| - **Hugging Face:** [PyLate models on Hugging Face](https://huggingface.co/models?library=PyLate) |
|
|
| ### Full Model Architecture |
|
|
| ``` |
| ColBERT( |
| (0): Transformer({'max_seq_length': 31, 'do_lower_case': False, 'architecture': 'ModernBertModel'}) |
| (1): Dense({'in_features': 256, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'use_residual': False}) |
| ) |
| ``` |
|
|
| ## Usage |
| First install the PyLate library: |
|
|
| ```bash |
| pip install -U pylate |
| ``` |
|
|
| ### Retrieval |
|
|
| Use this model with PyLate to index and retrieve documents. The index uses [FastPLAID](https://github.com/lightonai/fast-plaid) for efficient similarity search. |
|
|
| #### Indexing documents |
|
|
| Load the ColBERT model and initialize the PLAID index, then encode and index your documents: |
|
|
| ```python |
| from pylate import indexes, models, retrieve |
| |
| # Step 1: Load the ColBERT model |
| model = models.ColBERT( |
| model_name_or_path="colbert-code-17m", |
| ) |
| |
| # Step 2: Initialize the PLAID index |
| index = indexes.PLAID( |
| index_folder="pylate-index", |
| index_name="index", |
| override=True, # This overwrites the existing index if any |
| ) |
| |
| # Step 3: Encode the documents |
| documents_ids = ["1", "2", "3"] |
| documents = ["document 1 text", "document 2 text", "document 3 text"] |
| |
| documents_embeddings = model.encode( |
| documents, |
| batch_size=32, |
| is_query=False, # Ensure that it is set to False to indicate that these are documents, not queries |
| show_progress_bar=True, |
| ) |
| |
| # Step 4: Add document embeddings to the index by providing embeddings and corresponding ids |
| index.add_documents( |
| documents_ids=documents_ids, |
| documents_embeddings=documents_embeddings, |
| ) |
| ``` |
|
|
| Note that you do not have to recreate the index and encode the documents every time. Once you have created an index and added the documents, you can re-use the index later by loading it: |
|
|
| ```python |
| # To load an index, simply instantiate it with the correct folder/name and without overriding it |
| index = indexes.PLAID( |
| index_folder="pylate-index", |
| index_name="index", |
| ) |
| ``` |
|
|
| #### Retrieving top-k documents for queries |
|
|
| Once the documents are indexed, you can retrieve the top-k most relevant documents for a given set of queries. |
| To do so, initialize the ColBERT retriever with the index you want to search in, encode the queries and then retrieve the top-k documents to get the top matches ids and relevance scores: |
|
|
| ```python |
| # Step 1: Initialize the ColBERT retriever |
| retriever = retrieve.ColBERT(index=index) |
| |
| # Step 2: Encode the queries |
| queries_embeddings = model.encode( |
| ["query for document 3", "query for document 1"], |
| batch_size=32, |
| is_query=True, # # Ensure that it is set to False to indicate that these are queries |
| show_progress_bar=True, |
| ) |
| |
| # Step 3: Retrieve top-k documents |
| scores = retriever.retrieve( |
| queries_embeddings=queries_embeddings, |
| k=10, # Retrieve the top 10 matches for each query |
| ) |
| ``` |
|
|
| ### Reranking |
| If you only want to use the ColBERT model to perform reranking on top of your first-stage retrieval pipeline without building an index, you can simply use rank function and pass the queries and documents to rerank: |
|
|
| ```python |
| from pylate import rank, models |
| |
| queries = [ |
| "query A", |
| "query B", |
| ] |
| |
| documents = [ |
| ["document A", "document B"], |
| ["document 1", "document C", "document B"], |
| ] |
| |
| documents_ids = [ |
| [1, 2], |
| [1, 3, 2], |
| ] |
| |
| model = models.ColBERT( |
| model_name_or_path="colbert-code-17m", |
| ) |
| |
| queries_embeddings = model.encode( |
| queries, |
| is_query=True, |
| ) |
| |
| documents_embeddings = model.encode( |
| documents, |
| is_query=False, |
| ) |
| |
| reranked_documents = rank.rerank( |
| documents_ids=documents_ids, |
| queries_embeddings=queries_embeddings, |
| documents_embeddings=documents_embeddings, |
| ) |
| ``` |
|
|
| <!-- |
| ### Direct Usage (Transformers) |
|
|
| <details><summary>Click to see the direct usage in Transformers</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Downstream Usage (Sentence Transformers) |
|
|
| You can finetune this model on your own dataset. |
|
|
| <details><summary>Click to expand</summary> |
|
|
| </details> |
| --> |
|
|
| <!-- |
| ### Out-of-Scope Use |
|
|
| *List how the model may foreseeably be misused and address what users ought not to do with the model.* |
| --> |
|
|
| <!-- |
| ## Bias, Risks and Limitations |
|
|
| *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
| --> |
|
|
| <!-- |
| ### Recommendations |
|
|
| *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
| --> |
|
|
| ## Training Details |
|
|
| ### Training Dataset |
|
|
| #### code-retrieval-combined-v2-llm-negatives |
|
|
| * Dataset: [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives) at [1917069](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives/tree/19170694c88339259e15b3dcc7f99db2136bd00a) |
| * Size: 1,188,486 training samples |
| * Columns: <code>query</code>, <code>positive</code>, <code>source</code>, <code>hard_negatives</code>, and <code>negatives</code> |
| * Approximate statistics based on the first 1000 samples: |
| | | query | positive | source | hard_negatives | negatives | |
| |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:-----------------------------------|:-----------------------------------------------------------------------------------| |
| | type | string | string | string | list | string | |
| | details | <ul><li>min: 6 tokens</li><li>mean: 23.61 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>min: 14 tokens</li><li>mean: 31.43 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 7.46 tokens</li><li>max: 10 tokens</li></ul> | <ul><li>size: 1 elements</li></ul> | <ul><li>min: 16 tokens</li><li>mean: 31.15 tokens</li><li>max: 32 tokens</li></ul> | |
| * Samples: |
| | query | positive | source | hard_negatives | negatives | |
| |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | <code>wait for AWS PCA CSR propagation before issuing certificate</code> | <code>func (c *ACMPCA) WaitUntilCertificateAuthorityCSRCreated(input *GetCertificateAuthorityCsrInput) error {<br> return c.WaitUntilCertificateAuthorityCSRCreatedWithContext(aws.BackgroundContext(), input)<br>}</code> | <code>csn_syntethic</code> | <code>['func (c *ACMPCA) WaitUntilAuditReportCreated(input *DescribeCertificateAuthorityAuditReportInput) error {\n\treturn c.WaitUntilAuditReportCreatedWithContext(aws.BackgroundContext(), input)\n}']</code> | <code>func (c *ACMPCA) WaitUntilAuditReportCreated(input *DescribeCertificateAuthorityAuditReportInput) error {<br> return c.WaitUntilAuditReportCreatedWithContext(aws.BackgroundContext(), input)<br>}</code> | |
| | <code>func (f *Filter) Gather() ([]*dto.MetricFamily, error) {<br> mfs, err := f.Gatherer.Gather()<br> if err != nil {<br></code> | <code> return nil, err<br> }<br> return f.Matcher.Match(mfs), nil<br>}</code> | <code>csn_ccr</code> | <code>['\n\tf.err = err\n\tif err != nil {\n\t\tfor _, node := range f.cells {\n\t\t\tnode.PropagateWatchError(err)\n\t\t}\n\t}\n}']</code> | <code><br> f.err = err<br> if err != nil {<br> for _, node := range f.cells {<br> node.PropagateWatchError(err)<br> }<br> }<br>}</code> | |
| | <code>def latent_to_dist(name, x, hparams, output_channels=None):<br> """Map latent to the mean and log-scale of a Gaussian.<br><br> Args:<br> name: variable scope.<br> x: 4-D Tensor of shape (NHWC)<br> hparams: HParams.<br> latent_architecture - can be "single_conv", "glow_nn" or "glow_resnet",<br> default = single_conv<br> latent_encoder_depth - int, depth of architecture, valid if<br> latent_architecture is "glow_nn" or "glow_resnet".<br> latent_pre_output_channels - 512, valid only when latent_architecture<br> is "glow_nn".<br> latent_encoder_width - 512, maximum width of the network<br> output_channels: int, number of output channels of the mean (and std).<br> if not provided, set it to be the output channels of x.<br> Returns:<br> dist: instance of tfp.distributions.Normal<br> Raises:<br> ValueError: If architecture not in ["single_conv", "glow_nn"]<br> """<br> architecture = hparams.get("latent_a...</code> | <code> mid_channels=mid_channels)<br> mean_log_scale = conv("glow_nn_zeros", mean_log_scale,<br> filter_size=[3, 3], stride=[1, 1],<br> output_channels=2*output_channels,<br> apply_actnorm=False, conv_init="zeros")<br> elif architecture == "glow_resnet":<br> h = x<br> for layer in range(depth):<br> h3 = conv_stack("latent_resnet_%d" % layer, h,<br> mid_channels=width, output_channels=x_shape[-1],<br> dropout=hparams.coupling_dropout)<br> h += h3<br> mean_log_scale = conv("glow_res_final", h, conv_init="zeros",<br> output_channels=2*output_channels,<br> apply_actnorm=False)<br> else:<br> raise ValueError("expected architecture to be single_conv or glow_nn "<br> "got %s" % architecture)<br><br> mean = mean_log_scale[:, :, :, 0::2]<br> log_scale = mean_log_scale[:, :, :, 1::2]<br> return tfp.distribu...</code> | <code>csn_ccr</code> | <code>[' first_relu=False,\n padding="SAME",\n strides=(2, 2),\n force2d=True,\n name="conv0")\n x = common_layers.conv_block(\n x, 64, [((1, 1), (3, 3))], padding="SAME", force2d=True, name="conv1")\n x = xnet_resblock(x, min(128, hidden_dim), True, "block0")\n x = xnet_resblock(x, min(256, hidden_dim), False, "block1")\n return xnet_resblock(x, hidden_dim, False, "block2")']</code> | <code> first_relu=False,<br> padding="SAME",<br> strides=(2, 2),<br> force2d=True,<br> name="conv0")<br> x = common_layers.conv_block(<br> x, 64, [((1, 1), (3, 3))], padding="SAME", force2d=True, name="conv1")<br> x = xnet_resblock(x, min(128, hidden_dim), True, "block0")<br> x = xnet_resblock(x, min(256, hidden_dim), False, "block1")<br> return xnet_resblock(x, hidden_dim, False, "block2")</code> | |
| * Loss: <code>pylate.losses.contrastive.Contrastive</code> |
|
|
| ### Evaluation Dataset |
|
|
| #### code-retrieval-combined-v2-llm-negatives |
|
|
| * Dataset: [code-retrieval-combined-v2-llm-negatives](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives) at [1917069](https://huggingface.co/datasets/benjamintli/code-retrieval-combined-v2-llm-negatives/tree/19170694c88339259e15b3dcc7f99db2136bd00a) |
| * Size: 12,005 evaluation samples |
| * Columns: <code>query</code>, <code>positive</code>, <code>source</code>, <code>hard_negatives</code>, and <code>negatives</code> |
| * Approximate statistics based on the first 1000 samples: |
| | | query | positive | source | hard_negatives | negatives | |
| |:--------|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:---------------------------------------------------------------------------------|:-----------------------------------|:-----------------------------------------------------------------------------------| |
| | type | string | string | string | list | string | |
| | details | <ul><li>min: 6 tokens</li><li>mean: 23.78 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>min: 14 tokens</li><li>mean: 31.45 tokens</li><li>max: 32 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 7.55 tokens</li><li>max: 10 tokens</li></ul> | <ul><li>size: 1 elements</li></ul> | <ul><li>min: 14 tokens</li><li>mean: 31.15 tokens</li><li>max: 32 tokens</li></ul> | |
| * Samples: |
| | query | positive | source | hard_negatives | negatives | |
| |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
| | <code>public static function list_templates($competencyid, $onlyvisible) {<br> global $DB;<br><br> $sql = 'SELECT tpl.*<br> FROM {' . template::TABLE . '} tpl<br> JOIN {' . self::TABLE . '} tplcomp<br> ON tplcomp.templateid = tpl.id<br> WHERE tplcomp.competencyid = ? ';<br> $params = array($competencyid);<br><br> if ($onlyvisible) {<br> $sql .= ' AND tpl.visible = ?';<br> $params[] = 1;<br> }<br><br> </code> | <code> $sql .= ' ORDER BY tpl.id ASC';<br><br> $results = $DB->get_records_sql($sql, $params);<br><br> $instances = array();<br> foreach ($results as $result) {<br> array_push($instances, new template(0, $result));<br> }<br><br> return $instances;<br> }</code> | <code>csn_ccr</code> | <code>[" FROM {assign_grades} g\n JOIN(' . $esql . ') e ON e.id = g.userid\n WHERE g.assignment = :assignid';\n\n return $DB->count_records_sql($sql, $params);\n }"]</code> | <code> FROM {assign_grades} g<br> JOIN(' . $esql . ') e ON e.id = g.userid<br> WHERE g.assignment = :assignid';<br><br> return $DB->count_records_sql($sql, $params);<br> }</code> | |
| | <code>python multiprocessing sandboxed process isolation file system network</code> | <code>def start(self):<br> '''Create a process in which the isolated code will be run.'''<br> assert self._client is None<br><br> logger.debug('IsolationContext[%d] starting', id(self))<br><br> # Create the queues<br> request_queue = multiprocessing.Queue()<br> response_queue = multiprocessing.Queue()<br><br> # Launch the server process<br> server = Server(request_queue, response_queue) # Do not keep a reference to this object!<br> server_process = multiprocessing.Process(target=server.loop)<br> server_process.start()<br><br> # Create a client to talk to the server<br> self._client = Client(server_process, request_queue, response_queue)</code> | <code>csn_syntethic</code> | <code>['def start(cls, _init_logging=True):\n """\n Arrange for the subprocess to be started, if it is not already running.\n\n The parent process picks a UNIX socket path the child will use prior to\n fork, creates a socketpair used essentially as a semaphore, then blocks\n waiting for the child to indicate the UNIX socket is ready for use.\n\n :param bool _init_logging:\n For testing, if :data:`False`, don\'t initialize logging.\n """\n if cls.worker_sock is not None:\n return\n\n if faulthandler is not None:\n faulthandler.enable()\n\n mitogen.utils.setup_gil()\n cls.unix_listener_path = mitogen.unix.make_socket_path()\n cls.worker_sock, cls.child_sock = socket.socketpair()\n atexit.register(lambda: clean_shutdown(cls.worker_sock))\n mitogen.core.set_cloexec(cls.worker_sock.fileno())\n mitogen.core.set_cloexec(cls.child_sock.fileno())\n\n cls.profiling = os.environ.get(\'MITOGEN_PROFILING\') is not None\n if cls.profiling:\n mitogen.core.enable_profiling()\n if _init_logging:\n ansible_mitogen.logging.setup()\n\n cls.original_env = dict(os.environ)\n cls.child_pid = os.fork()\n if cls.child_pid:\n save_pid(\'controller\')\n ansible_mitogen.logging.set_process_name(\'top\')\n ansible_mitogen.affinity.policy.assign_controller()\n cls.child_sock.close()\n cls.child_sock = None\n mitogen.core.io_op(cls.worker_sock.recv, 1)\n else:\n save_pid(\'mux\')\n ansible_mitogen.logging.set_process_name(\'mux\')\n ansible_mitogen.affinity.policy.assign_muxprocess()\n cls.worker_sock.close()\n cls.worker_sock = None\n self = cls()\n self.worker_main()']</code> | <code>def start(cls, _init_logging=True):<br> """<br> Arrange for the subprocess to be started, if it is not already running.<br><br> The parent process picks a UNIX socket path the child will use prior to<br> fork, creates a socketpair used essentially as a semaphore, then blocks<br> waiting for the child to indicate the UNIX socket is ready for use.<br><br> :param bool _init_logging:<br> For testing, if :data:`False`, don't initialize logging.<br> """<br> if cls.worker_sock is not None:<br> return<br><br> if faulthandler is not None:<br> faulthandler.enable()<br><br> mitogen.utils.setup_gil()<br> cls.unix_listener_path = mitogen.unix.make_socket_path()<br> cls.worker_sock, cls.child_sock = socket.socketpair()<br> atexit.register(lambda: clean_shutdown(cls.worker_sock))<br> mitogen.core.set_cloexec(cls.worker_sock.fileno())<br> mitogen.core.set_cloexec(cls.child_sock.fileno())<br><br> cls.profiling = os.environ.get('MI...</code> | |
| | <code>updates the current cookies with a new set<br><br>@param array $cookies new cookies with which to update current ones<br><br>@return boolean always return true<br>@access private</code> | <code>function UpdateCookies($cookies)
<br> {
<br> if (sizeof($this->cookies) == 0) {
<br> // no existing cookies: take whatever is new
<br> if (sizeof($cookies) > 0) {
<br> $this->debug('Setting new cookie(s)');
<br> $this->cookies = $cookies;
<br> }
<br>
<br> return TRUE;
<br> }
<br> if (sizeof($cookies) == 0) {
<br> // no new cookies: keep what we've got
<br> return TRUE;
<br> }
<br> // merge
<br> foreach ($cookies as $newCookie) {
<br> if (!is_array($newCookie)) {
<br> continue;
<br> }
<br> if ((!isset($newCookie['name'])) \|\| (!isset($newCookie['value']))) {
<br> continue;
<br> }
<br> $newName = $newCookie['name'];
<br>
<br> $found = FALSE;
<br> for ($i = 0; $i < count($this->cookies); $i++) {
<br> $cookie = $this->cookies[$i];
<br> if (!is_array($cookie)) {
<br> continue;
<br> ...</code> | <code>csn</code> | <code>["public function setCookies(array $cookies) : Request\n {\n $query = http_build_query($cookies);\n parse_str($query, $this->cookies);\n\n if ($cookies) {\n $cookie = str_replace('&', '; ', $query);\n $this->setHeader('Cookie', $cookie);\n } else {\n $this->removeHeader('Cookie');\n }\n\n return $this;\n }"]</code> | <code>public function setCookies(array $cookies) : Request<br> {<br> $query = http_build_query($cookies);<br> parse_str($query, $this->cookies);<br><br> if ($cookies) {<br> $cookie = str_replace('&', '; ', $query);<br> $this->setHeader('Cookie', $cookie);<br> } else {<br> $this->removeHeader('Cookie');<br> }<br><br> return $this;<br> }</code> | |
| * Loss: <code>pylate.losses.contrastive.Contrastive</code> |
|
|
| ### Training Hyperparameters |
| #### Non-Default Hyperparameters |
|
|
| - `eval_strategy`: steps |
| - `per_device_train_batch_size`: 256 |
| - `per_device_eval_batch_size`: 256 |
| - `learning_rate`: 3e-06 |
| - `num_train_epochs`: 1 |
| - `fp16`: True |
| - `hub_model_id`: colbert-code-17m |
|
|
| #### All Hyperparameters |
| <details><summary>Click to expand</summary> |
|
|
| - `overwrite_output_dir`: False |
| - `do_predict`: False |
| - `eval_strategy`: steps |
| - `prediction_loss_only`: True |
| - `per_device_train_batch_size`: 256 |
| - `per_device_eval_batch_size`: 256 |
| - `per_gpu_train_batch_size`: None |
| - `per_gpu_eval_batch_size`: None |
| - `gradient_accumulation_steps`: 1 |
| - `eval_accumulation_steps`: None |
| - `torch_empty_cache_steps`: None |
| - `learning_rate`: 3e-06 |
| - `weight_decay`: 0.0 |
| - `adam_beta1`: 0.9 |
| - `adam_beta2`: 0.999 |
| - `adam_epsilon`: 1e-08 |
| - `max_grad_norm`: 1.0 |
| - `num_train_epochs`: 1 |
| - `max_steps`: -1 |
| - `lr_scheduler_type`: linear |
| - `lr_scheduler_kwargs`: {} |
| - `warmup_ratio`: 0.0 |
| - `warmup_steps`: 0 |
| - `log_level`: passive |
| - `log_level_replica`: warning |
| - `log_on_each_node`: True |
| - `logging_nan_inf_filter`: True |
| - `save_safetensors`: True |
| - `save_on_each_node`: False |
| - `save_only_model`: False |
| - `restore_callback_states_from_checkpoint`: False |
| - `no_cuda`: False |
| - `use_cpu`: False |
| - `use_mps_device`: False |
| - `seed`: 42 |
| - `data_seed`: None |
| - `jit_mode_eval`: False |
| - `use_ipex`: False |
| - `bf16`: False |
| - `fp16`: True |
| - `fp16_opt_level`: O1 |
| - `half_precision_backend`: auto |
| - `bf16_full_eval`: False |
| - `fp16_full_eval`: False |
| - `tf32`: None |
| - `local_rank`: 0 |
| - `ddp_backend`: None |
| - `tpu_num_cores`: None |
| - `tpu_metrics_debug`: False |
| - `debug`: [] |
| - `dataloader_drop_last`: False |
| - `dataloader_num_workers`: 0 |
| - `dataloader_prefetch_factor`: None |
| - `past_index`: -1 |
| - `disable_tqdm`: False |
| - `remove_unused_columns`: True |
| - `label_names`: None |
| - `load_best_model_at_end`: False |
| - `ignore_data_skip`: False |
| - `fsdp`: [] |
| - `fsdp_min_num_params`: 0 |
| - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
| - `fsdp_transformer_layer_cls_to_wrap`: None |
| - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
| - `parallelism_config`: None |
| - `deepspeed`: None |
| - `label_smoothing_factor`: 0.0 |
| - `optim`: adamw_torch_fused |
| - `optim_args`: None |
| - `adafactor`: False |
| - `group_by_length`: False |
| - `length_column_name`: length |
| - `ddp_find_unused_parameters`: None |
| - `ddp_bucket_cap_mb`: None |
| - `ddp_broadcast_buffers`: False |
| - `dataloader_pin_memory`: True |
| - `dataloader_persistent_workers`: False |
| - `skip_memory_metrics`: True |
| - `use_legacy_prediction_loop`: False |
| - `push_to_hub`: False |
| - `resume_from_checkpoint`: None |
| - `hub_model_id`: colbert-code-17m |
| - `hub_strategy`: every_save |
| - `hub_private_repo`: None |
| - `hub_always_push`: False |
| - `hub_revision`: None |
| - `gradient_checkpointing`: False |
| - `gradient_checkpointing_kwargs`: None |
| - `include_inputs_for_metrics`: False |
| - `include_for_metrics`: [] |
| - `eval_do_concat_batches`: True |
| - `fp16_backend`: auto |
| - `push_to_hub_model_id`: None |
| - `push_to_hub_organization`: None |
| - `mp_parameters`: |
| - `auto_find_batch_size`: False |
| - `full_determinism`: False |
| - `torchdynamo`: None |
| - `ray_scope`: last |
| - `ddp_timeout`: 1800 |
| - `torch_compile`: False |
| - `torch_compile_backend`: None |
| - `torch_compile_mode`: None |
| - `include_tokens_per_second`: False |
| - `include_num_input_tokens_seen`: False |
| - `neftune_noise_alpha`: None |
| - `optim_target_modules`: None |
| - `batch_eval_metrics`: False |
| - `eval_on_start`: False |
| - `use_liger_kernel`: False |
| - `liger_kernel_config`: None |
| - `eval_use_gather_object`: False |
| - `average_tokens_across_devices`: False |
| - `prompts`: None |
| - `batch_sampler`: batch_sampler |
| - `multi_dataset_batch_sampler`: proportional |
| - `router_mapping`: {} |
| - `learning_rate_mapping`: {} |
|
|
| </details> |
|
|
| ### Framework Versions |
| - Python: 3.12.3 |
| - Sentence Transformers: 5.1.1 |
| - PyLate: 1.4.0 |
| - Transformers: 4.56.2 |
| - PyTorch: 2.9.0+cu128 |
| - Accelerate: 1.13.0 |
| - Datasets: 4.8.4 |
| - Tokenizers: 0.22.2 |
|
|
|
|
| ## Citation |
|
|
| ### BibTeX |
|
|
| #### Sentence Transformers |
| ```bibtex |
| @inproceedings{reimers-2019-sentence-bert, |
| title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
| author = "Reimers, Nils and Gurevych, Iryna", |
| booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
| month = "11", |
| year = "2019", |
| publisher = "Association for Computational Linguistics", |
| url = "https://arxiv.org/abs/1908.10084" |
| } |
| ``` |
|
|
| #### PyLate |
| ```bibtex |
| @inproceedings{DBLP:conf/cikm/ChaffinS25, |
| author = {Antoine Chaffin and |
| Rapha{"{e}}l Sourty}, |
| editor = {Meeyoung Cha and |
| Chanyoung Park and |
| Noseong Park and |
| Carl Yang and |
| Senjuti Basu Roy and |
| Jessie Li and |
| Jaap Kamps and |
| Kijung Shin and |
| Bryan Hooi and |
| Lifang He}, |
| title = {PyLate: Flexible Training and Retrieval for Late Interaction Models}, |
| booktitle = {Proceedings of the 34th {ACM} International Conference on Information |
| and Knowledge Management, {CIKM} 2025, Seoul, Republic of Korea, November |
| 10-14, 2025}, |
| pages = {6334--6339}, |
| publisher = {{ACM}}, |
| year = {2025}, |
| url = {https://github.com/lightonai/pylate}, |
| doi = {10.1145/3746252.3761608}, |
| } |
| ``` |
|
|
| <!-- |
| ## Glossary |
|
|
| *Clearly define terms in order to be accessible across audiences.* |
| --> |
|
|
| <!-- |
| ## Model Card Authors |
|
|
| *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| --> |
|
|
| <!-- |
| ## Model Card Contact |
|
|
| *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| --> |
