| | --- |
| | tags: |
| | - sentence-transformers |
| | - code |
| | - code-retrieval |
| | - retrieval-augmented-generation |
| | - rag |
| | - python |
| | - java |
| | - go |
| | - php |
| | - sentence-similarity |
| | - feature-extraction |
| | - generated_from_trainer |
| | - loss:MultipleNegativesRankingLoss |
| | widget: |
| | - source_sentence: |- |
| | search_query: Finds the top long, short, and absolute positions. |
| | |
| | Parameters |
| | ---------- |
| | positions : pd.DataFrame |
| | The positions that the strategy takes over time. |
| | top : int, optional |
| | How many of each to find (default 10). |
| |
|
| | Returns |
| | ------- |
| | df_top_long : pd.DataFrame |
| | Top long positions. |
| | df_top_short : pd.DataFrame |
| | Top short positions. |
| | df_top_abs : pd.DataFrame |
| | Top absolute positions. |
| | sentences: |
| | - >- |
| | search_document: def symmetric_ema(xolds, yolds, low=None, high=None, n=512, |
| | decay_steps=1., low_counts_threshold=1e-8): |
| | ''' |
| | perform symmetric EMA (exponential moving average) |
| | smoothing and resampling to an even grid with n points. |
| | Does not do extrapolation, so we assume |
| | xolds[0] <= low && high <= xolds[-1] |
| | |
| | Arguments: |
| |
|
| | xolds: array or list - x values of data. Needs to be sorted in ascending order |
| | yolds: array of list - y values of data. Has to have the same length as xolds |
| |
|
| | low: float - min value of the new x grid. By default equals to xolds[0] |
| | high: float - max value of the new x grid. By default equals to xolds[-1] |
| |
|
| | n: int - number of points in new x grid |
| |
|
| | decay_steps: float - EMA decay factor, expressed in new x grid steps. |
| |
|
| | low_counts_threshold: float or int |
| | - y values with counts less than this value will be set to NaN |
| |
|
| | Returns: |
| | tuple sum_ys, count_ys where |
| | xs - array with new x grid |
| | ys - array of EMA of y at each point of the new x grid |
| | count_ys - array of EMA of y counts at each point of the new x grid |
| |
|
| | ''' |
| | xs, ys1, count_ys1 = one_sided_ema(xolds, yolds, low, high, n, decay_steps, low_counts_threshold=0) |
| | _, ys2, count_ys2 = one_sided_ema(-xolds[::-1], yolds[::-1], -high, -low, n, decay_steps, low_counts_threshold=0) |
| | ys2 = ys2[::-1] |
| | count_ys2 = count_ys2[::-1] |
| | count_ys = count_ys1 + count_ys2 |
| | ys = (ys1 * count_ys1 + ys2 * count_ys2) / count_ys |
| | ys[count_ys < low_counts_threshold] = np.nan |
| | return xs, ys, count_ys |
| | - |- |
| | search_document: def project(self, from_shape, to_shape): |
| | """ |
| | Project the polygon onto an image with different shape. |
| | |
| | The relative coordinates of all points remain the same. |
| | E.g. a point at (x=20, y=20) on an image (width=100, height=200) will be |
| | projected on a new image (width=200, height=100) to (x=40, y=10). |
| | |
| | This is intended for cases where the original image is resized. |
| | It cannot be used for more complex changes (e.g. padding, cropping). |
| | |
| | Parameters |
| | ---------- |
| | from_shape : tuple of int |
| | Shape of the original image. (Before resize.) |
| | |
| | to_shape : tuple of int |
| | Shape of the new image. (After resize.) |
| | |
| | Returns |
| | ------- |
| | imgaug.Polygon |
| | Polygon object with new coordinates. |
| | |
| | """ |
| | if from_shape[0:2] == to_shape[0:2]: |
| | return self.copy() |
| | ls_proj = self.to_line_string(closed=False).project( |
| | from_shape, to_shape) |
| | return self.copy(exterior=ls_proj.coords) |
| | - |- |
| | search_document: def get_top_long_short_abs(positions, top=10): |
| | """ |
| | Finds the top long, short, and absolute positions. |
| | |
| | Parameters |
| | ---------- |
| | positions : pd.DataFrame |
| | The positions that the strategy takes over time. |
| | top : int, optional |
| | How many of each to find (default 10). |
| | |
| | Returns |
| | ------- |
| | df_top_long : pd.DataFrame |
| | Top long positions. |
| | df_top_short : pd.DataFrame |
| | Top short positions. |
| | df_top_abs : pd.DataFrame |
| | Top absolute positions. |
| | """ |
| | |
| | positions = positions.drop('cash', axis='columns') |
| | df_max = positions.max() |
| | df_min = positions.min() |
| | df_abs_max = positions.abs().max() |
| | df_top_long = df_max[df_max > 0].nlargest(top) |
| | df_top_short = df_min[df_min < 0].nsmallest(top) |
| | df_top_abs = df_abs_max.nlargest(top) |
| | return df_top_long, df_top_short, df_top_abs |
| | - source_sentence: |- |
| | search_query: Draw text on an image. |
| | |
| | This uses by default DejaVuSans as its font, which is included in this library. |
| |
|
| | dtype support:: |
| |
|
| | * ``uint8``: yes; fully tested |
| | * ``uint16``: no |
| | * ``uint32``: no |
| | * ``uint64``: no |
| | * ``int8``: no |
| | * ``int16``: no |
| | * ``int32``: no |
| | * ``int64``: no |
| | * ``float16``: no |
| | * ``float32``: yes; not tested |
| | * ``float64``: no |
| | * ``float128``: no |
| | * ``bool``: no |
| |
|
| | TODO check if other dtypes could be enabled |
| |
|
| | Parameters |
| | ---------- |
| | img : (H,W,3) ndarray |
| | The image array to draw text on. |
| | Expected to be of dtype uint8 or float32 (value range 0.0 to 255.0). |
| |
|
| | y : int |
| | x-coordinate of the top left corner of the text. |
| |
|
| | x : int |
| | y- coordinate of the top left corner of the text. |
| |
|
| | text : str |
| | The text to draw. |
| |
|
| | color : iterable of int, optional |
| | Color of the text to draw. For RGB-images this is expected to be an RGB color. |
| |
|
| | size : int, optional |
| | Font size of the text to draw. |
| |
|
| | Returns |
| | ------- |
| | img_np : (H,W,3) ndarray |
| | Input image with text drawn on it. |
| | sentences: |
| | - >- |
| | search_document: def cross_entropy_seq_with_mask(logits, target_seqs, |
| | input_mask, return_details=False, name=None): |
| | """Returns the expression of cross-entropy of two sequences, implement |
| | softmax internally. Normally be used for Dynamic RNN with Synced sequence input and output. |
| | |
| | Parameters |
| | ----------- |
| | logits : Tensor |
| | 2D tensor with shape of [batch_size * ?, n_classes], `?` means dynamic IDs for each example. |
| | - Can be get from `DynamicRNNLayer` by setting ``return_seq_2d`` to `True`. |
| | target_seqs : Tensor |
| | int of tensor, like word ID. [batch_size, ?], `?` means dynamic IDs for each example. |
| | input_mask : Tensor |
| | The mask to compute loss, it has the same size with `target_seqs`, normally 0 or 1. |
| | return_details : boolean |
| | Whether to return detailed losses. |
| | - If False (default), only returns the loss. |
| | - If True, returns the loss, losses, weights and targets (see source code). |
| |
|
| | Examples |
| | -------- |
| | >>> batch_size = 64 |
| | >>> vocab_size = 10000 |
| | >>> embedding_size = 256 |
| | >>> input_seqs = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="input") |
| | >>> target_seqs = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="target") |
| | >>> input_mask = tf.placeholder(dtype=tf.int64, shape=[batch_size, None], name="mask") |
| | >>> net = tl.layers.EmbeddingInputlayer( |
| | ... inputs = input_seqs, |
| | ... vocabulary_size = vocab_size, |
| | ... embedding_size = embedding_size, |
| | ... name = 'seq_embedding') |
| | >>> net = tl.layers.DynamicRNNLayer(net, |
| | ... cell_fn = tf.contrib.rnn.BasicLSTMCell, |
| | ... n_hidden = embedding_size, |
| | ... dropout = (0.7 if is_train else None), |
| | ... sequence_length = tl.layers.retrieve_seq_length_op2(input_seqs), |
| | ... return_seq_2d = True, |
| | ... name = 'dynamicrnn') |
| | >>> print(net.outputs) |
| | (?, 256) |
| | >>> net = tl.layers.DenseLayer(net, n_units=vocab_size, name="output") |
| | >>> print(net.outputs) |
| | (?, 10000) |
| | >>> loss = tl.cost.cross_entropy_seq_with_mask(net.outputs, target_seqs, input_mask) |
| |
|
| | """ |
| | targets = tf.reshape(target_seqs, [-1]) # to one vector |
| | weights = tf.to_float(tf.reshape(input_mask, [-1])) # to one vector like targets |
| | losses = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=targets, name=name) * weights |
| | # losses = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=targets, name=name)) # for TF1.0 and others |
| | |
| | loss = tf.divide( |
| | tf.reduce_sum(losses), # loss from mask. reduce_sum before element-wise mul with mask !! |
| | tf.reduce_sum(weights), |
| | name="seq_loss_with_mask" |
| | ) |
| |
|
| | if return_details: |
| | return loss, losses, weights, targets |
| | else: |
| | return loss |
| | - |- |
| | search_document: def pickle_load(path, compression=False): |
| | """Unpickle a possible compressed pickle. |
| | |
| | Parameters |
| | ---------- |
| | path: str |
| | path to the output file |
| | compression: bool |
| | if true assumes that pickle was compressed when created and attempts decompression. |
| |
|
| | Returns |
| | ------- |
| | obj: object |
| | the unpickled object |
| | """ |
| | |
| | if compression: |
| | with zipfile.ZipFile(path, "r", compression=zipfile.ZIP_DEFLATED) as myzip: |
| | with myzip.open("data") as f: |
| | return pickle.load(f) |
| | else: |
| | with open(path, "rb") as f: |
| | return pickle.load(f) |
| | - |- |
| | search_document: def draw_text(img, y, x, text, color=(0, 255, 0), size=25): |
| | """ |
| | Draw text on an image. |
| | |
| | This uses by default DejaVuSans as its font, which is included in this library. |
| |
|
| | dtype support:: |
| |
|
| | * ``uint8``: yes; fully tested |
| | * ``uint16``: no |
| | * ``uint32``: no |
| | * ``uint64``: no |
| | * ``int8``: no |
| | * ``int16``: no |
| | * ``int32``: no |
| | * ``int64``: no |
| | * ``float16``: no |
| | * ``float32``: yes; not tested |
| | * ``float64``: no |
| | * ``float128``: no |
| | * ``bool``: no |
| |
|
| | TODO check if other dtypes could be enabled |
| |
|
| | Parameters |
| | ---------- |
| | img : (H,W,3) ndarray |
| | The image array to draw text on. |
| | Expected to be of dtype uint8 or float32 (value range 0.0 to 255.0). |
| |
|
| | y : int |
| | x-coordinate of the top left corner of the text. |
| |
|
| | x : int |
| | y- coordinate of the top left corner of the text. |
| |
|
| | text : str |
| | The text to draw. |
| |
|
| | color : iterable of int, optional |
| | Color of the text to draw. For RGB-images this is expected to be an RGB color. |
| |
|
| | size : int, optional |
| | Font size of the text to draw. |
| |
|
| | Returns |
| | ------- |
| | img_np : (H,W,3) ndarray |
| | Input image with text drawn on it. |
| |
|
| | """ |
| | do_assert(img.dtype in [np.uint8, np.float32]) |
| | |
| | input_dtype = img.dtype |
| | if img.dtype == np.float32: |
| | img = img.astype(np.uint8) |
| | |
| | img = PIL_Image.fromarray(img) |
| | font = PIL_ImageFont.truetype(DEFAULT_FONT_FP, size) |
| | context = PIL_ImageDraw.Draw(img) |
| | context.text((x, y), text, fill=tuple(color), font=font) |
| | img_np = np.asarray(img) |
| | |
| | # PIL/asarray returns read only array |
| | if not img_np.flags["WRITEABLE"]: |
| | try: |
| | |
| | img_np.setflags(write=True) |
| | except ValueError as ex: |
| | if "cannot set WRITEABLE flag to True of this array" in str(ex): |
| | img_np = np.copy(img_np) |
| |
|
| | if img_np.dtype != input_dtype: |
| | img_np = img_np.astype(input_dtype) |
| |
|
| | return img_np |
| | - source_sentence: >- |
| | search_query: Choice and return an an action by given the action probability |
| | distribution. |
| | |
| | Parameters |
| | ------------ |
| | probs : list of float. |
| | The probability distribution of all actions. |
| | action_list : None or a list of int or others |
| | A list of action in integer, string or others. If None, returns an integer range between 0 and len(probs)-1. |
| |
|
| | Returns |
| | -------- |
| | float int or str |
| | The chosen action. |
| |
|
| | Examples |
| | ---------- |
| | >>> for _ in range(5): |
| | >>> a = choice_action_by_probs([0.2, 0.4, 0.4]) |
| | >>> print(a) |
| | 0 |
| | 1 |
| | 1 |
| | 2 |
| | 1 |
| | >>> for _ in range(3): |
| | >>> a = choice_action_by_probs([0.5, 0.5], ['a', 'b']) |
| | >>> print(a) |
| | a |
| | b |
| | b |
| | sentences: |
| | - >- |
| | search_document: def from_keypoint_image(image, if_not_found_coords={"x": |
| | -1, "y": -1}, threshold=1, nb_channels=None): # pylint: |
| | disable=locally-disabled, dangerous-default-value, line-too-long |
| | """ |
| | Converts an image generated by ``to_keypoint_image()`` back to a KeypointsOnImage object. |
| | |
| | Parameters |
| | ---------- |
| | image : (H,W,N) ndarray |
| | The keypoints image. N is the number of keypoints. |
| |
|
| | if_not_found_coords : tuple or list or dict or None, optional |
| | Coordinates to use for keypoints that cannot be found in `image`. |
| | If this is a list/tuple, it must have two integer values. |
| | If it is a dictionary, it must have the keys ``x`` and ``y`` with |
| | each containing one integer value. |
| | If this is None, then the keypoint will not be added to the final |
| | KeypointsOnImage object. |
| |
|
| | threshold : int, optional |
| | The search for keypoints works by searching for the argmax in |
| | each channel. This parameters contains the minimum value that |
| | the max must have in order to be viewed as a keypoint. |
| |
|
| | nb_channels : None or int, optional |
| | Number of channels of the image on which the keypoints are placed. |
| | Some keypoint augmenters require that information. |
| | If set to None, the keypoint's shape will be set |
| | to ``(height, width)``, otherwise ``(height, width, nb_channels)``. |
| |
|
| | Returns |
| | ------- |
| | out : KeypointsOnImage |
| | The extracted keypoints. |
| |
|
| | """ |
| | ia.do_assert(len(image.shape) == 3) |
| | height, width, nb_keypoints = image.shape |
| | |
| | drop_if_not_found = False |
| | if if_not_found_coords is None: |
| | drop_if_not_found = True |
| | if_not_found_x = -1 |
| | if_not_found_y = -1 |
| | elif isinstance(if_not_found_coords, (tuple, list)): |
| | ia.do_assert(len(if_not_found_coords) == 2) |
| | if_not_found_x = if_not_found_coords[0] |
| | if_not_found_y = if_not_found_coords[1] |
| | elif isinstance(if_not_found_coords, dict): |
| | if_not_found_x = if_not_found_coords["x"] |
| | if_not_found_y = if_not_found_coords["y"] |
| | else: |
| | raise Exception("Expected if_not_found_coords to be None or tuple or list or dict, got %s." % ( |
| | type(if_not_found_coords),)) |
| |
|
| | keypoints = [] |
| | for i in sm.xrange(nb_keypoints): |
| | maxidx_flat = np.argmax(image[..., i]) |
| | maxidx_ndim = np.unravel_index(maxidx_flat, (height, width)) |
| | found = (image[maxidx_ndim[0], maxidx_ndim[1], i] >= threshold) |
| | if found: |
| | keypoints.append(Keypoint(x=maxidx_ndim[1], y=maxidx_ndim[0])) |
| | else: |
| | if drop_if_not_found: |
| | pass |
| | else: |
| | keypoints.append(Keypoint(x=if_not_found_x, y=if_not_found_y)) |
| |
|
| | out_shape = (height, width) |
| | if nb_channels is not None: |
| | out_shape += (nb_channels,) |
| | return KeypointsOnImage(keypoints, shape=out_shape) |
| | - >- |
| | search_document: def choice_action_by_probs(probs=(0.5, 0.5), |
| | action_list=None): |
| | """Choice and return an an action by given the action probability distribution. |
| | |
| | Parameters |
| | ------------ |
| | probs : list of float. |
| | The probability distribution of all actions. |
| | action_list : None or a list of int or others |
| | A list of action in integer, string or others. If None, returns an integer range between 0 and len(probs)-1. |
| |
|
| | Returns |
| | -------- |
| | float int or str |
| | The chosen action. |
| |
|
| | Examples |
| | ---------- |
| | >>> for _ in range(5): |
| | >>> a = choice_action_by_probs([0.2, 0.4, 0.4]) |
| | >>> print(a) |
| | 0 |
| | 1 |
| | 1 |
| | 2 |
| | 1 |
| | >>> for _ in range(3): |
| | >>> a = choice_action_by_probs([0.5, 0.5], ['a', 'b']) |
| | >>> print(a) |
| | a |
| | b |
| | b |
| |
|
| | """ |
| | if action_list is None: |
| | n_action = len(probs) |
| | action_list = np.arange(n_action) |
| | else: |
| | if len(action_list) != len(probs): |
| | raise Exception("number of actions should equal to number of probabilities.") |
| | return np.random.choice(action_list, p=probs) |
| | - |- |
| | search_document: def __validateExperimentControl(self, control): |
| | """ Validates control dictionary for the experiment context""" |
| | # Validate task list |
| | taskList = control.get('tasks', None) |
| | if taskList is not None: |
| | taskLabelsList = [] |
| | |
| | for task in taskList: |
| | validateOpfJsonValue(task, "opfTaskSchema.json") |
| | validateOpfJsonValue(task['taskControl'], "opfTaskControlSchema.json") |
| |
|
| | taskLabel = task['taskLabel'] |
| |
|
| | assert isinstance(taskLabel, types.StringTypes), \ |
| | "taskLabel type: %r" % type(taskLabel) |
| | assert len(taskLabel) > 0, "empty string taskLabel not is allowed" |
| |
|
| | taskLabelsList.append(taskLabel.lower()) |
| |
|
| | taskLabelDuplicates = filter(lambda x: taskLabelsList.count(x) > 1, |
| | taskLabelsList) |
| | assert len(taskLabelDuplicates) == 0, \ |
| | "Duplcate task labels are not allowed: %s" % taskLabelDuplicates |
| |
|
| | return |
| | - source_sentence: |- |
| | search_query: Augment endlessly images in the source queue. |
| | |
| | This is a worker function for that endlessly queries the source queue (input batches), |
| | augments batches in it and sends the result to the output queue. |
| | sentences: |
| | - >- |
| | search_document: def _augment_images_worker(self, augseq, queue_source, |
| | queue_result, seedval): |
| | """ |
| | Augment endlessly images in the source queue. |
| | |
| | This is a worker function for that endlessly queries the source queue (input batches), |
| | augments batches in it and sends the result to the output queue. |
| |
|
| | """ |
| | np.random.seed(seedval) |
| | random.seed(seedval) |
| | augseq.reseed(seedval) |
| | ia.seed(seedval) |
| | |
| | loader_finished = False |
| | |
| | while not loader_finished: |
| | # wait for a new batch in the source queue and load it |
| | try: |
| | batch_str = queue_source.get(timeout=0.1) |
| | batch = pickle.loads(batch_str) |
| | if batch is None: |
| | loader_finished = True |
| | # put it back in so that other workers know that the loading queue is finished |
| | queue_source.put(pickle.dumps(None, protocol=-1)) |
| | else: |
| | batch_aug = augseq.augment_batch(batch) |
| | |
| | # send augmented batch to output queue |
| | batch_str = pickle.dumps(batch_aug, protocol=-1) |
| | queue_result.put(batch_str) |
| | except QueueEmpty: |
| | time.sleep(0.01) |
| | |
| | queue_result.put(pickle.dumps(None, protocol=-1)) |
| | time.sleep(0.01) |
| | - |- |
| | search_document: def show_perf_attrib_stats(returns, |
| | positions, |
| | factor_returns, |
| | factor_loadings, |
| | transactions=None, |
| | pos_in_dollars=True): |
| | """ |
| | Calls `perf_attrib` using inputs, and displays outputs using |
| | `utils.print_table`. |
| | """ |
| | risk_exposures, perf_attrib_data = perf_attrib( |
| | returns, |
| | positions, |
| | factor_returns, |
| | factor_loadings, |
| | transactions, |
| | pos_in_dollars=pos_in_dollars, |
| | ) |
| | |
| | perf_attrib_stats, risk_exposure_stats =\ |
| | create_perf_attrib_stats(perf_attrib_data, risk_exposures) |
| | |
| | percentage_formatter = '{:.2%}'.format |
| | float_formatter = '{:.2f}'.format |
| | |
| | summary_stats = perf_attrib_stats.loc[['Annualized Specific Return', |
| | 'Annualized Common Return', |
| | 'Annualized Total Return', |
| | 'Specific Sharpe Ratio']] |
| | |
| | # Format return rows in summary stats table as percentages. |
| | for col_name in ( |
| | 'Annualized Specific Return', |
| | 'Annualized Common Return', |
| | 'Annualized Total Return', |
| | ): |
| | summary_stats[col_name] = percentage_formatter(summary_stats[col_name]) |
| | |
| | # Display sharpe to two decimal places. |
| | summary_stats['Specific Sharpe Ratio'] = float_formatter( |
| | summary_stats['Specific Sharpe Ratio'] |
| | ) |
| | |
| | print_table(summary_stats, name='Summary Statistics') |
| | |
| | print_table( |
| | risk_exposure_stats, |
| | name='Exposures Summary', |
| | # In exposures table, format exposure column to 2 decimal places, and |
| | # return columns as percentages. |
| | formatters={ |
| | 'Average Risk Factor Exposure': float_formatter, |
| | 'Annualized Return': percentage_formatter, |
| | 'Cumulative Return': percentage_formatter, |
| | }, |
| | ) |
| | - >- |
| | search_document: def binary_cross_entropy(output, target, epsilon=1e-8, |
| | name='bce_loss'): |
| | """Binary cross entropy operation. |
| |
|
| | Parameters |
| | ---------- |
| | output : Tensor |
| | Tensor with type of `float32` or `float64`. |
| | target : Tensor |
| | The target distribution, format the same with `output`. |
| | epsilon : float |
| | A small value to avoid output to be zero. |
| | name : str |
| | An optional name to attach to this function. |
| |
|
| | References |
| | ----------- |
| | - `ericjang-DRAW <https://github.com/ericjang/draw/blob/master/draw.py#L73>`__ |
| |
|
| | """ |
| | # with ops.op_scope([output, target], name, "bce_loss") as name: |
| | |
| | |
| |
|
| | |
| | return tf.reduce_mean( |
| | tf.reduce_sum(-(target * tf.log(output + epsilon) + (1. - target) * tf.log(1. - output + epsilon)), axis=1), |
| | name=name |
| | ) |
| | - source_sentence: 'search_query: episode_batch: array(batch_size x (T or T+1) x dim_key)' |
| | sentences: |
| | - |- |
| | search_document: def get_txn_vol(transactions): |
| | """ |
| | Extract daily transaction data from set of transaction objects. |
| | |
| | Parameters |
| | ---------- |
| | transactions : pd.DataFrame |
| | Time series containing one row per symbol (and potentially |
| | duplicate datetime indices) and columns for amount and |
| | price. |
| |
|
| | Returns |
| | ------- |
| | pd.DataFrame |
| | Daily transaction volume and number of shares. |
| | - See full explanation in tears.create_full_tear_sheet. |
| | """ |
| | |
| | txn_norm = transactions.copy() |
| | txn_norm.index = txn_norm.index.normalize() |
| | amounts = txn_norm.amount.abs() |
| | prices = txn_norm.price |
| | values = amounts * prices |
| | daily_amounts = amounts.groupby(amounts.index).sum() |
| | daily_values = values.groupby(values.index).sum() |
| | daily_amounts.name = "txn_shares" |
| | daily_values.name = "txn_volume" |
| | return pd.concat([daily_values, daily_amounts], axis=1) |
| | - |- |
| | search_document: def deepcopy(self, exterior=None, label=None): |
| | """ |
| | Create a deep copy of the Polygon object. |
| | |
| | Parameters |
| | ---------- |
| | exterior : list of Keypoint or list of tuple or (N,2) ndarray, optional |
| | List of points defining the polygon. See `imgaug.Polygon.__init__` for details. |
| |
|
| | label : None or str |
| | If not None, then the label of the copied object will be set to this value. |
| |
|
| | Returns |
| | ------- |
| | imgaug.Polygon |
| | Deep copy. |
| |
|
| | """ |
| | return Polygon( |
| | exterior=np.copy(self.exterior) if exterior is None else exterior, |
| | label=self.label if label is None else label |
| | ) |
| | - |- |
| | search_document: def store_episode(self, episode_batch): |
| | """episode_batch: array(batch_size x (T or T+1) x dim_key) |
| | """ |
| | batch_sizes = [len(episode_batch[key]) for key in episode_batch.keys()] |
| | assert np.all(np.array(batch_sizes) == batch_sizes[0]) |
| | batch_size = batch_sizes[0] |
| | |
| | with self.lock: |
| | idxs = self._get_storage_idx(batch_size) |
| | |
| | # load inputs into buffers |
| | for key in self.buffers.keys(): |
| | self.buffers[key][idxs] = episode_batch[key] |
| | |
| | self.n_transitions_stored += batch_size * self.T |
| | pipeline_tag: sentence-similarity |
| | library_name: sentence-transformers |
| | metrics: |
| | - cosine_accuracy@1 |
| | - cosine_accuracy@3 |
| | - cosine_accuracy@5 |
| | - cosine_accuracy@10 |
| | - cosine_precision@1 |
| | - cosine_precision@3 |
| | - cosine_precision@5 |
| | - cosine_precision@10 |
| | - cosine_recall@1 |
| | - cosine_recall@3 |
| | - cosine_recall@5 |
| | - cosine_recall@10 |
| | - cosine_ndcg@10 |
| | - cosine_mrr@10 |
| | - cosine_map@100 |
| | model-index: |
| | - name: SentenceTransformer |
| | results: |
| | - task: |
| | type: information-retrieval |
| | name: Information Retrieval |
| | dataset: |
| | name: codesearchnet val |
| | type: codesearchnet_val |
| | metrics: |
| | - type: cosine_accuracy@1 |
| | value: 0.8926 |
| | name: Cosine Accuracy@1 |
| | - type: cosine_accuracy@3 |
| | value: 0.9453666666666667 |
| | name: Cosine Accuracy@3 |
| | - type: cosine_accuracy@5 |
| | value: 0.9545 |
| | name: Cosine Accuracy@5 |
| | - type: cosine_accuracy@10 |
| | value: 0.9637666666666667 |
| | name: Cosine Accuracy@10 |
| | - type: cosine_precision@1 |
| | value: 0.8926 |
| | name: Cosine Precision@1 |
| | - type: cosine_precision@3 |
| | value: 0.31512222222222214 |
| | name: Cosine Precision@3 |
| | - type: cosine_precision@5 |
| | value: 0.19090000000000004 |
| | name: Cosine Precision@5 |
| | - type: cosine_precision@10 |
| | value: 0.09637666666666667 |
| | name: Cosine Precision@10 |
| | - type: cosine_recall@1 |
| | value: 0.8926 |
| | name: Cosine Recall@1 |
| | - type: cosine_recall@3 |
| | value: 0.9453666666666667 |
| | name: Cosine Recall@3 |
| | - type: cosine_recall@5 |
| | value: 0.9545 |
| | name: Cosine Recall@5 |
| | - type: cosine_recall@10 |
| | value: 0.9637666666666667 |
| | name: Cosine Recall@10 |
| | - type: cosine_ndcg@10 |
| | value: 0.9313201256618757 |
| | name: Cosine Ndcg@10 |
| | - type: cosine_mrr@10 |
| | value: 0.9206047883597835 |
| | name: Cosine Mrr@10 |
| | - type: cosine_map@100 |
| | value: 0.9212040995599341 |
| | name: Cosine Map@100 |
| | license: mit |
| | datasets: |
| | - code-search-net/code_search_net |
| | language: |
| | - en |
| | base_model: |
| | - answerdotai/ModernBERT-base |
| | --- |
| | |
| | # SentenceTransformer |
| |
|
| | This is a [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model trained on the [code_search_net](https://huggingface.co/datasets/code-search-net/code_search_net) dataset with |
| | [<code>MultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#multiplenegativesrankingloss) with in-batch negatives. Model can be used for code retrieval and reranking. |
| |
|
| | ## Perfomance on code retrieval benchmarks |
| |
|
| | **RTEB** |
| |
|
| | On 14.10.2025 the model is **6th** on RTEB leaderbord among models with <500M parameters: |
| | <details> |
| | <summary>Click</summary> |
| | <figure> |
| | <img src="Rteb_top.jpg"> |
| | </figure> |
| | </details> |
| | |
| | Perfomance per task: |
| | | Model | AppsRetrieval | Code1Retrieval (Private) | DS1000Retrieval | FreshStackRetrieval | HumanEvalRetrieval | JapaneseCode1Retrieval (Private)| MBPPRetrieval | WikiSQLRetrieval | |
| | |-------|---------------|----------------|-----------------|---------------------|--------------------|------------------------|---------------|------------------| |
| | | english_code_retriever | 8.04 | 75.36 | 32.42 | 18.30 | 71.82 | 46.59 | 72.06 | 87.92 | |
| |
|
| | **COIR**: |
| | | Model | AppsRetrieval | COIRCodeSearchNetRetrieval | CodeFeedbackMT | CodeFeedbackST | CodeSearchNetCCRetrieval | CodeTransOceanContest | CodeTransOceanDL | CosQA | StackOverflowQA | SyntheticText2SQL | |
| | |-------|---------------|----------------------------|----------------|----------------|--------------------------|------------------------|------------------|-------|------------------|-------------------| |
| | | english_code_retriever | 8.04 | 74.23 | 44.01 | 57.79 | 42.71 | 60.68 | 35.16 | 25.56 | 56.53 | 42.79 | |
| |
|
| | more information you cand find [in MTEB leaderbord](https://huggingface.co/spaces/mteb/leaderboard) |
| |
|
| |
|
| | ## Model Details |
| |
|
| | ### Model Description |
| | - **Model Type:** Sentence Transformer |
| | - **Maximum Sequence Length:** 8192 tokens |
| | - **Output Dimensionality:** 768 |
| | - **Similarity Function:** Cosine Similarity |
| | - Mean pooling |
| |
|
| | ## Usage |
| |
|
| | Using is easy with Sentence Transformers. |
| |
|
| | Pay attention that model was trained with prefixes 'search_query' for queries and 'search_document' for docs with code. |
| | So using with prefixes will improve model retrieving abilities. |
| |
|
| | ```python |
| | import torch |
| | from sentence_transformers import SentenceTransformer, util |
| | |
| | device = "cuda" if torch.cuda.is_available() else "cpu" |
| | model = SentenceTransformer("fyaronskiy/english_code_retriever").to(device) |
| | |
| | queries = [ |
| | "Write a Python function that calculates the factorial of a number recursively.", |
| | "How to check if a given string reads the same backward and forward?", |
| | "Combine two sorted lists into a single sorted list." |
| | ] |
| | |
| | corpus = [ |
| | # Relevant for Q1 |
| | """def factorial(n): |
| | if n == 0: |
| | return 1 |
| | return n * factorial(n-1)""", |
| | |
| | # Hard negative for Q1 (similar structure but computes sum) |
| | """def sum_recursive(n): |
| | if n == 0: |
| | return 0 |
| | return n + sum_recursive(n-1)""", |
| | |
| | # Relevant for Q2 |
| | """def is_palindrome(s: str) -> bool: |
| | s = s.lower().replace(" ", "") |
| | return s == s[::-1]""", |
| | |
| | # Hard negative for Q2 (string reverse but not palindrome check) |
| | """def reverse_string(s: str) -> str: |
| | return s[::-1]""", |
| | |
| | # Relevant for Q3 |
| | """def merge_sorted_lists(a, b): |
| | result = [] |
| | i = j = 0 |
| | while i < len(a) and j < len(b): |
| | if a[i] < b[j]: |
| | result.append(a[i]) |
| | i += 1 |
| | else: |
| | result.append(b[j]) |
| | j += 1 |
| | result.extend(a[i:]) |
| | result.extend(b[j:]) |
| | return result""", |
| | |
| | # Hard negative for Q3 (similar iteration but sums two lists elementwise) |
| | """def add_lists(a, b): |
| | return [x + y for x, y in zip(a, b)]""" |
| | ] |
| | |
| | |
| | doc_embeddings = model.encode(corpus, prompt_name='search_query', convert_to_tensor=True, device=device) |
| | query_embeddings = model.encode(queries, prompt_name='search_document', convert_to_tensor=True, device=device) |
| | |
| | # Compute cosine similarity and retrieve top-1 |
| | for i, query in enumerate(queries): |
| | scores = util.cos_sim(query_embeddings[i], doc_embeddings)[0] |
| | best_idx = torch.argmax(scores).item() |
| | print(f"\n Query {i+1}: {query}") |
| | print(f"Top-1 match (score={scores[best_idx]:.4f}):\n{corpus[best_idx]}") |
| | |
| | ''' Query 1: Write a Python function that calculates the factorial of a number recursively. |
| | Top-1 match (score=0.5983): |
| | def factorial(n): |
| | if n == 0: |
| | return 1 |
| | return n * factorial(n-1) |
| | |
| | Query 2: How to check if a given string reads the same backward and forward? |
| | Top-1 match (score=0.4925): |
| | def is_palindrome(s: str) -> bool: |
| | s = s.lower().replace(" ", "") |
| | return s == s[::-1] |
| | |
| | Query 3: Combine two sorted lists into a single sorted list. |
| | Top-1 match (score=0.6524): |
| | def merge_sorted_lists(a, b): |
| | result = [] |
| | i = j = 0 |
| | while i < len(a) and j < len(b): |
| | if a[i] < b[j]: |
| | result.append(a[i]) |
| | i += 1 |
| | else: |
| | result.append(b[j]) |
| | j += 1 |
| | result.extend(a[i:]) |
| | result.extend(b[j:]) |
| | return result |
| | ''' |
| | ``` |
| |
|
| | Using with Transformers |
| | ```python |
| | import torch |
| | from transformers import AutoTokenizer, AutoModel |
| | |
| | device = "cuda" if torch.cuda.is_available() else "cpu" |
| | |
| | model_name = "fyaronskiy/english_code_retriever" |
| | tokenizer = AutoTokenizer.from_pretrained(model_name) |
| | model = AutoModel.from_pretrained(model_name).to(device) |
| | model.eval() |
| | |
| | |
| | |
| | queries = [ |
| | "function of addition of two numbers", |
| | "finding the maximum element in an array", |
| | "sorting a list in ascending order" |
| | ] |
| | |
| | corpus = [ |
| | "def add(a, b): return a + b", |
| | "def find_max(arr): return max(arr)", |
| | "def sort_list(lst): return sorted(lst)" |
| | ] |
| | |
| | def mean_pooling(model_output, attention_mask): |
| | token_embeddings = model_output[0] # (batch_size, seq_len, hidden_dim) |
| | input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float() |
| | return (token_embeddings * input_mask_expanded).sum(1) / input_mask_expanded.sum(1).clamp(min=1e-9) |
| | |
| | def encode_texts(texts): |
| | encoded = tokenizer( |
| | texts, |
| | padding=True, |
| | truncation=True, |
| | return_tensors="pt", |
| | max_length=8192 |
| | ).to(device) |
| | with torch.no_grad(): |
| | model_output = model(**encoded) |
| | return mean_pooling(model_output, encoded["attention_mask"]) |
| | |
| | doc_embeddings = encode_texts(["search_document: " + document for document in corpus]) |
| | query_embeddings = encode_texts(["search_query: " + query for query in queries]) |
| | |
| | # Normalize embeddings for cosine similarity |
| | doc_embeddings = torch.nn.functional.normalize(doc_embeddings, p=2, dim=1) |
| | query_embeddings = torch.nn.functional.normalize(query_embeddings, p=2, dim=1) |
| | |
| | # Compute cosine similarity and retrieve top-1 |
| | for i, query in enumerate(queries): |
| | scores = torch.matmul(query_embeddings[i], doc_embeddings.T) |
| | best_idx = torch.argmax(scores).item() |
| | print(f"\n Query {i+1}: {query}") |
| | print(f"Top-1 match (score={scores[best_idx]:.4f}):\n{corpus[best_idx]}") |
| | |
| | ''' Query 1: function of addition of two numbers |
| | Top-1 match (score=0.6047): |
| | def add(a, b): return a + b |
| | |
| | Query 2: finding the maximum element in an array |
| | Top-1 match (score=0.7772): |
| | def find_max(arr): return max(arr) |
| | |
| | Query 3: sorting a list in ascending order |
| | Top-1 match (score=0.7389): |
| | def sort_list(lst): return sorted(lst) |
| | ''' |
| | ``` |
| |
|
| | <!-- |
| | ### Downstream Usage (Sentence Transformers) |
| |
|
| | You can finetune this model on your own dataset. |
| |
|
| | <details><summary>Click to expand</summary> |
| |
|
| | </details> |
| | --> |
| |
|
| | <!-- |
| | ### Out-of-Scope Use |
| |
|
| | *List how the model may foreseeably be misused and address what users ought not to do with the model.* |
| | --> |
| |
|
| | ## Evaluation |
| |
|
| | ### Metrics |
| |
|
| | #### Information Retrieval |
| |
|
| | * Dataset: validation part of `codesearchnet_val` |
| | * Size: 30,000 evaluation samples |
| |
|
| | * Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
| |
|
| | | Metric | Value | |
| | |:--------------------|:-----------| |
| | | cosine_accuracy@1 | 0.8926 | |
| | | cosine_accuracy@3 | 0.9454 | |
| | | cosine_accuracy@5 | 0.9545 | |
| | | cosine_accuracy@10 | 0.9638 | |
| | | cosine_precision@1 | 0.8926 | |
| | | cosine_precision@3 | 0.3151 | |
| | | cosine_precision@5 | 0.1909 | |
| | | cosine_precision@10 | 0.0964 | |
| | | cosine_recall@1 | 0.8926 | |
| | | cosine_recall@3 | 0.9454 | |
| | | cosine_recall@5 | 0.9545 | |
| | | cosine_recall@10 | 0.9638 | |
| | | **cosine_ndcg@10** | **0.9313** | |
| | | cosine_mrr@10 | 0.9206 | |
| | | cosine_map@100 | 0.9212 | |
| | |
| | <!-- |
| | ## Bias, Risks and Limitations |
| | |
| | *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
| | --> |
| | |
| | <!-- |
| | ### Recommendations |
| | |
| | *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
| | --> |
| | |
| | ## Training Details |
| | |
| | ### Training Dataset |
| | |
| | #### code_search_net |
| | |
| | * Dataset: train part of code_search_net |
| | * Size: 1,880,853 training samples |
| | * queries - function docstrings in english, relevant document - code of function |
| | * negatives was sampled from batch |
| | * Distribution of programming languages: |
| | * |
| | |
| |  |
| | |
| | |
| | ### Training Hyperparameters |
| | #### Non-Default Hyperparameters |
| | |
| | - `batch_size`: 64 |
| | - `learning_rate`: 2e-05 |
| | - `num_epochs`: 2 |
| | - `warmup_ratio`: 0.1 |
| | |
| | |
| | |
| | ### Framework Versions |
| | - Python: 3.10.11 |
| | - Sentence Transformers: 5.1.0 |
| | - Transformers: 4.52.3 |
| | - PyTorch: 2.6.0+cu124 |
| | - Accelerate: 1.10.0 |
| | - Datasets: 3.6.0 |
| | - Tokenizers: 0.21.4 |
| | |
| | |
| | <!-- |
| | ## Glossary |
| | |
| | *Clearly define terms in order to be accessible across audiences.* |
| | --> |
| | |
| | <!-- |
| | ## Model Card Authors |
| | |
| | *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
| | --> |
| | |
| | <!-- |
| | ## Model Card Contact |
| | |
| | *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
| | --> |
