--- language: - ru - en tags: - sentence-transformers - sentence-similarity - feature-extraction - dense - generated_from_trainer - dataset_size:7211755 - loss:MatryoshkaLoss - loss:CachedMultipleNegativesRankingLoss - loss:CoSENTLoss widget: - source_sentence: Returns the number of parameters in the network. sentences: - |- Удаляет журнал обучения. Параметры ----------- kwargs : информация для логирования Находит элементы для удаления, оставьте пустым, чтобы удалить все логи. Примеры --------- Сохранение журнала обучения >>> db.save_training_log(accuracy=0.33) >>> db.save_training_log(accuracy=0.44) Удаление логов, соответствующих требованиям >>> db.delete_training_log(accuracy=0.33) Удаление всех логов >>> db.delete_training_log() - |- Создайте глубокую копию объекта Polygon. Параметры ---------- exterior : список Keypoint или список кортежей или (N,2) ndarray, необязательный Список точек, определяющих полигон. См. `imgaug.Polygon.__init__` для деталей. label : None или str Если не None, то метка скопированного объекта будет установлена в этом значении. Возвращает ------- imgaug.Polygon Глубокая копия. - Возвращает количество параметров в сети. - source_sentence: |- Plots total amount of stocks with an active position, either short or long. Displays daily total, daily average per month, and all-time daily average. Parameters ---------- returns : pd.Series Daily returns of the strategy, noncumulative. - See full explanation in tears.create_full_tear_sheet. positions : pd.DataFrame, optional Daily net position values. - See full explanation in tears.create_full_tear_sheet. legend_loc : matplotlib.loc, optional The location of the legend on the plot. ax : matplotlib.Axes, optional Axes upon which to plot. **kwargs, optional Passed to plotting function. Returns ------- ax : matplotlib.Axes The axes that were plotted on. sentences: - >- Графики накопленных скользящих возвратов по сравнению с некоторыми бенчмарками. Возвраты бэктеста отображаются зеленым цветом, а возвраты за период вне выборки (живое трейдинг) — красным цветом. Дополнительно может быть добавлен не параметрический конусный график в область возвратов вне выборки. Параметры ---------- returns : pd.Series Ежедневные возвраты стратегии, накапливаемые. - Полное объяснение см. в tears.create_full_tear_sheet. factor_returns : pd.Series, необязательный Ежедневные ненакапливаемые возвраты бенчмарка, к которому вычисляются беты. Обычно это бенчмарк, например, возвраты рынка. - Этот параметр имеет тот же стиль, что и returns. live_start_date : datetime, необязательный Дата, когда стратегия начала торговлю в режиме реального времени, после периода бэктеста. Эта дата должна быть нормализована. logy : bool, необязательный Включает ли логарифмический масштаб оси Y. cone_std : float или кортеж, необязательный Если float, стандартное отклонение для конусных графиков. Если кортеж, кортеж значений стандартного отклонения для конусных графиков - Подробнее см. timeseries.forecast_cone_bounds. legend_loc : matplotlib.loc, необязательный Расположение легенды на графике. volatility_match : bool, необязательный Нормализует ли волатильность возвратов к волатильности бенчмарка. Это помогает сравнивать стратегии с разной волатильностью. Требуется передача benchmark_rets. cone_function : функция, необязательная Функция, используемая для генерации прогнозного вероятностного конуса. Подпись функции должна соответствовать следующему формату: def cone(in_sample_returns (pd.Series), days_to_project_forward (int), cone_std= (float, или кортеж), starting_value= (int, или float)) Пример см. в timeseries.forecast_cone_bootstrap. ax : matplotlib.Axes, необязательный Оси, на которых будет выполнен график. **kwargs, необязательный Передается в функцию отрисовки. Возвращаемое значение ------- ax : matplotlib.Axes Оси, на которых был выполнен график. - >- Обработка партии данных с помощью заданной функции с использованием многопоточности. Обычно используется для увеличения данных. Параметры ----------- data : numpy.array или другие типы Данные, которые нужно обработать. thread_count : int Количество потоков для использования. fn : function Функция для обработки данных. more args : аргументы для `fn` См. Примеры ниже. Примеры -------- Обработка изображений. >>> images, _, _, _ = tl.files.load_cifar10_dataset(shape=(-1, 32, 32, 3)) >>> images = tl.prepro.threading_data(images[0:32], tl.prepro.zoom, zoom_range=[0.5, 1]) Настроенная функция предварительной обработки изображений. >>> def distort_img(x): >>> x = tl.prepro.flip_axis(x, axis=0, is_random=True) >>> x = tl.prepro.flip_axis(x, axis=1, is_random=True) >>> x = tl.prepro.crop(x, 100, 100, is_random=True) >>> return x >>> images = tl.prepro.threading_data(images, distort_img) Обработка изображений и масок вместе (обычно используется для задач изображений сегментации). >>> X, Y --> [batch_size, row, col, 1] >>> data = tl.prepro.threading_data([_ for _ in zip(X, Y)], tl.prepro.zoom_multi, zoom_range=[0.5, 1], is_random=True) data --> [batch_size, 2, row, col, 1] >>> X_, Y_ = data.transpose((1,0,2,3,4)) X_, Y_ --> [batch_size, row, col, 1] >>> tl.vis.save_image(X_, 'images.png') >>> tl.vis.save_image(Y_, 'masks.png') Обработка изображений и масок вместе с использованием ``thread_count``. >>> X, Y --> [batch_size, row, col, 1] >>> data = tl.prepro.threading_data(X, tl.prepro.zoom_multi, 8, zoom_range=[0.5, 1], is_random=True) data --> [batch_size, 2, row, col, 1] >>> X_, Y_ = data.transpose((1,0,2,3,4)) X_, Y_ --> [batch_size, row, col, 1] >>> tl.vis.save_image(X_, 'after.png') >>> tl.vis.save_image(Y_, 'before.png') Настроенная функция для обработки изображений и масок вместе. >>> def distort_img(data): >>> x, y = data >>> x, y = tl.prepro.flip_axis_multi([x, y], axis=0, is_random=True) >>> x, y = tl.prepro.flip_axis_multi([x, y], axis=1, is_random=True) >>> x, y = tl.prepro.crop_multi([x, y], 100, 100, is_random=True) >>> return x, y >>> X, Y --> [batch_size, row, col, channel] >>> data = tl.prepro.threading_data([_ for _ in zip(X, Y)], distort_img) >>> X_, Y_ = data.transpose((1,0,2,3,4)) Возвращает ------- list или numpyarray Обработанные результаты. Ссылки ---------- - `python queue `__ - `run with limited queue `__ - >- Графики общего объема акций с активным позиционированием, либо коротким, либо длинным. Отображает общий объем ежедневно, среднее ежемесячное значение и среднее значение за все время. Параметры ---------- returns : pd.Series Ежедневные возвраты стратегии, не накапливаемые. - Полное объяснение см. в tears.create_full_tear_sheet. positions : pd.DataFrame, опционально Ежедневные значения чистых позиций. - Полное объяснение см. в tears.create_full_tear_sheet. legend_loc : matplotlib.loc, опционально Расположение легенды на графике. ax : matplotlib.Axes, опционально Оси, на которых будет выполнен график. **kwargs, опционально Передается в функцию построения графика. Возвращает ------- ax : matplotlib.Axes Оси, на которых был выполнен график. - source_sentence: >- T.set_default(k[,d]) -> T.get(k,d), также устанавливает T[k]=d, если k не в T sentences: - |- def compute_volume_exposures(shares_held, volumes, percentile): """ Returns arrays of pth percentile of long, short and gross volume exposures of an algorithm's held shares Parameters ---------- shares_held : pd.DataFrame Daily number of shares held by an algorithm. - See full explanation in create_risk_tear_sheet volume : pd.DataFrame Daily volume per asset - See full explanation in create_risk_tear_sheet percentile : float Percentile to use when computing and plotting volume exposures - See full explanation in create_risk_tear_sheet """ shares_held = shares_held.replace(0, np.nan) shares_longed = shares_held[shares_held > 0] shares_shorted = -1 * shares_held[shares_held < 0] shares_grossed = shares_held.abs() longed_frac = shares_longed.divide(volumes) shorted_frac = shares_shorted.divide(volumes) grossed_frac = shares_grossed.divide(volumes) # NOTE: To work around a bug in `quantile` with nan-handling in # pandas 0.18, use np.nanpercentile by applying to each row of # the dataframe. This is fixed in pandas 0.19. # # longed_threshold = 100*longed_frac.quantile(percentile, axis='columns') # shorted_threshold = 100*shorted_frac.quantile(percentile, axis='columns') # grossed_threshold = 100*grossed_frac.quantile(percentile, axis='columns') longed_threshold = 100 * longed_frac.apply( partial(np.nanpercentile, q=100 * percentile), axis='columns', ) shorted_threshold = 100 * shorted_frac.apply( partial(np.nanpercentile, q=100 * percentile), axis='columns', ) grossed_threshold = 100 * grossed_frac.apply( partial(np.nanpercentile, q=100 * percentile), axis='columns', ) return longed_threshold, shorted_threshold, grossed_threshold - |- def set_default(self, key, default=None): """T.set_default(k[,d]) -> T.get(k,d), also set T[k]=d if k not in T""" try: return self.get_value(key) except KeyError: self.insert(key, default) return default - |- def find_intersections_with(self, other): """ Find all intersection points between the line string and `other`. Parameters ---------- other : tuple of number or list of tuple of number or \ list of LineString or LineString The other geometry to use during intersection tests. Returns ------- list of list of tuple of number All intersection points. One list per pair of consecutive start and end point, i.e. `N-1` lists of `N` points. Each list may be empty or may contain multiple points. """ import shapely.geometry geom = _convert_var_to_shapely_geometry(other) result = [] for p_start, p_end in zip(self.coords[:-1], self.coords[1:]): ls = shapely.geometry.LineString([p_start, p_end]) intersections = ls.intersection(geom) intersections = list(_flatten_shapely_collection(intersections)) intersections_points = [] for inter in intersections: if isinstance(inter, shapely.geometry.linestring.LineString): inter_start = (inter.coords[0][0], inter.coords[0][1]) inter_end = (inter.coords[-1][0], inter.coords[-1][1]) intersections_points.extend([inter_start, inter_end]) else: assert isinstance(inter, shapely.geometry.point.Point), ( "Expected to find shapely.geometry.point.Point or " "shapely.geometry.linestring.LineString intersection, " "actually found %s." % (type(inter),)) intersections_points.append((inter.x, inter.y)) # sort by distance to start point, this makes it later on easier # to remove duplicate points inter_sorted = sorted( intersections_points, key=lambda p: np.linalg.norm(np.float32(p) - p_start) ) result.append(inter_sorted) return result - source_sentence: |- Обертка для _log_counter_per_token. Аргументы: token: Токен, для которого нужно найти количество. Возвращает: Количество раз, когда эта функция вызывалась с *token* в качестве аргумента (начинается с 0) sentences: - |- def _GetNextLogCountPerToken(token): """Wrapper for _log_counter_per_token. Args: token: The token for which to look up the count. Returns: The number of times this function has been called with *token* as an argument (starting at 0) """ global _log_counter_per_token # pylint: disable=global-variable-not-assigned _log_counter_per_token[token] = 1 + _log_counter_per_token.get(token, -1) return _log_counter_per_token[token] - |- def remove_out_of_image(self, fully=True, partly=False): """ Remove all bounding boxes that are fully or partially outside of the image. Parameters ---------- fully : bool, optional Whether to remove bounding boxes that are fully outside of the image. partly : bool, optional Whether to remove bounding boxes that are partially outside of the image. Returns ------- imgaug.BoundingBoxesOnImage Reduced set of bounding boxes, with those that were fully/partially outside of the image removed. """ bbs_clean = [bb for bb in self.bounding_boxes if not bb.is_out_of_image(self.shape, fully=fully, partly=partly)] return BoundingBoxesOnImage(bbs_clean, shape=self.shape) - |- def noise4d(self, x, y, z, w): """ Generate 4D OpenSimplex noise from X,Y,Z,W coordinates. """ # Place input coordinates on simplectic honeycomb. stretch_offset = (x + y + z + w) * STRETCH_CONSTANT_4D xs = x + stretch_offset ys = y + stretch_offset zs = z + stretch_offset ws = w + stretch_offset # Floor to get simplectic honeycomb coordinates of rhombo-hypercube super-cell origin. xsb = floor(xs) ysb = floor(ys) zsb = floor(zs) wsb = floor(ws) # Skew out to get actual coordinates of stretched rhombo-hypercube origin. We'll need these later. squish_offset = (xsb + ysb + zsb + wsb) * SQUISH_CONSTANT_4D xb = xsb + squish_offset yb = ysb + squish_offset zb = zsb + squish_offset wb = wsb + squish_offset # Compute simplectic honeycomb coordinates relative to rhombo-hypercube origin. xins = xs - xsb yins = ys - ysb zins = zs - zsb wins = ws - wsb # Sum those together to get a value that determines which region we're in. in_sum = xins + yins + zins + wins # Positions relative to origin po. dx0 = x - xb dy0 = y - yb dz0 = z - zb dw0 = w - wb value = 0 extrapolate = self._extrapolate4d if in_sum <= 1: # We're inside the pentachoron (4-Simplex) at (0,0,0,0) # Determine which two of (0,0,0,1), (0,0,1,0), (0,1,0,0), (1,0,0,0) are closest. a_po = 0x01 a_score = xins b_po = 0x02 b_score = yins if a_score >= b_score and zins > b_score: b_score = zins b_po = 0x04 elif a_score < b_score and zins > a_score: a_score = zins a_po = 0x04 if a_score >= b_score and wins > b_score: b_score = wins b_po = 0x08 elif a_score < b_score and wins > a_score: a_score = wins a_po = 0x08 # Now we determine the three lattice pos not part of the pentachoron that may contribute. # This depends on the closest two pentachoron vertices, including (0,0,0,0) uins = 1 - in_sum if uins > a_score or uins > b_score: # (0,0,0,0) is one of the closest two pentachoron vertices. c = b_po if (b_score > a_score) else a_po # Our other closest vertex is the closest out of a and b. if (c & 0x01) == 0: xsv_ext0 = xsb - 1 xsv_ext1 = xsv_ext2 = xsb dx_ext0 = dx0 + 1 dx_ext1 = dx_ext2 = dx0 else: xsv_ext0 = xsv_ext1 = xsv_ext2 = xsb + 1 dx_ext0 = dx_ext1 = dx_ext2 = dx0 - 1 if (c & 0x02) == 0: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb dy_ext0 = dy_ext1 = dy_ext2 = dy0 if (c & 0x01) == 0x01: ysv_ext0 -= 1 dy_ext0 += 1 else: ysv_ext1 -= 1 dy_ext1 += 1 else: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb + 1 dy_ext0 = dy_ext1 = dy_ext2 = dy0 - 1 if (c & 0x04) == 0: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb dz_ext0 = dz_ext1 = dz_ext2 = dz0 if (c & 0x03) != 0: if (c & 0x03) == 0x03: zsv_ext0 -= 1 dz_ext0 += 1 else: zsv_ext1 -= 1 dz_ext1 += 1 else: zsv_ext2 -= 1 dz_ext2 += 1 else: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb + 1 dz_ext0 = dz_ext1 = dz_ext2 = dz0 - 1 if (c & 0x08) == 0: wsv_ext0 = wsv_ext1 = wsb wsv_ext2 = wsb - 1 dw_ext0 = dw_ext1 = dw0 dw_ext2 = dw0 + 1 else: wsv_ext0 = wsv_ext1 = wsv_ext2 = wsb + 1 dw_ext0 = dw_ext1 = dw_ext2 = dw0 - 1 else: # (0,0,0,0) is not one of the closest two pentachoron vertices. c = (a_po | b_po) # Our three extra vertices are determined by the closest two. if (c & 0x01) == 0: xsv_ext0 = xsv_ext2 = xsb xsv_ext1 = xsb - 1 dx_ext0 = dx0 - 2 * SQUISH_CONSTANT_4D dx_ext1 = dx0 + 1 - SQUISH_CONSTANT_4D dx_ext2 = dx0 - SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsv_ext2 = xsb + 1 dx_ext0 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dx_ext1 = dx_ext2 = dx0 - 1 - SQUISH_CONSTANT_4D if (c & 0x02) == 0: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb dy_ext0 = dy0 - 2 * SQUISH_CONSTANT_4D dy_ext1 = dy_ext2 = dy0 - SQUISH_CONSTANT_4D if (c & 0x01) == 0x01: ysv_ext1 -= 1 dy_ext1 += 1 else: ysv_ext2 -= 1 dy_ext2 += 1 else: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb + 1 dy_ext0 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dy_ext1 = dy_ext2 = dy0 - 1 - SQUISH_CONSTANT_4D if (c & 0x04) == 0: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb dz_ext0 = dz0 - 2 * SQUISH_CONSTANT_4D dz_ext1 = dz_ext2 = dz0 - SQUISH_CONSTANT_4D if (c & 0x03) == 0x03: zsv_ext1 -= 1 dz_ext1 += 1 else: zsv_ext2 -= 1 dz_ext2 += 1 else: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb + 1 dz_ext0 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dz_ext1 = dz_ext2 = dz0 - 1 - SQUISH_CONSTANT_4D if (c & 0x08) == 0: wsv_ext0 = wsv_ext1 = wsb wsv_ext2 = wsb - 1 dw_ext0 = dw0 - 2 * SQUISH_CONSTANT_4D dw_ext1 = dw0 - SQUISH_CONSTANT_4D dw_ext2 = dw0 + 1 - SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsv_ext2 = wsb + 1 dw_ext0 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D dw_ext1 = dw_ext2 = dw0 - 1 - SQUISH_CONSTANT_4D # Contribution (0,0,0,0) attn0 = 2 - dx0 * dx0 - dy0 * dy0 - dz0 * dz0 - dw0 * dw0 if attn0 > 0: attn0 *= attn0 value += attn0 * attn0 * extrapolate(xsb + 0, ysb + 0, zsb + 0, wsb + 0, dx0, dy0, dz0, dw0) # Contribution (1,0,0,0) dx1 = dx0 - 1 - SQUISH_CONSTANT_4D dy1 = dy0 - 0 - SQUISH_CONSTANT_4D dz1 = dz0 - 0 - SQUISH_CONSTANT_4D dw1 = dw0 - 0 - SQUISH_CONSTANT_4D attn1 = 2 - dx1 * dx1 - dy1 * dy1 - dz1 * dz1 - dw1 * dw1 if attn1 > 0: attn1 *= attn1 value += attn1 * attn1 * extrapolate(xsb + 1, ysb + 0, zsb + 0, wsb + 0, dx1, dy1, dz1, dw1) # Contribution (0,1,0,0) dx2 = dx0 - 0 - SQUISH_CONSTANT_4D dy2 = dy0 - 1 - SQUISH_CONSTANT_4D dz2 = dz1 dw2 = dw1 attn2 = 2 - dx2 * dx2 - dy2 * dy2 - dz2 * dz2 - dw2 * dw2 if attn2 > 0: attn2 *= attn2 value += attn2 * attn2 * extrapolate(xsb + 0, ysb + 1, zsb + 0, wsb + 0, dx2, dy2, dz2, dw2) # Contribution (0,0,1,0) dx3 = dx2 dy3 = dy1 dz3 = dz0 - 1 - SQUISH_CONSTANT_4D dw3 = dw1 attn3 = 2 - dx3 * dx3 - dy3 * dy3 - dz3 * dz3 - dw3 * dw3 if attn3 > 0: attn3 *= attn3 value += attn3 * attn3 * extrapolate(xsb + 0, ysb + 0, zsb + 1, wsb + 0, dx3, dy3, dz3, dw3) # Contribution (0,0,0,1) dx4 = dx2 dy4 = dy1 dz4 = dz1 dw4 = dw0 - 1 - SQUISH_CONSTANT_4D attn4 = 2 - dx4 * dx4 - dy4 * dy4 - dz4 * dz4 - dw4 * dw4 if attn4 > 0: attn4 *= attn4 value += attn4 * attn4 * extrapolate(xsb + 0, ysb + 0, zsb + 0, wsb + 1, dx4, dy4, dz4, dw4) elif in_sum >= 3: # We're inside the pentachoron (4-Simplex) at (1,1,1,1) # Determine which two of (1,1,1,0), (1,1,0,1), (1,0,1,1), (0,1,1,1) are closest. a_po = 0x0E a_score = xins b_po = 0x0D b_score = yins if a_score <= b_score and zins < b_score: b_score = zins b_po = 0x0B elif a_score > b_score and zins < a_score: a_score = zins a_po = 0x0B if a_score <= b_score and wins < b_score: b_score = wins b_po = 0x07 elif a_score > b_score and wins < a_score: a_score = wins a_po = 0x07 # Now we determine the three lattice pos not part of the pentachoron that may contribute. # This depends on the closest two pentachoron vertices, including (0,0,0,0) uins = 4 - in_sum if uins < a_score or uins < b_score: # (1,1,1,1) is one of the closest two pentachoron vertices. c = b_po if (b_score < a_score) else a_po # Our other closest vertex is the closest out of a and b. if (c & 0x01) != 0: xsv_ext0 = xsb + 2 xsv_ext1 = xsv_ext2 = xsb + 1 dx_ext0 = dx0 - 2 - 4 * SQUISH_CONSTANT_4D dx_ext1 = dx_ext2 = dx0 - 1 - 4 * SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsv_ext2 = xsb dx_ext0 = dx_ext1 = dx_ext2 = dx0 - 4 * SQUISH_CONSTANT_4D if (c & 0x02) != 0: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb + 1 dy_ext0 = dy_ext1 = dy_ext2 = dy0 - 1 - 4 * SQUISH_CONSTANT_4D if (c & 0x01) != 0: ysv_ext1 += 1 dy_ext1 -= 1 else: ysv_ext0 += 1 dy_ext0 -= 1 else: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb dy_ext0 = dy_ext1 = dy_ext2 = dy0 - 4 * SQUISH_CONSTANT_4D if (c & 0x04) != 0: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb + 1 dz_ext0 = dz_ext1 = dz_ext2 = dz0 - 1 - 4 * SQUISH_CONSTANT_4D if (c & 0x03) != 0x03: if (c & 0x03) == 0: zsv_ext0 += 1 dz_ext0 -= 1 else: zsv_ext1 += 1 dz_ext1 -= 1 else: zsv_ext2 += 1 dz_ext2 -= 1 else: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb dz_ext0 = dz_ext1 = dz_ext2 = dz0 - 4 * SQUISH_CONSTANT_4D if (c & 0x08) != 0: wsv_ext0 = wsv_ext1 = wsb + 1 wsv_ext2 = wsb + 2 dw_ext0 = dw_ext1 = dw0 - 1 - 4 * SQUISH_CONSTANT_4D dw_ext2 = dw0 - 2 - 4 * SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsv_ext2 = wsb dw_ext0 = dw_ext1 = dw_ext2 = dw0 - 4 * SQUISH_CONSTANT_4D else: # (1,1,1,1) is not one of the closest two pentachoron vertices. c = (a_po & b_po) # Our three extra vertices are determined by the closest two. if (c & 0x01) != 0: xsv_ext0 = xsv_ext2 = xsb + 1 xsv_ext1 = xsb + 2 dx_ext0 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dx_ext1 = dx0 - 2 - 3 * SQUISH_CONSTANT_4D dx_ext2 = dx0 - 1 - 3 * SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsv_ext2 = xsb dx_ext0 = dx0 - 2 * SQUISH_CONSTANT_4D dx_ext1 = dx_ext2 = dx0 - 3 * SQUISH_CONSTANT_4D if (c & 0x02) != 0: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb + 1 dy_ext0 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dy_ext1 = dy_ext2 = dy0 - 1 - 3 * SQUISH_CONSTANT_4D if (c & 0x01) != 0: ysv_ext2 += 1 dy_ext2 -= 1 else: ysv_ext1 += 1 dy_ext1 -= 1 else: ysv_ext0 = ysv_ext1 = ysv_ext2 = ysb dy_ext0 = dy0 - 2 * SQUISH_CONSTANT_4D dy_ext1 = dy_ext2 = dy0 - 3 * SQUISH_CONSTANT_4D if (c & 0x04) != 0: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb + 1 dz_ext0 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dz_ext1 = dz_ext2 = dz0 - 1 - 3 * SQUISH_CONSTANT_4D if (c & 0x03) != 0: zsv_ext2 += 1 dz_ext2 -= 1 else: zsv_ext1 += 1 dz_ext1 -= 1 else: zsv_ext0 = zsv_ext1 = zsv_ext2 = zsb dz_ext0 = dz0 - 2 * SQUISH_CONSTANT_4D dz_ext1 = dz_ext2 = dz0 - 3 * SQUISH_CONSTANT_4D if (c & 0x08) != 0: wsv_ext0 = wsv_ext1 = wsb + 1 wsv_ext2 = wsb + 2 dw_ext0 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D dw_ext1 = dw0 - 1 - 3 * SQUISH_CONSTANT_4D dw_ext2 = dw0 - 2 - 3 * SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsv_ext2 = wsb dw_ext0 = dw0 - 2 * SQUISH_CONSTANT_4D dw_ext1 = dw_ext2 = dw0 - 3 * SQUISH_CONSTANT_4D # Contribution (1,1,1,0) dx4 = dx0 - 1 - 3 * SQUISH_CONSTANT_4D dy4 = dy0 - 1 - 3 * SQUISH_CONSTANT_4D dz4 = dz0 - 1 - 3 * SQUISH_CONSTANT_4D dw4 = dw0 - 3 * SQUISH_CONSTANT_4D attn4 = 2 - dx4 * dx4 - dy4 * dy4 - dz4 * dz4 - dw4 * dw4 if attn4 > 0: attn4 *= attn4 value += attn4 * attn4 * extrapolate(xsb + 1, ysb + 1, zsb + 1, wsb + 0, dx4, dy4, dz4, dw4) # Contribution (1,1,0,1) dx3 = dx4 dy3 = dy4 dz3 = dz0 - 3 * SQUISH_CONSTANT_4D dw3 = dw0 - 1 - 3 * SQUISH_CONSTANT_4D attn3 = 2 - dx3 * dx3 - dy3 * dy3 - dz3 * dz3 - dw3 * dw3 if attn3 > 0: attn3 *= attn3 value += attn3 * attn3 * extrapolate(xsb + 1, ysb + 1, zsb + 0, wsb + 1, dx3, dy3, dz3, dw3) # Contribution (1,0,1,1) dx2 = dx4 dy2 = dy0 - 3 * SQUISH_CONSTANT_4D dz2 = dz4 dw2 = dw3 attn2 = 2 - dx2 * dx2 - dy2 * dy2 - dz2 * dz2 - dw2 * dw2 if attn2 > 0: attn2 *= attn2 value += attn2 * attn2 * extrapolate(xsb + 1, ysb + 0, zsb + 1, wsb + 1, dx2, dy2, dz2, dw2) # Contribution (0,1,1,1) dx1 = dx0 - 3 * SQUISH_CONSTANT_4D dz1 = dz4 dy1 = dy4 dw1 = dw3 attn1 = 2 - dx1 * dx1 - dy1 * dy1 - dz1 * dz1 - dw1 * dw1 if attn1 > 0: attn1 *= attn1 value += attn1 * attn1 * extrapolate(xsb + 0, ysb + 1, zsb + 1, wsb + 1, dx1, dy1, dz1, dw1) # Contribution (1,1,1,1) dx0 = dx0 - 1 - 4 * SQUISH_CONSTANT_4D dy0 = dy0 - 1 - 4 * SQUISH_CONSTANT_4D dz0 = dz0 - 1 - 4 * SQUISH_CONSTANT_4D dw0 = dw0 - 1 - 4 * SQUISH_CONSTANT_4D attn0 = 2 - dx0 * dx0 - dy0 * dy0 - dz0 * dz0 - dw0 * dw0 if attn0 > 0: attn0 *= attn0 value += attn0 * attn0 * extrapolate(xsb + 1, ysb + 1, zsb + 1, wsb + 1, dx0, dy0, dz0, dw0) elif in_sum <= 2: # We're inside the first dispentachoron (Rectified 4-Simplex) a_is_bigger_side = True b_is_bigger_side = True # Decide between (1,1,0,0) and (0,0,1,1) if xins + yins > zins + wins: a_score = xins + yins a_po = 0x03 else: a_score = zins + wins a_po = 0x0C # Decide between (1,0,1,0) and (0,1,0,1) if xins + zins > yins + wins: b_score = xins + zins b_po = 0x05 else: b_score = yins + wins b_po = 0x0A # Closer between (1,0,0,1) and (0,1,1,0) will replace the further of a and b, if closer. if xins + wins > yins + zins: score = xins + wins if a_score >= b_score and score > b_score: b_score = score b_po = 0x09 elif a_score < b_score and score > a_score: a_score = score a_po = 0x09 else: score = yins + zins if a_score >= b_score and score > b_score: b_score = score b_po = 0x06 elif a_score < b_score and score > a_score: a_score = score a_po = 0x06 # Decide if (1,0,0,0) is closer. p1 = 2 - in_sum + xins if a_score >= b_score and p1 > b_score: b_score = p1 b_po = 0x01 b_is_bigger_side = False elif a_score < b_score and p1 > a_score: a_score = p1 a_po = 0x01 a_is_bigger_side = False # Decide if (0,1,0,0) is closer. p2 = 2 - in_sum + yins if a_score >= b_score and p2 > b_score: b_score = p2 b_po = 0x02 b_is_bigger_side = False elif a_score < b_score and p2 > a_score: a_score = p2 a_po = 0x02 a_is_bigger_side = False # Decide if (0,0,1,0) is closer. p3 = 2 - in_sum + zins if a_score >= b_score and p3 > b_score: b_score = p3 b_po = 0x04 b_is_bigger_side = False elif a_score < b_score and p3 > a_score: a_score = p3 a_po = 0x04 a_is_bigger_side = False # Decide if (0,0,0,1) is closer. p4 = 2 - in_sum + wins if a_score >= b_score and p4 > b_score: b_po = 0x08 b_is_bigger_side = False elif a_score < b_score and p4 > a_score: a_po = 0x08 a_is_bigger_side = False # Where each of the two closest pos are determines how the extra three vertices are calculated. if a_is_bigger_side == b_is_bigger_side: if a_is_bigger_side: # Both closest pos on the bigger side c1 = (a_po | b_po) c2 = (a_po & b_po) if (c1 & 0x01) == 0: xsv_ext0 = xsb xsv_ext1 = xsb - 1 dx_ext0 = dx0 - 3 * SQUISH_CONSTANT_4D dx_ext1 = dx0 + 1 - 2 * SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsb + 1 dx_ext0 = dx0 - 1 - 3 * SQUISH_CONSTANT_4D dx_ext1 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D if (c1 & 0x02) == 0: ysv_ext0 = ysb ysv_ext1 = ysb - 1 dy_ext0 = dy0 - 3 * SQUISH_CONSTANT_4D dy_ext1 = dy0 + 1 - 2 * SQUISH_CONSTANT_4D else: ysv_ext0 = ysv_ext1 = ysb + 1 dy_ext0 = dy0 - 1 - 3 * SQUISH_CONSTANT_4D dy_ext1 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D if (c1 & 0x04) == 0: zsv_ext0 = zsb zsv_ext1 = zsb - 1 dz_ext0 = dz0 - 3 * SQUISH_CONSTANT_4D dz_ext1 = dz0 + 1 - 2 * SQUISH_CONSTANT_4D else: zsv_ext0 = zsv_ext1 = zsb + 1 dz_ext0 = dz0 - 1 - 3 * SQUISH_CONSTANT_4D dz_ext1 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D if (c1 & 0x08) == 0: wsv_ext0 = wsb wsv_ext1 = wsb - 1 dw_ext0 = dw0 - 3 * SQUISH_CONSTANT_4D dw_ext1 = dw0 + 1 - 2 * SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsb + 1 dw_ext0 = dw0 - 1 - 3 * SQUISH_CONSTANT_4D dw_ext1 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D # One combination is a _permutation of (0,0,0,2) based on c2 xsv_ext2 = xsb ysv_ext2 = ysb zsv_ext2 = zsb wsv_ext2 = wsb dx_ext2 = dx0 - 2 * SQUISH_CONSTANT_4D dy_ext2 = dy0 - 2 * SQUISH_CONSTANT_4D dz_ext2 = dz0 - 2 * SQUISH_CONSTANT_4D dw_ext2 = dw0 - 2 * SQUISH_CONSTANT_4D if (c2 & 0x01) != 0: xsv_ext2 += 2 dx_ext2 -= 2 elif (c2 & 0x02) != 0: ysv_ext2 += 2 dy_ext2 -= 2 elif (c2 & 0x04) != 0: zsv_ext2 += 2 dz_ext2 -= 2 else: wsv_ext2 += 2 dw_ext2 -= 2 else: # Both closest pos on the smaller side # One of the two extra pos is (0,0,0,0) xsv_ext2 = xsb ysv_ext2 = ysb zsv_ext2 = zsb wsv_ext2 = wsb dx_ext2 = dx0 dy_ext2 = dy0 dz_ext2 = dz0 dw_ext2 = dw0 # Other two pos are based on the omitted axes. c = (a_po | b_po) if (c & 0x01) == 0: xsv_ext0 = xsb - 1 xsv_ext1 = xsb dx_ext0 = dx0 + 1 - SQUISH_CONSTANT_4D dx_ext1 = dx0 - SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsb + 1 dx_ext0 = dx_ext1 = dx0 - 1 - SQUISH_CONSTANT_4D if (c & 0x02) == 0: ysv_ext0 = ysv_ext1 = ysb dy_ext0 = dy_ext1 = dy0 - SQUISH_CONSTANT_4D if (c & 0x01) == 0x01: ysv_ext0 -= 1 dy_ext0 += 1 else: ysv_ext1 -= 1 dy_ext1 += 1 else: ysv_ext0 = ysv_ext1 = ysb + 1 dy_ext0 = dy_ext1 = dy0 - 1 - SQUISH_CONSTANT_4D if (c & 0x04) == 0: zsv_ext0 = zsv_ext1 = zsb dz_ext0 = dz_ext1 = dz0 - SQUISH_CONSTANT_4D if (c & 0x03) == 0x03: zsv_ext0 -= 1 dz_ext0 += 1 else: zsv_ext1 -= 1 dz_ext1 += 1 else: zsv_ext0 = zsv_ext1 = zsb + 1 dz_ext0 = dz_ext1 = dz0 - 1 - SQUISH_CONSTANT_4D if (c & 0x08) == 0: wsv_ext0 = wsb wsv_ext1 = wsb - 1 dw_ext0 = dw0 - SQUISH_CONSTANT_4D dw_ext1 = dw0 + 1 - SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsb + 1 dw_ext0 = dw_ext1 = dw0 - 1 - SQUISH_CONSTANT_4D else: # One po on each "side" if a_is_bigger_side: c1 = a_po c2 = b_po else: c1 = b_po c2 = a_po # Two contributions are the bigger-sided po with each 0 replaced with -1. if (c1 & 0x01) == 0: xsv_ext0 = xsb - 1 xsv_ext1 = xsb dx_ext0 = dx0 + 1 - SQUISH_CONSTANT_4D dx_ext1 = dx0 - SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsb + 1 dx_ext0 = dx_ext1 = dx0 - 1 - SQUISH_CONSTANT_4D if (c1 & 0x02) == 0: ysv_ext0 = ysv_ext1 = ysb dy_ext0 = dy_ext1 = dy0 - SQUISH_CONSTANT_4D if (c1 & 0x01) == 0x01: ysv_ext0 -= 1 dy_ext0 += 1 else: ysv_ext1 -= 1 dy_ext1 += 1 else: ysv_ext0 = ysv_ext1 = ysb + 1 dy_ext0 = dy_ext1 = dy0 - 1 - SQUISH_CONSTANT_4D if (c1 & 0x04) == 0: zsv_ext0 = zsv_ext1 = zsb dz_ext0 = dz_ext1 = dz0 - SQUISH_CONSTANT_4D if (c1 & 0x03) == 0x03: zsv_ext0 -= 1 dz_ext0 += 1 else: zsv_ext1 -= 1 dz_ext1 += 1 else: zsv_ext0 = zsv_ext1 = zsb + 1 dz_ext0 = dz_ext1 = dz0 - 1 - SQUISH_CONSTANT_4D if (c1 & 0x08) == 0: wsv_ext0 = wsb wsv_ext1 = wsb - 1 dw_ext0 = dw0 - SQUISH_CONSTANT_4D dw_ext1 = dw0 + 1 - SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsb + 1 dw_ext0 = dw_ext1 = dw0 - 1 - SQUISH_CONSTANT_4D # One contribution is a _permutation of (0,0,0,2) based on the smaller-sided po xsv_ext2 = xsb ysv_ext2 = ysb zsv_ext2 = zsb wsv_ext2 = wsb dx_ext2 = dx0 - 2 * SQUISH_CONSTANT_4D dy_ext2 = dy0 - 2 * SQUISH_CONSTANT_4D dz_ext2 = dz0 - 2 * SQUISH_CONSTANT_4D dw_ext2 = dw0 - 2 * SQUISH_CONSTANT_4D if (c2 & 0x01) != 0: xsv_ext2 += 2 dx_ext2 -= 2 elif (c2 & 0x02) != 0: ysv_ext2 += 2 dy_ext2 -= 2 elif (c2 & 0x04) != 0: zsv_ext2 += 2 dz_ext2 -= 2 else: wsv_ext2 += 2 dw_ext2 -= 2 # Contribution (1,0,0,0) dx1 = dx0 - 1 - SQUISH_CONSTANT_4D dy1 = dy0 - 0 - SQUISH_CONSTANT_4D dz1 = dz0 - 0 - SQUISH_CONSTANT_4D dw1 = dw0 - 0 - SQUISH_CONSTANT_4D attn1 = 2 - dx1 * dx1 - dy1 * dy1 - dz1 * dz1 - dw1 * dw1 if attn1 > 0: attn1 *= attn1 value += attn1 * attn1 * extrapolate(xsb + 1, ysb + 0, zsb + 0, wsb + 0, dx1, dy1, dz1, dw1) # Contribution (0,1,0,0) dx2 = dx0 - 0 - SQUISH_CONSTANT_4D dy2 = dy0 - 1 - SQUISH_CONSTANT_4D dz2 = dz1 dw2 = dw1 attn2 = 2 - dx2 * dx2 - dy2 * dy2 - dz2 * dz2 - dw2 * dw2 if attn2 > 0: attn2 *= attn2 value += attn2 * attn2 * extrapolate(xsb + 0, ysb + 1, zsb + 0, wsb + 0, dx2, dy2, dz2, dw2) # Contribution (0,0,1,0) dx3 = dx2 dy3 = dy1 dz3 = dz0 - 1 - SQUISH_CONSTANT_4D dw3 = dw1 attn3 = 2 - dx3 * dx3 - dy3 * dy3 - dz3 * dz3 - dw3 * dw3 if attn3 > 0: attn3 *= attn3 value += attn3 * attn3 * extrapolate(xsb + 0, ysb + 0, zsb + 1, wsb + 0, dx3, dy3, dz3, dw3) # Contribution (0,0,0,1) dx4 = dx2 dy4 = dy1 dz4 = dz1 dw4 = dw0 - 1 - SQUISH_CONSTANT_4D attn4 = 2 - dx4 * dx4 - dy4 * dy4 - dz4 * dz4 - dw4 * dw4 if attn4 > 0: attn4 *= attn4 value += attn4 * attn4 * extrapolate(xsb + 0, ysb + 0, zsb + 0, wsb + 1, dx4, dy4, dz4, dw4) # Contribution (1,1,0,0) dx5 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy5 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz5 = dz0 - 0 - 2 * SQUISH_CONSTANT_4D dw5 = dw0 - 0 - 2 * SQUISH_CONSTANT_4D attn5 = 2 - dx5 * dx5 - dy5 * dy5 - dz5 * dz5 - dw5 * dw5 if attn5 > 0: attn5 *= attn5 value += attn5 * attn5 * extrapolate(xsb + 1, ysb + 1, zsb + 0, wsb + 0, dx5, dy5, dz5, dw5) # Contribution (1,0,1,0) dx6 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy6 = dy0 - 0 - 2 * SQUISH_CONSTANT_4D dz6 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw6 = dw0 - 0 - 2 * SQUISH_CONSTANT_4D attn6 = 2 - dx6 * dx6 - dy6 * dy6 - dz6 * dz6 - dw6 * dw6 if attn6 > 0: attn6 *= attn6 value += attn6 * attn6 * extrapolate(xsb + 1, ysb + 0, zsb + 1, wsb + 0, dx6, dy6, dz6, dw6) # Contribution (1,0,0,1) dx7 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy7 = dy0 - 0 - 2 * SQUISH_CONSTANT_4D dz7 = dz0 - 0 - 2 * SQUISH_CONSTANT_4D dw7 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D attn7 = 2 - dx7 * dx7 - dy7 * dy7 - dz7 * dz7 - dw7 * dw7 if attn7 > 0: attn7 *= attn7 value += attn7 * attn7 * extrapolate(xsb + 1, ysb + 0, zsb + 0, wsb + 1, dx7, dy7, dz7, dw7) # Contribution (0,1,1,0) dx8 = dx0 - 0 - 2 * SQUISH_CONSTANT_4D dy8 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz8 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw8 = dw0 - 0 - 2 * SQUISH_CONSTANT_4D attn8 = 2 - dx8 * dx8 - dy8 * dy8 - dz8 * dz8 - dw8 * dw8 if attn8 > 0: attn8 *= attn8 value += attn8 * attn8 * extrapolate(xsb + 0, ysb + 1, zsb + 1, wsb + 0, dx8, dy8, dz8, dw8) # Contribution (0,1,0,1) dx9 = dx0 - 0 - 2 * SQUISH_CONSTANT_4D dy9 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz9 = dz0 - 0 - 2 * SQUISH_CONSTANT_4D dw9 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D attn9 = 2 - dx9 * dx9 - dy9 * dy9 - dz9 * dz9 - dw9 * dw9 if attn9 > 0: attn9 *= attn9 value += attn9 * attn9 * extrapolate(xsb + 0, ysb + 1, zsb + 0, wsb + 1, dx9, dy9, dz9, dw9) # Contribution (0,0,1,1) dx10 = dx0 - 0 - 2 * SQUISH_CONSTANT_4D dy10 = dy0 - 0 - 2 * SQUISH_CONSTANT_4D dz10 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw10 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D attn10 = 2 - dx10 * dx10 - dy10 * dy10 - dz10 * dz10 - dw10 * dw10 if attn10 > 0: attn10 *= attn10 value += attn10 * attn10 * extrapolate(xsb + 0, ysb + 0, zsb + 1, wsb + 1, dx10, dy10, dz10, dw10) else: # We're inside the second dispentachoron (Rectified 4-Simplex) a_is_bigger_side = True b_is_bigger_side = True # Decide between (0,0,1,1) and (1,1,0,0) if xins + yins < zins + wins: a_score = xins + yins a_po = 0x0C else: a_score = zins + wins a_po = 0x03 # Decide between (0,1,0,1) and (1,0,1,0) if xins + zins < yins + wins: b_score = xins + zins b_po = 0x0A else: b_score = yins + wins b_po = 0x05 # Closer between (0,1,1,0) and (1,0,0,1) will replace the further of a and b, if closer. if xins + wins < yins + zins: score = xins + wins if a_score <= b_score and score < b_score: b_score = score b_po = 0x06 elif a_score > b_score and score < a_score: a_score = score a_po = 0x06 else: score = yins + zins if a_score <= b_score and score < b_score: b_score = score b_po = 0x09 elif a_score > b_score and score < a_score: a_score = score a_po = 0x09 # Decide if (0,1,1,1) is closer. p1 = 3 - in_sum + xins if a_score <= b_score and p1 < b_score: b_score = p1 b_po = 0x0E b_is_bigger_side = False elif a_score > b_score and p1 < a_score: a_score = p1 a_po = 0x0E a_is_bigger_side = False # Decide if (1,0,1,1) is closer. p2 = 3 - in_sum + yins if a_score <= b_score and p2 < b_score: b_score = p2 b_po = 0x0D b_is_bigger_side = False elif a_score > b_score and p2 < a_score: a_score = p2 a_po = 0x0D a_is_bigger_side = False # Decide if (1,1,0,1) is closer. p3 = 3 - in_sum + zins if a_score <= b_score and p3 < b_score: b_score = p3 b_po = 0x0B b_is_bigger_side = False elif a_score > b_score and p3 < a_score: a_score = p3 a_po = 0x0B a_is_bigger_side = False # Decide if (1,1,1,0) is closer. p4 = 3 - in_sum + wins if a_score <= b_score and p4 < b_score: b_po = 0x07 b_is_bigger_side = False elif a_score > b_score and p4 < a_score: a_po = 0x07 a_is_bigger_side = False # Where each of the two closest pos are determines how the extra three vertices are calculated. if a_is_bigger_side == b_is_bigger_side: if a_is_bigger_side: # Both closest pos on the bigger side c1 = (a_po & b_po) c2 = (a_po | b_po) # Two contributions are _permutations of (0,0,0,1) and (0,0,0,2) based on c1 xsv_ext0 = xsv_ext1 = xsb ysv_ext0 = ysv_ext1 = ysb zsv_ext0 = zsv_ext1 = zsb wsv_ext0 = wsv_ext1 = wsb dx_ext0 = dx0 - SQUISH_CONSTANT_4D dy_ext0 = dy0 - SQUISH_CONSTANT_4D dz_ext0 = dz0 - SQUISH_CONSTANT_4D dw_ext0 = dw0 - SQUISH_CONSTANT_4D dx_ext1 = dx0 - 2 * SQUISH_CONSTANT_4D dy_ext1 = dy0 - 2 * SQUISH_CONSTANT_4D dz_ext1 = dz0 - 2 * SQUISH_CONSTANT_4D dw_ext1 = dw0 - 2 * SQUISH_CONSTANT_4D if (c1 & 0x01) != 0: xsv_ext0 += 1 dx_ext0 -= 1 xsv_ext1 += 2 dx_ext1 -= 2 elif (c1 & 0x02) != 0: ysv_ext0 += 1 dy_ext0 -= 1 ysv_ext1 += 2 dy_ext1 -= 2 elif (c1 & 0x04) != 0: zsv_ext0 += 1 dz_ext0 -= 1 zsv_ext1 += 2 dz_ext1 -= 2 else: wsv_ext0 += 1 dw_ext0 -= 1 wsv_ext1 += 2 dw_ext1 -= 2 # One contribution is a _permutation of (1,1,1,-1) based on c2 xsv_ext2 = xsb + 1 ysv_ext2 = ysb + 1 zsv_ext2 = zsb + 1 wsv_ext2 = wsb + 1 dx_ext2 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy_ext2 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz_ext2 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw_ext2 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D if (c2 & 0x01) == 0: xsv_ext2 -= 2 dx_ext2 += 2 elif (c2 & 0x02) == 0: ysv_ext2 -= 2 dy_ext2 += 2 elif (c2 & 0x04) == 0: zsv_ext2 -= 2 dz_ext2 += 2 else: wsv_ext2 -= 2 dw_ext2 += 2 else: # Both closest pos on the smaller side # One of the two extra pos is (1,1,1,1) xsv_ext2 = xsb + 1 ysv_ext2 = ysb + 1 zsv_ext2 = zsb + 1 wsv_ext2 = wsb + 1 dx_ext2 = dx0 - 1 - 4 * SQUISH_CONSTANT_4D dy_ext2 = dy0 - 1 - 4 * SQUISH_CONSTANT_4D dz_ext2 = dz0 - 1 - 4 * SQUISH_CONSTANT_4D dw_ext2 = dw0 - 1 - 4 * SQUISH_CONSTANT_4D # Other two pos are based on the shared axes. c = (a_po & b_po) if (c & 0x01) != 0: xsv_ext0 = xsb + 2 xsv_ext1 = xsb + 1 dx_ext0 = dx0 - 2 - 3 * SQUISH_CONSTANT_4D dx_ext1 = dx0 - 1 - 3 * SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsb dx_ext0 = dx_ext1 = dx0 - 3 * SQUISH_CONSTANT_4D if (c & 0x02) != 0: ysv_ext0 = ysv_ext1 = ysb + 1 dy_ext0 = dy_ext1 = dy0 - 1 - 3 * SQUISH_CONSTANT_4D if (c & 0x01) == 0: ysv_ext0 += 1 dy_ext0 -= 1 else: ysv_ext1 += 1 dy_ext1 -= 1 else: ysv_ext0 = ysv_ext1 = ysb dy_ext0 = dy_ext1 = dy0 - 3 * SQUISH_CONSTANT_4D if (c & 0x04) != 0: zsv_ext0 = zsv_ext1 = zsb + 1 dz_ext0 = dz_ext1 = dz0 - 1 - 3 * SQUISH_CONSTANT_4D if (c & 0x03) == 0: zsv_ext0 += 1 dz_ext0 -= 1 else: zsv_ext1 += 1 dz_ext1 -= 1 else: zsv_ext0 = zsv_ext1 = zsb dz_ext0 = dz_ext1 = dz0 - 3 * SQUISH_CONSTANT_4D if (c & 0x08) != 0: wsv_ext0 = wsb + 1 wsv_ext1 = wsb + 2 dw_ext0 = dw0 - 1 - 3 * SQUISH_CONSTANT_4D dw_ext1 = dw0 - 2 - 3 * SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsb dw_ext0 = dw_ext1 = dw0 - 3 * SQUISH_CONSTANT_4D else: # One po on each "side" if a_is_bigger_side: c1 = a_po c2 = b_po else: c1 = b_po c2 = a_po # Two contributions are the bigger-sided po with each 1 replaced with 2. if (c1 & 0x01) != 0: xsv_ext0 = xsb + 2 xsv_ext1 = xsb + 1 dx_ext0 = dx0 - 2 - 3 * SQUISH_CONSTANT_4D dx_ext1 = dx0 - 1 - 3 * SQUISH_CONSTANT_4D else: xsv_ext0 = xsv_ext1 = xsb dx_ext0 = dx_ext1 = dx0 - 3 * SQUISH_CONSTANT_4D if (c1 & 0x02) != 0: ysv_ext0 = ysv_ext1 = ysb + 1 dy_ext0 = dy_ext1 = dy0 - 1 - 3 * SQUISH_CONSTANT_4D if (c1 & 0x01) == 0: ysv_ext0 += 1 dy_ext0 -= 1 else: ysv_ext1 += 1 dy_ext1 -= 1 else: ysv_ext0 = ysv_ext1 = ysb dy_ext0 = dy_ext1 = dy0 - 3 * SQUISH_CONSTANT_4D if (c1 & 0x04) != 0: zsv_ext0 = zsv_ext1 = zsb + 1 dz_ext0 = dz_ext1 = dz0 - 1 - 3 * SQUISH_CONSTANT_4D if (c1 & 0x03) == 0: zsv_ext0 += 1 dz_ext0 -= 1 else: zsv_ext1 += 1 dz_ext1 -= 1 else: zsv_ext0 = zsv_ext1 = zsb dz_ext0 = dz_ext1 = dz0 - 3 * SQUISH_CONSTANT_4D if (c1 & 0x08) != 0: wsv_ext0 = wsb + 1 wsv_ext1 = wsb + 2 dw_ext0 = dw0 - 1 - 3 * SQUISH_CONSTANT_4D dw_ext1 = dw0 - 2 - 3 * SQUISH_CONSTANT_4D else: wsv_ext0 = wsv_ext1 = wsb dw_ext0 = dw_ext1 = dw0 - 3 * SQUISH_CONSTANT_4D # One contribution is a _permutation of (1,1,1,-1) based on the smaller-sided po xsv_ext2 = xsb + 1 ysv_ext2 = ysb + 1 zsv_ext2 = zsb + 1 wsv_ext2 = wsb + 1 dx_ext2 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy_ext2 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz_ext2 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw_ext2 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D if (c2 & 0x01) == 0: xsv_ext2 -= 2 dx_ext2 += 2 elif (c2 & 0x02) == 0: ysv_ext2 -= 2 dy_ext2 += 2 elif (c2 & 0x04) == 0: zsv_ext2 -= 2 dz_ext2 += 2 else: wsv_ext2 -= 2 dw_ext2 += 2 # Contribution (1,1,1,0) dx4 = dx0 - 1 - 3 * SQUISH_CONSTANT_4D dy4 = dy0 - 1 - 3 * SQUISH_CONSTANT_4D dz4 = dz0 - 1 - 3 * SQUISH_CONSTANT_4D dw4 = dw0 - 3 * SQUISH_CONSTANT_4D attn4 = 2 - dx4 * dx4 - dy4 * dy4 - dz4 * dz4 - dw4 * dw4 if attn4 > 0: attn4 *= attn4 value += attn4 * attn4 * extrapolate(xsb + 1, ysb + 1, zsb + 1, wsb + 0, dx4, dy4, dz4, dw4) # Contribution (1,1,0,1) dx3 = dx4 dy3 = dy4 dz3 = dz0 - 3 * SQUISH_CONSTANT_4D dw3 = dw0 - 1 - 3 * SQUISH_CONSTANT_4D attn3 = 2 - dx3 * dx3 - dy3 * dy3 - dz3 * dz3 - dw3 * dw3 if attn3 > 0: attn3 *= attn3 value += attn3 * attn3 * extrapolate(xsb + 1, ysb + 1, zsb + 0, wsb + 1, dx3, dy3, dz3, dw3) # Contribution (1,0,1,1) dx2 = dx4 dy2 = dy0 - 3 * SQUISH_CONSTANT_4D dz2 = dz4 dw2 = dw3 attn2 = 2 - dx2 * dx2 - dy2 * dy2 - dz2 * dz2 - dw2 * dw2 if attn2 > 0: attn2 *= attn2 value += attn2 * attn2 * extrapolate(xsb + 1, ysb + 0, zsb + 1, wsb + 1, dx2, dy2, dz2, dw2) # Contribution (0,1,1,1) dx1 = dx0 - 3 * SQUISH_CONSTANT_4D dz1 = dz4 dy1 = dy4 dw1 = dw3 attn1 = 2 - dx1 * dx1 - dy1 * dy1 - dz1 * dz1 - dw1 * dw1 if attn1 > 0: attn1 *= attn1 value += attn1 * attn1 * extrapolate(xsb + 0, ysb + 1, zsb + 1, wsb + 1, dx1, dy1, dz1, dw1) # Contribution (1,1,0,0) dx5 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy5 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz5 = dz0 - 0 - 2 * SQUISH_CONSTANT_4D dw5 = dw0 - 0 - 2 * SQUISH_CONSTANT_4D attn5 = 2 - dx5 * dx5 - dy5 * dy5 - dz5 * dz5 - dw5 * dw5 if attn5 > 0: attn5 *= attn5 value += attn5 * attn5 * extrapolate(xsb + 1, ysb + 1, zsb + 0, wsb + 0, dx5, dy5, dz5, dw5) # Contribution (1,0,1,0) dx6 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy6 = dy0 - 0 - 2 * SQUISH_CONSTANT_4D dz6 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw6 = dw0 - 0 - 2 * SQUISH_CONSTANT_4D attn6 = 2 - dx6 * dx6 - dy6 * dy6 - dz6 * dz6 - dw6 * dw6 if attn6 > 0: attn6 *= attn6 value += attn6 * attn6 * extrapolate(xsb + 1, ysb + 0, zsb + 1, wsb + 0, dx6, dy6, dz6, dw6) # Contribution (1,0,0,1) dx7 = dx0 - 1 - 2 * SQUISH_CONSTANT_4D dy7 = dy0 - 0 - 2 * SQUISH_CONSTANT_4D dz7 = dz0 - 0 - 2 * SQUISH_CONSTANT_4D dw7 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D attn7 = 2 - dx7 * dx7 - dy7 * dy7 - dz7 * dz7 - dw7 * dw7 if attn7 > 0: attn7 *= attn7 value += attn7 * attn7 * extrapolate(xsb + 1, ysb + 0, zsb + 0, wsb + 1, dx7, dy7, dz7, dw7) # Contribution (0,1,1,0) dx8 = dx0 - 0 - 2 * SQUISH_CONSTANT_4D dy8 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz8 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw8 = dw0 - 0 - 2 * SQUISH_CONSTANT_4D attn8 = 2 - dx8 * dx8 - dy8 * dy8 - dz8 * dz8 - dw8 * dw8 if attn8 > 0: attn8 *= attn8 value += attn8 * attn8 * extrapolate(xsb + 0, ysb + 1, zsb + 1, wsb + 0, dx8, dy8, dz8, dw8) # Contribution (0,1,0,1) dx9 = dx0 - 0 - 2 * SQUISH_CONSTANT_4D dy9 = dy0 - 1 - 2 * SQUISH_CONSTANT_4D dz9 = dz0 - 0 - 2 * SQUISH_CONSTANT_4D dw9 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D attn9 = 2 - dx9 * dx9 - dy9 * dy9 - dz9 * dz9 - dw9 * dw9 if attn9 > 0: attn9 *= attn9 value += attn9 * attn9 * extrapolate(xsb + 0, ysb + 1, zsb + 0, wsb + 1, dx9, dy9, dz9, dw9) # Contribution (0,0,1,1) dx10 = dx0 - 0 - 2 * SQUISH_CONSTANT_4D dy10 = dy0 - 0 - 2 * SQUISH_CONSTANT_4D dz10 = dz0 - 1 - 2 * SQUISH_CONSTANT_4D dw10 = dw0 - 1 - 2 * SQUISH_CONSTANT_4D attn10 = 2 - dx10 * dx10 - dy10 * dy10 - dz10 * dz10 - dw10 * dw10 if attn10 > 0: attn10 *= attn10 value += attn10 * attn10 * extrapolate(xsb + 0, ysb + 0, zsb + 1, wsb + 1, dx10, dy10, dz10, dw10) # First extra vertex attn_ext0 = 2 - dx_ext0 * dx_ext0 - dy_ext0 * dy_ext0 - dz_ext0 * dz_ext0 - dw_ext0 * dw_ext0 if attn_ext0 > 0: attn_ext0 *= attn_ext0 value += attn_ext0 * attn_ext0 * extrapolate(xsv_ext0, ysv_ext0, zsv_ext0, wsv_ext0, dx_ext0, dy_ext0, dz_ext0, dw_ext0) # Second extra vertex attn_ext1 = 2 - dx_ext1 * dx_ext1 - dy_ext1 * dy_ext1 - dz_ext1 * dz_ext1 - dw_ext1 * dw_ext1 if attn_ext1 > 0: attn_ext1 *= attn_ext1 value += attn_ext1 * attn_ext1 * extrapolate(xsv_ext1, ysv_ext1, zsv_ext1, wsv_ext1, dx_ext1, dy_ext1, dz_ext1, dw_ext1) # Third extra vertex attn_ext2 = 2 - dx_ext2 * dx_ext2 - dy_ext2 * dy_ext2 - dz_ext2 * dz_ext2 - dw_ext2 * dw_ext2 if attn_ext2 > 0: attn_ext2 *= attn_ext2 value += attn_ext2 * attn_ext2 * extrapolate(xsv_ext2, ysv_ext2, zsv_ext2, wsv_ext2, dx_ext2, dy_ext2, dz_ext2, dw_ext2) return value / NORM_CONSTANT_4D - source_sentence: |- Method which returns a dictionary of field statistics received from the input source. Returns: fieldStats: dict of dicts where the first level is the field name and the second level is the statistic. ie. fieldStats['pounds']['min'] sentences: - |- def customize(func): """ Decorator to set plotting context and axes style during function call. """ @wraps(func) def call_w_context(*args, **kwargs): set_context = kwargs.pop('set_context', True) if set_context: with plotting_context(), axes_style(): return func(*args, **kwargs) else: return func(*args, **kwargs) return call_w_context - |- def Vgg19_simple_api(rgb): """ Build the VGG 19 Model Parameters ----------- rgb : rgb image placeholder [batch, height, width, 3] values scaled [0, 1] """ start_time = time.time() print("build model started") rgb_scaled = rgb * 255.0 # Convert RGB to BGR red, green, blue = tf.split(rgb_scaled, 3, 3) if red.get_shape().as_list()[1:] != [224, 224, 1]: raise Exception("image size unmatch") if green.get_shape().as_list()[1:] != [224, 224, 1]: raise Exception("image size unmatch") if blue.get_shape().as_list()[1:] != [224, 224, 1]: raise Exception("image size unmatch") bgr = tf.concat([ blue - VGG_MEAN[0], green - VGG_MEAN[1], red - VGG_MEAN[2], ], axis=3) if bgr.get_shape().as_list()[1:] != [224, 224, 3]: raise Exception("image size unmatch") # input layer net_in = InputLayer(bgr, name='input') # conv1 net = Conv2d(net_in, 64, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv1_1') net = Conv2d(net, n_filter=64, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv1_2') net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool1') # conv2 net = Conv2d(net, n_filter=128, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv2_1') net = Conv2d(net, n_filter=128, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv2_2') net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool2') # conv3 net = Conv2d(net, n_filter=256, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv3_1') net = Conv2d(net, n_filter=256, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv3_2') net = Conv2d(net, n_filter=256, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv3_3') net = Conv2d(net, n_filter=256, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv3_4') net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool3') # conv4 net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv4_1') net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv4_2') net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv4_3') net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv4_4') net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool4') # conv5 net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv5_1') net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv5_2') net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv5_3') net = Conv2d(net, n_filter=512, filter_size=(3, 3), strides=(1, 1), act=tf.nn.relu, padding='SAME', name='conv5_4') net = MaxPool2d(net, filter_size=(2, 2), strides=(2, 2), padding='SAME', name='pool5') # fc 6~8 net = FlattenLayer(net, name='flatten') net = DenseLayer(net, n_units=4096, act=tf.nn.relu, name='fc6') net = DenseLayer(net, n_units=4096, act=tf.nn.relu, name='fc7') net = DenseLayer(net, n_units=1000, act=None, name='fc8') print("build model finished: %fs" % (time.time() - start_time)) return net - |- def _getFieldStats(self): """ Method which returns a dictionary of field statistics received from the input source. Returns: fieldStats: dict of dicts where the first level is the field name and the second level is the statistic. ie. fieldStats['pounds']['min'] """ fieldStats = dict() fieldNames = self._inputSource.getFieldNames() for field in fieldNames: curStats = dict() curStats['min'] = self._inputSource.getFieldMin(field) curStats['max'] = self._inputSource.getFieldMax(field) fieldStats[field] = curStats return fieldStats datasets: - fyaronskiy/cornstack_python_ru_en - fyaronskiy/code_search_net_ru_en - ai-forever/solyanka pipeline_tag: text-ranking library_name: sentence-transformers metrics: - cosine_accuracy@1 - cosine_accuracy@3 - cosine_accuracy@5 - cosine_accuracy@10 - cosine_precision@1 - cosine_recall@1 - cosine_recall@3 - cosine_recall@5 - cosine_recall@10 - cosine_ndcg@10 - cosine_mrr@10 model-index: - name: SentenceTransformer results: - task: type: information-retrieval name: Information Retrieval dataset: name: Unknown type: unknown metrics: - type: cosine_accuracy@1 value: 0.8683666666666666 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9439333333333333 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9566333333333333 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9668333333333333 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8683666666666666 name: Cosine Precision@1 - type: cosine_recall@1 value: 0.8683666666666666 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9439333333333333 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9566333333333333 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9668333333333333 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9224025873017736 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9076358333333253 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9082959802184539 name: Cosine Map@100 - type: cosine_accuracy@1 value: 0.8741666666666666 name: Cosine Accuracy@1 - type: cosine_accuracy@3 value: 0.9425 name: Cosine Accuracy@3 - type: cosine_accuracy@5 value: 0.9548666666666666 name: Cosine Accuracy@5 - type: cosine_accuracy@10 value: 0.9644333333333334 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.8741666666666666 name: Cosine Precision@1 - type: cosine_recall@1 value: 0.8741666666666666 name: Cosine Recall@1 - type: cosine_recall@3 value: 0.9425 name: Cosine Recall@3 - type: cosine_recall@5 value: 0.9548666666666666 name: Cosine Recall@5 - type: cosine_recall@10 value: 0.9644333333333334 name: Cosine Recall@10 - type: cosine_ndcg@10 value: 0.9234437208756444 name: Cosine Ndcg@10 - type: cosine_mrr@10 value: 0.9098453571428485 name: Cosine Mrr@10 - type: cosine_map@100 value: 0.9105416505961587 name: Cosine Map@100 license: apache-2.0 base_model: - deepvk/RuModernBERT-base --- # SentenceTransformer This is a [sentence-transformers](https://www.SBERT.net) model trained on the [cornstack_python](https://huggingface.co/datasets/fyaronskiy/cornstack_python_ru_en), cornstack_python_pairs, [codesearchnet](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en), [codesearchnet_pairs](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) and [solyanka_qa](https://huggingface.co/datasets/ai-forever/solyanka) datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space. Model can be used for text-to-code, code-to-text retrieval tasks where text is in **Russian/English** and code is in **Python/Java/Javascript/Go/Php/Ruby**. Queries, documents also can be mix of natural language text and code. Perfomance of code-to-code tasks wasn't measured. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [RuModernBERT-base](https://huggingface.co/deepvk/RuModernBERT-base) - **Maximum Sequence Length:** 8192 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Datasets:** - [cornstack_python](https://huggingface.co/datasets/fyaronskiy/cornstack_python_ru_en) - cornstack_python_pairs - [codesearchnet](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) - [codesearchnet_pairs](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) - [solyanka_qa](https://huggingface.co/datasets/ai-forever/solyanka) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'}) (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python import torch from sentence_transformers import SentenceTransformer, util device = "cuda" if torch.cuda.is_available() else "cpu" model = SentenceTransformer("fyaronskiy/code_retriever_ru_en").to(device) queries_ru = [ "Напиши функцию на Python, которая рекурсивно вычисляет факториал числа.", "Как проверить, является ли строка палиндромом?", "Объедини два отсортированных списка в один отсортированный список." ] corpus_ru = [ # Релевантный для Q1 """def factorial(n): if n == 0: return 1 return n * factorial(n - 1)""", # Hard negative для Q1 """def sum_recursive(n): if n == 0: return 0 return n + sum_recursive(n - 1)""", # Релевантный для Q2 """def is_palindrome(s: str) -> bool: s = s.lower().replace(" ", "") return s == s[::-1]""", # Hard negative для Q2 """def reverse_string(s: str) -> str: return s[::-1]""", # Релевантный для Q3 """def merge_sorted_lists(a, b): result = [] i = j = 0 while i < len(a) and j < len(b): if a[i] < b[j]: result.append(a[i]) i += 1 else: result.append(b[j]) j += 1 result.extend(a[i:]) result.extend(b[j:]) return result""", # Hard negative для Q3 """def add_lists(a, b): return [x + y for x, y in zip(a, b)]""" ] doc_embeddings = model.encode(corpus_ru, convert_to_tensor=True, device=device) query_embeddings = model.encode(queries_ru, convert_to_tensor=True, device=device) # Выполняем поиск по каждому запросу for i, query in enumerate(queries_ru): scores = util.cos_sim(query_embeddings[i], doc_embeddings)[0] best_idx = torch.argmax(scores).item() print(f"\nЗапрос {i+1}: {query}") print('Скоры всех документов в корпусе: ', scores) print(f"Наиболее подходящий документ (Скор={scores[best_idx]:.4f}):\n{corpus_ru[best_idx]}") ``` Model was trained with Matryoshka Loss with dims: 768, 512, 256, 128, 64. So for decreasing memory for your vector databaset and make inference faster you can truncate embeddings. To do this you need to initialize model as follows: ```python matryoshka_dim = 128 model = SentenceTransformer("fyaronskiy/code_retriever_ru_en", truncate_dim=matryoshka_dim).to(device) ``` ## Evaluation ### Metrics #### Information Retrieval * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8684 | | cosine_accuracy@3 | 0.9439 | | cosine_accuracy@5 | 0.9566 | | cosine_accuracy@10 | 0.9668 | | cosine_precision@1 | 0.8684 | | cosine_precision@3 | 0.3146 | | cosine_precision@5 | 0.1913 | | cosine_precision@10 | 0.0967 | | cosine_recall@1 | 0.8684 | | cosine_recall@3 | 0.9439 | | cosine_recall@5 | 0.9566 | | cosine_recall@10 | 0.9668 | | **cosine_ndcg@10** | **0.9224** | | cosine_mrr@10 | 0.9076 | | cosine_map@100 | 0.9083 | #### Information Retrieval * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.8742 | | cosine_accuracy@3 | 0.9425 | | cosine_accuracy@5 | 0.9549 | | cosine_accuracy@10 | 0.9644 | | cosine_precision@1 | 0.8742 | | cosine_precision@3 | 0.3142 | | cosine_precision@5 | 0.191 | | cosine_precision@10 | 0.0964 | | cosine_recall@1 | 0.8742 | | cosine_recall@3 | 0.9425 | | cosine_recall@5 | 0.9549 | | cosine_recall@10 | 0.9644 | | **cosine_ndcg@10** | **0.9234** | | cosine_mrr@10 | 0.9098 | | cosine_map@100 | 0.9105 | ## Training Details ### Training Datasets
cornstack_python #### cornstack_python * Dataset: cornstack_python * Size: 2,869,969 training samples * Columns: ru_query, document, negative_0, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, negative_7, negative_8, negative_9, negative_10, negative_11, negative_12, negative_13, negative_14, and negative_15 * Approximate statistics based on the first 1000 samples: | | ru_query | document | negative_0 | negative_1 | negative_2 | negative_3 | negative_4 | negative_5 | negative_6 | negative_7 | negative_8 | negative_9 | negative_10 | negative_11 | negative_12 | negative_13 | negative_14 | negative_15 | |:--------|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | string | | details |
  • min: 7 tokens
  • mean: 27.46 tokens
  • max: 162 tokens
|
  • min: 6 tokens
  • mean: 304.38 tokens
  • max: 5574 tokens
|
  • min: 6 tokens
  • mean: 237.08 tokens
  • max: 3627 tokens
|
  • min: 6 tokens
  • mean: 229.94 tokens
  • max: 6691 tokens
|
  • min: 6 tokens
  • mean: 230.06 tokens
  • max: 6229 tokens
|
  • min: 7 tokens
  • mean: 230.7 tokens
  • max: 4876 tokens
|
  • min: 8 tokens
  • mean: 220.57 tokens
  • max: 4876 tokens
|
  • min: 7 tokens
  • mean: 236.08 tokens
  • max: 5880 tokens
|
  • min: 6 tokens
  • mean: 247.91 tokens
  • max: 6621 tokens
|
  • min: 6 tokens
  • mean: 207.62 tokens
  • max: 3350 tokens
|
  • min: 6 tokens
  • mean: 222.54 tokens
  • max: 6863 tokens
|
  • min: 6 tokens
  • mean: 221.53 tokens
  • max: 4976 tokens
|
  • min: 7 tokens
  • mean: 216.06 tokens
  • max: 4876 tokens
|
  • min: 7 tokens
  • mean: 197.03 tokens
  • max: 4763 tokens
|
  • min: 6 tokens
  • mean: 200.83 tokens
  • max: 8192 tokens
|
  • min: 6 tokens
  • mean: 204.94 tokens
  • max: 3210 tokens
|
  • min: 6 tokens
  • mean: 188.51 tokens
  • max: 2754 tokens
|
  • min: 6 tokens
  • mean: 188.27 tokens
  • max: 4876 tokens
| * Samples: | ru_query | document | negative_0 | negative_1 | negative_2 | negative_3 | negative_4 | negative_5 | negative_6 | negative_7 | negative_8 | negative_9 | negative_10 | negative_11 | negative_12 | negative_13 | negative_14 | negative_15 | |:------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | установите значение business_id сообщения данных в конкретное значение | def step_impl_the_ru_is_set_to(context, business_id):
context.bdd_helper.message_data["business_id"] = business_id | def business_id(self, business_id):

self._business_id = business_id | def business_phone(self, business_phone):

self._business_phone = business_phone | def business_phone_number(self, business_phone_number):

self._business_phone_number = business_phone_number | def bus_ob_id(self, bus_ob_id):

self._bus_ob_id = bus_ob_id | def bus_ob_id(self, bus_ob_id):

self._bus_ob_id = bus_ob_id | def _set_id(self, value):
pass | def business_email(self, business_email):

self._business_email = business_email | def mailing_id(self, val: str):
self._mailing_id = val | def message_id(self, val: str):
self._message_id = val | def business_model(self, business_model):

self._business_model = business_model | def business_account(self, business_account):

self._business_account = business_account | def update_business(current_user, businessId):
business = Business.query.get(int(businessId))

if not business:
return make_json_reply('message', 'Business id does not exist'), 404

if business.user_id != current_user.id:
return make_json_reply('message', 'Cannot update business'), 400

data = request.get_json(force=True)
name = location = category = description = None

if 'name' in data.keys():
name = data['name']

if 'location' in data.keys():
location = data['location']

if 'category' in data.keys():
category = data['category']

if 'description' in data.keys():
description = data['description']

if check_validity_of_input(name=name):
business.name = name

if check_validity_of_input(location=location):
business.location = location

if check_validity_of_input(category=category):
business.category = category

if check_validity_of_input(description=description):
... | def set_company_id_value(self, company_id_value):
self.company_id_value = company_id_value | def id(self, value):
self._id = value | def set_bribe(self, bribe_amount):
self.bribe = bribe_amount | def business_owner(self, business_owner):

self._business_owner = business_owner | | Установить состояние правил sid | def set_state_sid_request(ruleset_name, sid):
message = json.loads(request.stream.read().decode('utf-8'))
message['sid'] = sid
result = host.patch_state(ruleset_name, message)
return jsonify(result) | def sid(self, sid):
self._sid = sid | def set_state(self,s):
self.state = s | def set_state(self, state: int): | def __setstate__(self, state):

self.set(DER = state) | def set_rule(self, rule):
self.rule.load_state_dict(rule, strict=True) | def _set_state(self, state):
#print("** set state from %d to %d" % (self.state, state))
self.state = state | def set_state( self ): | def set_ident(self, new_ident: int):
if not isinstance(new_ident, int):
raise TypeError("Spectrum set identifiers may ONLY be positive integers")
self._set_ident = new_ident | def set_state(self, state):
#print("ComponentBase.set_state")
for k,v in state.items():
#print(" Set {:14s} to {:s}".format(k,str(v)))
if k == "connectors":
for con_state in v:
self.add_connector()
self.connectors[-1].set_state(con_state)
else:
setattr(self, k, v) | def __setstate__(self, state):

self.list = state | def __setstate__(self, state):

self.list = state | def state_id(self, state_id):

self._state_id = state_id | def set_state(self, state: int):
self.state = state | def set_domain_sid(self, sid):
dsdb._samdb_set_domain_sid(self, sid) | def set_state(self,state):
self.__state = state | def set_srid(self, srid: ir.IntegerValue) -> GeoSpatialValue:
return ops.GeoSetSRID(self, srid=srid).to_expr() | | Отправить события sid в ruleset | def post_sid_events(ruleset_name, sid):
message = json.loads(request.stream.read().decode('utf-8'))
message['sid'] = sid
result = host.post(ruleset_name, message)
return jsonify(result) | def post_events(ruleset_name):
message = json.loads(request.stream.read().decode('utf-8'))
result = host.post(ruleset_name, message)
return jsonify(result) | def set_state_sid_request(ruleset_name, sid):
message = json.loads(request.stream.read().decode('utf-8'))
message['sid'] = sid
result = host.patch_state(ruleset_name, message)
return jsonify(result) | def sid(self, sid):
self._sid = sid | def post(self, request, *args, **kwargs):

id = args[0] if args else list(kwargs.values())[0]
try:
ssn = Subscription.objects.get(id=id)
except Subscription.DoesNotExist:
logger.error(
f'Received unwanted subscription {id} POST request! Sending status '
'410 back to hub.'
)
return Response('Unwanted subscription', status=410)

ssn.update(time_last_event_received=now())
self.handler_task.delay(request.data)
return Response('') # TODO | def informed_consent_on_post_save(sender, instance, raw, created, **kwargs):
if not raw:
if created:
pass
# instance.registration_update_or_create()
# update_model_fields(instance=instance,
# model_cls=['subject_identifier', instance.subject_identifier])
try:
OnSchedule.objects.get(
subject_identifier=instance.subject_identifier, )
except OnSchedule.DoesNotExist:
onschedule_model = 'training_subject.onschedule'
put_on_schedule(schedule_name='training_subject_visit_schedule', instance=instance, onschedule_model=onschedule_model) | def post_event(self, event):
from evennia.scripts.models import ScriptDB

if event.public_event:
event_manager = ScriptDB.objects.get(db_key="Event Manager")
event_manager.post_event(event, self.owner.player, event.display()) | def post(self, event, *args, **kwargs):
self.inq.Signal((event, args, kwargs)) | def post(self, request):
return self.serviceHandler.addEvent(request.data) | def register_to_event(request):
pass | def setFilterOnRule(request):

logger = logging.getLogger(__name__)

# Get some initial post values for processing.
ruleIds = request.POST.getlist('id')
sensors = request.POST.getlist('sensors')
commentString = request.POST['comment']
force = request.POST['force']
response = []

# If the ruleIds list is empty, it means a SID has been entered manually.
if len(ruleIds) == 0:
# Grab the value from the POST.
ruleSID = request.POST['sid']

# Match the GID:SID pattern, if its not there, throw exception.
try:
matchPattern = r"(\d+):(\d+)"
pattern = re.compile(matchPattern)
result = pattern.match(ruleSID)

ruleGID = result.group(1)
ruleSID = result.group(2)
except:
response.append({'response': 'invalidGIDSIDFormat', 'text': 'Please format in the GID:SID syntax.'})
logger.warning("Invalid GID:SID syntax provided: "+str(ruleSID)+".")
return HttpResponse(json.dumps(response))

# Try to find a generator object with the GID supplied, if it does... | def store_event(self, violations):
current_time = datetime.now().strftime("%Y/%m/%d %H:%M:%S")
insert_query = """INSERT INTO social_distancing (Location, Local_Time, Violations) VALUES ('{}', '{}', {})""".format(self.location, current_time, violations)
self.off_chain.insert(insert_query)

event_id = self.off_chain.select("""SELECT LAST_INSERT_ID() FROM social_distancing""")[0][0]
self.on_chain.store_hash(event_id, self.location, current_time, violations) | def test_post_event_on_schedule_page(self):
json_data = {
'title': 'Test Event',
'start': '2017-8-8T12:00:00',
'end': '2017-8-8T12:00:00',
'group': '3'
}

response = self.app.post("/saveEvent", data=json.dumps(json_data),
content_type='application/json')
self.assertTrue(response.status_code, 200) | def _push(self, server):
defns = [self.get_id(ident) for ident in list(self.ids)]
#for ident in list(self.ids):
# defn = self.get_id(ident)
if len(defns) == 0:
return
self.app.logger.info(f"Updating {server} with {len(defns)} records")
url = f"{server}/add_record"
try:
resp = requests.post(url, json=defns)
except Exception as e:
self.app.logger.error(str(e))
return
if not resp.ok:
self.app.logger.error(f"{resp.reason} {resp.content}")
return
self._server_updated[server] = True | def post(self, slug = None, eid = None):
uid = self.request.form.get("uid")
status = self.request.form.get("status") # can be join, maybe, notgoubg
event = self.barcamp.get_event(eid)

user = self.app.module_map.userbase.get_user_by_id(uid)

reg = RegistrationService(self, user)
try:
status = reg.set_status(eid, status, force=True)
except RegistrationError, e:
print "a registration error occurred", e
raise ProcessingError(str(e))
return

return {'status' : 'success', 'reload' : True} | def events(self): | def post(self):

# we need a unique tx number so we can look these back up again
# as well as for logging
# FIXME: how can we guarantee uniqueness here?
tx = int(time.time() * 100000) + random.randrange(10000, 99999)

log.info("EVENTS [{}]: Creating events".format(tx))

try:
user = self.jbody["user"]
if not EMAIL_REGEX.match(user):
user += "@" + self.domain
event_type_id = self.jbody.get("eventTypeId", None)
category = self.jbody.get("category", None)
state = self.jbody.get("state", None)
note = self.jbody.get("note", None)
except KeyError as err:
raise exc.BadRequest(
"Missing Required Argument: {}".format(err.message)
)
except ValueError as err:
raise exc.BadRequest(err.message)

if not event_type_id and (not category and not state):
raise exc.BadRequest(
... | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CachedMultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
cornstack_python_pairs #### cornstack_python_pairs * Dataset: cornstack_python_pairs * Size: 1,434,984 training samples * Columns: en_query, ru_query, and label * Approximate statistics based on the first 1000 samples: | | en_query | ru_query | label | |:--------|:-----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:--------------------------------------------------------------| | type | string | string | float | | details |
  • min: 7 tokens
  • mean: 26.96 tokens
  • max: 150 tokens
|
  • min: 7 tokens
  • mean: 27.46 tokens
  • max: 162 tokens
|
  • min: 1.0
  • mean: 1.0
  • max: 1.0
| * Samples: | en_query | ru_query | label | |:------------------------------------------------------------------|:------------------------------------------------------------------------------------|:-----------------| | set the message data business_id to a specific value | установите значение business_id сообщения данных в конкретное значение | 1.0 | | Set ruleset state sid | Установить состояние правил sid | 1.0 | | Post sid events to the ruleset | Отправить события sid в ruleset | 1.0 | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CoSENTLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
codesearchnet #### codesearchnet * Dataset: [codesearchnet](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) at [3f90200](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en/tree/3f9020072f2e6d5ac5445b39e566e5b669a1661b) * Size: 1,880,853 training samples * Columns: ru_func_documentation_string and func_code_string * Approximate statistics based on the first 1000 samples: | | ru_func_documentation_string | func_code_string | |:--------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| | type | string | string | | details |
  • min: 5 tokens
  • mean: 95.0 tokens
  • max: 619 tokens
|
  • min: 62 tokens
  • mean: 522.56 tokens
  • max: 8192 tokens
| * Samples: | ru_func_documentation_string | func_code_string | |:--------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Мультипроцессинг-целевой объект для устройства очереди zmq | def zmq_device(self):
'''
Multiprocessing target for the zmq queue device
'''
self.__setup_signals()
salt.utils.process.appendproctitle('MWorkerQueue')
self.context = zmq.Context(self.opts['worker_threads'])
# Prepare the zeromq sockets
self.uri = 'tcp://{interface}:{ret_port}'.format(**self.opts)
self.clients = self.context.socket(zmq.ROUTER)
if self.opts['ipv6'] is True and hasattr(zmq, 'IPV4ONLY'):
# IPv6 sockets work for both IPv6 and IPv4 addresses
self.clients.setsockopt(zmq.IPV4ONLY, 0)
self.clients.setsockopt(zmq.BACKLOG, self.opts.get('zmq_backlog', 1000))
self._start_zmq_monitor()
self.workers = self.context.socket(zmq.DEALER)

if self.opts.get('ipc_mode', '') == 'tcp':
self.w_uri = 'tcp://127.0.0.1:{0}'.format(
self.opts.get('tcp_master_workers', 4515)
)
else:
self.w_uri = 'ipc:... | | Чисто завершите работу сокета роутера | def close(self):
'''
Cleanly shutdown the router socket
'''
if self._closing:
return
log.info('MWorkerQueue under PID %s is closing', os.getpid())
self._closing = True
# pylint: disable=E0203
if getattr(self, '_monitor', None) is not None:
self._monitor.stop()
self._monitor = None
if getattr(self, '_w_monitor', None) is not None:
self._w_monitor.stop()
self._w_monitor = None
if hasattr(self, 'clients') and self.clients.closed is False:
self.clients.close()
if hasattr(self, 'workers') and self.workers.closed is False:
self.workers.close()
if hasattr(self, 'stream'):
self.stream.close()
if hasattr(self, '_socket') and self._socket.closed is False:
self._socket.close()
if hasattr(self, 'context') and self.context.closed is False:
self.context.term() | | До форка нам нужно создать устройство zmq роутера

:param func process_manager: Экземпляр класса salt.utils.process.ProcessManager | def pre_fork(self, process_manager):
'''
Pre-fork we need to create the zmq router device

:param func process_manager: An instance of salt.utils.process.ProcessManager
'''
salt.transport.mixins.auth.AESReqServerMixin.pre_fork(self, process_manager)
process_manager.add_process(self.zmq_device) | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CachedMultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
codesearchnet_pairs #### codesearchnet_pairs * Dataset: [codesearchnet_pairs](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) at [3f90200](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en/tree/3f9020072f2e6d5ac5445b39e566e5b669a1661b) * Size: 940,426 training samples * Columns: en_func_documentation_string, ru_func_documentation_string, and label * Approximate statistics based on the first 1000 samples: | | en_func_documentation_string | ru_func_documentation_string | label | |:--------|:-------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:--------------------------------------------------------------| | type | string | string | float | | details |
  • min: 5 tokens
  • mean: 102.69 tokens
  • max: 1485 tokens
|
  • min: 5 tokens
  • mean: 95.0 tokens
  • max: 619 tokens
|
  • min: 1.0
  • mean: 1.0
  • max: 1.0
| * Samples: | en_func_documentation_string | ru_func_documentation_string | label | |:-----------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------| | Multiprocessing target for the zmq queue device | Мультипроцессинг-целевой объект для устройства очереди zmq | 1.0 | | Cleanly shutdown the router socket | Чисто завершите работу сокета роутера | 1.0 | | Pre-fork we need to create the zmq router device

:param func process_manager: An instance of salt.utils.process.ProcessManager | До форка нам нужно создать устройство zmq роутера

:param func process_manager: Экземпляр класса salt.utils.process.ProcessManager | 1.0 | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CoSENTLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
solyanka_qa #### solyanka_qa * Dataset: [solyanka_qa](https://huggingface.co/datasets/ai-forever/solyanka) at [deeac62](https://huggingface.co/datasets/ai-forever/solyanka/tree/deeac621d4142d2754fa28f0eb58502b966383c3) * Size: 85,523 training samples * Columns: anchor and positive * Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details |
  • min: 19 tokens
  • mean: 202.49 tokens
  • max: 518 tokens
|
  • min: 16 tokens
  • mean: 196.36 tokens
  • max: 524 tokens
| * Samples: | anchor | positive | |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Как происходит взаимодействие нескольких языков программирования? Понятно, что большинство (если не все) крупные энтерпрайз сервисы, приложения и тд. (не только веб) написаны с использованием не одного языка программирования, а нескольких. И эти составные части, написанные на разных языках, как-то взаимодействуют между собой (фронт, бизнес-логика, еще что-то).
Опыта разработки подобных систем у меня нет, поэтому не совсем могу представить, как это происходит. Подозреваю, что взаимодействие идет через независимые от языков средства. Например, нечто написанное на одном языке, шлет через TCP-IP пакет, который ловится и обрабатывается чем-то написанным на другом языке. Либо через HTTP запросы. Либо через запись/чтение из БД. Либо через файловый обмен, XML например.
Хотелось бы, чтобы знающие люди привели пару примеров, как это обычно происходит. Не просто в двух словах, мол "фронт на яваскрипте, бэк на яве", а с техническими нюансами. Заранее спасибо. | Несколько языков могут сосуществовать как в рамках одного процесса, так и в рамках нескольких.
Проще всего сосуществовать в рамках нескольких процессов: если процессы обмениваются данными, то совершенно всё равно (ну, в известных рамках), на каком языке эти данные были созданы, и какой язык их читает. Например, вы можете генерировать данные в виде HTML сервером на ASP.NET, а читать браузером, написанным на C++. (Да, пара из сервера и клиента — тоже взаимодействие языков.)
Теперь, если мы хотим взаимодействие в рамках одного процесса, нам нужно уметь вызывать друг друга. Для этого нужен общий стандарт вызова. Часто таким общим стандартом являются бинарные соглашения C (`extern "C"`, экспорт из DLL в Windows).
Ещё пример общего стандарта — COM: COM-объекты можно писать на многих языках, так что если в языке есть часть, реализующая стандарт COM, он может вполне пользоваться им.
Отдельная возможность, популярная сейчас — языки, компилирующиеся в общий промежуточный код. Например, Java и Sc... | | Слэши и ковычки после использования stringify Есть подобный скрипт:
[code]
var output = {
lol: [
{name: "hahaha"}
]
};
console.log(output);
output = JSON.stringify(output);
console.log(output);
[/code]
в итоге получаем
почему он вставил слэши и кавычки там, где не надо? | Может сразу сделать валидный JSON
[code]
var output = {
lol: {name: "hahaha"}
};
console.log(output);
output = JSON.stringify(output);
console.log(output);
[/code]
Правда я незнаю что за переменная `name` | | Оптимизация поиска числа в списке Есть функция. Она принимает число от 1 до 9 (мы ищем, есть ли оно в списке), и список, в котором мы его ищем)
[code]
def is_number_already_in(number, line):
equality = False
for i in line:
if i == number:
equality = True
if equality:
return True
else:
return False
[/code]
Как можно этот код оптимизировать и как называется способ (тема) оптимизации, чтобы я мог загуглить
Только не через лямбду, пожалуйста) | >
[code]
> if equality:
> return True
> else:
> return False
>
[/code]
[code]
return equality
[/code]
>
[code]
> equality = False
> for i in line:
> if i == number:
> equality = True
>
[/code]
[code]
equality = any(i == number for i in line)
[/code]
Всё целиком:
[code]
def is_number_already_in(number, line):
return any(i == number for i in line)
[/code]
Хотя на самом деле вроде бы можно гораздо проще
[code]
def is_number_already_in(number, line):
return number in line
[/code]
PS: Не проверял, но в любом случае идея должна быть понятна. | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CachedMultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
### Evaluation Datasets
codesearchnet #### codesearchnet * Dataset: [codesearchnet](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) at [3f90200](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en/tree/3f9020072f2e6d5ac5445b39e566e5b669a1661b) * Size: 30,000 evaluation samples * Columns: ru_func_documentation_string and func_code_string * Approximate statistics based on the first 1000 samples: | | ru_func_documentation_string | func_code_string | |:--------|:-------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| | type | string | string | | details |
  • min: 6 tokens
  • mean: 194.76 tokens
  • max: 1278 tokens
|
  • min: 58 tokens
  • mean: 580.66 tokens
  • max: 8192 tokens
| * Samples: | ru_func_documentation_string | func_code_string | |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Обучить модель deepq.

Параметры
-------
env: gym.Env
среда для обучения
network: строка или функция
нейронная сеть, используемая в качестве аппроксиматора функции Q. Если строка, она должна быть одной из имен зарегистрированных моделей в baselines.common.models
(mlp, cnn, conv_only). Если функция, она должна принимать тензор наблюдения и возвращать тензор скрытой переменной, которая
будет отображена в головы функции Q (см. build_q_func в baselines.deepq.models для деталей по этому поводу)
seed: int или None
seed генератора случайных чисел. Запуски с одинаковым seed "должны" давать одинаковые результаты. Если None, используется отсутствие семени.
lr: float
скорость обучения для оптимизатора Adam
total_timesteps: int
количество шагов среды для оптимизации
buffer_size: int
размер буфера воспроизведения
exploration_fraction: float
доля всего периода обучения, в течение которого прои... | def learn(env,
network,
seed=None,
lr=5e-4,
total_timesteps=100000,
buffer_size=50000,
exploration_fraction=0.1,
exploration_final_eps=0.02,
train_freq=1,
batch_size=32,
print_freq=100,
checkpoint_freq=10000,
checkpoint_path=None,
learning_starts=1000,
gamma=1.0,
target_network_update_freq=500,
prioritized_replay=False,
prioritized_replay_alpha=0.6,
prioritized_replay_beta0=0.4,
prioritized_replay_beta_iters=None,
prioritized_replay_eps=1e-6,
param_noise=False,
callback=None,
load_path=None,
**network_kwargs
):
"""Train a deepq model.

Parameters
-------
env: gym.Env
environment to train on
network: string or a function
neural network to use as a q function approximator. If string, has to be one of the ... | | Сохранить модель в pickle, расположенный по пути `path` | def save_act(self, path=None):
"""Save model to a pickle located at `path`"""
if path is None:
path = os.path.join(logger.get_dir(), "model.pkl")

with tempfile.TemporaryDirectory() as td:
save_variables(os.path.join(td, "model"))
arc_name = os.path.join(td, "packed.zip")
with zipfile.ZipFile(arc_name, 'w') as zipf:
for root, dirs, files in os.walk(td):
for fname in files:
file_path = os.path.join(root, fname)
if file_path != arc_name:
zipf.write(file_path, os.path.relpath(file_path, td))
with open(arc_name, "rb") as f:
model_data = f.read()
with open(path, "wb") as f:
cloudpickle.dump((model_data, self._act_params), f) | | CNN из статьи Nature. | def nature_cnn(unscaled_images, **conv_kwargs):
"""
CNN from Nature paper.
"""
scaled_images = tf.cast(unscaled_images, tf.float32) / 255.
activ = tf.nn.relu
h = activ(conv(scaled_images, 'c1', nf=32, rf=8, stride=4, init_scale=np.sqrt(2),
**conv_kwargs))
h2 = activ(conv(h, 'c2', nf=64, rf=4, stride=2, init_scale=np.sqrt(2), **conv_kwargs))
h3 = activ(conv(h2, 'c3', nf=64, rf=3, stride=1, init_scale=np.sqrt(2), **conv_kwargs))
h3 = conv_to_fc(h3)
return activ(fc(h3, 'fc1', nh=512, init_scale=np.sqrt(2))) | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CachedMultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
codesearchnet_en #### codesearchnet_en * Dataset: [codesearchnet_en](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) at [3f90200](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en/tree/3f9020072f2e6d5ac5445b39e566e5b669a1661b) * Size: 30,000 evaluation samples * Columns: en_func_documentation_string and func_code_string * Approximate statistics based on the first 1000 samples: | | en_func_documentation_string | func_code_string | |:--------|:-------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------| | type | string | string | | details |
  • min: 6 tokens
  • mean: 200.33 tokens
  • max: 2498 tokens
|
  • min: 58 tokens
  • mean: 580.66 tokens
  • max: 8192 tokens
| * Samples: | en_func_documentation_string | func_code_string | |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Train a deepq model.

Parameters
-------
env: gym.Env
environment to train on
network: string or a function
neural network to use as a q function approximator. If string, has to be one of the names of registered models in baselines.common.models
(mlp, cnn, conv_only). If a function, should take an observation tensor and return a latent variable tensor, which
will be mapped to the Q function heads (see build_q_func in baselines.deepq.models for details on that)
seed: int or None
prng seed. The runs with the same seed "should" give the same results. If None, no seeding is used.
lr: float
learning rate for adam optimizer
total_timesteps: int
number of env steps to optimizer for
buffer_size: int
size of the replay buffer
exploration_fraction: float
fraction of entire training period over which the exploration rate is annealed
exploration_final_eps: float
final value of ra... | def learn(env,
network,
seed=None,
lr=5e-4,
total_timesteps=100000,
buffer_size=50000,
exploration_fraction=0.1,
exploration_final_eps=0.02,
train_freq=1,
batch_size=32,
print_freq=100,
checkpoint_freq=10000,
checkpoint_path=None,
learning_starts=1000,
gamma=1.0,
target_network_update_freq=500,
prioritized_replay=False,
prioritized_replay_alpha=0.6,
prioritized_replay_beta0=0.4,
prioritized_replay_beta_iters=None,
prioritized_replay_eps=1e-6,
param_noise=False,
callback=None,
load_path=None,
**network_kwargs
):
"""Train a deepq model.

Parameters
-------
env: gym.Env
environment to train on
network: string or a function
neural network to use as a q function approximator. If string, has to be one of the ... | | Save model to a pickle located at `path` | def save_act(self, path=None):
"""Save model to a pickle located at `path`"""
if path is None:
path = os.path.join(logger.get_dir(), "model.pkl")

with tempfile.TemporaryDirectory() as td:
save_variables(os.path.join(td, "model"))
arc_name = os.path.join(td, "packed.zip")
with zipfile.ZipFile(arc_name, 'w') as zipf:
for root, dirs, files in os.walk(td):
for fname in files:
file_path = os.path.join(root, fname)
if file_path != arc_name:
zipf.write(file_path, os.path.relpath(file_path, td))
with open(arc_name, "rb") as f:
model_data = f.read()
with open(path, "wb") as f:
cloudpickle.dump((model_data, self._act_params), f) | | CNN from Nature paper. | def nature_cnn(unscaled_images, **conv_kwargs):
"""
CNN from Nature paper.
"""
scaled_images = tf.cast(unscaled_images, tf.float32) / 255.
activ = tf.nn.relu
h = activ(conv(scaled_images, 'c1', nf=32, rf=8, stride=4, init_scale=np.sqrt(2),
**conv_kwargs))
h2 = activ(conv(h, 'c2', nf=64, rf=4, stride=2, init_scale=np.sqrt(2), **conv_kwargs))
h3 = activ(conv(h2, 'c3', nf=64, rf=3, stride=1, init_scale=np.sqrt(2), **conv_kwargs))
h3 = conv_to_fc(h3)
return activ(fc(h3, 'fc1', nh=512, init_scale=np.sqrt(2))) | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CachedMultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
codesearchnet_pairs #### codesearchnet_pairs * Dataset: [codesearchnet_pairs](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en) at [3f90200](https://huggingface.co/datasets/fyaronskiy/code_search_net_ru_en/tree/3f9020072f2e6d5ac5445b39e566e5b669a1661b) * Size: 30,000 evaluation samples * Columns: en_func_documentation_string, ru_func_documentation_string, and label * Approximate statistics based on the first 1000 samples: | | en_func_documentation_string | ru_func_documentation_string | label | |:--------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:--------------------------------------------------------------| | type | string | string | float | | details |
  • min: 6 tokens
  • mean: 200.33 tokens
  • max: 2498 tokens
|
  • min: 6 tokens
  • mean: 194.76 tokens
  • max: 1278 tokens
|
  • min: 1.0
  • mean: 1.0
  • max: 1.0
| * Samples: | en_func_documentation_string | ru_func_documentation_string | label | |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-----------------| | Train a deepq model.

Parameters
-------
env: gym.Env
environment to train on
network: string or a function
neural network to use as a q function approximator. If string, has to be one of the names of registered models in baselines.common.models
(mlp, cnn, conv_only). If a function, should take an observation tensor and return a latent variable tensor, which
will be mapped to the Q function heads (see build_q_func in baselines.deepq.models for details on that)
seed: int or None
prng seed. The runs with the same seed "should" give the same results. If None, no seeding is used.
lr: float
learning rate for adam optimizer
total_timesteps: int
number of env steps to optimizer for
buffer_size: int
size of the replay buffer
exploration_fraction: float
fraction of entire training period over which the exploration rate is annealed
exploration_final_eps: float
final value of ra... | Обучить модель deepq.

Параметры
-------
env: gym.Env
среда для обучения
network: строка или функция
нейронная сеть, используемая в качестве аппроксиматора функции Q. Если строка, она должна быть одной из имен зарегистрированных моделей в baselines.common.models
(mlp, cnn, conv_only). Если функция, она должна принимать тензор наблюдения и возвращать тензор скрытой переменной, которая
будет отображена в головы функции Q (см. build_q_func в baselines.deepq.models для деталей по этому поводу)
seed: int или None
seed генератора случайных чисел. Запуски с одинаковым seed "должны" давать одинаковые результаты. Если None, используется отсутствие семени.
lr: float
скорость обучения для оптимизатора Adam
total_timesteps: int
количество шагов среды для оптимизации
buffer_size: int
размер буфера воспроизведения
exploration_fraction: float
доля всего периода обучения, в течение которого прои... | 1.0 | | Save model to a pickle located at `path` | Сохранить модель в pickle, расположенный по пути `path` | 1.0 | | CNN from Nature paper. | CNN из статьи Nature. | 1.0 | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CoSENTLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
solyanka_qa #### solyanka_qa * Dataset: [solyanka_qa](https://huggingface.co/datasets/ai-forever/solyanka) at [deeac62](https://huggingface.co/datasets/ai-forever/solyanka/tree/deeac621d4142d2754fa28f0eb58502b966383c3) * Size: 5,000 evaluation samples * Columns: anchor and positive * Approximate statistics based on the first 1000 samples: | | anchor | positive | |:--------|:-------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| | type | string | string | | details |
  • min: 17 tokens
  • mean: 200.35 tokens
  • max: 533 tokens
|
  • min: 19 tokens
  • mean: 202.53 tokens
  • max: 525 tokens
| * Samples: | anchor | positive | |:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Atom IDE произвольное изменение строк Пользуюсь Atom IDE, установлены плагины для GIT'а, использую тему Material theme (может быть кому то это что то даст), в общем проблема такая, что в php файлах при сохранении файла, даже если я изменил всего один символ, он добавляет изменения очень странные,берет 2-3 строки (хз как выбирает) и удаляет их, а потом вставялет их же, без каких то либо изменений. При этом GIT фиксирует это изменение...
Вот скрин в blob формате: "blob:https://web.telegram.org/04094604-204d-47b0-a083-f8cd090bdfa0" | Проблема заключалась в том, что все IDE испльзуют свой символ перехода на следующую строку, если в команде разработчики используют разные IDE, у которых разный перенос строки, то при сохранении файла чужие переносы строк будут заменяться на свои :) | | print() с частью текста и форматированием как переменная Python3 Есть повторяющаяся функция `print('\n' + f'{" ЗАПУСКАЕМ ТЕСТ ":=^120}' + '\n')`
на выходе получаем чтото типа
================ ЗАПУСКАЕМ ТЕСТ ================
или с другим текстом
================= КОНЕЦ ТЕСТА ==================
Текст внутри может меняться, форматирование - нет.
Как обернуть `print('\n' + f'{"":=^120}' + '\n')` в переменную, с возможностью подставлять нужный текст, типа `print_var('ПРИМЕР ТЕКСТА')`? | [code]
def print_var(str):
print(f'\n{" " + str + " ":=^120}\n')
[/code]
В результате:
[code]
>>> print_var('КАКОЙ_ТО ТЕКСТ')
===================================================== КАКОЙ_ТО ТЕКСТ =====================================================
[/code] | | Не получается перегрузить оператор присваивания в шаблонном классе Нужно перегрузить оператор присваивания в шаблонном классе, не могу понять, почему не работает стандартный синтаксис, при реализации выдает эту ошибку (/home/anton/Programming/tree/tree.h:96: ошибка: overloaded 'operator=' must be a binary operator (has 1 parameter)). Объявление и реализация в одном .h файле.
Объявление:
[code]
tree& operator = (tree &other);
[/code]
реалицация:
[code]
template
tree& operator = (tree &other)
{
}
[/code] | Ну надо указать, какому классу он принадлежит... А так вы пытались реализовать унарный оператор `=`...
[code]
template
tree& tree::operator = (tree &other)
{
}
[/code]
И еще - вы точно планируете при присваивании менять присваиваемое? Может, лучше
[code]
template
tree& tree::operator = (const tree &other)
{
}
[/code] | * Loss: [MatryoshkaLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: ```json { "loss": "CachedMultipleNegativesRankingLoss", "matryoshka_dims": [ 768, 512, 256, 128, 64 ], "matryoshka_weights": [ 1, 1, 1, 1, 1 ], "n_dims_per_step": -1 } ```
### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 4 - `per_device_eval_batch_size`: 16 - `gradient_accumulation_steps`: 32 - `learning_rate`: 2e-05 - `num_train_epochs`: 2 - `warmup_ratio`: 0.1 - `bf16`: True - `resume_from_checkpoint`: ../models/RuModernBERT-base_bs128_lr_2e-05_2nd_epoch/checkpoint-27400 - `auto_find_batch_size`: True - `batch_sampler`: no_duplicates #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 4 - `per_device_eval_batch_size`: 16 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 32 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 2 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 42 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 0 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: False - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: ../models/RuModernBERT-base_bs128_lr_2e-05_2nd_epoch/checkpoint-27400 - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: True - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: no_duplicates - `multi_dataset_batch_sampler`: proportional - `router_mapping`: {} - `learning_rate_mapping`: {}
### Framework Versions - Python: 3.10.11 - Sentence Transformers: 5.1.2 - Transformers: 4.52.3 - PyTorch: 2.6.0+cu124 - Accelerate: 1.12.0 - Datasets: 4.0.0 - Tokenizers: 0.21.4 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### MatryoshkaLoss ```bibtex @misc{kusupati2024matryoshka, title={Matryoshka Representation Learning}, author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, year={2024}, eprint={2205.13147}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### CachedMultipleNegativesRankingLoss ```bibtex @misc{gao2021scaling, title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan}, year={2021}, eprint={2101.06983}, archivePrefix={arXiv}, primaryClass={cs.LG} } ``` #### CoSENTLoss ```bibtex @article{10531646, author={Huang, Xiang and Peng, Hao and Zou, Dongcheng and Liu, Zhiwei and Li, Jianxin and Liu, Kay and Wu, Jia and Su, Jianlin and Yu, Philip S.}, journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing}, title={CoSENT: Consistent Sentence Embedding via Similarity Ranking}, year={2024}, doi={10.1109/TASLP.2024.3402087} } ```