{"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the internal colors of the entry set.", "response": "def set_color_list(self, color_list, offset=0):\n \"\"\"\n Set the internal colors starting at an optional offset.\n\n If `color_list` is a list or other 1-dimensional array, it is reshaped\n into an N x 3 list.\n\n If `color_list` too long it is truncated; if it is too short then only\n the initial colors are set.\n \"\"\"\n if not len(color_list):\n return\n color_list = make.colors(color_list)\n\n size = len(self._colors) - offset\n if len(color_list) > size:\n color_list = color_list[:size]\n self._colors[offset:offset + len(color_list)] = color_list"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setRGB(self, pixel, r, g, b):\n self._set_base(pixel, (r, g, b))", "response": "Set single pixel using individual RGB values instead of tuple"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset single pixel to HSV tuple", "response": "def setHSV(self, pixel, hsv):\n \"\"\"Set single pixel to HSV tuple\"\"\"\n color = conversions.hsv2rgb(hsv)\n self._set_base(pixel, color)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fill(self, color, start=0, end=-1):\n start = max(start, 0)\n if end < 0 or end >= self.numLEDs:\n end = self.numLEDs - 1\n for led in range(start, end + 1): # since 0-index include end in range\n self._set_base(led, color)", "response": "Fill the entire strip with RGB color tuple"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fillRGB(self, r, g, b, start=0, end=-1):\n self.fill((r, g, b), start, end)", "response": "Fill the strip by giving individual RGB values instead of tuple"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfill the entire strip with HSV color tuple", "response": "def fillHSV(self, hsv, start=0, end=-1):\n \"\"\"Fill the entire strip with HSV color tuple\"\"\"\n self.fill(conversions.hsv2rgb(hsv), start, end)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nset the color list for the current locale.", "response": "def set_colors(self, buf):\n \"\"\"\n DEPRECATED: use self.color_list\n\n Use with extreme caution!\n Directly sets the internal buffer and bypasses all brightness and\n rotation control buf must also be in the exact format required by the\n display type.\n \"\"\"\n deprecated.deprecated('layout.set_colors')\n if len(self._colors) != len(buf):\n raise IOError(\"Data buffer size incorrect! \"\n \"Expected: {} bytes / Received: {} bytes\"\n .format(len(self._colors), len(buf)))\n self._colors[:] = buf"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef wheel_helper(pos, length, cycle_step):\n return wheel_color((pos * len(_WHEEL) / length) + cycle_step)", "response": "Helper for wheel_color that distributes colors over length and\n allows shifting position."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef multi(method):\n @functools.wraps(method)\n def multi(self, address=''):\n values = flask.request.values\n address = urllib.parse.unquote_plus(address)\n if address and values and not address.endswith('.'):\n address += '.'\n\n result = {}\n for a in values or '':\n try:\n if not self.project:\n raise ValueError('No Project is currently loaded')\n\n ed = editor.Editor(address + a, self.project)\n result[address + a] = {'value': method(self, ed, a)}\n except:\n if self.project:\n traceback.print_exc()\n result[address + a] = {'error': 'Could not multi addr %s' % a}\n\n return flask.jsonify(result)\n\n return multi", "response": "Decorator for RestServer methods that take multiple addresses"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef advance_permutation(a, increasing=True, forward=True):\n\n if not forward:\n a.reverse()\n\n cmp = operator.lt if increasing else operator.gt\n try:\n i = next(i for i in reversed(range(len(a) - 1)) if cmp(a[i], a[i + 1]))\n j = next(j for j in reversed(range(i + 1, len(a))) if cmp(a[i], a[j]))\n except StopIteration:\n # This is the lexicographically last permutation.\n if forward:\n a.reverse()\n return False\n\n a[i], a[j] = a[j], a[i]\n a[i + 1:] = reversed(a[i + 1:])\n if not forward:\n a.reverse()\n\n return True", "response": "Advance a list of unique ordered elements in - place in - place increasing or backward by rightmost digit."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _on_index(self, old_index):\n if self.animation:\n log.debug('%s: %s',\n self.__class__.__name__, self.current_animation.title)\n self.frames = self.animation.generate_frames(False)", "response": "Called when the index of the record is changed."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef animation(self):\n if 0 <= self._index < len(self.animations):\n return self.animations[self._index]", "response": "returns the animation based on self. index"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef apply(self, function):\n for cut in self.cuts:\n value = self.read(cut)\n function(value)\n self.write(cut, value)", "response": "Apply a function to each of the cuts in the current layout."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef setRGB(self, pixel, r, g, b):\n self.set(pixel, (r, g, b))", "response": "Set single pixel using individual RGB values instead of tuple"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setHSV(self, pixel, hsv):\n color = conversions.hsv2rgb(hsv)\n self.set(pixel, color)", "response": "Set single pixel to HSV tuple"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef compose_events(events, condition=all):\n events = list(events)\n master_event = threading.Event()\n\n def changed():\n if condition(e.is_set() for e in events):\n master_event.set()\n else:\n master_event.clear()\n\n def add_changed(f):\n @functools.wraps(f)\n def wrapped():\n f()\n changed()\n\n return wrapped\n\n for e in events:\n e.set = add_changed(e.set)\n e.clear = add_changed(e.clear)\n\n changed()\n return master_event", "response": "Compose a sequence of events into one event."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nadding redundant arguments to the parser.", "response": "def _add_redundant_arguments(parser):\n \"\"\"\n These arguments are redundant with just using a project, and we should\n encouraging that as you don't have to learn any dumb flags!\n\n For example, instead of\n\n bp foo.yml --animation=wombat --numbers=float\n\n use\n\n bp foo.yml + '{animation: wombat, numbers: float}'\n\n \"\"\"\n parser.add_argument(\n '-a', '--animation', default=None,\n help='Default animation type if no animation is specified')\n\n if deprecated.allowed(): # pragma: no cover\n parser.add_argument(\n '--dimensions', '--dim', default=None,\n help='DEPRECATED: x, (x, y) or (x, y, z) dimensions for project')\n\n parser.add_argument(\n '--shape', default=None,\n help='x, (x, y) or (x, y, z) dimensions for project')\n\n parser.add_argument(\n '-l', '--layout', default=None,\n help='Default layout class if no layout is specified')\n\n parser.add_argument(\n '--numbers', '-n', default='python', choices=NUMBER_TYPES,\n help=NUMBERS_HELP)\n\n parser.add_argument('-p', '--path', default=None, help=PATH_HELP)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndrawing a circle at point x0 y0 with radius r of the specified RGB color", "response": "def draw_circle(setter, x0, y0, r, color=None):\n \"\"\"\n Draws a circle at point x0, y0 with radius r of the specified RGB color\n \"\"\"\n f = 1 - r\n ddF_x = 1\n ddF_y = -2 * r\n x = 0\n y = r\n\n setter(x0, y0 + r, color)\n setter(x0, y0 - r, color)\n setter(x0 + r, y0, color)\n setter(x0 - r, y0, color)\n\n while x < y:\n if f >= 0:\n y -= 1\n ddF_y += 2\n f += ddF_y\n x += 1\n ddF_x += 2\n f += ddF_x\n\n setter(x0 + x, y0 + y, color)\n setter(x0 - x, y0 + y, color)\n setter(x0 + x, y0 - y, color)\n setter(x0 - x, y0 - y, color)\n setter(x0 + y, y0 + x, color)\n setter(x0 - y, y0 + x, color)\n setter(x0 + y, y0 - x, color)\n setter(x0 - y, y0 - x, color)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fill_circle(setter, x0, y0, r, color=None):\n _draw_fast_vline(setter, x0, y0 - r, 2 * r + 1, color)\n _fill_circle_helper(setter, x0, y0, r, 3, 0, color)", "response": "Draws a filled circle at point x0 y0 with radius r and specified color"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndrawing a line from point x0 y0 to x1 y1. Will draw beyond matrix bounds.", "response": "def bresenham_line(setter, x0, y0, x1, y1, color=None, colorFunc=None):\n \"\"\"Draw line from point x0,y0 to x,1,y1. Will draw beyond matrix bounds.\"\"\"\n steep = abs(y1 - y0) > abs(x1 - x0)\n if steep:\n x0, y0 = y0, x0\n x1, y1 = y1, x1\n\n if x0 > x1:\n x0, x1 = x1, x0\n y0, y1 = y1, y0\n\n dx = x1 - x0\n dy = abs(y1 - y0)\n\n err = dx / 2\n\n if y0 < y1:\n ystep = 1\n else:\n ystep = -1\n\n count = 0\n for x in range(x0, x1 + 1):\n if colorFunc:\n color = colorFunc(count)\n count += 1\n\n if steep:\n setter(y0, x, color)\n else:\n setter(x, y0, color)\n\n err -= dy\n if err < 0:\n y0 += ystep\n err += dx"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndraws a rectangle at x y w and h.", "response": "def draw_rect(setter, x, y, w, h, color=None, aa=False):\n \"\"\"Draw rectangle with top-left corner at x,y, width w and height h\"\"\"\n _draw_fast_hline(setter, x, y, w, color, aa)\n _draw_fast_hline(setter, x, y + h - 1, w, color, aa)\n _draw_fast_vline(setter, x, y, h, color, aa)\n _draw_fast_vline(setter, x + w - 1, y, h, color, aa)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fill_rect(setter, x, y, w, h, color=None, aa=False):\n for i in range(x, x + w):\n _draw_fast_vline(setter, i, y, h, color, aa)", "response": "Draw a solid rectangle with top - left corner at x y w and h."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndraws a round rectangle at x y w h with radius r.", "response": "def draw_round_rect(setter, x, y, w, h, r, color=None, aa=False):\n \"\"\"Draw rectangle with top-left corner at x,y, width w, height h,\n and corner radius r.\n \"\"\"\n _draw_fast_hline(setter, x + r, y, w - 2 * r, color, aa) # Top\n _draw_fast_hline(setter, x + r, y + h - 1, w - 2 * r, color, aa) # Bottom\n _draw_fast_vline(setter, x, y + r, h - 2 * r, color, aa) # Left\n _draw_fast_vline(setter, x + w - 1, y + r, h - 2 * r, color, aa) # Right\n # draw four corners\n _draw_circle_helper(setter, x + r, y + r, r, 1, color, aa)\n _draw_circle_helper(setter, x + w - r - 1, y + r, r, 2, color, aa)\n _draw_circle_helper(setter, x + w - r - 1, y + h - r - 1, r, 4, color, aa)\n _draw_circle_helper(setter, x + r, y + h - r - 1, r, 8, color, aa)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndraw a round rectangle with top - left corner at x y width w height h and corner radius r", "response": "def fill_round_rect(setter, x, y, w, h, r, color=None, aa=False):\n \"\"\"Draw solid rectangle with top-left corner at x,y, width w, height h,\n and corner radius r\"\"\"\n fill_rect(setter, x + r, y, w - 2 * r, h, color, aa)\n _fill_circle_helper(setter, x + w - r - 1, y + r, r,\n 1, h - 2 * r - 1, color, aa)\n _fill_circle_helper(setter, x + r, y + r, r, 2, h - 2 * r - 1, color, aa)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndraw a triangle with points x0 y0 x1 y1 x2 y2", "response": "def draw_triangle(setter, x0, y0, x1, y1, x2, y2, color=None, aa=False):\n \"\"\"Draw triangle with points x0,y0 - x1,y1 - x2,y2\"\"\"\n draw_line(setter, x0, y0, x1, y1, color, aa)\n draw_line(setter, x1, y1, x2, y2, color, aa)\n draw_line(setter, x2, y2, x0, y0, color, aa)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndrawing a solid triangle with points x0 y0 x1 y1 x2 y2", "response": "def fill_triangle(setter, x0, y0, x1, y1, x2, y2, color=None, aa=False):\n \"\"\"Draw solid triangle with points x0,y0 - x1,y1 - x2,y2\"\"\"\n a = b = y = last = 0\n\n if y0 > y1:\n y0, y1 = y1, y0\n x0, x1 = x1, x0\n if y1 > y2:\n y2, y1 = y1, y2\n x2, x1 = x1, x2\n if y0 > y1:\n y0, y1 = y1, y0\n x0, x1 = x1, x0\n\n if y0 == y2: # Handle awkward all-on-same-line case as its own thing\n a = b = x0\n if x1 < a:\n a = x1\n elif x1 > b:\n b = x1\n if x2 < a:\n a = x2\n elif x2 > b:\n b = x2\n _draw_fast_hline(setter, a, y0, b - a + 1, color, aa)\n\n dx01 = x1 - x0\n dy01 = y1 - y0\n dx02 = x2 - x0\n dy02 = y2 - y0\n dx12 = x2 - x1\n dy12 = y2 - y1\n sa = 0\n sb = 0\n\n # For upper part of triangle, find scanline crossings for segments\n # 0-1 and 0-2. If y1=y2 (flat-bottomed triangle), the scanline y1\n # is included here (and second loop will be skipped, avoiding a /0\n # error there), otherwise scanline y1 is skipped here and handled\n # in the second loop...which also avoids a /0 error here if y0=y1\n # (flat-topped triangle).\n\n if y1 == y2:\n last = y1 # include y1 scanline\n else:\n last = y1 - 1 # skip it\n\n for y in range(y, last + 1):\n a = x0 + sa / dy01\n b = x0 + sb / dy02\n sa += dx01\n sb += dx02\n\n if a > b:\n a, b = b, a\n _draw_fast_hline(setter, a, y, b - a + 1, color, aa)\n\n # For lower part of triangle, find scanline crossings for segments\n # 0-2 and 1-2. This loop is skipped if y1=y2.\n sa = dx12 * (y - y1)\n sb = dx02 * (y - y0)\n\n for y in range(y, y2 + 1):\n a = x1 + sa / dy12\n b = x0 + sb / dy02\n sa += dx12\n sb += dx02\n\n if a > b:\n a, b = b, a\n _draw_fast_hline(setter, a, y, b - a + 1, color, aa)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_colors(self, colors, pos):\n self._colors = colors\n self._pos = pos\n\n end = self._pos + self.numLEDs\n if end > len(self._colors):\n raise ValueError('Needed %d colors but found %d' % (\n end, len(self._colors)))", "response": "Set the color buffers for the current item."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update_colors(self):\n start = self.clock.time()\n\n with self.brightness_lock:\n # Swap in a new brightness.\n brightness, self._waiting_brightness = (\n self._waiting_brightness, None)\n\n if brightness is not None:\n self._brightness = brightness\n if self.set_device_brightness:\n self.set_device_brightness(brightness)\n\n self._compute_packet()\n self._send_packet()\n\n self.lastUpdate = self.clock.time() - start", "response": "Update the color list with any corrections applied to the current color list and send the results to the driver output."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _render(self):\n if self.set_device_brightness:\n level = 1.0\n else:\n level = self._brightness / 255.0\n gam, (r, g, b) = self.gamma.get, self.c_order\n for i in range(min(self.numLEDs, len(self._buf) / 3)):\n c = [int(level * x) for x in self._colors[i + self._pos]]\n self._buf[i * 3:(i + 1) * 3] = gam(c[r]), gam(c[g]), gam(c[b])", "response": "This method is called by the _compute_packet method to render the image of the current color set."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pointOnCircle(cx, cy, radius, angle):\n angle = math.radians(angle) - (math.pi / 2)\n x = cx + radius * math.cos(angle)\n if x < cx:\n x = math.ceil(x)\n else:\n x = math.floor(x)\n\n y = cy + radius * math.sin(angle)\n\n if y < cy:\n y = math.ceil(y)\n else:\n y = math.floor(y)\n\n return (int(x), int(y))", "response": "Calculates the coordinates of a point on a circle given the center point radius and angle."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef genVector(width, height, x_mult=1, y_mult=1):\n center_x = (width - 1) / 2\n center_y = (height - 1) / 2\n\n def length(x, y):\n dx = math.pow(x - center_x, 2 * x_mult)\n dy = math.pow(y - center_y, 2 * y_mult)\n return int(math.sqrt(dx + dy))\n\n return [[length(x, y) for x in range(width)] for y in range(height)]", "response": "Generates a map of vector lengths from the center point to each coordinate."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn an iteration over all named colors in tables", "response": "def all_named_colors():\n \"\"\"Return an iteration over all name, color pairs in tables\"\"\"\n yield from _TO_COLOR_USER.items()\n for name, color in _TO_COLOR.items():\n if name not in _TO_COLOR_USER:\n yield name, color"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning true if this string or integer tuple appears in tables", "response": "def contains(x):\n \"\"\"Return true if this string or integer tuple appears in tables\"\"\"\n if isinstance(x, str):\n x = canonical_name(x)\n return x in _TO_COLOR_USER or x in _TO_COLOR\n else:\n x = tuple(x)\n return x in _TO_NAME_USER or x in _TO_NAME"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of Segments that evenly split the strip.", "response": "def make_segments(strip, length):\n \"\"\"Return a list of Segments that evenly split the strip.\"\"\"\n if len(strip) % length:\n raise ValueError('The length of strip must be a multiple of length')\n\n s = []\n try:\n while True:\n s.append(s[-1].next(length) if s else Segment(strip, length))\n except ValueError:\n return s"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef next(self, length):\n return Segment(self.strip, length, self.offset + self.length)", "response": "Return a new segment starting right after self in the same buffer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sender(address, use_queue=True, **kwds):\n return QueuedSender(address, **kwds) if use_queue else Sender(address)", "response": "create a sender for the given address"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef start(self, threaded=None):\n if threaded is not None:\n self.threaded = threaded\n run = {'run': {'threaded': False}}\n self.project = project.project(\n self.desc, run, root_file=self.project_file)\n self._run = self.project.run\n self._runner.start(self.threaded)", "response": "Creates and starts the project."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef stop(self=None):\n if not self:\n instance = getattr(Runner.instance(), 'builder', None)\n self = instance and instance()\n if not self:\n return\n\n self._runner.stop()\n if self.project:\n self.project.stop()\n self.project = None", "response": "Stop the builder if it s running."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nopen an instance of simpixel in the browser", "response": "def simpixel(new=0, autoraise=True):\n \"\"\"Open an instance of simpixel in the browser\"\"\"\n simpixel_driver.open_browser(new=new, autoraise=autoraise)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef recurse(desc, pre='pre_recursion', post=None, python_path=None):\n def call(f, desc):\n if isinstance(f, str):\n # f is the name of a static class method on the datatype.\n f = getattr(datatype, f, None)\n return f and f(desc)\n\n # Automatically load strings that look like JSON or Yaml filenames.\n desc = load.load_if_filename(desc) or desc\n\n desc = construct.to_type_constructor(desc, python_path)\n datatype = desc.get('datatype')\n\n desc = call(pre, desc) or desc\n\n for child_name in getattr(datatype, 'CHILDREN', []):\n child = desc.get(child_name)\n if child:\n is_plural = child_name.endswith('s')\n remove_s = is_plural and child_name != 'drivers'\n # This is because it's the \"drivers\" directory, whereas\n # the others are animation, control, layout, project\n # without the s. TODO: rename drivers/ to driver/ in v4\n\n cname = child_name[:-1] if remove_s else child_name\n new_path = python_path or ('bibliopixel.' + cname)\n if is_plural:\n if isinstance(child, (dict, str)):\n child = [child]\n for i, c in enumerate(child):\n child[i] = recurse(c, pre, post, new_path)\n desc[child_name] = child\n else:\n desc[child_name] = recurse(child, pre, post, new_path)\n\n d = call(post, desc)\n return desc if d is None else d", "response": "Recursive function that recursively walks the dictionary and returns a new tree of nodes."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the classname of the class c in the order of the name", "response": "def class_name(c):\n \"\"\"\n :param c: either an object or a class\n :return: the classname as a string\n \"\"\"\n if not isinstance(c, type):\n c = type(c)\n\n return '%s.%s' % (c.__module__, c.__name__)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_type_constructor(value, python_path=None):\n if not value:\n return value\n\n if callable(value):\n return {'datatype': value}\n\n value = to_type(value)\n typename = value.get('typename')\n if typename:\n r = aliases.resolve(typename)\n try:\n value['datatype'] = importer.import_symbol(\n r, python_path=python_path)\n del value['typename']\n except Exception as e:\n value['_exception'] = e\n\n return value", "response": "Tries to convert a value to a type constructor."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfilling a portion of a strip with a given item.", "response": "def fill(strip, item, start=0, stop=None, step=1):\n \"\"\"Fill a portion of a strip from start to stop by step with a given item.\n If stop is not given, it defaults to the length of the strip.\n \"\"\"\n if stop is None:\n stop = len(strip)\n\n for i in range(start, stop, step):\n strip[i] = item"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsquaring of the euclidean distance between two sequences.", "response": "def euclidean(c1, c2):\n \"\"\"Square of the euclidean distance\"\"\"\n diffs = ((i - j) for i, j in zip(c1, c2))\n return sum(x * x for x in diffs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting a series of frames as a single animated GIF.", "response": "def _write(self, filename, frames, fps, loop=0, palette=256):\n \"\"\"\n Write a series of frames as a single animated GIF.\n\n :param str filename: the name of the GIF file to write\n\n :param list frames: a list of filenames, each of which represents a single\n frame of the animation. Each frame must have exactly the same\n dimensions, and the code has only been tested with .gif files.\n\n :param float fps:\n The number of frames per second.\n\n :param int loop:\n The number of iterations. Default 0 (meaning loop indefinitely).\n\n :param int palette:\n The number of colors to quantize the image to. Is rounded to\n the nearest power of two. Default 256.\n \"\"\"\n\n from PIL import Image\n images = []\n for f in frames:\n data = open(f, 'rb').read()\n images.append(Image.open(io.BytesIO(data)))\n\n # GIF duration is only measured to a hundredth of a second\n duration = round(1 / fps, 2)\n im = images.pop(0)\n im.save(filename,\n save_all=True,\n append_images=images,\n duration=duration,\n loop=loop,\n palette=palette)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndump a dictionary into a nicely formatted JSON string.", "response": "def dumps(data, use_yaml=None, safe=True, **kwds):\n \"\"\"\n Dumps data into a nicely formatted JSON string.\n\n :param dict data: a dictionary to dump\n :param kwds: keywords to pass to json.dumps\n :returns: a string with formatted data\n :rtype: str\n \"\"\"\n if use_yaml is None:\n use_yaml = ALWAYS_DUMP_YAML\n\n if use_yaml:\n dumps = yaml.safe_dump if safe else yaml.dump\n else:\n dumps = json.dumps\n kwds.update(indent=4, sort_keys=True)\n if not safe:\n kwds.update(default=repr)\n return dumps(data, **kwds)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndump data as nicely formatted JSON string to a file or handle.", "response": "def dump(data, file=sys.stdout, use_yaml=None, **kwds):\n \"\"\"\n Dumps data as nicely formatted JSON string to a file or file handle\n\n :param dict data: a dictionary to dump\n :param file: a filename or file handle to write to\n :param kwds: keywords to pass to json.dump\n \"\"\"\n if use_yaml is None:\n use_yaml = ALWAYS_DUMP_YAML\n\n def dump(fp):\n if use_yaml:\n yaml.safe_dump(data, stream=fp, **kwds)\n else:\n json.dump(data, fp, indent=4, sort_keys=True, **kwds)\n\n if not isinstance(file, str):\n return dump(file)\n\n if os.path.isabs(file):\n parent = os.path.dirname(file)\n if not os.path.exists(parent):\n os.makedirs(parent, exist_ok=True)\n\n with open(file, 'w') as fp:\n return dump(fp)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load(file, use_yaml=None):\n if isinstance(file, str):\n fp = open(file)\n filename = file\n else:\n fp = file\n filename = getattr(fp, 'name', '')\n\n try:\n return loads(fp.read(), use_yaml, filename)\n\n except Exception as e:\n e.args = ('There was a error in the data file', filename) + e.args\n raise", "response": "Loads the data from a file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load_if(s):\n is_data_file = s.endswith('.json') or s.endswith('.yml')\n return load(s) if is_data_file else loads(s)", "response": "Load either a filename or a string representation of yml / json."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef adapt_animation_layout(animation):\n layout = animation.layout\n required = getattr(animation, 'LAYOUT_CLASS', None)\n\n if not required or isinstance(layout, required):\n return\n\n msg = LAYOUT_WARNING % (\n type(animation).__name__, required.__name__, type(layout).__name__)\n\n setter = layout.set\n adaptor = None\n\n if required is strip.Strip:\n if isinstance(layout, matrix.Matrix):\n width = layout.width\n\n def adaptor(pixel, color=None):\n y, x = divmod(pixel, width)\n setter(x, y, color or BLACK)\n\n elif isinstance(layout, cube.Cube):\n lx, ly = layout.x, layout.y\n\n def adaptor(pixel, color=None):\n yz, x = divmod(pixel, lx)\n z, y = divmod(yz, ly)\n setter(x, y, z, color or BLACK)\n\n elif isinstance(layout, circle.Circle):\n\n def adaptor(pixel, color=None):\n layout._set_base(pixel, color or BLACK)\n\n elif required is matrix.Matrix:\n if isinstance(layout, strip.Strip):\n width = animation.width\n\n def adaptor(x, y, color=None):\n setter(x + y * width, color or BLACK)\n\n if not adaptor:\n raise ValueError(msg)\n\n log.warning(msg)\n animation.layout.set = adaptor", "response": "Adapt the setter in an animation s layout so that strips can run on a Strip or Matrix or Cube or Circle layout."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert an HSV tuple to RGB. Intended for internal use.", "response": "def hsv2rgb_raw(hsv):\n \"\"\"\n Converts an HSV tuple to RGB. Intended for internal use.\n You should use hsv2rgb_spectrum or hsv2rgb_rainbow instead.\n \"\"\"\n\n HSV_SECTION_3 = 0x40\n\n h, s, v = hsv\n\n # The brightness floor is minimum number that all of\n # R, G, and B will be set to.\n invsat = 255 - s\n brightness_floor = (v * invsat) // 256\n\n # The color amplitude is the maximum amount of R, G, and B\n # that will be added on top of the brightness_floor to\n # create the specific hue desired.\n color_amplitude = v - brightness_floor\n\n # figure out which section of the hue wheel we're in,\n # and how far offset we are within that section\n section = h // HSV_SECTION_3 # 0..2\n offset = h % HSV_SECTION_3 # 0..63\n\n rampup = offset\n rampdown = (HSV_SECTION_3 - 1) - offset\n\n # compute color-amplitude-scaled-down versions of rampup and rampdown\n rampup_amp_adj = (rampup * color_amplitude) // (256 // 4)\n rampdown_amp_adj = (rampdown * color_amplitude) // (256 // 4)\n\n # add brightness_floor offset to everything\n rampup_adj_with_floor = rampup_amp_adj + brightness_floor\n rampdown_adj_with_floor = rampdown_amp_adj + brightness_floor\n\n r, g, b = (0, 0, 0)\n\n if section:\n if section == 1:\n # section 1: 0x40..0x7F\n r = brightness_floor\n g = rampdown_adj_with_floor\n b = rampup_adj_with_floor\n else:\n # section 2; 0x80..0xBF\n r = rampup_adj_with_floor\n g = brightness_floor\n b = rampdown_adj_with_floor\n else:\n # section 0: 0x00..0x3F\n r = rampdown_adj_with_floor\n g = rampup_adj_with_floor\n b = brightness_floor\n\n return (r, g, b)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate RGB values from HSV values in line with a typical light spectrum.", "response": "def hsv2rgb_spectrum(hsv):\n \"\"\"Generates RGB values from HSV values in line with a typical light\n spectrum.\"\"\"\n h, s, v = hsv\n return hsv2rgb_raw(((h * 192) >> 8, s, v))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate RGB values from an even visual Formula.", "response": "def hsv2rgb_rainbow(hsv):\n \"\"\"Generates RGB values from HSV that have an even visual\n distribution. Be careful as this method is only have as fast as\n hsv2rgb_spectrum.\"\"\"\n\n def nscale8x3_video(r, g, b, scale):\n nonzeroscale = 0\n if scale != 0:\n nonzeroscale = 1\n if r != 0:\n r = ((r * scale) >> 8) + nonzeroscale\n if g != 0:\n g = ((g * scale) >> 8) + nonzeroscale\n if b != 0:\n b = ((b * scale) >> 8) + nonzeroscale\n return (r, g, b)\n\n def scale8_video_LEAVING_R1_DIRTY(i, scale):\n nonzeroscale = 0\n if scale != 0:\n nonzeroscale = 1\n if i != 0:\n i = ((i * scale) >> 8) + nonzeroscale\n return i\n\n h, s, v = hsv\n offset = h & 0x1F # 0..31\n offset8 = offset * 8\n third = (offset8 * (256 // 3)) >> 8\n r, g, b = (0, 0, 0)\n\n if not (h & 0x80):\n if not (h & 0x40):\n if not (h & 0x20):\n r = 255 - third\n g = third\n b = 0\n else:\n r = 171\n g = 85 + third\n b = 0x00\n else:\n if not (h & 0x20):\n twothirds = (third << 1)\n r = 171 - twothirds\n g = 171 + third\n b = 0\n else:\n r = 0\n g = 255 - third\n b = third\n else:\n if not (h & 0x40):\n if not (h & 0x20):\n r = 0x00\n twothirds = (third << 1)\n g = 171 - twothirds\n b = 85 + twothirds\n else:\n r = third\n g = 0\n b = 255 - third\n else:\n if not (h & 0x20):\n r = 85 + third\n g = 0\n b = 171 - third\n else:\n r = 171 + third\n g = 0x00\n b = 85 - third\n\n if s != 255:\n r, g, b = nscale8x3_video(r, g, b, s)\n desat = 255 - s\n desat = (desat * desat) >> 8\n brightness_floor = desat\n r = r + brightness_floor\n g = g + brightness_floor\n b = b + brightness_floor\n\n if v != 255:\n v = scale8_video_LEAVING_R1_DIRTY(v, v)\n r, g, b = nscale8x3_video(r, g, b, v)\n\n return (r, g, b)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef hsv2rgb_360(hsv):\n\n h, s, v = hsv\n\n r, g, b = colorsys.hsv_to_rgb(h / 360.0, s, v)\n return (int(r * 255.0), int(g * 255.0), int(b * 255.0))", "response": "Python default hsv to rgb conversion for when hue values in the hsv range 0 - 359 are preferred."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsort colors by hue saturation and value.", "response": "def color_cmp(a, b):\n \"\"\"Order colors by hue, saturation and value, in that order.\n\n Returns -1 if a < b, 0 if a == b and 1 if a < b.\n \"\"\"\n if a == b:\n return 0\n\n a, b = rgb_to_hsv(a), rgb_to_hsv(b)\n return -1 if a < b else 1"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget a server for this key.", "response": "def get_server(self, key, **kwds):\n \"\"\"\n Get a new or existing server for this key.\n\n :param int key: key for the server to use\n \"\"\"\n kwds = dict(self.kwds, **kwds)\n server = self.servers.get(key)\n if server:\n # Make sure it's the right server.\n server.check_keywords(self.constructor, kwds)\n else:\n # Make a new server\n server = _CachedServer(self.constructor, key, kwds)\n self.servers[key] = server\n\n return server"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_one(desc, name, value):\n old_value = desc.get(name)\n if old_value is None:\n raise KeyError('No section \"%s\"' % name)\n\n if value is None:\n value = type(old_value)()\n\n elif name in CLASS_SECTIONS:\n if isinstance(value, str):\n value = {'typename': aliases.resolve(value)}\n elif isinstance(value, type):\n value = {'typename': class_name.class_name(value)}\n elif not isinstance(value, dict):\n raise TypeError('Expected dict, str or type, got \"%s\"' % value)\n\n typename = value.get('typename')\n if typename:\n s = 's' if name == 'driver' else ''\n path = 'bibliopixel.' + name + s\n importer.import_symbol(typename, path)\n\n elif name == 'shape':\n if not isinstance(value, (list, int, tuple, str)):\n raise TypeError('Expected shape, got \"%s\"' % value)\n\n elif type(old_value) is not type(value):\n raise TypeError('Expected %s but got \"%s\" of type %s' %\n (type(old_value), value, type(value)))\n\n desc[name] = value", "response": "Set one section in a Project description"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(desc, other=None, **kwds):\n other = other and _as_dict(other) or {}\n for i in other, kwds:\n for k, v in i.items():\n if isinstance(v, dict):\n # Only for dicts, merge instead of overwriting\n old_v = desc[k]\n for k2, v2 in v.items():\n if v2 is None:\n old_v.pop(k2, None)\n else:\n old_v[k2] = v2\n else:\n set_one(desc, k, v)", "response": "Update sections in a Project description"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef name_to_color(name):\n def to_color(name):\n name = name.lower()\n if ',' in name:\n if name.startswith('(') and name.endswith(')'):\n name = name[1:-1]\n if name.startswith('[') and name.endswith(']'):\n name = name[1:-1]\n\n r, g, b = name.split(',')\n return _from_number(r), _from_number(g), _from_number(b)\n\n try:\n n = _from_number(name)\n except:\n color = tables.get_color(name)\n if color:\n return color\n raise ValueError\n\n return tables.to_triplet(n)\n\n try:\n color = to_color(name)\n except:\n raise ValueError('Unknown color name %s' % str(name))\n\n if not all(0 <= i <= 255 for i in color):\n raise ValueError('Component out of range: %s' % color)\n\n return color", "response": "Converts a string name to a color in a list of RGB 3 - tuple."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a color to a name.", "response": "def color_to_name(color, use_hex=False):\n \"\"\"\n :param tuple color: an RGB 3-tuple of integer colors\n :returns: a string name for this color\n\n ``name_to_color(color_to_name(c)) == c`` is guaranteed to be true (but the\n reverse is not true, because name_to_color is a many-to-one function).\n \"\"\"\n if isinstance(color, list):\n color = tuple(color)\n elif not isinstance(color, tuple):\n raise ValueError('Not a color')\n\n if use_hex:\n return '#%02x%02x%02x' % color\n\n return tables.get_name(color) or str(color)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ntoggles back and forth between a name and a tuple representation.", "response": "def toggle(s):\n \"\"\"\n Toggle back and forth between a name and a tuple representation.\n\n :param str s: a string which is either a text name, or a tuple-string:\n a string with three numbers separated by commas\n\n :returns: if the string was a text name, return a tuple. If it's a\n tuple-string and it corresponds to a text name, return the text\n name, else return the original tuple-string.\n \"\"\"\n is_numeric = ',' in s or s.startswith('0x') or s.startswith('#')\n c = name_to_color(s)\n return color_to_name(c) if is_numeric else str(c)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef construct(cls, project, *, run=None, name=None, data=None, **desc):\n from . failed import Failed\n exception = desc.pop('_exception', None)\n if exception:\n a = Failed(project.layout, desc, exception)\n else:\n try:\n a = cls(project.layout, **desc)\n a._set_runner(run or {})\n except Exception as e:\n if cls.FAIL_ON_EXCEPTION:\n raise\n a = Failed(project.layout, desc, e)\n\n a.name = name\n a.data = data\n return a", "response": "Construct an animation from a project and desc."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef convert_mode(image, mode='RGB'):\n deprecated.deprecated('util.gif.convert_model')\n\n return image if (image.mode == mode) else image.convert(mode=mode)", "response": "Return an image in the given mode."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef image_to_colorlist(image, container=list):\n deprecated.deprecated('util.gif.image_to_colorlist')\n\n return container(convert_mode(image).getdata())", "response": "Given a PIL. Image returns a ColorList of its pixels."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef animated_gif_to_colorlists(image, container=list):\n deprecated.deprecated('util.gif.animated_gif_to_colorlists')\n\n from PIL import ImageSequence\n\n it = ImageSequence.Iterator(image)\n return [image_to_colorlist(i, container) for i in it]", "response": "Given an animated GIF return a list with a colorlist for each frame."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse(s):\n\n parts = s.replace(',', ' ').split()\n if not parts:\n raise ValueError('Cannot parse empty string')\n\n pieces = []\n for part in parts:\n m = PART_MATCH(part)\n pieces.extend(m.groups() if m else [part])\n\n if len(pieces) == 1:\n pieces.append('s')\n\n if len(pieces) % 2:\n raise ValueError('Malformed duration %s: %s: %s' % (s, parts, pieces))\n\n result = 0\n for number, units in zip(*[iter(pieces)] * 2):\n number = float(number)\n if number < 0:\n raise ValueError('Durations cannot have negative components')\n result += number * _get_units(units)\n\n return result", "response": "Parse a string representing a time interval or duration into seconds."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef opener(ip_address, port, delay=1):\n global WEBPAGE_OPENED\n if WEBPAGE_OPENED:\n return\n WEBPAGE_OPENED = True\n raw_opener(ip_address, port, delay)", "response": "Open a web browser page for the control panel."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef raw_opener(ip_address, port, delay=1):\n def target():\n time.sleep(delay)\n url = 'http://%s:%d' % (ip_address, port)\n webbrowser.open(url, new=0, autoraise=True)\n\n threading.Thread(target=target, daemon=True).start()", "response": "Open a web browser page for the control panel."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef start(self, threaded):\n self.stop()\n self.__class__._INSTANCE = weakref.ref(self)\n self.is_running = True\n\n if threaded:\n self.thread = runnable.LoopThread()\n self.thread.run_once = self._target\n self.thread.start()\n else:\n self._target()", "response": "Creates and starts the project."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nstopping the running instance of the current instance.", "response": "def stop(self):\n \"\"\"\n Stop the Runner if it's running.\n Called as a classmethod, stop the running instance if any.\n \"\"\"\n if self.is_running:\n log.info('Stopping')\n self.is_running = False\n self.__class__._INSTANCE = None\n\n try:\n self.thread and self.thread.stop()\n except:\n log.error('Error stopping thread')\n traceback.print_exc()\n self.thread = None\n return True"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef show_image(setter, width, height,\n image_path='', image_obj=None, offset=(0, 0),\n bgcolor=COLORS.Off, brightness=255):\n \"\"\"Display an image on a matrix.\"\"\"\n bgcolor = color_scale(bgcolor, brightness)\n\n img = image_obj\n if image_path and not img:\n from PIL import Image\n\n img = Image.open(image_path)\n elif not img:\n raise ValueError('Must provide either image_path or image_obj')\n\n w = min(width - offset[0], img.size[0])\n h = min(height - offset[1], img.size[1])\n ox = offset[0]\n oy = offset[1]\n\n for x in range(ox, w + ox):\n for y in range(oy, h + oy):\n r, g, b, a = (0, 0, 0, 255)\n rgba = img.getpixel((x - ox, y - oy))\n\n if isinstance(rgba, int):\n raise ValueError('Image must be in RGB or RGBA format!')\n if len(rgba) == 3:\n r, g, b = rgba\n elif len(rgba) == 4:\n r, g, b, a = rgba\n else:\n raise ValueError('Image must be in RGB or RGBA format!')\n\n if a == 0:\n r, g, b = bgcolor\n else:\n r, g, b = color_scale((r, g, b), a)\n\n if brightness != 255:\n r, g, b = color_scale((r, g, b), brightness)\n\n setter(x, y, (r, g, b))", "response": "Display an image on a matrix."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndisplaying an image on the matrix", "response": "def showImage(layout, imagePath=\"\", imageObj=None, offset=(0, 0),\n bgcolor=COLORS.Off, brightness=255):\n \"\"\"Display an image on the matrix\"\"\"\n if not isinstance(layout, Matrix):\n raise RuntimeError(\"Must use Matrix with showImage!\")\n\n layout.all_off()\n\n return show_image(layout.set, layout.width, layout.height, imagePath,\n imageObj, offset, bgcolor, brightness)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndisplaying an image on the matrix", "response": "def loadImage(layout, imagePath=\"\", imageObj=None, offset=(0, 0),\n bgcolor=COLORS.Off, brightness=255):\n \"\"\"Display an image on the matrix\"\"\"\n\n if not isinstance(layout, Matrix):\n raise RuntimeError(\"Must use Matrix with loadImage!\")\n\n texture = [[COLORS.Off for x in range(layout.width)]\n for y in range(layout.height)]\n\n def setter(x, y, pixel):\n if y >= 0 and x >= 0:\n texture[y][x] = pixel\n\n show_image(setter, layout.width, layout.height, imagePath, imageObj,\n offset, bgcolor, brightness)\n\n return texture"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncoerces a list into a list of triplets.", "response": "def to_triplets(colors):\n \"\"\"\n Coerce a list into a list of triplets.\n\n If `colors` is a list of lists or strings, return it as is. Otherwise,\n divide it into tuplets of length three, silently discarding any extra\n elements beyond a multiple of three.\n \"\"\"\n try:\n colors[0][0]\n return colors\n except:\n pass\n\n # It's a 1-dimensional list\n extra = len(colors) % 3\n if extra:\n colors = colors[:-extra]\n return list(zip(*[iter(colors)] * 3))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef colors_no_palette(colors=None, **kwds):\n if isinstance(colors, str):\n colors = _split_colors(colors)\n else:\n colors = to_triplets(colors or ())\n\n colors = (color(c) for c in colors or ())\n return palette.Palette(colors, **kwds)", "response": "Return a Palette but don t take into account Pallete Names."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nflattens a multilevel dictionary into a single - level one.", "response": "def flatten(master):\n \"\"\"\n :param dict master: a multilevel dictionary\n :return: a flattened dictionary\n :rtype: dict\n\n Flattens a multilevel dictionary into a single-level one so that::\n\n {'foo':\n {'bar':\n {\n 'a': 1,\n 'b': True,\n 'c': 'hello',\n },\n },\n }\n\n would become::\n\n {'foo.bar.a': 1,\n 'foo.bar.b': True,\n 'foo.bar.a': 1,\n }\n\n You can mix and match both input (hierarchical) and output (dotted) formats\n in the input without problems - and if you call flatten more than once, it\n has no effect.\n \"\"\"\n result = {}\n\n def add(value, *keys):\n if keys in result:\n raise ValueError('Duplicate key %s' % keys)\n result[keys] = value\n\n def recurse(value, *keys):\n if isinstance(value, dict):\n for k, v in value.items():\n recurse(v, k, *keys)\n else:\n key = '.'.join(reversed(keys))\n if key in result:\n raise ValueError('Duplicate key %s' % str(keys))\n result[key] = value\n\n recurse(master)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreceive a message and sets it immediately.", "response": "def receive(self, msg):\n \"\"\"\n Receives a message, and either sets it immediately, or puts it on the\n edit queue if there is one.\n\n \"\"\"\n if self.edit_queue:\n self.edit_queue.put_edit(self._set, msg)\n else:\n self._set(msg)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef make_matrix_coord_map(\n dx, dy, serpentine=True, offset=0, rotation=0, y_flip=False):\n \"\"\"Helper method to generate X,Y coordinate maps for strips\"\"\"\n result = []\n for y in range(dy):\n if not serpentine or y % 2 == 0:\n result.append([(dx * y) + x + offset for x in range(dx)])\n else:\n result.append([dx * (y + 1) - 1 - x + offset for x in range(dx)])\n\n result = rotate_and_flip(result, rotation, y_flip)\n\n return result", "response": "Helper method to generate X Y coordinate maps for strips"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_object(*args, typename=None, python_path=None, datatype=None, **kwds):\n datatype = datatype or import_symbol(typename, python_path)\n field_types = getattr(datatype, 'FIELD_TYPES', fields.FIELD_TYPES)\n return datatype(*args, **fields.component(kwds, field_types))", "response": "Make an object from a symbol."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pid_context(pid_filename=None):\n pid_filename = pid_filename or DEFAULT_PID_FILENAME\n if os.path.exists(pid_filename):\n contents = open(pid_filename).read(16)\n log.warning('pid_filename %s already exists with contents %s',\n pid_filename, contents)\n\n with open(pid_filename, 'w') as fp:\n fp.write(str(os.getpid()))\n fp.write('\\n')\n\n try:\n yield\n finally:\n try:\n os.remove(pid_filename)\n except Exception as e:\n log.error('Got an exception %s deleting the pid_filename %s',\n e, pid_filename)", "response": "A context manager that creates a new process with the PID of this process."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef index(self, i, length=None):\n if self.begin <= i <= self.end:\n index = i - self.BEGIN - self.offset\n if length is None:\n length = self.full_range()\n else:\n length = min(length, self.full_range())\n\n if 0 <= index < length:\n return index", "response": "Return an integer index or None"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a generator that yields the elements from the given data.", "response": "def read_from(self, data, pad=0):\n \"\"\"\n Returns a generator with the elements \"data\" taken by offset, restricted\n by self.begin and self.end, and padded on either end by `pad` to get\n back to the original length of `data`\n \"\"\"\n for i in range(self.BEGIN, self.END + 1):\n index = self.index(i, len(data))\n yield pad if index is None else data[index]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncleaning up the animation from a yaml file.", "response": "def _clean_animation(desc, parent):\n \"\"\"\n Cleans up all sorts of special cases that humans want when entering\n an animation from a yaml file.\n\n 1. Loading it from a file\n 2. Using just a typename instead of a dict\n 3. A single dict representing an animation, with a run: section.\n 4. (Legacy) Having a dict with parallel elements run: and animation:\n 5. (Legacy) A tuple or list: (animation, run )\n\n \"\"\"\n desc = load.load_if_filename(desc) or desc\n\n if isinstance(desc, str):\n animation = {'typename': desc}\n\n elif not isinstance(desc, dict):\n raise TypeError('Unexpected type %s in collection' % type(desc))\n\n elif 'typename' in desc or 'animation' not in desc:\n animation = desc\n\n else:\n animation = desc.pop('animation', {})\n if isinstance(animation, str):\n animation = {'typename': animation}\n\n animation['run'] = desc.pop('run', {})\n if desc:\n raise ValueError('Extra animation fields: ' + ', '.join(desc))\n\n animation.setdefault('typename', DEFAULT_ANIMATION)\n animation = construct.to_type_constructor(animation, ANIMATION_PATH)\n datatype = animation.setdefault('datatype', failed.Failed)\n animation.setdefault('name', datatype.__name__)\n\n # Children without fps or sleep_time get it from their parents.\n # TODO: We shouldn't have to rewrite our descriptions here! The\n # animation engine should be smart enough to figure out the right\n # speed to run a subanimation without a run: section.\n run = animation.setdefault('run', {})\n run_parent = parent.setdefault('run', {})\n if not ('fps' in run or 'sleep_time' in run):\n if 'fps' in run_parent:\n run.update(fps=run_parent['fps'])\n elif 'sleep_time' in run_parent:\n run.update(sleep_time=run_parent['sleep_time'])\n\n return animation"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngives a list of animations and a list of names rename them to be unique.", "response": "def _make_names_unique(animations):\n \"\"\"\n Given a list of animations, some of which might have duplicate names, rename\n the first one to be _0, the second _1,\n _2, etc.\"\"\"\n counts = {}\n for a in animations:\n c = counts.get(a['name'], 0) + 1\n counts[a['name']] = c\n if c > 1:\n a['name'] += '_' + str(c - 1)\n\n dupes = set(k for k, v in counts.items() if v > 1)\n for a in animations:\n if a['name'] in dupes:\n a['name'] += '_0'"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef detach(self, overlay):\n # See #868\n for i, a in enumerate(self.animations):\n a.layout = a.layout.clone()\n if overlay and i:\n a.preclear = False", "response": "Detaches all animations from the current layout."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmerges zero or more dictionaries representing projects with the default project dictionary and return the result", "response": "def merge(*projects):\n \"\"\"\n Merge zero or more dictionaries representing projects with the default\n project dictionary and return the result\n \"\"\"\n result = {}\n for project in projects:\n for name, section in (project or {}).items():\n if name not in PROJECT_SECTIONS:\n raise ValueError(UNKNOWN_SECTION_ERROR % name)\n\n if section is None:\n result[name] = type(result[name])()\n continue\n\n if name in NOT_MERGEABLE + SPECIAL_CASE:\n result[name] = section\n continue\n\n if section and not isinstance(section, (dict, str)):\n cname = section.__class__.__name__\n raise ValueError(SECTION_ISNT_DICT_ERROR % (name, cname))\n\n if name == 'animation':\n # Useful hack to allow you to load projects as animations.\n adesc = load.load_if_filename(section)\n if adesc:\n section = adesc.get('animation', {})\n section['run'] = adesc.get('run', {})\n\n result_section = result.setdefault(name, {})\n section = construct.to_type(section)\n for k, v in section.items():\n if v is None:\n result_section.pop(k, None)\n else:\n result_section[k] = v\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncopy the instance and make sure not to use a reference", "response": "def copy(self):\n \"\"\" Copy the instance and make sure not to use a reference\n \"\"\"\n return self.__class__(\n amount=self[\"amount\"],\n asset=self[\"asset\"].copy(),\n blockchain_instance=self.blockchain,\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the asset as instance of : class :. asset. Asset", "response": "def asset(self):\n \"\"\" Returns the asset as instance of :class:`.asset.Asset`\n \"\"\"\n if not self[\"asset\"]:\n self[\"asset\"] = self.asset_class(\n self[\"symbol\"], blockchain_instance=self.blockchain\n )\n return self[\"asset\"]"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nrefreshes the object with the data from the API server", "response": "def refresh(self):\n \"\"\" Refresh/Obtain an account's data from the API server\n \"\"\"\n import re\n\n if re.match(r\"^1\\.2\\.[0-9]*$\", self.identifier):\n account = self.blockchain.rpc.get_objects([self.identifier])[0]\n else:\n account = self.blockchain.rpc.lookup_account_names([self.identifier])[0]\n if not account:\n raise AccountDoesNotExistsException(self.identifier)\n self.store(account, \"name\")\n\n if self.full: # pragma: no cover\n accounts = self.blockchain.rpc.get_full_accounts([account[\"id\"]], False)\n if accounts and isinstance(accounts, list):\n account = accounts[0][1]\n else:\n raise AccountDoesNotExistsException(self.identifier)\n super(Account, self).__init__(\n account[\"account\"], blockchain_instance=self.blockchain\n )\n for k, v in account.items():\n if k != \"account\":\n self[k] = v\n else:\n super(Account, self).__init__(account, blockchain_instance=self.blockchain)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef balances(self):\n balances = self.blockchain.rpc.get_account_balances(self[\"id\"], [])\n return [\n self.amount_class(b, blockchain_instance=self.blockchain)\n for b in balances\n if int(b[\"amount\"]) > 0\n ]", "response": "List balances of an account. This call returns instances of\n . Amount."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef balance(self, symbol):\n if isinstance(symbol, dict) and \"symbol\" in symbol:\n symbol = symbol[\"symbol\"]\n balances = self.balances\n for b in balances:\n if b[\"symbol\"] == symbol:\n return b\n return self.amount_class(0, symbol, blockchain_instance=self.blockchain)", "response": "Obtain the balance of a specific Asset. This call returns instances of\n . Amount."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef history(self, first=0, last=0, limit=-1, only_ops=[], exclude_ops=[]):\n _limit = 100\n cnt = 0\n\n if first < 0:\n first = 0\n\n while True:\n # RPC call\n txs = self.blockchain.rpc.get_account_history(\n self[\"id\"],\n \"1.11.{}\".format(last),\n _limit,\n \"1.11.{}\".format(first - 1),\n api=\"history\",\n )\n for i in txs:\n if (\n exclude_ops\n and self.operations.getOperationNameForId(i[\"op\"][0]) in exclude_ops\n ):\n continue\n if (\n not only_ops\n or self.operations.getOperationNameForId(i[\"op\"][0]) in only_ops\n ):\n cnt += 1\n yield i\n if limit >= 0 and cnt >= limit: # pragma: no cover\n return\n\n if not txs:\n log.info(\"No more history returned from API node\")\n break\n if len(txs) < _limit:\n log.info(\"Less than {} have been returned.\".format(_limit))\n break\n first = int(txs[-1][\"id\"].split(\".\")[2])", "response": "Returns a generator for individual account transactions."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nupgrades the account to life time member.", "response": "def upgrade(self): # pragma: no cover\n \"\"\" Upgrade account to life time member\n \"\"\"\n assert callable(self.blockchain.upgrade_account)\n return self.blockchain.upgrade_account(account=self)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef whitelist(self, account): # pragma: no cover\n assert callable(self.blockchain.account_whitelist)\n return self.blockchain.account_whitelist(account, lists=[\"white\"], account=self)", "response": "Add an account to the whitelist of this account"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds an account to the blacklist of this account", "response": "def blacklist(self, account): # pragma: no cover\n \"\"\" Add an other account to the blacklist of this account\n \"\"\"\n assert callable(self.blockchain.account_whitelist)\n return self.blockchain.account_whitelist(account, lists=[\"black\"], account=self)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nremoving an account from any other account in any list of this account.", "response": "def nolist(self, account): # pragma: no cover\n \"\"\" Remove an other account from any list of this account\n \"\"\"\n assert callable(self.blockchain.account_whitelist)\n return self.blockchain.account_whitelist(account, lists=[], account=self)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the actual account object for this canon.", "response": "def account(self):\n \"\"\" In oder to obtain the actual\n :class:`account.Account` from this class, you can\n use the ``account`` attribute.\n \"\"\"\n account = self.account_class(self[\"owner\"], blockchain_instance=self.blockchain)\n # account.refresh()\n return account"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef recover_public_key(digest, signature, i, message=None):\n\n # See http: //www.secg.org/download/aid-780/sec1-v2.pdf section 4.1.6 primarily\n curve = ecdsa.SECP256k1.curve\n G = ecdsa.SECP256k1.generator\n order = ecdsa.SECP256k1.order\n yp = i % 2\n r, s = ecdsa.util.sigdecode_string(signature, order)\n # 1.1\n x = r + (i // 2) * order\n # 1.3. This actually calculates for either effectively 02||X or 03||X depending on 'k' instead of always for 02||X as specified.\n # This substitutes for the lack of reversing R later on. -R actually is defined to be just flipping the y-coordinate in the elliptic curve.\n alpha = ((x * x * x) + (curve.a() * x) + curve.b()) % curve.p()\n beta = ecdsa.numbertheory.square_root_mod_prime(alpha, curve.p())\n y = beta if (beta - yp) % 2 == 0 else curve.p() - beta\n # 1.4 Constructor of Point is supposed to check if nR is at infinity.\n R = ecdsa.ellipticcurve.Point(curve, x, y, order)\n # 1.5 Compute e\n e = ecdsa.util.string_to_number(digest)\n # 1.6 Compute Q = r^-1(sR - eG)\n Q = ecdsa.numbertheory.inverse_mod(r, order) * (s * R + (-e % order) * G)\n\n if SECP256K1_MODULE == \"cryptography\" and message is not None:\n if not isinstance(message, bytes):\n message = bytes(message, \"utf-8\") # pragma: no cover\n sigder = encode_dss_signature(r, s)\n public_key = ec.EllipticCurvePublicNumbers(\n Q._Point__x, Q._Point__y, ec.SECP256K1()\n ).public_key(default_backend())\n public_key.verify(sigder, message, ec.ECDSA(hashes.SHA256()))\n return public_key\n else:\n # Not strictly necessary, but let's verify the message for paranoia's sake.\n if not ecdsa.VerifyingKey.from_public_point(\n Q, curve=ecdsa.SECP256k1\n ).verify_digest(\n signature, digest, sigdecode=ecdsa.util.sigdecode_string\n ): # pragma: no cover\n return None # pragma: no cover\n return ecdsa.VerifyingKey.from_public_point(\n Q, curve=ecdsa.SECP256k1\n )", "response": "Recover the public key from the signature and the digest."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nuse to derive the number that allows to easily recover the base64 - encoded public key from the signature", "response": "def recoverPubkeyParameter(message, digest, signature, pubkey):\n \"\"\" Use to derive a number that allows to easily recover the\n public key from the signature\n \"\"\"\n if not isinstance(message, bytes):\n message = bytes(message, \"utf-8\") # pragma: no cover\n for i in range(0, 4):\n if SECP256K1_MODULE == \"secp256k1\": # pragma: no cover\n sig = pubkey.ecdsa_recoverable_deserialize(signature, i)\n p = secp256k1.PublicKey(pubkey.ecdsa_recover(message, sig))\n if p.serialize() == pubkey.serialize():\n return i\n elif SECP256K1_MODULE == \"cryptography\" and not isinstance(pubkey, PublicKey):\n p = recover_public_key(digest, signature, i, message)\n p_comp = hexlify(compressedPubkey(p))\n pubkey_comp = hexlify(compressedPubkey(pubkey))\n if p_comp == pubkey_comp:\n return i\n else: # pragma: no cover\n p = recover_public_key(digest, signature, i)\n p_comp = hexlify(compressedPubkey(p))\n p_string = hexlify(p.to_string())\n if isinstance(pubkey, PublicKey): # pragma: no cover\n pubkey_string = bytes(repr(pubkey), \"ascii\")\n else: # pragma: no cover\n pubkey_string = hexlify(pubkey.to_string())\n if p_string == pubkey_string or p_comp == pubkey_string: # pragma: no cover\n return i"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sign_message(message, wif, hashfn=hashlib.sha256):\n\n if not isinstance(message, bytes):\n message = bytes(message, \"utf-8\")\n\n digest = hashfn(message).digest()\n priv_key = PrivateKey(wif)\n p = bytes(priv_key)\n\n if SECP256K1_MODULE == \"secp256k1\":\n ndata = secp256k1.ffi.new(\"const int *ndata\")\n ndata[0] = 0\n while True:\n ndata[0] += 1\n privkey = secp256k1.PrivateKey(p, raw=True)\n sig = secp256k1.ffi.new(\"secp256k1_ecdsa_recoverable_signature *\")\n signed = secp256k1.lib.secp256k1_ecdsa_sign_recoverable(\n privkey.ctx, sig, digest, privkey.private_key, secp256k1.ffi.NULL, ndata\n )\n if not signed == 1: # pragma: no cover\n raise AssertionError()\n signature, i = privkey.ecdsa_recoverable_serialize(sig)\n if _is_canonical(signature):\n i += 4 # compressed\n i += 27 # compact\n break\n elif SECP256K1_MODULE == \"cryptography\":\n cnt = 0\n private_key = ec.derive_private_key(\n int(repr(priv_key), 16), ec.SECP256K1(), default_backend()\n )\n public_key = private_key.public_key()\n while True:\n cnt += 1\n if not cnt % 20: # pragma: no cover\n log.info(\n \"Still searching for a canonical signature. Tried %d times already!\"\n % cnt\n )\n order = ecdsa.SECP256k1.order\n # signer = private_key.signer(ec.ECDSA(hashes.SHA256()))\n # signer.update(message)\n # sigder = signer.finalize()\n sigder = private_key.sign(message, ec.ECDSA(hashes.SHA256()))\n r, s = decode_dss_signature(sigder)\n signature = ecdsa.util.sigencode_string(r, s, order)\n # Make sure signature is canonical!\n #\n sigder = bytearray(sigder)\n lenR = sigder[3]\n lenS = sigder[5 + lenR]\n if lenR is 32 and lenS is 32:\n # Derive the recovery parameter\n #\n i = recoverPubkeyParameter(message, digest, signature, public_key)\n i += 4 # compressed\n i += 27 # compact\n break\n else: # pragma: no branch # pragma: no cover\n cnt = 0\n sk = ecdsa.SigningKey.from_string(p, curve=ecdsa.SECP256k1)\n while 1:\n cnt += 1\n if not cnt % 20: # pragma: no branch\n log.info(\n \"Still searching for a canonical signature. Tried %d times already!\"\n % cnt\n )\n\n # Deterministic k\n #\n k = ecdsa.rfc6979.generate_k(\n sk.curve.generator.order(),\n sk.privkey.secret_multiplier,\n hashlib.sha256,\n hashlib.sha256(\n digest\n + struct.pack(\n \"d\", time.time()\n ) # use the local time to randomize the signature\n ).digest(),\n )\n\n # Sign message\n #\n sigder = sk.sign_digest(digest, sigencode=ecdsa.util.sigencode_der, k=k)\n\n # Reformating of signature\n #\n r, s = ecdsa.util.sigdecode_der(sigder, sk.curve.generator.order())\n signature = ecdsa.util.sigencode_string(r, s, sk.curve.generator.order())\n\n # Make sure signature is canonical!\n #\n sigder = bytearray(sigder)\n lenR = sigder[3]\n lenS = sigder[5 + lenR]\n if lenR is 32 and lenS is 32:\n # Derive the recovery parameter\n #\n i = recoverPubkeyParameter(\n message, digest, signature, sk.get_verifying_key()\n )\n i += 4 # compressed\n i += 27 # compact\n break\n\n # pack signature\n #\n sigstr = struct.pack(\" stop:\n # raise StopIteration\n return\n\n # Sleep for one block\n time.sleep(self.block_interval)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwaiting for a block to be returned and return it.", "response": "def wait_for_and_get_block(self, block_number, blocks_waiting_for=None):\n \"\"\" Get the desired block from the chain, if the current head block is\n smaller (for both head and irreversible) then we wait, but a\n maxmimum of blocks_waiting_for * max_block_wait_repetition time\n before failure.\n\n :param int block_number: desired block number\n :param int blocks_waiting_for: (default) difference between\n block_number and current head how many blocks we are willing to\n wait, positive int\n \"\"\"\n if not blocks_waiting_for:\n blocks_waiting_for = max(1, block_number - self.get_current_block_num())\n\n repetition = 0\n # can't return the block before the chain has reached it (support\n # future block_num)\n while self.get_current_block_num() < block_number:\n repetition += 1\n time.sleep(self.block_interval)\n if repetition > blocks_waiting_for * self.max_block_wait_repetition:\n raise Exception(\"Wait time for new block exceeded, aborting\")\n # block has to be returned properly\n block = self.blockchain.rpc.get_block(block_number)\n repetition = 0\n while not block:\n repetition += 1\n time.sleep(self.block_interval)\n if repetition > self.max_block_wait_repetition:\n raise Exception(\"Wait time for new block exceeded, aborting\")\n block = self.blockchain.rpc.get_block(block_number)\n return block"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nyielding all operations starting from start and ending at stop.", "response": "def ops(self, start=None, stop=None, **kwargs):\n \"\"\" Yields all operations (excluding virtual operations) starting from\n ``start``.\n\n :param int start: Starting block\n :param int stop: Stop at this block\n :param str mode: We here have the choice between\n \"head\" (the last block) and \"irreversible\" (the block that is\n confirmed by 2/3 of all block producers and is thus irreversible)\n :param bool only_virtual_ops: Only yield virtual operations\n\n This call returns a list that only carries one operation and\n its type!\n \"\"\"\n\n for block in self.blocks(start=start, stop=stop, **kwargs):\n for tx in block[\"transactions\"]:\n for op in tx[\"operations\"]:\n # Replace opid by op name\n op[0] = self.operationids.getOperationName(op[0])\n yield {\n \"block_num\": block[\"block_num\"],\n \"op\": op,\n \"timestamp\": block[\"timestamp\"],\n }"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nyield specific operations from all the block store entries in the database.", "response": "def stream(self, opNames=[], *args, **kwargs):\n \"\"\" Yield specific operations (e.g. comments) only\n\n :param array opNames: List of operations to filter for\n :param int start: Start at this block\n :param int stop: Stop at this block\n :param str mode: We here have the choice between\n * \"head\": the last block\n * \"irreversible\": the block that is confirmed by 2/3 of all\n block producers and is thus irreversible!\n\n The dict output is formated such that ``type`` caries the\n operation type, timestamp and block_num are taken from the\n block the operation was stored in and the other key depend\n on the actualy operation.\n \"\"\"\n for op in self.ops(**kwargs):\n if not opNames or op[\"op\"][0] in opNames:\n r = {\n \"type\": op[\"op\"][0],\n \"timestamp\": op.get(\"timestamp\"),\n \"block_num\": op.get(\"block_num\"),\n }\n r.update(op[\"op\"][1])\n yield r"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwait until a transaction is included in a block.", "response": "def awaitTxConfirmation(self, transaction, limit=10):\n \"\"\" Returns the transaction as seen by the blockchain after being\n included into a block\n\n .. note:: If you want instant confirmation, you need to instantiate\n class:`.blockchain.Blockchain` with\n ``mode=\"head\"``, otherwise, the call will wait until\n confirmed in an irreversible block.\n\n .. note:: This method returns once the blockchain has included a\n transaction with the **same signature**. Even though the\n signature is not usually used to identify a transaction,\n it still cannot be forfeited and is derived from the\n transaction contented and thus identifies a transaction\n uniquely.\n \"\"\"\n counter = 10\n for block in self.blocks():\n counter += 1\n for tx in block[\"transactions\"]:\n if sorted(tx[\"signatures\"]) == sorted(transaction[\"signatures\"]):\n return tx\n if counter > limit:\n raise Exception(\"The operation has not been added after 10 blocks!\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_all_accounts(self, start=\"\", stop=\"\", steps=1e3, **kwargs):\n lastname = start\n while True:\n ret = self.blockchain.rpc.lookup_accounts(lastname, steps)\n for account in ret:\n yield account[0]\n if account[0] == stop:\n raise StopIteration\n if lastname == ret[-1][0]:\n raise StopIteration\n lastname = ret[-1][0]\n if len(ret) < steps:\n raise StopIteration", "response": "Get all the account names between start and stop."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef refresh(self):\n asset = self.blockchain.rpc.get_asset(self.identifier)\n if not asset:\n raise AssetDoesNotExistsException(self.identifier)\n super(Asset, self).__init__(asset, blockchain_instance=self.blockchain)\n if self.full:\n if \"bitasset_data_id\" in asset:\n self[\"bitasset_data\"] = self.blockchain.rpc.get_object(\n asset[\"bitasset_data_id\"]\n )\n self[\"dynamic_asset_data\"] = self.blockchain.rpc.get_object(\n asset[\"dynamic_asset_data_id\"]\n )", "response": "Refresh the data from the API server"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_cer(self, cer, account=None, **kwargs):\n assert callable(self.blockchain.update_cer)\n return self.blockchain.update_cer(\n self[\"symbol\"], cer, account=account, **kwargs\n )", "response": "Update the Core Exchange Rate of an asset."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rpcexec(self, payload):\n try:\n response = requests.post(\n \"http://{}:{}/rpc\".format(self.host, self.port),\n data=json.dumps(payload, ensure_ascii=False).encode(\"utf8\"),\n headers=self.headers,\n auth=(self.username, self.password),\n )\n if response.status_code == 401:\n raise UnauthorizedError\n ret = json.loads(response.text)\n if \"error\" in ret:\n if \"detail\" in ret[\"error\"]:\n raise RPCError(ret[\"error\"][\"detail\"])\n else:\n raise RPCError(ret[\"error\"][\"message\"])\n except requests.exceptions.RequestException:\n raise RPCConnection(\"Error connecting to Client!\")\n except UnauthorizedError:\n raise UnauthorizedError(\"Invalid login credentials!\")\n except ValueError:\n raise ValueError(\"Client returned invalid format. Expected JSON!\")\n except RPCError as err:\n raise err\n else:\n return ret[\"result\"]", "response": "Manual execute a command on the API and return the response."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nformat time for permlinks", "response": "def formatTime(t):\n \"\"\" Properly Format Time for permlinks\n \"\"\"\n if isinstance(t, float):\n return datetime.utcfromtimestamp(t).strftime(timeFormat)\n if isinstance(t, datetime):\n return t.strftime(timeFormat)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef formatTimeFromNow(secs=None):\n return datetime.utcfromtimestamp(time.time() + int(secs or 0)).strftime(timeFormat)", "response": "Properly Format time that is x seconds in the future\n past."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_time(block_time):\n return datetime.strptime(block_time, timeFormat).replace(tzinfo=timezone.utc)", "response": "Take a string representation of time from the blockchain and parse it\n into a datetime object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting an operation id into a corresponding string", "response": "def getOperationNameForId(i: int):\n \"\"\" Convert an operation id into the corresponding string\n \"\"\"\n assert isinstance(i, (int)), \"This method expects an integer argument\"\n for key in operations:\n if int(operations[key]) is int(i):\n return key\n raise ValueError(\"Unknown Operation ID %d\" % i)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef unlocked(self):\n if self.password is not None:\n return bool(self.password)\n else:\n if (\n \"UNLOCK\" in os.environ\n and os.environ[\"UNLOCK\"]\n and self.config_key in self.config\n and self.config[self.config_key]\n ):\n log.debug(\"Trying to use environmental \" \"variable to unlock wallet\")\n self.unlock(os.environ.get(\"UNLOCK\"))\n return bool(self.password)\n return False", "response": "Is the store unlocked?"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef unlock(self, password):\n self.password = password\n if self.config_key in self.config and self.config[self.config_key]:\n self._decrypt_masterpassword()\n else:\n self._new_masterpassword(password)\n self._save_encrypted_masterpassword()", "response": "Unlocks the masterpassword of the current node."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a new random masterkey encrypt it with the password and store it in the store.", "response": "def _new_masterpassword(self, password):\n \"\"\" Generate a new random masterkey, encrypt it with the password and\n store it in the store.\n\n :param str password: Password to use for en-/de-cryption\n \"\"\"\n # make sure to not overwrite an existing key\n if self.config_key in self.config and self.config[self.config_key]:\n raise Exception(\"Storage already has a masterpassword!\")\n\n self.decrypted_master = hexlify(os.urandom(32)).decode(\"ascii\")\n\n # Encrypt and save master\n self.password = password\n self._save_encrypted_masterpassword()\n return self.masterkey"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _derive_checksum(self, s):\n checksum = hashlib.sha256(bytes(s, \"ascii\")).hexdigest()\n return checksum[:4]", "response": "Derive the checksum from the given random string"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_encrypted_masterpassword(self):\n if not self.unlocked():\n raise WalletLocked\n aes = AESCipher(self.password)\n return \"{}${}\".format(\n self._derive_checksum(self.masterkey), aes.encrypt(self.masterkey)\n )", "response": "Obtain the encrypted masterpassword for the current object."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef change_password(self, newpassword):\n if not self.unlocked():\n raise WalletLocked\n self.password = newpassword\n self._save_encrypted_masterpassword()", "response": "Change the password that allows to decrypt the master key."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef decrypt(self, wif):\n if not self.unlocked():\n raise WalletLocked\n return format(bip38.decrypt(wif, self.masterkey), \"wif\")", "response": "Decrypt the content according to BIP38\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nencrypts the content according to BIP38", "response": "def encrypt(self, wif):\n \"\"\" Encrypt the content according to BIP38\n\n :param str wif: Unencrypted key\n \"\"\"\n if not self.unlocked():\n raise WalletLocked\n return format(bip38.encrypt(str(wif), self.masterkey), \"encwif\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nderiving private key from the brain key and the current sequence number", "response": "def get_private(self):\n \"\"\" Derive private key from the brain key and the current sequence\n number\n \"\"\"\n encoded = \"%s %d\" % (self.brainkey, self.sequence)\n a = _bytes(encoded)\n s = hashlib.sha256(hashlib.sha512(a).digest()).digest()\n return PrivateKey(hexlify(s).decode(\"ascii\"), prefix=self.prefix)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nderive private key from the brain key and no sequence number", "response": "def get_blind_private(self):\n \"\"\" Derive private key from the brain key (and no sequence number)\n \"\"\"\n a = _bytes(self.brainkey)\n return PrivateKey(hashlib.sha256(a).hexdigest(), prefix=self.prefix)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nsuggest a new random brain key.", "response": "def suggest():\n \"\"\" Suggest a new random brain key. Randomness is provided by the\n operating system using ``os.urandom()``.\n \"\"\"\n word_count = 16\n brainkey = [None] * word_count\n dict_lines = BrainKeyDictionary.split(\",\")\n assert len(dict_lines) == 49744\n for j in range(0, word_count):\n num = int.from_bytes(os.urandom(2), byteorder=\"little\")\n rndMult = num / 2 ** 16 # returns float between 0..1 (inclusive)\n wIdx = round(len(dict_lines) * rndMult)\n brainkey[j] = dict_lines[wIdx]\n return \" \".join(brainkey).upper()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_pubkey(cls, pubkey, compressed=True, version=56, prefix=None):\n # Ensure this is a public key\n pubkey = PublicKey(pubkey, prefix=prefix or Prefix.prefix)\n if compressed:\n pubkey_plain = pubkey.compressed()\n else:\n pubkey_plain = pubkey.uncompressed()\n sha = hashlib.sha256(unhexlify(pubkey_plain)).hexdigest()\n rep = hexlify(ripemd160(sha)).decode(\"ascii\")\n s = (\"%.2x\" % version) + rep\n result = s + hexlify(doublesha256(s)[:4]).decode(\"ascii\")\n result = hexlify(ripemd160(result)).decode(\"ascii\")\n return cls(result, prefix=pubkey.prefix)", "response": "Load an address provided the public key."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a new instance of the class from a public key.", "response": "def from_pubkey(cls, pubkey, compressed=True, version=56, prefix=None):\n # Ensure this is a public key\n pubkey = PublicKey(pubkey, prefix=prefix or Prefix.prefix)\n if compressed:\n pubkey_plain = pubkey.compressed()\n else:\n pubkey_plain = pubkey.uncompressed()\n\n \"\"\" Derive address using ``RIPEMD160(SHA512(x))`` \"\"\"\n addressbin = ripemd160(hashlib.sha512(unhexlify(pubkey_plain)).hexdigest())\n result = Base58(hexlify(addressbin).decode(\"ascii\"))\n return cls(result, prefix=pubkey.prefix)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _derive_y_from_x(self, x, is_even):\n curve = ecdsa.SECP256k1.curve\n # The curve equation over F_p is:\n # y^2 = x^3 + ax + b\n a, b, p = curve.a(), curve.b(), curve.p()\n alpha = (pow(x, 3, p) + a * x + b) % p\n beta = ecdsa.numbertheory.square_root_mod_prime(alpha, p)\n if (beta % 2) == is_even:\n beta = p - beta\n return beta", "response": "Derive y point from x point"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef point(self):\n string = unhexlify(self.unCompressed())\n return ecdsa.VerifyingKey.from_string(\n string[1:], curve=ecdsa.SECP256k1\n ).pubkey.point", "response": "Return the point for the public key"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef child(self, offset256):\n a = bytes(self) + offset256\n s = hashlib.sha256(a).digest()\n return self.add(s)", "response": "Derive new public key from this key and a sha256 offset"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nderiving uncompressed public key from a private key.", "response": "def from_privkey(cls, privkey, prefix=None):\n \"\"\" Derive uncompressed public key \"\"\"\n privkey = PrivateKey(privkey, prefix=prefix or Prefix.prefix)\n secret = unhexlify(repr(privkey))\n order = ecdsa.SigningKey.from_string(\n secret, curve=ecdsa.SECP256k1\n ).curve.generator.order()\n p = ecdsa.SigningKey.from_string(\n secret, curve=ecdsa.SECP256k1\n ).verifying_key.pubkey.point\n x_str = ecdsa.util.number_to_string(p.x(), order)\n # y_str = ecdsa.util.number_to_string(p.y(), order)\n compressed = hexlify(chr(2 + (p.y() & 1)).encode(\"ascii\") + x_str).decode(\n \"ascii\"\n )\n # uncompressed = hexlify(\n # chr(4).encode('ascii') + x_str + y_str).decode('ascii')\n return cls(compressed, prefix=prefix or Prefix.prefix)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef derive_private_key(self, sequence):\n encoded = \"%s %d\" % (str(self), sequence)\n a = bytes(encoded, \"ascii\")\n s = hashlib.sha256(hashlib.sha512(a).digest()).digest()\n return PrivateKey(hexlify(s).decode(\"ascii\"), prefix=self.pubkey.prefix)", "response": "Derive a new private key from this private key and an arbitrary sequence number."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nderiving new private key from this key and a sha256 offset256", "response": "def child(self, offset256):\n \"\"\" Derive new private key from this key and a sha256 \"offset\"\n \"\"\"\n a = bytes(self.pubkey) + offset256\n s = hashlib.sha256(a).digest()\n return self.derive_from_seed(s)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nderives private key from seed and offset.", "response": "def derive_from_seed(self, offset):\n \"\"\" Derive private key using \"generate_from_seed\" method.\n Here, the key itself serves as a `seed`, and `offset`\n is expected to be a sha256 digest.\n \"\"\"\n seed = int(hexlify(bytes(self)).decode(\"ascii\"), 16)\n z = int(hexlify(offset).decode(\"ascii\"), 16)\n order = ecdsa.SECP256k1.order\n secexp = (seed + z) % order\n secret = \"%0x\" % secexp\n if len(secret) < 64: # left-pad with zeroes\n secret = (\"0\" * (64-len(secret))) + secret\n return PrivateKey(secret, prefix=self.pubkey.prefix)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_pubkey(cls, pubkey, compressed=False, version=56, prefix=None):\n # Ensure this is a public key\n pubkey = PublicKey(pubkey)\n if compressed:\n pubkey = pubkey.compressed()\n else:\n pubkey = pubkey.uncompressed()\n\n \"\"\" Derive address using ``RIPEMD160(SHA256(x))`` \"\"\"\n addressbin = ripemd160(hexlify(hashlib.sha256(unhexlify(pubkey)).digest()))\n return cls(hexlify(addressbin).decode(\"ascii\"))", "response": "Create a new instance of the class from a public key."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef rpcexec(self, payload):\n log.debug(json.dumps(payload))\n query = requests.post(self.url, json=payload, proxies=self.proxies())\n if query.status_code != 200: # pragma: no cover\n raise HttpInvalidStatusCode(\n \"Status code returned: {}\".format(query.status_code)\n )\n\n return query.text", "response": "Execute a call by sending the payload to the server and return the response."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef varint(n):\n data = b\"\"\n while n >= 0x80:\n data += bytes([(n & 0x7F) | 0x80])\n n >>= 7\n data += bytes([n])\n return data", "response": "Encode a Varint value."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef varintdecode(data): # pragma: no cover\n shift = 0\n result = 0\n for b in bytes(data):\n result |= (b & 0x7F) << shift\n if not (b & 0x80):\n break\n shift += 7\n return result", "response": "Decode a Varint from a byte string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nclaim a balance from the genesis block.", "response": "def claim(self, account=None, **kwargs):\n \"\"\" Claim a balance from the genesis block\n\n :param str balance_id: The identifier that identifies the balance\n to claim (1.15.x)\n :param str account: (optional) the account that owns the bet\n (defaults to ``default_account``)\n \"\"\"\n if not account:\n if \"default_account\" in self.blockchain.config:\n account = self.blockchain.config[\"default_account\"]\n if not account:\n raise ValueError(\"You need to provide an account\")\n account = self.account_class(account, blockchain_instance=self.blockchain)\n pubkeys = self.blockchain.wallet.getPublicKeys()\n addresses = dict()\n for p in pubkeys:\n if p[: len(self.blockchain.prefix)] != self.blockchain.prefix:\n continue\n pubkey = self.publickey_class(p, prefix=self.blockchain.prefix)\n addresses[\n str(\n self.address_class.from_pubkey(\n pubkey,\n compressed=False,\n version=0,\n prefix=self.blockchain.prefix,\n )\n )\n ] = pubkey\n addresses[\n str(\n self.address_class.from_pubkey(\n pubkey,\n compressed=True,\n version=0,\n prefix=self.blockchain.prefix,\n )\n )\n ] = pubkey\n addresses[\n str(\n self.address_class.from_pubkey(\n pubkey,\n compressed=False,\n version=56,\n prefix=self.blockchain.prefix,\n )\n )\n ] = pubkey\n addresses[\n str(\n self.address_class.from_pubkey(\n pubkey,\n compressed=True,\n version=56,\n prefix=self.blockchain.prefix,\n )\n )\n ] = pubkey\n\n if self[\"owner\"] not in addresses.keys():\n raise MissingKeyError(\"Need key for address {}\".format(self[\"owner\"]))\n\n op = self.operations.Balance_claim(\n **{\n \"fee\": {\"amount\": 0, \"asset_id\": \"1.3.0\"},\n \"deposit_to_account\": account[\"id\"],\n \"balance_to_claim\": self[\"id\"],\n \"balance_owner_key\": addresses[self[\"owner\"]],\n \"total_claimed\": self[\"balance\"],\n \"prefix\": self.blockchain.prefix,\n }\n )\n signers = [\n account[\"name\"], # The fee payer and receiver account\n addresses.get(self[\"owner\"]), # The genesis balance!\n ]\n return self.blockchain.finalizeOp(op, signers, \"active\", **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef getOperationNameForId(self, i):\n for key in self.ops:\n if int(self.ops[key]) is int(i):\n return key\n raise ValueError(\"Unknown Operation ID %d\" % i)", "response": "Convert an operation id into a corresponding string"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_shared_config(cls, config):\n assert isinstance(config, dict)\n cls._sharedInstance.config.update(config)\n # if one is already set, delete\n if cls._sharedInstance.instance:\n cls._sharedInstance.instance = None", "response": "Sets the configuration for the current instance of the current class to the given dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds the next URL in the list of available URLS.", "response": "def find_next(self):\n \"\"\" Find the next url in the list\n \"\"\"\n if int(self.num_retries) < 0: # pragma: no cover\n self._cnt_retries += 1\n sleeptime = (self._cnt_retries - 1) * 2 if self._cnt_retries < 10 else 10\n if sleeptime:\n log.warning(\n \"Lost connection to node during rpcexec(): %s (%d/%d) \"\n % (self.url, self._cnt_retries, self.num_retries)\n + \"Retrying in %d seconds\" % sleeptime\n )\n sleep(sleeptime)\n return next(self.urls)\n\n urls = [\n k\n for k, v in self._url_counter.items()\n if (\n # Only provide URLS if num_retries is bigger equal 0,\n # i.e. we want to do reconnects at all\n int(self.num_retries) >= 0\n # the counter for this host/endpoint should be smaller than\n # num_retries\n and v <= self.num_retries\n # let's not retry with the same URL *if* we have others\n # available\n and (k != self.url or len(self._url_counter) == 1)\n )\n ]\n if not len(urls):\n raise NumRetriesReached\n url = urls[0]\n return url"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reset_counter(self):\n self._cnt_retries = 0\n for i in self._url_counter:\n self._url_counter[i] = 0", "response": "reset the failed connection counters\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrefresh the block with the latest contents of the block.", "response": "def refresh(self):\n \"\"\" Even though blocks never change, you freshly obtain its contents\n from an API with this method\n \"\"\"\n block = self.blockchain.rpc.get_block(self.identifier)\n if not block:\n raise BlockDoesNotExistsException\n super(Block, self).__init__(\n block, blockchain_instance=self.blockchain, use_cache=self._use_cache\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrefreshing the block header with the latest contents.", "response": "def refresh(self):\n \"\"\" Even though blocks never change, you freshly obtain its contents\n from an API with this method\n \"\"\"\n block = self.blockchain.rpc.get_block_header(self.identifier)\n if not block:\n raise BlockDoesNotExistsException\n super(BlockHeader, self).__init__(\n block, blockchain_instance=self.blockchain, use_cache=self._use_cache\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _haveKey(self, key):\n query = (\n \"SELECT {} FROM {} WHERE {}=?\".format(\n self.__value__, self.__tablename__, self.__key__\n ),\n (key,),\n )\n connection = sqlite3.connect(self.sqlite_file)\n cursor = connection.cursor()\n cursor.execute(*query)\n return True if cursor.fetchone() else False", "response": "Is the key available?"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef items(self):\n query = \"SELECT {}, {} from {}\".format(\n self.__key__, self.__value__, self.__tablename__\n )\n connection = sqlite3.connect(self.sqlite_file)\n cursor = connection.cursor()\n cursor.execute(query)\n r = []\n for key, value in cursor.fetchall():\n r.append((key, value))\n return r", "response": "returns all items off the store as tuples\n "} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the key if exists or a default value if not present", "response": "def get(self, key, default=None):\n \"\"\" Return the key if exists or a default value\n\n :param str value: Value\n :param str default: Default value if key not present\n \"\"\"\n if key in self:\n return self.__getitem__(key)\n else:\n return default"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef delete(self, key):\n query = (\n \"DELETE FROM {} WHERE {}=?\".format(self.__tablename__, self.__key__),\n (key,),\n )\n connection = sqlite3.connect(self.sqlite_file)\n cursor = connection.cursor()\n cursor.execute(*query)\n connection.commit()", "response": "Delete a key from the store."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef exists(self):\n query = (\n \"SELECT name FROM sqlite_master \" + \"WHERE type='table' AND name=?\",\n (self.__tablename__,),\n )\n connection = sqlite3.connect(self.sqlite_file)\n cursor = connection.cursor()\n cursor.execute(*query)\n return True if cursor.fetchone() else False", "response": "Check if the database table exists in the database."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create(self): # pragma: no cover\n query = (\n \"\"\"\n CREATE TABLE {} (\n id INTEGER PRIMARY KEY AUTOINCREMENT,\n {} STRING(256),\n {} STRING(256)\n )\"\"\"\n ).format(self.__tablename__, self.__key__, self.__value__)\n connection = sqlite3.connect(self.sqlite_file)\n cursor = connection.cursor()\n cursor.execute(query)\n connection.commit()", "response": "Create the new table in the SQLite database."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef appendOps(self, ops, append_to=None):\n if isinstance(ops, list):\n self.ops.extend(ops)\n else:\n self.ops.append(ops)\n parent = self.parent\n if parent:\n parent._set_require_reconstruction()", "response": "Append operations to the transaction builder\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning an instance of base Operations for further processing", "response": "def get_raw(self):\n \"\"\" Returns an instance of base \"Operations\" for further processing\n \"\"\"\n if not self.ops:\n return\n ops = [self.operations.Op_wrapper(op=o) for o in list(self.ops)]\n proposer = self.account_class(\n self.proposer, blockchain_instance=self.blockchain\n )\n data = {\n \"fee\": {\"amount\": 0, \"asset_id\": \"1.3.0\"},\n \"fee_paying_account\": proposer[\"id\"],\n \"expiration_time\": formatTimeFromNow(self.proposal_expiration),\n \"proposed_ops\": [o.json() for o in ops],\n \"extensions\": [],\n }\n if self.proposal_review:\n data.update({\"review_period_seconds\": self.proposal_review})\n ops = self.operations.Proposal_create(**data)\n return self.operation_class(ops)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef json(self):\n if not self._is_constructed() or self._is_require_reconstruction():\n self.constructTx()\n return dict(self)", "response": "Show the transaction as plain json"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef appendSigner(self, accounts, permission):\n assert permission in self.permission_types, \"Invalid permission\"\n\n if self.blockchain.wallet.locked():\n raise WalletLocked()\n if not isinstance(accounts, (list, tuple, set)):\n accounts = [accounts]\n\n for account in accounts:\n # Now let's actually deal with the accounts\n if account not in self.signing_accounts:\n # is the account an instance of public key?\n if isinstance(account, self.publickey_class):\n self.appendWif(\n self.blockchain.wallet.getPrivateKeyForPublicKey(str(account))\n )\n # ... or should we rather obtain the keys from an account name\n else:\n accountObj = self.account_class(\n account, blockchain_instance=self.blockchain\n )\n required_treshold = accountObj[permission][\"weight_threshold\"]\n keys = self._fetchkeys(\n accountObj, permission, required_treshold=required_treshold\n )\n # If we couldn't find an active key, let's try overwrite it\n # with an owner key\n if not keys and permission != \"owner\":\n keys.extend(\n self._fetchkeys(\n accountObj, \"owner\", required_treshold=required_treshold\n )\n )\n for x in keys:\n self.appendWif(x[0])\n\n self.signing_accounts.append(account)", "response": "Append a signer to the wallet by telling which accounts are supposed to sign the transaction."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding a wif that should be used for signing of the transaction.", "response": "def appendWif(self, wif):\n \"\"\" Add a wif that should be used for signing of the transaction.\n \"\"\"\n if wif:\n try:\n self.privatekey_class(wif)\n self.wifs.add(wif)\n except Exception:\n raise InvalidWifError"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsetting asset to fee_asset", "response": "def set_fee_asset(self, fee_asset):\n \"\"\" Set asset to fee\n \"\"\"\n if isinstance(fee_asset, self.amount_class):\n self.fee_asset_id = fee_asset[\"id\"]\n elif isinstance(fee_asset, self.asset_class):\n self.fee_asset_id = fee_asset[\"id\"]\n elif fee_asset:\n self.fee_asset_id = fee_asset\n else:\n self.fee_asset_id = \"1.3.0\""} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd the required fees to the operations that are required for a witness node.", "response": "def add_required_fees(self, ops, asset_id=\"1.3.0\"):\n \"\"\" Auxiliary method to obtain the required fees for a set of\n operations. Requires a websocket connection to a witness node!\n \"\"\"\n ws = self.blockchain.rpc\n fees = ws.get_required_fees([i.json() for i in ops], asset_id)\n for i, d in enumerate(ops):\n if isinstance(fees[i], list):\n # Operation is a proposal\n ops[i].op.data[\"fee\"] = Asset(\n amount=fees[i][0][\"amount\"], asset_id=fees[i][0][\"asset_id\"]\n )\n for j, _ in enumerate(ops[i].op.data[\"proposed_ops\"].data):\n ops[i].op.data[\"proposed_ops\"].data[j].data[\"op\"].op.data[\n \"fee\"\n ] = Asset(\n amount=fees[i][1][j][\"amount\"],\n asset_id=fees[i][1][j][\"asset_id\"],\n )\n else:\n # Operation is a regular operation\n ops[i].op.data[\"fee\"] = Asset(\n amount=fees[i][\"amount\"], asset_id=fees[i][\"asset_id\"]\n )\n return ops"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef constructTx(self):\n ops = list()\n for op in self.ops:\n if isinstance(op, ProposalBuilder):\n # This operation is a proposal an needs to be deal with\n # differently\n proposal = op.get_raw()\n if proposal:\n ops.append(proposal)\n elif isinstance(op, self.operation_class):\n ops.extend([op])\n else:\n # otherwise, we simply wrap ops into Operations\n ops.extend([self.operation_class(op)])\n\n # We now wrap everything into an actual transaction\n ops = self.add_required_fees(ops, asset_id=self.fee_asset_id)\n expiration = formatTimeFromNow(\n self.expiration\n or self.blockchain.expiration\n or 30 # defaults to 30 seconds\n )\n ref_block_num, ref_block_prefix = self.get_block_params()\n self.tx = self.signed_transaction_class(\n ref_block_num=ref_block_num,\n ref_block_prefix=ref_block_prefix,\n expiration=expiration,\n operations=ops,\n )\n dict.update(self, self.tx.json())\n self._unset_require_reconstruction()", "response": "Construct the actual transaction and store it in the class s dict\n store"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_block_params(self):\n ws = self.blockchain.rpc\n dynBCParams = ws.get_dynamic_global_properties()\n ref_block_num = dynBCParams[\"head_block_number\"] & 0xFFFF\n ref_block_prefix = struct.unpack_from(\n \" 2, \"Incorrect number of message parts\"\n\n # Strip away all whitespaces before and after the message\n message = parts[0].strip()\n signature = parts[2].strip()\n # Parse the meta data\n meta = dict(re.findall(r\"(\\S+)=(.*)\", parts[1]))\n\n log.info(\"Message is: {}\".format(message))\n log.info(\"Meta is: {}\".format(json.dumps(meta)))\n log.info(\"Signature is: {}\".format(signature))\n\n # Ensure we have all the data in meta\n assert \"account\" in meta, \"No 'account' could be found in meta data\"\n assert \"memokey\" in meta, \"No 'memokey' could be found in meta data\"\n assert \"block\" in meta, \"No 'block' could be found in meta data\"\n assert \"timestamp\" in meta, \"No 'timestamp' could be found in meta data\"\n\n account_name = meta.get(\"account\").strip()\n memo_key = meta[\"memokey\"].strip()\n\n try:\n self.publickey_class(memo_key, prefix=self.blockchain.prefix)\n except Exception:\n raise InvalidMemoKeyException(\"The memo key in the message is invalid\")\n\n # Load account from blockchain\n try:\n account = self.account_class(\n account_name, blockchain_instance=self.blockchain\n )\n except AccountDoesNotExistsException:\n raise AccountDoesNotExistsException(\n \"Could not find account {}. Are you connected to the right chain?\".format(\n account_name\n )\n )\n\n # Test if memo key is the same as on the blockchain\n if not account[\"options\"][\"memo_key\"] == memo_key:\n raise WrongMemoKey(\n \"Memo Key of account {} on the Blockchain \".format(account[\"name\"])\n + \"differs from memo key in the message: {} != {}\".format(\n account[\"options\"][\"memo_key\"], memo_key\n )\n )\n\n # Reformat message\n enc_message = self.SIGNED_MESSAGE_META.format(**locals())\n\n # Verify Signature\n pubkey = verify_message(enc_message, unhexlify(signature))\n\n # Verify pubky\n pk = self.publickey_class(\n hexlify(pubkey).decode(\"ascii\"), prefix=self.blockchain.prefix\n )\n if format(pk, self.blockchain.prefix) != memo_key:\n raise InvalidMessageSignature(\"The signature doesn't match the memo key\")\n\n self.signed_by_account = account\n self.signed_by_name = account[\"name\"]\n self.meta = meta\n self.plain_message = message\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sign(self, account=None, **kwargs):\n if not account:\n if \"default_account\" in self.blockchain.config:\n account = self.blockchain.config[\"default_account\"]\n if not account:\n raise ValueError(\"You need to provide an account\")\n\n # Data for message\n account = self.account_class(account, blockchain_instance=self.blockchain)\n\n # wif key\n wif = self.blockchain.wallet.getPrivateKeyForPublicKey(\n account[\"options\"][\"memo_key\"]\n )\n\n payload = [\n \"from\",\n account[\"name\"],\n \"key\",\n account[\"options\"][\"memo_key\"],\n \"time\",\n str(datetime.utcnow()),\n \"text\",\n self.message,\n ]\n enc_message = json.dumps(payload, separators=(\",\", \":\"))\n\n # signature\n signature = hexlify(sign_message(enc_message, wif)).decode(\"ascii\")\n\n return dict(signed=enc_message, payload=payload, signature=signature)", "response": "Sign a message with an account s memo key"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef verify(self, **kwargs):\n if not isinstance(self.message, dict):\n try:\n self.message = json.loads(self.message)\n except Exception:\n raise ValueError(\"Message must be valid JSON\")\n\n payload = self.message.get(\"payload\")\n assert payload, \"Missing payload\"\n payload_dict = {k[0]: k[1] for k in zip(payload[::2], payload[1::2])}\n signature = self.message.get(\"signature\")\n\n account_name = payload_dict.get(\"from\").strip()\n memo_key = payload_dict.get(\"key\").strip()\n\n assert account_name, \"Missing account name 'from'\"\n assert memo_key, \"missing 'key'\"\n\n try:\n self.publickey_class(memo_key, prefix=self.blockchain.prefix)\n except Exception:\n raise InvalidMemoKeyException(\"The memo key in the message is invalid\")\n\n # Load account from blockchain\n try:\n account = self.account_class(\n account_name, blockchain_instance=self.blockchain\n )\n except AccountDoesNotExistsException:\n raise AccountDoesNotExistsException(\n \"Could not find account {}. Are you connected to the right chain?\".format(\n account_name\n )\n )\n\n # Test if memo key is the same as on the blockchain\n if not account[\"options\"][\"memo_key\"] == memo_key:\n raise WrongMemoKey(\n \"Memo Key of account {} on the Blockchain \".format(account[\"name\"])\n + \"differs from memo key in the message: {} != {}\".format(\n account[\"options\"][\"memo_key\"], memo_key\n )\n )\n\n # Ensure payload and signed match\n signed_target = json.dumps(self.message.get(\"payload\"), separators=(\",\", \":\"))\n signed_actual = self.message.get(\"signed\")\n assert (\n signed_target == signed_actual\n ), \"payload doesn't match signed message: \\n{}\\n{}\".format(\n signed_target, signed_actual\n )\n\n # Reformat message\n enc_message = self.message.get(\"signed\")\n\n # Verify Signature\n pubkey = verify_message(enc_message, unhexlify(signature))\n\n # Verify pubky\n pk = self.publickey_class(\n hexlify(pubkey).decode(\"ascii\"), prefix=self.blockchain.prefix\n )\n if format(pk, self.blockchain.prefix) != memo_key:\n raise InvalidMessageSignature(\"The signature doesn't match the memo key\")\n\n self.signed_by_account = account\n self.signed_by_name = account[\"name\"]\n self.plain_message = payload_dict.get(\"text\")\n\n return True", "response": "Verify a message with an account s memo key"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef cmd(command):\n\n env()\n\n ipmi = cij.env_to_dict(PREFIX, EXPORTED + REQUIRED)\n\n command = \"ipmitool -U %s -P %s -H %s -p %s %s\" % (\n ipmi[\"USER\"], ipmi[\"PASS\"], ipmi[\"HOST\"], ipmi[\"PORT\"], command)\n cij.info(\"ipmi.command: %s\" % command)\n\n return cij.util.execute(command, shell=True, echo=True)", "response": "Send IPMI command via ipmitool"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfinds the given pattern in content", "response": "def regex_find(pattern, content):\n \"\"\"Find the given 'pattern' in 'content'\"\"\"\n\n find = re.findall(pattern, content)\n if not find:\n cij.err(\"pattern <%r> is invalid, no matches!\" % pattern)\n cij.err(\"content: %r\" % content)\n return ''\n\n if len(find) >= 2:\n cij.err(\"pattern <%r> is too simple, matched more than 2!\" % pattern)\n cij.err(\"content: %r\" % content)\n return ''\n\n return find[0]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nexecutes the given cmd and return the return code stdout and stderr.", "response": "def execute(cmd=None, shell=True, echo=True):\n \"\"\"\n Execute the given 'cmd'\n\n @returns (rcode, stdout, stderr)\n \"\"\"\n if echo:\n cij.emph(\"cij.util.execute: shell: %r, cmd: %r\" % (shell, cmd))\n\n rcode = 1\n stdout, stderr = (\"\", \"\")\n\n if cmd:\n if shell:\n cmd = \" \".join(cmd)\n\n proc = Popen(cmd, stdout=PIPE, stderr=PIPE, shell=shell, close_fds=True)\n stdout, stderr = proc.communicate()\n rcode = proc.returncode\n\n if rcode and echo:\n cij.warn(\"cij.util.execute: stdout: %s\" % stdout)\n cij.err(\"cij.util.execute: stderr: %s\" % stderr)\n cij.err(\"cij.util.execute: rcode: %s\" % rcode)\n\n return rcode, stdout, stderr"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef env():\n\n if cij.ssh.env():\n cij.err(\"board.env: invalid SSH environment\")\n return 1\n\n board = cij.env_to_dict(PREFIX, REQUIRED) # Verify REQUIRED variables\n if board is None:\n cij.err(\"board.env: invalid BOARD environment\")\n return 1\n\n board[\"CLASS\"] = \"_\".join([board[r] for r in REQUIRED[:-1]])\n board[\"IDENT\"] = \"-\".join([board[\"CLASS\"], board[\"ALIAS\"]])\n\n cij.env_export(PREFIX, EXPORTED, board) # Export EXPORTED variables\n\n return 0", "response": "Verify BOARD variables and construct exported variables"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncat file and return content", "response": "def cat_file(path):\n \"\"\"Cat file and return content\"\"\"\n\n cmd = [\"cat\", path]\n status, stdout, _ = cij.ssh.command(cmd, shell=True, echo=True)\n if status:\n raise RuntimeError(\"cij.nvme.env: cat %s failed\" % path)\n return stdout.strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nverifies NVME variables and construct exported variables", "response": "def env():\n \"\"\"Verify NVME variables and construct exported variables\"\"\"\n\n if cij.ssh.env():\n cij.err(\"cij.nvme.env: invalid SSH environment\")\n return 1\n\n nvme = cij.env_to_dict(PREFIX, REQUIRED)\n\n nvme[\"DEV_PATH\"] = os.path.join(\"/dev\", nvme[\"DEV_NAME\"])\n\n # get version, chunks, luns and chs\n try:\n sysfs = os.path.join(\"/sys/class/block\", nvme[\"DEV_NAME\"], \"lightnvm\")\n\n nvme[\"LNVM_VERSION\"] = cat_file(os.path.join(sysfs, \"version\"))\n if nvme[\"LNVM_VERSION\"] == \"2.0\":\n luns = \"punits\"\n chs = \"groups\"\n elif nvme[\"LNVM_VERSION\"] == \"1.2\":\n luns = \"num_luns\"\n chs = \"num_channels\"\n else:\n raise RuntimeError(\"cij.nvme.env: invalid lnvm version: %s\" % nvme[\"LNVM_VERSION\"])\n\n nvme[\"LNVM_NUM_CHUNKS\"] = cat_file(os.path.join(sysfs, \"chunks\"))\n nvme[\"LNVM_NUM_LUNS\"] = cat_file(os.path.join(sysfs, luns))\n nvme[\"LNVM_NUM_CHS\"] = cat_file(os.path.join(sysfs, chs))\n\n nvme[\"LNVM_TOTAL_LUNS\"] = str(int(nvme[\"LNVM_NUM_LUNS\"]) * int(nvme[\"LNVM_NUM_CHS\"]))\n nvme[\"LNVM_TOTAL_CHUNKS\"] = str(int(nvme[\"LNVM_TOTAL_LUNS\"]) * int(nvme[\"LNVM_NUM_CHUNKS\"]))\n\n # get spec version by identify namespace data struct\n if nvme[\"LNVM_VERSION\"] == \"2.0\":\n cmd = [\"nvme\", \"id-ctrl\", nvme[\"DEV_PATH\"], \"--raw-binary\"]\n status, stdout, _ = cij.ssh.command(cmd, shell=True)\n if status:\n raise RuntimeError(\"cij.nvme.env: nvme id-ctrl fail\")\n\n buff = cij.bin.Buffer(types=IdentifyCDS, length=1)\n buff.memcopy(stdout)\n\n if buff[0].VS[1023] == 0x5a:\n nvme[\"SPEC_VERSION\"] = \"Denali\"\n else:\n nvme[\"SPEC_VERSION\"] = \"Spec20\"\n else:\n nvme[\"SPEC_VERSION\"] = \"Spec12\"\n\n # get chunk meta information\n nvme[\"LNVM_CHUNK_META_LENGTH\"] = str(get_sizeof_descriptor_table(nvme[\"SPEC_VERSION\"]))\n nvme[\"LNVM_CHUNK_META_SIZE\"] = str(int(nvme[\"LNVM_CHUNK_META_LENGTH\"]) *\n int(nvme[\"LNVM_TOTAL_CHUNKS\"]))\n\n except StandardError:\n traceback.print_exc()\n return 1\n\n cij.env_export(PREFIX, EXPORTED, nvme)\n\n return 0"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fmt(lbaf=3):\n\n if env():\n cij.err(\"cij.nvme.exists: Invalid NVMe ENV.\")\n return 1\n\n nvme = cij.env_to_dict(PREFIX, EXPORTED + REQUIRED)\n\n cmd = [\"nvme\", \"format\", nvme[\"DEV_PATH\"], \"-l\", str(lbaf)]\n rcode, _, _ = cij.ssh.command(cmd, shell=True)\n\n return rcode", "response": "Do format for NVMe device"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nget the meta of the NVMe device.", "response": "def get_meta(offset, length, output):\n \"\"\"Get chunk meta of NVMe device\"\"\"\n\n if env():\n cij.err(\"cij.nvme.meta: Invalid NVMe ENV.\")\n return 1\n\n nvme = cij.env_to_dict(PREFIX, EXPORTED + REQUIRED)\n\n max_size = 0x40000\n with open(output, \"wb\") as fout:\n for off in range(offset, length, max_size):\n size = min(length - off, max_size)\n cmd = [\"nvme get-log\",\n nvme[\"DEV_PATH\"],\n \"-i 0xca\",\n \"-o 0x%x\" % off,\n \"-l 0x%x\" % size,\n \"-b\"]\n status, stdout, _ = cij.ssh.command(cmd, shell=True)\n if status:\n cij.err(\"cij.nvme.meta: Error get chunk meta\")\n return 1\n\n fout.write(stdout)\n\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncompares chunk meta files", "response": "def comp_meta(file_bef, file_aft, mode=\"pfail\"):\n \"\"\"Compare chunk meta, mode=[pfail, power, reboot]\"\"\"\n if env():\n cij.err(\"cij.nvme.comp_meta: Invalid NVMe ENV.\")\n return 1\n\n nvme = cij.env_to_dict(PREFIX, EXPORTED + REQUIRED)\n num_chk = int(nvme[\"LNVM_TOTAL_CHUNKS\"])\n\n meta_bef = cij.bin.Buffer(types=get_descriptor_table(nvme['SPEC_VERSION']), length=num_chk)\n meta_aft = cij.bin.Buffer(types=get_descriptor_table(nvme['SPEC_VERSION']), length=num_chk)\n meta_bef.read(file_bef)\n meta_aft.read(file_aft)\n\n for chk in range(num_chk):\n ignore = [\"WL\", \"RSV0\"]\n\n # PFAIL: BEFORE IS OPEN CHUNK, WRITE POINTER IS NOT SURE, IGNORE\n if mode == \"pfail\" and meta_bef[chk].CS == 4:\n ignore.append(\"WP\")\n\n # COMPARE CHUNK META\n if meta_bef.compare(meta_aft, chk, ignore=ignore):\n cij.warn(\"META_BUFF_BEF[%s]:\" % chk)\n meta_bef.dump(chk)\n cij.warn(\"META_BUFF_AFT[%s]:\" % chk)\n meta_aft.dump(chk)\n cij.err(\"Error compare, CHUNK: %s\" % chk)\n return 1\n\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_sizeof_descriptor_table(version=\"Denali\"):\r\n if version == \"Denali\":\r\n return sizeof(DescriptorTableDenali)\r\n elif version == \"Spec20\":\r\n return sizeof(DescriptorTableSpec20)\r\n elif version == \"Spec12\":\r\n return 0\r\n else:\r\n raise RuntimeError(\"Error version!\")", "response": "Get the size of the descriptor table."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef env():\n\n if cij.ssh.env():\n cij.err(\"cij.lnvm.env: invalid SSH environment\")\n return 1\n\n lnvm = cij.env_to_dict(PREFIX, REQUIRED)\n nvme = cij.env_to_dict(\"NVME\", [\"DEV_NAME\"])\n\n if \"BGN\" not in lnvm.keys():\n cij.err(\"cij.lnvm.env: invalid LNVM_BGN\")\n return 1\n if \"END\" not in lnvm.keys():\n cij.err(\"cij.lnvm.env: invalid LNVM_END\")\n return 1\n if \"DEV_TYPE\" not in lnvm.keys():\n cij.err(\"cij.lnvm.env: invalid LNVM_DEV_TYPE\")\n return 1\n\n lnvm[\"DEV_NAME\"] = \"%sb%03de%03d\" % (nvme[\"DEV_NAME\"], int(lnvm[\"BGN\"]), int(lnvm[\"END\"]))\n lnvm[\"DEV_PATH\"] = \"/dev/%s\" % lnvm[\"DEV_NAME\"]\n\n cij.env_export(PREFIX, EXPORTED, lnvm)\n\n return 0", "response": "Verify LNVM variables and construct exported variables"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndump UnionType or StructType to STDOUT", "response": "def dump(buf, indent=0, skip=\"\"):\n \"\"\"Dump UnionType/StructType to STDOUT\"\"\"\n if not isinstance(type(buf), (type(Union), type(Structure))):\n raise RuntimeError(\"Error type(%s)\" % type(buf))\n\n for field in getattr(buf, '_fields_'):\n name, types = field[0], field[1]\n if name in skip:\n return\n value = getattr(buf, name)\n\n if isinstance(types, (type(Union), type(Structure))):\n cij.info(\"%s%s:\" % (\" \" * indent, name))\n dump(value, indent+2, skip)\n elif isinstance(types, type(Array)):\n for i, item in enumerate(value):\n name_index = \"%s[%s]\" % (name, i)\n\n if isinstance(types, (type(Union), type(Structure))):\n cij.info(\"%s%s:\" % (\" \" * indent, name_index))\n dump(item, indent + 2, skip)\n else:\n cij.info(\"%s%-12s: 0x%x\" % (\" \" * indent, name_index, item))\n else:\n cij.info(\"%s%-12s: 0x%x\" % (\" \" * indent, name, value))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncompares of two Buffer item", "response": "def compare(buf_a, buf_b, ignore):\n \"\"\"Compare of two Buffer item\"\"\"\n for field in getattr(buf_a, '_fields_'):\n name, types = field[0], field[1]\n\n if name in ignore:\n continue\n\n val_a = getattr(buf_a, name)\n val_b = getattr(buf_b, name)\n\n if isinstance(types, (type(Union), type(Structure))):\n if compare(val_a, val_b, ignore):\n return 1\n elif isinstance(types, type(Array)):\n for i, _ in enumerate(val_a):\n if isinstance(types, (type(Union), type(Structure))):\n if compare(val_a[i], val_b[i], ignore):\n return 1\n else:\n if val_a[i] != val_b[i]:\n return 1\n else:\n if val_a != val_b:\n return 1\n\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef memcopy(self, stream, offset=0, length=float(\"inf\")):\n data = [ord(i) for i in list(stream)]\n size = min(length, len(data), self.m_size)\n buff = cast(self.m_buf, POINTER(c_uint8))\n for i in range(size):\n buff[offset + i] = data[i]", "response": "Copy stream to buffer"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrite buffer to file", "response": "def write(self, path):\n \"\"\"Write buffer to file\"\"\"\n\n with open(path, \"wb\") as fout:\n fout.write(self.m_buf)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef read(self, path):\n\n with open(path, \"rb\") as fout:\n memmove(self.m_buf, fout.read(self.m_size), self.m_size)", "response": "Read file to buffer"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dump(self, offset=0, length=1):\n\n for i in range(offset, offset + length):\n if \"ctypes\" in str(self.m_types):\n cij.info(\"Buff[%s]: %s\" % (i, self.m_buf[i]))\n else:\n cij.info(\"Buff[%s]:\" % i)\n dump(self.m_buf[i], 2)", "response": "Dump the contents of the buffer to the console."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomparing the buffer with the current one.", "response": "def compare(self, buf, offset=0, length=1, ignore=\"\"):\n \"\"\"Compare buffer\"\"\"\n\n for i in range(offset, offset + length):\n if isinstance(self.m_types, (type(Union), type(Structure))):\n if compare(self.m_buf[i], buf[i], ignore=ignore):\n return 1\n elif self.m_buf[i] != buf[i]:\n return 1\n\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks the chunk is offline or not", "response": "def is_bad_chunk(self, chk, yml):\r\n \"\"\"Check the chunk is offline or not\"\"\"\r\n cs = self.get_chunk_status(chk, yml)\r\n if cs >= 8:\r\n return True\r\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck the chunk is free or not", "response": "def is_free_chunk(self, chk):\r\n \"\"\"Check the chunk is free or not\"\"\"\r\n cs = self.get_chunk_status(chk)\r\n if cs & 0x1 != 0:\r\n return True\r\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks the chunk is free or not", "response": "def is_closed_chunk(self, chk):\r\n \"\"\"Check the chunk is free or not\"\"\"\r\n cs = self.get_chunk_status(chk)\r\n if cs & 0x2 != 0:\r\n return True\r\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_open_chunk(self, chk):\r\n cs = self.get_chunk_status(chk)\r\n if cs & 0x4 != 0:\r\n return True\r\n return False", "response": "Check the chunk is free or not"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef vblk_erase(self, address):\r\n cmd = [\"nvm_vblk erase\", self.envs, \"0x%x\" % address]\r\n status, _, _ = cij.ssh.command(cmd, shell=True)\r\n return status", "response": "Erase the vblk entry at the given address."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nerasing the current set of pages from the specified address.", "response": "def slc_erase(self, address, BE_ID=0x1, PMODE=0x0100):\r\n \"\"\"slc erase\"\"\"\r\n cmd = [\"NVM_CLI_BE_ID=0x%x\" % BE_ID, \"NVM_CLI_PMODE=0x%x\" % PMODE, \"nvm_cmd erase\", self.envs, \"0x%x\" % address]\r\n status, _, _ = cij.ssh.command(cmd, shell=True)\r\n return status"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef env():\n\n if cij.ssh.env():\n cij.err(\"cij.block.env: invalid SSH environment\")\n return 1\n\n block = cij.env_to_dict(PREFIX, REQUIRED)\n\n block[\"DEV_PATH\"] = \"/dev/%s\" % block[\"DEV_NAME\"]\n\n cij.env_export(PREFIX, EXPORTED, block)\n\n return 0", "response": "Verify BLOCK variables and construct exported variables"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting a script or testcase", "response": "def script_run(trun, script):\n \"\"\"Execute a script or testcase\"\"\"\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:script:run { script: %s }\" % script)\n cij.emph(\"rnr:script:run:evars: %s\" % script[\"evars\"])\n\n launchers = {\n \".py\": \"python\",\n \".sh\": \"source\"\n }\n\n ext = os.path.splitext(script[\"fpath\"])[-1]\n if not ext in launchers.keys():\n cij.err(\"rnr:script:run { invalid script[\\\"fpath\\\"]: %r }\" % script[\"fpath\"])\n return 1\n\n launch = launchers[ext]\n\n with open(script[\"log_fpath\"], \"a\") as log_fd:\n log_fd.write(\"# script_fpath: %r\\n\" % script[\"fpath\"])\n log_fd.flush()\n\n bgn = time.time()\n cmd = [\n 'bash', '-c',\n 'CIJ_ROOT=$(cij_root) && '\n 'source $CIJ_ROOT/modules/cijoe.sh && '\n 'source %s && '\n 'CIJ_TEST_RES_ROOT=\"%s\" %s %s ' % (\n trun[\"conf\"][\"ENV_FPATH\"],\n script[\"res_root\"],\n launch,\n script[\"fpath\"]\n )\n ]\n if trun[\"conf\"][\"VERBOSE\"] > 1:\n cij.emph(\"rnr:script:run { cmd: %r }\" % \" \".join(cmd))\n\n evars = os.environ.copy()\n evars.update({k: str(script[\"evars\"][k]) for k in script[\"evars\"]})\n\n process = Popen(\n cmd,\n stdout=log_fd,\n stderr=STDOUT,\n cwd=script[\"res_root\"],\n env=evars\n )\n process.wait()\n\n script[\"rcode\"] = process.returncode\n script[\"wallc\"] = time.time() - bgn\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:script:run { wallc: %02f }\" % script[\"wallc\"])\n cij.emph(\n \"rnr:script:run { rcode: %r } \" % script[\"rcode\"],\n script[\"rcode\"]\n )\n\n return script[\"rcode\"]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef trun_to_file(trun, fpath=None):\n\n if fpath is None:\n fpath = yml_fpath(trun[\"conf\"][\"OUTPUT\"])\n\n with open(fpath, 'w') as yml_file:\n data = yaml.dump(trun, explicit_start=True, default_flow_style=False)\n yml_file.write(data)", "response": "Dump the given trun to file"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef trun_emph(trun):\n\n if trun[\"conf\"][\"VERBOSE\"] > 1: # Print environment variables\n cij.emph(\"rnr:CONF {\")\n for cvar in sorted(trun[\"conf\"].keys()):\n cij.emph(\" % 16s: %r\" % (cvar, trun[\"conf\"][cvar]))\n cij.emph(\"}\")\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:INFO {\")\n cij.emph(\" OUTPUT: %r\" % trun[\"conf\"][\"OUTPUT\"])\n cij.emph(\" yml_fpath: %r\" % yml_fpath(trun[\"conf\"][\"OUTPUT\"]))\n cij.emph(\"}\")", "response": "Print essential info on"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates and initialize a testcase", "response": "def tcase_setup(trun, parent, tcase_fname):\n \"\"\"\n Create and initialize a testcase\n \"\"\"\n #pylint: disable=locally-disabled, unused-argument\n\n case = copy.deepcopy(TESTCASE)\n\n case[\"fname\"] = tcase_fname\n case[\"fpath_orig\"] = os.sep.join([trun[\"conf\"][\"TESTCASES\"], case[\"fname\"]])\n\n if not os.path.exists(case[\"fpath_orig\"]):\n cij.err('rnr:tcase_setup: !case[\"fpath_orig\"]: %r' % case[\"fpath_orig\"])\n return None\n\n case[\"name\"] = os.path.splitext(case[\"fname\"])[0]\n case[\"ident\"] = \"/\".join([parent[\"ident\"], case[\"fname\"]])\n\n case[\"res_root\"] = os.sep.join([parent[\"res_root\"], case[\"fname\"]])\n case[\"aux_root\"] = os.sep.join([case[\"res_root\"], \"_aux\"])\n case[\"log_fpath\"] = os.sep.join([case[\"res_root\"], \"run.log\"])\n\n case[\"fpath\"] = os.sep.join([case[\"res_root\"], case[\"fname\"]])\n\n case[\"evars\"].update(copy.deepcopy(parent[\"evars\"]))\n\n # Initalize\n os.makedirs(case[\"res_root\"]) # Create DIRS\n os.makedirs(case[\"aux_root\"])\n shutil.copyfile(case[\"fpath_orig\"], case[\"fpath\"]) # Copy testcase\n\n # Initialize hooks\n case[\"hooks\"] = hooks_setup(trun, case, parent.get(\"hooks_pr_tcase\"))\n\n return case"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntriggers when exiting the given testsuite", "response": "def tsuite_exit(trun, tsuite):\n \"\"\"Triggers when exiting the given testsuite\"\"\"\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:tsuite:exit\")\n\n rcode = 0\n for hook in reversed(tsuite[\"hooks\"][\"exit\"]): # EXIT-hooks\n rcode = script_run(trun, hook)\n if rcode:\n break\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:tsuite:exit { rcode: %r } \" % rcode, rcode)\n\n return rcode"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntrigger when entering the given testsuite", "response": "def tsuite_enter(trun, tsuite):\n \"\"\"Triggers when entering the given testsuite\"\"\"\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:tsuite:enter { name: %r }\" % tsuite[\"name\"])\n\n rcode = 0\n for hook in tsuite[\"hooks\"][\"enter\"]: # ENTER-hooks\n rcode = script_run(trun, hook)\n if rcode:\n break\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:tsuite:enter { rcode: %r } \" % rcode, rcode)\n\n return rcode"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate and initialize a new TESTSUITE struct and site - effects such as creating a TESTSUITE output directories and forwarding initialization of testcases.", "response": "def tsuite_setup(trun, declr, enum):\n \"\"\"\n Creates and initialized a TESTSUITE struct and site-effects such as creating\n output directories and forwarding initialization of testcases\n \"\"\"\n\n suite = copy.deepcopy(TESTSUITE) # Setup the test-suite\n\n suite[\"name\"] = declr.get(\"name\")\n if suite[\"name\"] is None:\n cij.err(\"rnr:tsuite_setup: no testsuite is given\")\n return None\n\n suite[\"alias\"] = declr.get(\"alias\")\n suite[\"ident\"] = \"%s_%d\" % (suite[\"name\"], enum)\n\n suite[\"res_root\"] = os.sep.join([trun[\"conf\"][\"OUTPUT\"], suite[\"ident\"]])\n suite[\"aux_root\"] = os.sep.join([suite[\"res_root\"], \"_aux\"])\n\n suite[\"evars\"].update(copy.deepcopy(trun[\"evars\"]))\n suite[\"evars\"].update(copy.deepcopy(declr.get(\"evars\", {})))\n\n # Initialize\n os.makedirs(suite[\"res_root\"])\n os.makedirs(suite[\"aux_root\"])\n\n # Setup testsuite-hooks\n suite[\"hooks\"] = hooks_setup(trun, suite, declr.get(\"hooks\"))\n\n # Forward from declaration\n suite[\"hooks_pr_tcase\"] = declr.get(\"hooks_pr_tcase\", [])\n\n suite[\"fname\"] = \"%s.suite\" % suite[\"name\"]\n suite[\"fpath\"] = os.sep.join([trun[\"conf\"][\"TESTSUITES\"], suite[\"fname\"]])\n\n #\n # Load testcases from .suite file OR from declaration\n #\n tcase_fpaths = [] # Load testcase fpaths\n if os.path.exists(suite[\"fpath\"]): # From suite-file\n suite_lines = (\n l.strip() for l in open(suite[\"fpath\"]).read().splitlines()\n )\n tcase_fpaths.extend(\n (l for l in suite_lines if len(l) > 1 and l[0] != \"#\")\n )\n else: # From declaration\n tcase_fpaths.extend(declr.get(\"testcases\", []))\n\n # NOTE: fix duplicates; allow them\n # NOTE: Currently hot-fixed here\n if len(set(tcase_fpaths)) != len(tcase_fpaths):\n cij.err(\"rnr:suite: failed: duplicate tcase in suite not supported\")\n return None\n\n for tcase_fname in tcase_fpaths: # Setup testcases\n tcase = tcase_setup(trun, suite, tcase_fname)\n if not tcase:\n cij.err(\"rnr:suite: failed: tcase_setup\")\n return None\n\n suite[\"testcases\"].append(tcase)\n\n return suite"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexits tcase - exit", "response": "def tcase_exit(trun, tsuite, tcase):\n \"\"\"...\"\"\"\n #pylint: disable=locally-disabled, unused-argument\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:tcase:exit { fname: %r }\" % tcase[\"fname\"])\n\n rcode = 0\n for hook in reversed(tcase[\"hooks\"][\"exit\"]): # tcase EXIT-hooks\n rcode = script_run(trun, hook)\n if rcode:\n break\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:tcase:exit { rcode: %r }\" % rcode, rcode)\n\n return rcode"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef trun_exit(trun):\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:trun:exit\")\n\n rcode = 0\n for hook in reversed(trun[\"hooks\"][\"exit\"]): # EXIT-hooks\n rcode = script_run(trun, hook)\n if rcode:\n break\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:trun::exit { rcode: %r }\" % rcode, rcode)\n\n return rcode", "response": "Triggers when exiting the given testrun"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntrigger when entering the given testrun", "response": "def trun_enter(trun):\n \"\"\"Triggers when entering the given testrun\"\"\"\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:trun::enter\")\n\n trun[\"stamp\"][\"begin\"] = int(time.time()) # Record start timestamp\n\n rcode = 0\n for hook in trun[\"hooks\"][\"enter\"]: # ENTER-hooks\n rcode = script_run(trun, hook)\n if rcode:\n break\n\n if trun[\"conf\"][\"VERBOSE\"]:\n cij.emph(\"rnr:trun::enter { rcode: %r }\" % rcode, rcode)\n\n return rcode"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef main(conf):\n\n fpath = yml_fpath(conf[\"OUTPUT\"])\n if os.path.exists(fpath): # YAML exists, we exit, it might be RUNNING!\n cij.err(\"main:FAILED { fpath: %r }, exists\" % fpath)\n return 1\n\n trun = trun_setup(conf) # Construct 'trun' from 'conf'\n if not trun:\n return 1\n\n trun_to_file(trun) # Persist trun\n trun_emph(trun) # Print trun before run\n\n tr_err = 0\n tr_ent_err = trun_enter(trun)\n for tsuite in (ts for ts in trun[\"testsuites\"] if not tr_ent_err):\n\n ts_err = 0\n ts_ent_err = tsuite_enter(trun, tsuite)\n for tcase in (tc for tc in tsuite[\"testcases\"] if not ts_ent_err):\n\n tc_err = tcase_enter(trun, tsuite, tcase)\n if not tc_err:\n tc_err += script_run(trun, tcase)\n tc_err += tcase_exit(trun, tsuite, tcase)\n\n tcase[\"status\"] = \"FAIL\" if tc_err else \"PASS\"\n\n trun[\"progress\"][tcase[\"status\"]] += 1 # Update progress\n trun[\"progress\"][\"UNKN\"] -= 1\n\n ts_err += tc_err # Accumulate errors\n\n trun_to_file(trun) # Persist trun\n\n if not ts_ent_err:\n ts_err += tsuite_exit(trun, tsuite)\n\n ts_err += ts_ent_err # Accumulate errors\n tr_err += ts_err\n\n tsuite[\"status\"] = \"FAIL\" if ts_err else \"PASS\"\n\n cij.emph(\"rnr:tsuite %r\" % tsuite[\"status\"], tsuite[\"status\"] != \"PASS\")\n\n if not tr_ent_err:\n trun_exit(trun)\n\n tr_err += tr_ent_err\n trun[\"status\"] = \"FAIL\" if tr_err else \"PASS\"\n\n trun[\"stamp\"][\"end\"] = int(time.time()) + 1 # END STAMP\n trun_to_file(trun) # PERSIST\n\n cij.emph(\"rnr:main:progress %r\" % trun[\"progress\"])\n cij.emph(\"rnr:main:trun %r\" % trun[\"status\"], trun[\"status\"] != \"PASS\")\n\n return trun[\"progress\"][\"UNKN\"] + trun[\"progress\"][\"FAIL\"]", "response": "Main entry point for CIJ Test Runner."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets chunk meta table", "response": "def get_chunk_meta(self, meta_file):\n \"\"\"Get chunk meta table\"\"\"\n chunks = self.envs[\"CHUNKS\"]\n if cij.nvme.get_meta(0, chunks * self.envs[\"CHUNK_META_SIZEOF\"], meta_file):\n raise RuntimeError(\"cij.liblight.get_chunk_meta: fail\")\n\n chunk_meta = cij.bin.Buffer(types=self.envs[\"CHUNK_META_STRUCT\"], length=chunks)\n chunk_meta.read(meta_file)\n return chunk_meta"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_chunk_meta_item(self, chunk_meta, grp, pug, chk):\n num_chk = self.envs[\"NUM_CHK\"]\n num_pu = self.envs[\"NUM_PU\"]\n index = grp * num_pu * num_chk + pug * num_chk + chk\n return chunk_meta[index]", "response": "Get item of chunk meta table"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck the chunk is offline or not", "response": "def is_bad_chunk(self, chunk_meta, grp, pug, chk):\n \"\"\"Check the chunk is offline or not\"\"\"\n meta = self.get_chunk_meta_item(chunk_meta, grp, pug, chk)\n if meta.CS & 0x8 != 0:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a s20 unit to generic address", "response": "def s20_to_gen(self, pugrp, punit, chunk, sectr):\n \"\"\"S20 unit to generic address\"\"\"\n cmd = [\"nvm_addr s20_to_gen\", self.envs[\"DEV_PATH\"],\n \"%d %d %d %d\" % (pugrp, punit, chunk, sectr)]\n status, stdout, _ = cij.ssh.command(cmd, shell=True)\n if status:\n raise RuntimeError(\"cij.liblight.s20_to_gen: cmd fail\")\n\n return int(re.findall(r\"val: ([0-9a-fx]+)\", stdout)[0], 16)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef gen_to_dev(self, address):\n cmd = [\"nvm_addr gen2dev\", self.envs[\"DEV_PATH\"], \"0x{:x}\".format(address)]\n status, stdout, _ = cij.ssh.command(cmd, shell=True)\n if status:\n raise RuntimeError(\"cij.liblight.gen_to_dev: cmd fail\")\n\n return int(re.findall(r\"dev: ([0-9a-fx]+)\", stdout)[0], 16)", "response": "Generic address to device address"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef vblk_write(self, address, meta=False):\n cmd = list()\n if meta:\n cmd.append(\"NVM_CLI_META_MODE=1\")\n cmd += [\"nvm_vblk write\", self.envs[\"DEV_PATH\"], \"0x%x\" % address]\n status, _, _ = cij.ssh.command(cmd, shell=True)\n return status", "response": "Write a set of VBLK entries to the local machine."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef vector_read(self, address_list, file_name=None):\n address = [\"0x{:x}\".format(i) for i in address_list]\n cmd = [\"nvm_cmd read\", self.envs[\"DEV_PATH\"], \" \".join(address)]\n if file_name:\n cmd += [\"-o {}\".format(file_name)]\n status, _, _ = cij.ssh.command(cmd, shell=True)\n return status", "response": "read a set of attributes from the device"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef scalar_write(self, address, block_count, data_file, meta_file):\n cmd = [\"nvme\", \"write\", self.envs[\"DEV_PATH\"], \"-s 0x{:x}\".format(address),\n \"-c {}\".format(block_count-1), \"-d {}\".format(data_file), \"-M {}\".format(meta_file),\n \"-z 0x{:x}\".format(block_count * self.envs[\"NBYTES\"]),\n \"-y 0x{:x}\".format(block_count * self.envs[\"NBYTES_OOB\"])]\n status, _, _ = cij.ssh.command(cmd, shell=True)\n return status", "response": "write a scalar file to a local file"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nstarting DMESG job in thread", "response": "def start(self):\n \"\"\"Start DMESG job in thread\"\"\"\n\n self.__thread = Thread(target=self.__run, args=(True, False))\n self.__thread.setDaemon(True)\n self.__thread.start()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate_steady_rt_pic(process_data, para_meter, scale, steady_time):\r\n pic_path_steady = para_meter['filename'] + '_steady.png'\r\n plt.figure(figsize=(4 * scale, 2.5 * scale))\r\n for key in process_data.keys():\r\n if len(process_data[key]) < steady_time:\r\n steady_time = len(process_data[key])\r\n plt.scatter(process_data[key][-1 * steady_time:, 0],\r\n process_data[key][-1 * steady_time:, 1], label=str(key), s=10)\r\n steady_value = np.mean(process_data[key][-1 * steady_time:, 1])\r\n steady_value_5 = steady_value * (1 + 0.05)\r\n steady_value_10 = steady_value * (1 + 0.1)\r\n steady_value_ng_5 = steady_value * (1 - 0.05)\r\n steady_value_ng_10 = steady_value * (1 - 0.1)\r\n plt.plot(process_data[key][-1 * steady_time:, 0], [steady_value] * steady_time, 'b')\r\n plt.plot(process_data[key][-1 * steady_time:, 0], [steady_value_5] * steady_time, 'g')\r\n plt.plot(process_data[key][-1 * steady_time:, 0],\r\n [steady_value_ng_5] * steady_time, 'g')\r\n plt.plot(process_data[key][-1 * steady_time:, 0], [steady_value_10] * steady_time, 'r')\r\n plt.plot(process_data[key][-1 * steady_time:, 0],\r\n [steady_value_ng_10] * steady_time, 'r')\r\n plt.title(para_meter['title'] + '(steady)')\r\n plt.xlabel(para_meter['x_axis_name'] + '(steady)')\r\n plt.ylabel(para_meter['y_axis_name'] + '(steady)')\r\n plt.legend(loc='upper left')\r\n plt.savefig(pic_path_steady)\r\n return pic_path_steady", "response": "generate a new image for the steady time"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfunctions to import a file into a specific sheet and save it into the specific worksheet", "response": "def import_source(self, sheet, source, delimiter=\",\"):\r\n \"\"\"\r\n Function:\r\n Save original data into specific sheet, and try to translate data to float type\r\n Input:\r\n sheet: Must be a non exists sheet\r\n source: File path of source\r\n \"\"\"\r\n # check input parameters\r\n if ' ' in sheet:\r\n raise RuntimeError(\"Error sheet name: %s\" % sheet)\r\n\r\n if not source.endswith(\"txt\") and not source.endswith(\"csv\"):\r\n raise RuntimeError(\"Error source name: %s\" % source)\r\n\r\n self.source_sheet = sheet\r\n\r\n source_data = np.loadtxt(source, dtype=str, delimiter=delimiter)\r\n self.source_data = {\"title\": source_data[0].tolist(),\r\n \"data\": source_data[1:]}\r\n\r\n cell_format_title = self.workbook.add_format({'bold': True,\r\n 'font_name': u'\u7b49\u7ebf',\r\n 'bg_color': '#c5d9f1',\r\n 'rotation': 45})\r\n cell_format = self.workbook.add_format({'bold': False,\r\n 'font_name': u'\u7b49\u7ebf',\r\n 'num_format': 0})\r\n\r\n worksheet = self.workbook.add_worksheet(sheet)\r\n worksheet.write_row('A1', self.source_data['title'], cell_format_title)\r\n\r\n _, col_num = self.source_data['data'].shape\r\n for i in range(col_num):\r\n try:\r\n data_array = self.source_data['data'][:, i].astype(float)\r\n except ValueError:\r\n data_array = self.source_data['data'][:, i]\r\n\r\n worksheet.write_column(1, i, data_array.tolist(), cell_format)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_chart(self, properties):\r\n # check input parameters\r\n if not {'x_axis', 'y_axis', 'series', 'filters'}.issubset(set(properties.keys())):\r\n raise RuntimeError(\"Error properties: %s\" % properties.keys())\r\n\r\n # generate chart\r\n mask = self.__filter_data(properties['filters'])\r\n chart = self.__generate_chart(mask, properties)\r\n sheet = properties['sheet']\r\n\r\n # Add work sheet\r\n if sheet in self.sheet_dict.keys():\r\n self.sheet_dict[sheet] += 1\r\n worksheet = self.workbook.get_worksheet_by_name(sheet)\r\n else:\r\n self.sheet_dict[sheet] = 1\r\n worksheet = self.workbook.add_worksheet(sheet)\r\n worksheet.insert_chart('B%d' % (5 + (self.sheet_dict[sheet] - 1) * 35), chart)", "response": "Function to generate and save a chart for a specific date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef gen_data_sheet(self, datafile, para_meter, scale=1.75, steady_time=300):\r\n filename = os.path.splitext(os.path.split(datafile)[1])[0][:-5]\r\n para_meter['filename'] = filename\r\n source_data = np.loadtxt(datafile, dtype=int, delimiter=',')[:, :3]\r\n is_bw = 'bw'in para_meter['title'].lower()\r\n file_data = process_rt_data(source_data, is_bw)\r\n pic_path = generate_rt_pic(file_data, para_meter, scale)\r\n pic_path_steady = generate_steady_rt_pic(file_data, para_meter, scale, steady_time)\r\n if para_meter['sheetname'] in self.sheetname_dict.keys():\r\n self.sheetname_dict[para_meter['sheetname']] = \\\r\n self.sheetname_dict[para_meter['sheetname']] + 1\r\n chart_sheet = self.workbook.get_worksheet_by_name(para_meter['sheetname'])\r\n else:\r\n self.sheetname_dict[para_meter['sheetname']] = 1\r\n chart_sheet = self.workbook.add_worksheet(para_meter['sheetname'])\r\n chart_sheet.insert_image('B%d' %\r\n (5 + (self.sheetname_dict[para_meter['sheetname']] - 1) * 30),\r\n pic_path)\r\n chart_sheet.insert_image('Q%d' %\r\n (5 + (self.sheetname_dict[para_meter['sheetname']] - 1) * 30),\r\n pic_path_steady)\r\n self.__insert_value(chart_sheet, file_data,\r\n 5 + (self.sheetname_dict[para_meter['sheetname']] - 1) * 30,\r\n steady_time)\r\n self.pic_list.append(pic_path)\r\n self.pic_list.append(pic_path_steady)", "response": "This function generates a new data sheet from a realtime bw file and saves it into a specific one."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef env():\n\n if cij.ssh.env():\n cij.err(\"cij.pci.env: invalid SSH environment\")\n return 1\n\n pci = cij.env_to_dict(PREFIX, REQUIRED)\n\n pci[\"BUS_PATH\"] = \"/sys/bus/pci\"\n pci[\"DEV_PATH\"] = os.sep.join([pci[\"BUS_PATH\"], \"devices\", pci[\"DEV_NAME\"]])\n\n cij.env_export(PREFIX, EXPORTED, pci)\n\n return 0", "response": "Verify PCI variables and construct exported variables"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef info(txt):\n\n print(\"%s# %s%s%s\" % (PR_EMPH_CC, get_time_stamp(), txt, PR_NC))\n sys.stdout.flush()", "response": "Print emphasized neutral and the given txt message"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nprints emphasized good message", "response": "def good(txt):\n \"\"\"Print, emphasized 'good', the given 'txt' message\"\"\"\n\n print(\"%s# %s%s%s\" % (PR_GOOD_CC, get_time_stamp(), txt, PR_NC))\n sys.stdout.flush()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef warn(txt):\n\n print(\"%s# %s%s%s\" % (PR_WARN_CC, get_time_stamp(), txt, PR_NC))\n sys.stdout.flush()", "response": "Print emphasized warning message"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef err(txt):\n\n print(\"%s# %s%s%s\" % (PR_ERR_CC, get_time_stamp(), txt, PR_NC))\n sys.stdout.flush()", "response": "Print emphasized error message"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef emph(txt, rval=None):\n\n if rval is None: # rval is not specified, use 'neutral'\n info(txt)\n elif rval == 0: # rval is 0, by convention, this is 'good'\n good(txt)\n else: # any other value, considered 'bad'\n err(txt)", "response": "Print emphasized based on rval"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconstruct dict of paths from environment variables", "response": "def paths_from_env(prefix=None, names=None):\n \"\"\"Construct dict of paths from environment variables'\"\"\"\n\n\n def expand_path(path):\n \"\"\"Expands variables in 'path' and turns it into absolute path\"\"\"\n\n return os.path.abspath(os.path.expanduser(os.path.expandvars(path)))\n\n\n if prefix is None:\n prefix = \"CIJ\"\n if names is None:\n names = [\n \"ROOT\", \"ENVS\", \"TESTPLANS\", \"TESTCASES\", \"TESTSUITES\", \"MODULES\",\n \"HOOKS\", \"TEMPLATES\"\n ]\n\n conf = {v: os.environ.get(\"_\".join([prefix, v])) for v in names}\n\n for env in (e for e in conf.keys() if e[:len(prefix)] in names and conf[e]):\n conf[env] = expand_path(conf[env])\n if not os.path.exists(conf[env]):\n err(\"%s_%s: %r, does not exist\" % (prefix, env, conf[env]))\n\n return conf"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef env_to_dict(prefix, names):\n\n env = {}\n for name in names:\n env[name] = ENV.get(\"_\".join([prefix, name]))\n if env[name] is None:\n return None\n\n return env", "response": "Construct dict from environment variables named in names"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef env_export(prefix, exported, env):\n\n for exp in exported:\n ENV[\"_\".join([prefix, exp])] = env[exp]", "response": "Define the list of exported variables with prefix with values from env"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef env():\n\n if cij.ssh.env():\n cij.err(\"cij.nvm.env: invalid SSH environment\")\n return 1\n\n nvm = cij.env_to_dict(PREFIX, REQUIRED)\n\n if \"nvme\" in nvm[\"DEV_NAME\"]:\n nvm[\"DEV_PATH\"] = \"/dev/%s\" % nvm[\"DEV_NAME\"]\n else:\n nvm[\"DEV_PATH\"] = \"traddr:%s\" % nvm[\"DEV_NAME\"]\n\n cij.env_export(PREFIX, EXPORTED, nvm)\n\n return 0", "response": "Verify NVME variables and construct exported variables"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef exists():\n\n if env():\n cij.err(\"cij.nvm.exists: Invalid NVMe ENV.\")\n return 1\n\n nvm = cij.env_to_dict(PREFIX, EXPORTED + REQUIRED)\n\n cmd = ['[[ -b \"%s\" ]]' % nvm[\"DEV_PATH\"]]\n rcode, _, _ = cij.ssh.command(cmd, shell=True, echo=False)\n\n return rcode", "response": "Verify that the ENV defined NVMe device exists"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the first log page chunk information for a given device.", "response": "def dev_get_rprt(dev_name, pugrp=None, punit=None):\n \"\"\"\n Get-log-page chunk information\n\n If the pugrp and punit is set, then provide report only for that pugrp/punit\n\n @returns the first chunk in the given state if one exists, None otherwise\n \"\"\"\n\n cmd = [\"nvm_cmd\", \"rprt_all\", dev_name]\n if not (pugrp is None and punit is None):\n cmd = [\"nvm_cmd\", \"rprt_lun\", dev_name, str(pugrp), str(punit)]\n\n _, _, _, struct = cij.test.command_to_struct(cmd)\n if not struct:\n return None\n\n return struct[\"rprt_descr\"]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dev_get_chunk(dev_name, state, pugrp=None, punit=None):\n\n rprt = dev_get_rprt(dev_name, pugrp, punit)\n if not rprt:\n return None\n\n return next((d for d in rprt if d[\"cs\"] == state), None)", "response": "Get a chunk - descriptor for the first chunk in the given state."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nkill all of FIO processes", "response": "def pkill():\n \"\"\"Kill all of FIO processes\"\"\"\n\n if env():\n return 1\n\n cmd = [\"ps -aux | grep fio | grep -v grep\"]\n status, _, _ = cij.ssh.command(cmd, shell=True, echo=False)\n if not status:\n status, _, _ = cij.ssh.command([\"pkill -f fio\"], shell=True)\n if status:\n return 1\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntranslate dict parameters to string", "response": "def __parse_parms(self):\n \"\"\"Translate dict parameters to string\"\"\"\n\n args = list()\n for key, val in self.__parm.items():\n key = key.replace(\"FIO_\", \"\").lower()\n\n if key == \"runtime\":\n args.append(\"--time_based\")\n\n if val is None:\n args.append(\"--%s\" % key)\n else:\n args.append(\"--%s=%s\" % (key, val))\n return args"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nimport external dict to internal dict", "response": "def import_parms(self, args):\n \"\"\"Import external dict to internal dict\"\"\"\n\n for key, val in args.items():\n self.set_parm(key, val)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting parameter of FIO", "response": "def get_parm(self, key):\n \"\"\"Get parameter of FIO\"\"\"\n\n if key in self.__parm.keys():\n return self.__parm[key]\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nstart FIO job in thread", "response": "def start(self):\n \"\"\"Run FIO job in thread\"\"\"\n\n self.__thread = Threads(target=self.run, args=(True, True, False))\n self.__thread.setDaemon(True)\n self.__thread.start()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef extract_hook_names(ent):\n\n hnames = []\n for hook in ent[\"hooks\"][\"enter\"] + ent[\"hooks\"][\"exit\"]:\n hname = os.path.basename(hook[\"fpath_orig\"])\n hname = os.path.splitext(hname)[0]\n hname = hname.strip()\n hname = hname.replace(\"_enter\", \"\")\n hname = hname.replace(\"_exit\", \"\")\n if hname in hnames:\n continue\n\n hnames.append(hname)\n\n hnames.sort()\n\n return hnames", "response": "Extract hook names from the given entity"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nextract testcase comment section and testcase description from the file at tcase[\"fpath\"]", "response": "def tcase_comment(tcase):\n \"\"\"\n Extract testcase comment section / testcase description\n\n @returns the testcase-comment from the tcase[\"fpath\"] as a list of strings\n \"\"\"\n\n src = open(tcase[\"fpath\"]).read()\n if len(src) < 3:\n cij.err(\"rprtr::tcase_comment: invalid src, tcase: %r\" % tcase[\"name\"])\n return None\n\n ext = os.path.splitext(tcase[\"fpath\"])[-1]\n if ext not in [\".sh\", \".py\"]:\n cij.err(\"rprtr::tcase_comment: invalid ext: %r, tcase: %r\" % (\n ext, tcase[\"name\"]\n ))\n return None\n\n comment = []\n for line in src.splitlines()[2:]:\n if ext == \".sh\" and not line.startswith(\"#\"):\n break\n elif ext == \".py\" and not '\"\"\"' in line:\n break\n\n comment.append(line)\n\n return comment"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tcase_parse_descr(tcase):\n\n descr_short = \"SHORT\"\n descr_long = \"LONG\"\n\n try:\n comment = tcase_comment(tcase)\n except (IOError, OSError, ValueError) as exc:\n comment = []\n cij.err(\"tcase_parse_descr: failed: %r, tcase: %r\" % (exc, tcase))\n\n comment = [l for l in comment if l.strip()] # Remove empty lines\n\n for line_number, line in enumerate(comment):\n if line.startswith(\"#\"):\n comment[line_number] = line[1:]\n\n if comment:\n descr_short = comment[0]\n\n if len(comment) > 1:\n descr_long = \"\\n\".join(comment[1:])\n\n return descr_short, descr_long", "response": "Parse the descriptions from the given tcase"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the content of the given filepath with HTML annotations", "response": "def runlogs_to_html(run_root):\n \"\"\"\n Returns content of the given 'fpath' with HTML annotations, currently simply\n a conversion of ANSI color codes to HTML elements\n \"\"\"\n\n if not os.path.isdir(run_root):\n return \"CANNOT_LOCATE_LOGFILES\"\n\n hook_enter = []\n hook_exit = []\n tcase = []\n for fpath in glob.glob(os.sep.join([run_root, \"*.log\"])):\n if \"exit\" in fpath:\n hook_exit.append(fpath)\n continue\n\n if \"hook\" in fpath:\n hook_enter.append(fpath)\n continue\n\n tcase.append(fpath)\n\n content = \"\"\n for fpath in hook_enter + tcase + hook_exit:\n content += \"# BEGIN: run-log from log_fpath: %s\\n\" % fpath\n content += open(fpath, \"r\").read()\n content += \"# END: run-log from log_fpath: %s\\n\\n\" % fpath\n\n return content"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the content of the given fpath with HTML annotations for syntax highlighting", "response": "def src_to_html(fpath):\n \"\"\"\n Returns content of the given 'fpath' with HTML annotations for syntax\n highlighting\n \"\"\"\n\n if not os.path.exists(fpath):\n return \"COULD-NOT-FIND-TESTCASE-SRC-AT-FPATH:%r\" % fpath\n\n # NOTE: Do SYNTAX highlight?\n\n return open(fpath, \"r\").read()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef aux_listing(aux_root):\n\n listing = []\n\n for root, _, fnames in os.walk(aux_root):\n count = len(aux_root.split(os.sep))\n prefix = root.split(os.sep)[count:]\n\n for fname in fnames:\n listing.append(os.sep.join(prefix + [fname]))\n\n return listing", "response": "Returns a list of all files in the aux_root directory."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngoes through the tsuite and processes the. log files.", "response": "def process_tsuite(tsuite):\n \"\"\"Goes through the tsuite and processes \"*.log\" \"\"\"\n\n # scoop of output from all run-logs\n\n tsuite[\"log_content\"] = runlogs_to_html(tsuite[\"res_root\"])\n tsuite[\"aux_list\"] = aux_listing(tsuite[\"aux_root\"])\n tsuite[\"hnames\"] = extract_hook_names(tsuite)\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprocess a single TCase.", "response": "def process_tcase(tcase):\n \"\"\"Goes through the trun and processes \"run.log\" \"\"\"\n\n tcase[\"src_content\"] = src_to_html(tcase[\"fpath\"])\n tcase[\"log_content\"] = runlogs_to_html(tcase[\"res_root\"])\n tcase[\"aux_list\"] = aux_listing(tcase[\"aux_root\"])\n tcase[\"descr_short\"], tcase[\"descr_long\"] = tcase_parse_descr(tcase)\n tcase[\"hnames\"] = extract_hook_names(tcase)\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngoing through the trun and processes run. log", "response": "def process_trun(trun):\n \"\"\"Goes through the trun and processes \"run.log\" \"\"\"\n\n trun[\"log_content\"] = runlogs_to_html(trun[\"res_root\"])\n trun[\"aux_list\"] = aux_listing(trun[\"aux_root\"])\n trun[\"hnames\"] = extract_hook_names(trun)\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef postprocess(trun):\n\n plog = []\n plog.append((\"trun\", process_trun(trun)))\n\n for tsuite in trun[\"testsuites\"]:\n plog.append((\"tsuite\", process_tsuite(tsuite)))\n\n for tcase in tsuite[\"testcases\"]:\n plog.append((\"tcase\", process_tcase(tcase)))\n\n for task, success in plog:\n if not success:\n cij.err(\"rprtr::postprocess: FAILED for %r\" % task)\n\n return sum((success for task, success in plog))", "response": "Perform postprocessing of the given test run"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreplace all absolute paths to re - home", "response": "def rehome(old, new, struct):\n \"\"\"\n Replace all absolute paths to \"re-home\" it\n \"\"\"\n\n if old == new:\n return\n\n if isinstance(struct, list):\n for item in struct:\n rehome(old, new, item)\n elif isinstance(struct, dict):\n for key, val in struct.iteritems():\n if isinstance(val, (dict, list)):\n rehome(old, new, val)\n elif \"conf\" in key:\n continue\n elif \"orig\" in key:\n continue\n elif \"root\" in key or \"path\" in key:\n struct[key] = struct[key].replace(old, new)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef main(args):\n\n trun = cij.runner.trun_from_file(args.trun_fpath)\n\n rehome(trun[\"conf\"][\"OUTPUT\"], args.output, trun)\n\n postprocess(trun)\n\n cij.emph(\"main: reports are uses tmpl_fpath: %r\" % args.tmpl_fpath)\n cij.emph(\"main: reports are here args.output: %r\" % args.output)\n\n html_fpath = os.sep.join([args.output, \"%s.html\" % args.tmpl_name])\n cij.emph(\"html_fpath: %r\" % html_fpath)\n try: # Create and store HTML report\n with open(html_fpath, 'w') as html_file:\n html_file.write(dset_to_html(trun, args.tmpl_fpath))\n except (IOError, OSError, ValueError) as exc:\n import traceback\n traceback.print_exc()\n cij.err(\"rprtr:main: exc: %s\" % exc)\n return 1\n\n return 0", "response": "Main entry point for the report generator."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nverifying SSH variables and construct exported variables", "response": "def env():\n \"\"\"Verify SSH variables and construct exported variables\"\"\"\n\n ssh = cij.env_to_dict(PREFIX, REQUIRED)\n if \"KEY\" in ssh:\n ssh[\"KEY\"] = cij.util.expand_path(ssh[\"KEY\"])\n\n if cij.ENV.get(\"SSH_PORT\") is None:\n cij.ENV[\"SSH_PORT\"] = \"22\"\n cij.warn(\"cij.ssh.env: SSH_PORT was not set, assigned: %r\" % (\n cij.ENV.get(\"SSH_PORT\")\n ))\n\n if cij.ENV.get(\"SSH_CMD_TIME\") is None:\n cij.ENV[\"SSH_CMD_TIME\"] = \"1\"\n cij.warn(\"cij.ssh.env: SSH_CMD_TIME was not set, assigned: %r\" % (\n cij.ENV.get(\"SSH_CMD_TIME\")\n ))\n\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning a command over SSH", "response": "def command(cmd, shell=True, echo=True, suffix=None):\n \"\"\"SSH: Run the given command over SSH as defined in environment\"\"\"\n\n if env():\n cij.err(\"cij.ssh.command: Invalid SSH environment\")\n return 1\n\n prefix = []\n\n if cij.ENV.get(\"SSH_CMD_TIME\") == \"1\":\n prefix.append(\"/usr/bin/time\")\n\n if cij.ENV.get(\"SSH_CMD_TIMEOUT\"):\n prefix.append(\"timeout\")\n prefix.append(cij.ENV.get(\"SSH_CMD_TIMEOUT\"))\n\n prefix.append(\"ssh\")\n\n args = []\n\n if cij.ENV.get(\"SSH_KEY\"):\n args.append(\"-i\")\n args.append(cij.ENV.get(\"SSH_KEY\"))\n\n if cij.ENV.get(\"SSH_PORT\"):\n args.append(\"-p\")\n args.append(cij.ENV.get(\"SSH_PORT\"))\n\n args.append(\"@\".join([cij.ENV.get(\"SSH_USER\"), cij.ENV.get(\"SSH_HOST\")]))\n\n wrapped = prefix + args + [\"'%s'\" % \" \".join(cmd)]\n if suffix:\n wrapped += suffix\n\n return cij.util.execute(wrapped, shell, echo)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef pull(src, dst, folder=False):\n\n if env():\n cij.err(\"cij.ssh.pull: Invalid SSH environment\")\n return 1\n\n args = []\n\n if cij.ENV.get(\"SSH_KEY\"):\n args.append(\"-i\")\n args.append(cij.ENV.get(\"SSH_KEY\"))\n\n if cij.ENV.get(\"SSH_PORT\"):\n args.append(\"-P\")\n args.append(cij.ENV.get(\"SSH_PORT\"))\n\n if folder:\n args.append(\"-r\")\n\n target = \"%s:%s\" % (\"@\".join([cij.ENV.get(\"SSH_USER\"), cij.ENV.get(\"SSH_HOST\")]), src)\n wrapped = [\"scp\", \" \".join(args), target, dst]\n\n return cij.util.execute(wrapped, shell=True, echo=True)", "response": "SSH pull data from remote linux"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef wait(timeout=300):\n\n if env():\n cij.err(\"cij.ssh.wait: Invalid SSH environment\")\n return 1\n\n timeout_backup = cij.ENV.get(\"SSH_CMD_TIMEOUT\")\n\n try:\n time_start = time.time()\n\n cij.ENV[\"SSH_CMD_TIMEOUT\"] = \"3\"\n\n while True:\n time_current = time.time()\n if (time_current - time_start) > timeout:\n cij.err(\"cij.ssh.wait: Timeout\")\n return 1\n\n status, _, _ = command([\"exit\"], shell=True, echo=False)\n if not status:\n break\n\n cij.info(\"cij.ssh.wait: Time elapsed: %d seconds\" % (time_current - time_start))\n\n finally:\n if timeout_backup is None:\n del cij.ENV[\"SSH_CMD_TIMEOUT\"]\n else:\n cij.ENV[\"SSH_CMD_TIMEOUT\"] = timeout_backup\n\n return 0", "response": "Wait util target connected"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reboot(timeout=300, extra=\"\"):\n\n if env():\n cij.err(\"cij.ssh.reboot: Invalid SSH environment\")\n return 1\n\n timeout_backup = cij.ENV.get(\"SSH_CMD_TIMEOUT\")\n\n try:\n time_start = time.time()\n status, last_uptime, _ = command([\"/usr/bin/uptime -s\"], shell=True, echo=False)\n if status:\n return 1\n\n cij.ENV[\"SSH_CMD_TIMEOUT\"] = \"3\"\n cij.info(\"cij.ssh.reboot: Target: %s\" % cij.ENV.get(\"SSH_HOST\"))\n command([\"reboot %s\" % extra], shell=True, echo=False)\n\n while True:\n time_current = time.time()\n if (time_current - time_start) > timeout:\n cij.err(\"cij.ssh.reboot: Timeout\")\n return 1\n\n status, current_uptime, _ = command([\"/usr/bin/uptime -s\"], shell=True, echo=False)\n if not status and current_uptime != last_uptime:\n break\n\n cij.info(\"cij.ssh.reboot: Time elapsed: %d seconds\" % (time_current - time_start))\n\n finally:\n if timeout_backup is None:\n del cij.ENV[\"SSH_CMD_TIMEOUT\"]\n else:\n cij.ENV[\"SSH_CMD_TIMEOUT\"] = timeout_backup\n\n return 0", "response": "Reboot the specified target."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef assert_that(val, description=''):\n global _soft_ctx\n if _soft_ctx:\n return AssertionBuilder(val, description, 'soft')\n return AssertionBuilder(val, description)", "response": "Factory method for the assertion builder with value to be tested and optional description."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef contents_of(f, encoding='utf-8'):\n\n try:\n contents = f.read()\n except AttributeError:\n try:\n with open(f, 'r') as fp:\n contents = fp.read()\n except TypeError:\n raise ValueError('val must be file or path, but was type <%s>' % type(f).__name__)\n except OSError:\n if not isinstance(f, str_types):\n raise ValueError('val must be file or path, but was type <%s>' % type(f).__name__)\n raise\n\n if sys.version_info[0] == 3 and type(contents) is bytes:\n # in PY3 force decoding of bytes to target encoding\n return contents.decode(encoding, 'replace')\n elif sys.version_info[0] == 2 and encoding == 'ascii':\n # in PY2 force encoding back to ascii\n return contents.encode('ascii', 'replace')\n else:\n # in all other cases, try to decode to target encoding\n try:\n return contents.decode(encoding, 'replace')\n except AttributeError:\n pass\n # if all else fails, just return the contents \"as is\"\n return contents", "response": "Helper to read the contents of the given file or path into a string with the given encoding."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef soft_fail(msg=''):\n global _soft_ctx\n if _soft_ctx:\n global _soft_err\n _soft_err.append('Fail: %s!' % msg if msg else 'Fail!')\n return\n fail(msg)", "response": "Adds error message to soft errors list if within soft assertions context.\n Either just force test failure with the given message."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_equal_to(self, other, **kwargs):\n if self._check_dict_like(self.val, check_values=False, return_as_bool=True) and \\\n self._check_dict_like(other, check_values=False, return_as_bool=True):\n if self._dict_not_equal(self.val, other, ignore=kwargs.get('ignore'), include=kwargs.get('include')):\n self._dict_err(self.val, other, ignore=kwargs.get('ignore'), include=kwargs.get('include'))\n else:\n if self.val != other:\n self._err('Expected <%s> to be equal to <%s>, but was not.' % (self.val, other))\n return self", "response": "Asserts that val is equal to other."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nasserts that val is not equal to other.", "response": "def is_not_equal_to(self, other):\n \"\"\"Asserts that val is not equal to other.\"\"\"\n if self.val == other:\n self._err('Expected <%s> to be not equal to <%s>, but was.' % (self.val, other))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_same_as(self, other):\n if self.val is not other:\n self._err('Expected <%s> to be identical to <%s>, but was not.' % (self.val, other))\n return self", "response": "Asserts that the val is identical to other via is compare."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nasserting that the val is not identical to other via is compare.", "response": "def is_not_same_as(self, other):\n \"\"\"Asserts that the val is not identical to other, via 'is' compare.\"\"\"\n if self.val is other:\n self._err('Expected <%s> to be not identical to <%s>, but was.' % (self.val, other))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef is_type_of(self, some_type):\n if type(some_type) is not type and\\\n not issubclass(type(some_type), type):\n raise TypeError('given arg must be a type')\n if type(self.val) is not some_type:\n if hasattr(self.val, '__name__'):\n t = self.val.__name__\n elif hasattr(self.val, '__class__'):\n t = self.val.__class__.__name__\n else:\n t = 'unknown'\n self._err('Expected <%s:%s> to be of type <%s>, but was not.' % (self.val, t, some_type.__name__))\n return self", "response": "Asserts that val is of the given type."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_instance_of(self, some_class):\n try:\n if not isinstance(self.val, some_class):\n if hasattr(self.val, '__name__'):\n t = self.val.__name__\n elif hasattr(self.val, '__class__'):\n t = self.val.__class__.__name__\n else:\n t = 'unknown'\n self._err('Expected <%s:%s> to be instance of class <%s>, but was not.' % (self.val, t, some_class.__name__))\n except TypeError:\n raise TypeError('given arg must be a class')\n return self", "response": "Asserts that val is an instance of the given class."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nasserting that val is the given length.", "response": "def is_length(self, length):\n \"\"\"Asserts that val is the given length.\"\"\"\n if type(length) is not int:\n raise TypeError('given arg must be an int')\n if length < 0:\n raise ValueError('given arg must be a positive int')\n if len(self.val) != length:\n self._err('Expected <%s> to be of length <%d>, but was <%d>.' % (self.val, length, len(self.val)))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef contains(self, *items):\n if len(items) == 0:\n raise ValueError('one or more args must be given')\n elif len(items) == 1:\n if items[0] not in self.val:\n if self._check_dict_like(self.val, return_as_bool=True):\n self._err('Expected <%s> to contain key <%s>, but did not.' % (self.val, items[0]))\n else:\n self._err('Expected <%s> to contain item <%s>, but did not.' % (self.val, items[0]))\n else:\n missing = []\n for i in items:\n if i not in self.val:\n missing.append(i)\n if missing:\n if self._check_dict_like(self.val, return_as_bool=True):\n self._err('Expected <%s> to contain keys %s, but did not contain key%s %s.' % (self.val, self._fmt_items(items), '' if len(missing) == 0 else 's', self._fmt_items(missing)))\n else:\n self._err('Expected <%s> to contain items %s, but did not contain %s.' % (self.val, self._fmt_items(items), self._fmt_items(missing)))\n return self", "response": "Asserts that val contains the given item or items."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nasserting that val does not contain the given item or items.", "response": "def does_not_contain(self, *items):\n \"\"\"Asserts that val does not contain the given item or items.\"\"\"\n if len(items) == 0:\n raise ValueError('one or more args must be given')\n elif len(items) == 1:\n if items[0] in self.val:\n self._err('Expected <%s> to not contain item <%s>, but did.' % (self.val, items[0]))\n else:\n found = []\n for i in items:\n if i in self.val:\n found.append(i)\n if found:\n self._err('Expected <%s> to not contain items %s, but did contain %s.' % (self.val, self._fmt_items(items), self._fmt_items(found)))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nasserting that val contains only the given item or items.", "response": "def contains_only(self, *items):\n \"\"\"Asserts that val contains only the given item or items.\"\"\"\n if len(items) == 0:\n raise ValueError('one or more args must be given')\n else:\n extra = []\n for i in self.val:\n if i not in items:\n extra.append(i)\n if extra:\n self._err('Expected <%s> to contain only %s, but did contain %s.' % (self.val, self._fmt_items(items), self._fmt_items(extra)))\n\n missing = []\n for i in items:\n if i not in self.val:\n missing.append(i)\n if missing:\n self._err('Expected <%s> to contain only %s, but did not contain %s.' % (self.val, self._fmt_items(items), self._fmt_items(missing)))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nasserts that val contains the given sequence of items in order.", "response": "def contains_sequence(self, *items):\n \"\"\"Asserts that val contains the given sequence of items in order.\"\"\"\n if len(items) == 0:\n raise ValueError('one or more args must be given')\n else:\n try:\n for i in xrange(len(self.val) - len(items) + 1):\n for j in xrange(len(items)):\n if self.val[i+j] != items[j]:\n break\n else:\n return self\n except TypeError:\n raise TypeError('val is not iterable')\n self._err('Expected <%s> to contain sequence %s, but did not.' % (self.val, self._fmt_items(items)))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nassert that val is iterable and contains duplicate items.", "response": "def contains_duplicates(self):\n \"\"\"Asserts that val is iterable and contains duplicate items.\"\"\"\n try:\n if len(self.val) != len(set(self.val)):\n return self\n except TypeError:\n raise TypeError('val is not iterable')\n self._err('Expected <%s> to contain duplicates, but did not.' % self.val)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef does_not_contain_duplicates(self):\n try:\n if len(self.val) == len(set(self.val)):\n return self\n except TypeError:\n raise TypeError('val is not iterable')\n self._err('Expected <%s> to not contain duplicates, but did.' % self.val)", "response": "Asserts that val is iterable and does not contain any duplicate items."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_empty(self):\n if len(self.val) != 0:\n if isinstance(self.val, str_types):\n self._err('Expected <%s> to be empty string, but was not.' % self.val)\n else:\n self._err('Expected <%s> to be empty, but was not.' % self.val)\n return self", "response": "Asserts that val is empty."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nasserting that val is not empty.", "response": "def is_not_empty(self):\n \"\"\"Asserts that val is not empty.\"\"\"\n if len(self.val) == 0:\n if isinstance(self.val, str_types):\n self._err('Expected not empty string, but was empty.')\n else:\n self._err('Expected not empty, but was empty.')\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nasserting that val is one of the given items.", "response": "def is_in(self, *items):\n \"\"\"Asserts that val is equal to one of the given items.\"\"\"\n if len(items) == 0:\n raise ValueError('one or more args must be given')\n else:\n for i in items:\n if self.val == i:\n return self\n self._err('Expected <%s> to be in %s, but was not.' % (self.val, self._fmt_items(items)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_nan(self):\n self._validate_number()\n self._validate_real()\n if not math.isnan(self.val):\n self._err('Expected <%s> to be , but was not.' % self.val)\n return self", "response": "Asserts that val is real number and NaN."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_not_nan(self):\n self._validate_number()\n self._validate_real()\n if math.isnan(self.val):\n self._err('Expected not , but was.')\n return self", "response": "Asserts that val is real number and not NaN."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nasserting that val is real number and Inf.", "response": "def is_inf(self):\n \"\"\"Asserts that val is real number and Inf (infinity).\"\"\"\n self._validate_number()\n self._validate_real()\n if not math.isinf(self.val):\n self._err('Expected <%s> to be , but was not.' % self.val)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_not_inf(self):\n self._validate_number()\n self._validate_real()\n if math.isinf(self.val):\n self._err('Expected not , but was.')\n return self", "response": "Asserts that val is real number and not Inf."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef is_less_than(self, other):\n self._validate_compareable(other)\n if self.val >= other:\n if type(self.val) is datetime.datetime:\n self._err('Expected <%s> to be less than <%s>, but was not.' % (self.val.strftime('%Y-%m-%d %H:%M:%S'), other.strftime('%Y-%m-%d %H:%M:%S')))\n else:\n self._err('Expected <%s> to be less than <%s>, but was not.' % (self.val, other))\n return self", "response": "Asserts that val is numeric and is less than other."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nasserts that val is numeric and is between low and high.", "response": "def is_between(self, low, high):\n \"\"\"Asserts that val is numeric and is between low and high.\"\"\"\n val_type = type(self.val)\n self._validate_between_args(val_type, low, high)\n\n if self.val < low or self.val > high:\n if val_type is datetime.datetime:\n self._err('Expected <%s> to be between <%s> and <%s>, but was not.' % (self.val.strftime('%Y-%m-%d %H:%M:%S'), low.strftime('%Y-%m-%d %H:%M:%S'), high.strftime('%Y-%m-%d %H:%M:%S')))\n else:\n self._err('Expected <%s> to be between <%s> and <%s>, but was not.' % (self.val, low, high))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nassert that val is numeric and is close to other within tolerance.", "response": "def is_close_to(self, other, tolerance):\n \"\"\"Asserts that val is numeric and is close to other within tolerance.\"\"\"\n self._validate_close_to_args(self.val, other, tolerance)\n\n if self.val < (other-tolerance) or self.val > (other+tolerance):\n if type(self.val) is datetime.datetime:\n tolerance_seconds = tolerance.days * 86400 + tolerance.seconds + tolerance.microseconds / 1000000\n h, rem = divmod(tolerance_seconds, 3600)\n m, s = divmod(rem, 60)\n self._err('Expected <%s> to be close to <%s> within tolerance <%d:%02d:%02d>, but was not.' % (self.val.strftime('%Y-%m-%d %H:%M:%S'), other.strftime('%Y-%m-%d %H:%M:%S'), h, m, s))\n else:\n self._err('Expected <%s> to be close to <%s> within tolerance <%s>, but was not.' % (self.val, other, tolerance))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nassert that val is case - insensitive equal to other.", "response": "def is_equal_to_ignoring_case(self, other):\n \"\"\"Asserts that val is case-insensitive equal to other.\"\"\"\n if not isinstance(self.val, str_types):\n raise TypeError('val is not a string')\n if not isinstance(other, str_types):\n raise TypeError('given arg must be a string')\n if self.val.lower() != other.lower():\n self._err('Expected <%s> to be case-insensitive equal to <%s>, but was not.' % (self.val, other))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef contains_ignoring_case(self, *items):\n if len(items) == 0:\n raise ValueError('one or more args must be given')\n if isinstance(self.val, str_types):\n if len(items) == 1:\n if not isinstance(items[0], str_types):\n raise TypeError('given arg must be a string')\n if items[0].lower() not in self.val.lower():\n self._err('Expected <%s> to case-insensitive contain item <%s>, but did not.' % (self.val, items[0]))\n else:\n missing = []\n for i in items:\n if not isinstance(i, str_types):\n raise TypeError('given args must all be strings')\n if i.lower() not in self.val.lower():\n missing.append(i)\n if missing:\n self._err('Expected <%s> to case-insensitive contain items %s, but did not contain %s.' % (self.val, self._fmt_items(items), self._fmt_items(missing)))\n elif isinstance(self.val, Iterable):\n missing = []\n for i in items:\n if not isinstance(i, str_types):\n raise TypeError('given args must all be strings')\n found = False\n for v in self.val:\n if not isinstance(v, str_types):\n raise TypeError('val items must all be strings')\n if i.lower() == v.lower():\n found = True\n break\n if not found:\n missing.append(i)\n if missing:\n self._err('Expected <%s> to case-insensitive contain items %s, but did not contain %s.' % (self.val, self._fmt_items(items), self._fmt_items(missing)))\n else:\n raise TypeError('val is not a string or iterable')\n return self", "response": "Asserts that val is string and contains the given item or items."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nasserting that val starts with prefix.", "response": "def starts_with(self, prefix):\n \"\"\"Asserts that val is string or iterable and starts with prefix.\"\"\"\n if prefix is None:\n raise TypeError('given prefix arg must not be none')\n if isinstance(self.val, str_types):\n if not isinstance(prefix, str_types):\n raise TypeError('given prefix arg must be a string')\n if len(prefix) == 0:\n raise ValueError('given prefix arg must not be empty')\n if not self.val.startswith(prefix):\n self._err('Expected <%s> to start with <%s>, but did not.' % (self.val, prefix))\n elif isinstance(self.val, Iterable):\n if len(self.val) == 0:\n raise ValueError('val must not be empty')\n first = next(iter(self.val))\n if first != prefix:\n self._err('Expected %s to start with <%s>, but did not.' % (self.val, prefix))\n else:\n raise TypeError('val is not a string or iterable')\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nasserting that val ends with suffix.", "response": "def ends_with(self, suffix):\n \"\"\"Asserts that val is string or iterable and ends with suffix.\"\"\"\n if suffix is None:\n raise TypeError('given suffix arg must not be none')\n if isinstance(self.val, str_types):\n if not isinstance(suffix, str_types):\n raise TypeError('given suffix arg must be a string')\n if len(suffix) == 0:\n raise ValueError('given suffix arg must not be empty')\n if not self.val.endswith(suffix):\n self._err('Expected <%s> to end with <%s>, but did not.' % (self.val, suffix))\n elif isinstance(self.val, Iterable):\n if len(self.val) == 0:\n raise ValueError('val must not be empty')\n last = None\n for last in self.val:\n pass\n if last != suffix:\n self._err('Expected %s to end with <%s>, but did not.' % (self.val, suffix))\n else:\n raise TypeError('val is not a string or iterable')\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef matches(self, pattern):\n if not isinstance(self.val, str_types):\n raise TypeError('val is not a string')\n if not isinstance(pattern, str_types):\n raise TypeError('given pattern arg must be a string')\n if len(pattern) == 0:\n raise ValueError('given pattern arg must not be empty')\n if re.search(pattern, self.val) is None:\n self._err('Expected <%s> to match pattern <%s>, but did not.' % (self.val, pattern))\n return self", "response": "Asserts that val is string and matches regex pattern."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nassert that val is non - empty string and all characters are alphabetic.", "response": "def is_alpha(self):\n \"\"\"Asserts that val is non-empty string and all characters are alphabetic.\"\"\"\n if not isinstance(self.val, str_types):\n raise TypeError('val is not a string')\n if len(self.val) == 0:\n raise ValueError('val is empty')\n if not self.val.isalpha():\n self._err('Expected <%s> to contain only alphabetic chars, but did not.' % self.val)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nasserts that val is non - empty string and all characters are digits.", "response": "def is_digit(self):\n \"\"\"Asserts that val is non-empty string and all characters are digits.\"\"\"\n if not isinstance(self.val, str_types):\n raise TypeError('val is not a string')\n if len(self.val) == 0:\n raise ValueError('val is empty')\n if not self.val.isdigit():\n self._err('Expected <%s> to contain only digits, but did not.' % self.val)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_lower(self):\n if not isinstance(self.val, str_types):\n raise TypeError('val is not a string')\n if len(self.val) == 0:\n raise ValueError('val is empty')\n if self.val != self.val.lower():\n self._err('Expected <%s> to contain only lowercase chars, but did not.' % self.val)\n return self", "response": "Asserts that val is non - empty string and all characters are lowercase."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nasserts that val is non - empty string and all characters are uppercase.", "response": "def is_upper(self):\n \"\"\"Asserts that val is non-empty string and all characters are uppercase.\"\"\"\n if not isinstance(self.val, str_types):\n raise TypeError('val is not a string')\n if len(self.val) == 0:\n raise ValueError('val is empty')\n if self.val != self.val.upper():\n self._err('Expected <%s> to contain only uppercase chars, but did not.' % self.val)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nassert that val is a unicode string.", "response": "def is_unicode(self):\n \"\"\"Asserts that val is a unicode string.\"\"\"\n if type(self.val) is not unicode:\n self._err('Expected <%s> to be unicode, but was <%s>.' % (self.val, type(self.val).__name__))\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nassert that val is iterable and a subset of the given supersets or flattened superset if multiple supersets are given.", "response": "def is_subset_of(self, *supersets):\n \"\"\"Asserts that val is iterable and a subset of the given superset or flattened superset if multiple supersets are given.\"\"\"\n if not isinstance(self.val, Iterable):\n raise TypeError('val is not iterable')\n if len(supersets) == 0:\n raise ValueError('one or more superset args must be given')\n\n missing = []\n if hasattr(self.val, 'keys') and callable(getattr(self.val, 'keys')) and hasattr(self.val, '__getitem__'):\n # flatten superset dicts\n superdict = {}\n for l,j in enumerate(supersets):\n self._check_dict_like(j, check_values=False, name='arg #%d' % (l+1))\n for k in j.keys():\n superdict.update({k: j[k]})\n\n for i in self.val.keys():\n if i not in superdict:\n missing.append({i: self.val[i]}) # bad key\n elif self.val[i] != superdict[i]:\n missing.append({i: self.val[i]}) # bad val\n if missing:\n self._err('Expected <%s> to be subset of %s, but %s %s missing.' % (self.val, self._fmt_items(superdict), self._fmt_items(missing), 'was' if len(missing) == 1 else 'were'))\n else:\n # flatten supersets\n superset = set()\n for j in supersets:\n try:\n for k in j:\n superset.add(k)\n except Exception:\n superset.add(j)\n\n for i in self.val:\n if i not in superset:\n missing.append(i)\n if missing:\n self._err('Expected <%s> to be subset of %s, but %s %s missing.' % (self.val, self._fmt_items(superset), self._fmt_items(missing), 'was' if len(missing) == 1 else 'were'))\n\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef contains_key(self, *keys):\n self._check_dict_like(self.val, check_values=False, check_getitem=False)\n return self.contains(*keys)", "response": "Asserts the val is a dict and contains the given key or keys."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef does_not_contain_key(self, *keys):\n self._check_dict_like(self.val, check_values=False, check_getitem=False)\n return self.does_not_contain(*keys)", "response": "Asserts the val is a dict and does not contain the given keys."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef contains_value(self, *values):\n self._check_dict_like(self.val, check_getitem=False)\n if len(values) == 0:\n raise ValueError('one or more value args must be given')\n missing = []\n for v in values:\n if v not in self.val.values():\n missing.append(v)\n if missing:\n self._err('Expected <%s> to contain values %s, but did not contain %s.' % (self.val, self._fmt_items(values), self._fmt_items(missing)))\n return self", "response": "Asserts that val is a dict and contains the given value or values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef does_not_contain_value(self, *values):\n self._check_dict_like(self.val, check_getitem=False)\n if len(values) == 0:\n raise ValueError('one or more value args must be given')\n else:\n found = []\n for v in values:\n if v in self.val.values():\n found.append(v)\n if found:\n self._err('Expected <%s> to not contain values %s, but did contain %s.' % (self.val, self._fmt_items(values), self._fmt_items(found)))\n return self", "response": "Asserts that val does not contain the given value or values."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nassert that val is a dict and contains the given entry or entries.", "response": "def contains_entry(self, *args, **kwargs):\n \"\"\"Asserts that val is a dict and contains the given entry or entries.\"\"\"\n self._check_dict_like(self.val, check_values=False)\n entries = list(args) + [{k:v} for k,v in kwargs.items()]\n if len(entries) == 0:\n raise ValueError('one or more entry args must be given')\n missing = []\n for e in entries:\n if type(e) is not dict:\n raise TypeError('given entry arg must be a dict')\n if len(e) != 1:\n raise ValueError('given entry args must contain exactly one key-value pair')\n k = next(iter(e))\n if k not in self.val:\n missing.append(e) # bad key\n elif self.val[k] != e[k]:\n missing.append(e) # bad val\n if missing:\n self._err('Expected <%s> to contain entries %s, but did not contain %s.' % (self.val, self._fmt_items(entries), self._fmt_items(missing)))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef is_before(self, other):\n if type(self.val) is not datetime.datetime:\n raise TypeError('val must be datetime, but was type <%s>' % type(self.val).__name__)\n if type(other) is not datetime.datetime:\n raise TypeError('given arg must be datetime, but was type <%s>' % type(other).__name__)\n if self.val >= other:\n self._err('Expected <%s> to be before <%s>, but was not.' % (self.val.strftime('%Y-%m-%d %H:%M:%S'), other.strftime('%Y-%m-%d %H:%M:%S')))\n return self", "response": "Asserts that val is a date and is before other date."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef exists(self):\n if not isinstance(self.val, str_types):\n raise TypeError('val is not a path')\n if not os.path.exists(self.val):\n self._err('Expected <%s> to exist, but was not found.' % self.val)\n return self", "response": "Asserts that val is a path and that it exists."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nasserting that val is an existing path to a file.", "response": "def is_file(self):\n \"\"\"Asserts that val is an existing path to a file.\"\"\"\n self.exists()\n if not os.path.isfile(self.val):\n self._err('Expected <%s> to be a file, but was not.' % self.val)\n return self"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_directory(self):\n self.exists()\n if not os.path.isdir(self.val):\n self._err('Expected <%s> to be a directory, but was not.' % self.val)\n return self", "response": "Asserts that val is an existing path to a directory."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nasserts that val is an existing path to a file and that file is named filename.", "response": "def is_named(self, filename):\n \"\"\"Asserts that val is an existing path to a file and that file is named filename.\"\"\"\n self.is_file()\n if not isinstance(filename, str_types):\n raise TypeError('given filename arg must be a path')\n val_filename = os.path.basename(os.path.abspath(self.val))\n if val_filename != filename:\n self._err('Expected filename <%s> to be equal to <%s>, but was not.' % (val_filename, filename))\n return self"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nasserting that val is an existing path to a file and that file is a child of parent.", "response": "def is_child_of(self, parent):\n \"\"\"Asserts that val is an existing path to a file and that file is a child of parent.\"\"\"\n self.is_file()\n if not isinstance(parent, str_types):\n raise TypeError('given parent directory arg must be a path')\n val_abspath = os.path.abspath(self.val)\n parent_abspath = os.path.abspath(parent)\n if not val_abspath.startswith(parent_abspath):\n self._err('Expected file <%s> to be a child of <%s>, but was not.' % (val_abspath, parent_abspath))\n return self"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef extracting(self, *names, **kwargs):\n if not isinstance(self.val, Iterable):\n raise TypeError('val is not iterable')\n if isinstance(self.val, str_types):\n raise TypeError('val must not be string')\n if len(names) == 0:\n raise ValueError('one or more name args must be given')\n\n def _extract(x, name):\n if self._check_dict_like(x, check_values=False, return_as_bool=True):\n if name in x:\n return x[name]\n else:\n raise ValueError('item keys %s did not contain key <%s>' % (list(x.keys()), name))\n elif isinstance(x, Iterable):\n self._check_iterable(x, name='item')\n return x[name]\n elif hasattr(x, name):\n attr = getattr(x, name)\n if callable(attr):\n try:\n return attr()\n except TypeError:\n raise ValueError('val method <%s()> exists, but is not zero-arg method' % name)\n else:\n return attr\n else:\n raise ValueError('val does not have property or zero-arg method <%s>' % name)\n\n def _filter(x):\n if 'filter' in kwargs:\n if isinstance(kwargs['filter'], str_types):\n return bool(_extract(x, kwargs['filter']))\n elif self._check_dict_like(kwargs['filter'], check_values=False, return_as_bool=True):\n for k in kwargs['filter']:\n if isinstance(k, str_types):\n if _extract(x, k) != kwargs['filter'][k]:\n return False\n return True\n elif callable(kwargs['filter']):\n return kwargs['filter'](x)\n return False\n return True\n\n def _sort(x):\n if 'sort' in kwargs:\n if isinstance(kwargs['sort'], str_types):\n return _extract(x, kwargs['sort'])\n elif isinstance(kwargs['sort'], Iterable):\n items = []\n for k in kwargs['sort']:\n if isinstance(k, str_types):\n items.append(_extract(x, k))\n return tuple(items)\n elif callable(kwargs['sort']):\n return kwargs['sort'](x)\n return 0\n\n extracted = []\n for i in sorted(self.val, key=lambda x: _sort(x)):\n if _filter(i):\n items = [_extract(i, name) for name in names]\n extracted.append(tuple(items) if len(items) > 1 else items[0])\n return AssertionBuilder(extracted, self.description, self.kind)", "response": "Asserts that val is collection then extracts the named properties or named zero - arg methods into a list or list of tuples if multiple names are given."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef raises(self, ex):\n if not callable(self.val):\n raise TypeError('val must be callable')\n if not issubclass(ex, BaseException):\n raise TypeError('given arg must be exception')\n return AssertionBuilder(self.val, self.description, self.kind, ex)", "response": "Asserts that val is callable and that when called raises the given exception."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nasserts the val callable when invoked with the given args and kwargs raises the expected exception.", "response": "def when_called_with(self, *some_args, **some_kwargs):\n \"\"\"Asserts the val callable when invoked with the given args and kwargs raises the expected exception.\"\"\"\n if not self.expected:\n raise TypeError('expected exception not set, raises() must be called first')\n try:\n self.val(*some_args, **some_kwargs)\n except BaseException as e:\n if issubclass(type(e), self.expected):\n # chain on with exception message as val\n return AssertionBuilder(str(e), self.description, self.kind)\n else:\n # got exception, but wrong type, so raise\n self._err('Expected <%s> to raise <%s> when called with (%s), but raised <%s>.' % (\n self.val.__name__,\n self.expected.__name__,\n self._fmt_args_kwargs(*some_args, **some_kwargs),\n type(e).__name__))\n\n # didn't fail as expected, so raise\n self._err('Expected <%s> to raise <%s> when called with (%s).' % (\n self.val.__name__,\n self.expected.__name__,\n self._fmt_args_kwargs(*some_args, **some_kwargs)))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _fmt_args_kwargs(self, *some_args, **some_kwargs):\n if some_args:\n out_args = str(some_args).lstrip('(').rstrip(',)')\n if some_kwargs:\n out_kwargs = ', '.join([str(i).lstrip('(').rstrip(')').replace(', ',': ') for i in [\n (k,some_kwargs[k]) for k in sorted(some_kwargs.keys())]])\n\n if some_args and some_kwargs:\n return out_args + ', ' + out_kwargs\n elif some_args:\n return out_args\n elif some_kwargs:\n return out_kwargs\n else:\n return ''", "response": "Helper to convert the given args and kwargs into a string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntransform list of files to list of words", "response": "def generate_words(files):\n \"\"\"\n Transform list of files to list of words,\n removing new line character\n and replace name entity '...' and abbreviation '...' symbol\n \"\"\"\n\n repls = {'' : '','' : '','': '','': ''}\n\n words_all = []\n for i, file in enumerate(files):\n lines = open(file, 'r')\n for line in lines:\n line = reduce(lambda a, kv: a.replace(*kv), repls.items(), line)\n words = [word for word in line.split(\"|\") if word is not '\\n']\n words_all.extend(words)\n return words_all"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates dataframe of character tags for a list of input tokenized words", "response": "def create_char_dataframe(words):\n \"\"\"\n Give list of input tokenized words,\n create dataframe of characters where first character of\n the word is tagged as 1, otherwise 0\n\n Example\n =======\n ['\u0e01\u0e34\u0e19', '\u0e2b\u0e21\u0e14'] to dataframe of\n [{'char': '\u0e01', 'type': ..., 'target': 1}, ...,\n {'char': '\u0e14', 'type': ..., 'target': 0}]\n \"\"\"\n char_dict = []\n for word in words:\n for i, char in enumerate(word):\n if i == 0:\n char_dict.append({'char': char,\n 'type': CHAR_TYPE_FLATTEN.get(char, 'o'),\n 'target': True})\n else:\n char_dict.append({'char': char,\n 'type': CHAR_TYPE_FLATTEN.get(char, 'o'),\n 'target': False})\n return pd.DataFrame(char_dict)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates CSV file for training and testing set for the best article type.", "response": "def generate_best_dataset(best_path, output_path='cleaned_data', create_val=False):\n \"\"\"\n Generate CSV file for training and testing data\n\n Input\n =====\n best_path: str, path to BEST folder which contains unzipped subfolder\n 'article', 'encyclopedia', 'news', 'novel'\n\n cleaned_data: str, path to output folder, the cleaned data will be saved\n in the given folder name where training set will be stored in `train` folder\n and testing set will be stored on `test` folder\n\n create_val: boolean, True or False, if True, divide training set into training set and\n validation set in `val` folder\n \"\"\"\n if not os.path.isdir(output_path):\n os.mkdir(output_path)\n if not os.path.isdir(os.path.join(output_path, 'train')):\n os.makedirs(os.path.join(output_path, 'train'))\n if not os.path.isdir(os.path.join(output_path, 'test')):\n os.makedirs(os.path.join(output_path, 'test'))\n if not os.path.isdir(os.path.join(output_path, 'val')) and create_val:\n os.makedirs(os.path.join(output_path, 'val'))\n\n for article_type in article_types:\n files = glob(os.path.join(best_path, article_type, '*.txt'))\n files_train, files_test = train_test_split(files, random_state=0, test_size=0.1)\n if create_val:\n files_train, files_val = train_test_split(files_train, random_state=0, test_size=0.1)\n val_words = generate_words(files_val)\n val_df = create_char_dataframe(val_words)\n val_df.to_csv(os.path.join(output_path, 'val', 'df_best_{}_val.csv'.format(article_type)), index=False)\n train_words = generate_words(files_train)\n test_words = generate_words(files_test)\n train_df = create_char_dataframe(train_words)\n test_df = create_char_dataframe(test_words)\n train_df.to_csv(os.path.join(output_path, 'train', 'df_best_{}_train.csv'.format(article_type)), index=False)\n test_df.to_csv(os.path.join(output_path, 'test', 'df_best_{}_test.csv'.format(article_type)), index=False)\n print(\"Save {} to CSV file\".format(article_type))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntransforming processed path into feature matrix and output array", "response": "def prepare_feature(best_processed_path, option='train'):\n \"\"\"\n Transform processed path into feature matrix and output array\n\n Input\n =====\n best_processed_path: str, path to processed BEST dataset\n\n option: str, 'train' or 'test'\n \"\"\"\n # padding for training and testing set\n n_pad = 21\n n_pad_2 = int((n_pad - 1)/2)\n pad = [{'char': ' ', 'type': 'p', 'target': True}]\n df_pad = pd.DataFrame(pad * n_pad_2)\n\n df = []\n for article_type in article_types:\n df.append(pd.read_csv(os.path.join(best_processed_path, option, 'df_best_{}_{}.csv'.format(article_type, option))))\n df = pd.concat(df)\n df = pd.concat((df_pad, df, df_pad)) # pad with empty string feature\n\n df['char'] = df['char'].map(lambda x: CHARS_MAP.get(x, 80))\n df['type'] = df['type'].map(lambda x: CHAR_TYPES_MAP.get(x, 4))\n df_pad = create_n_gram_df(df, n_pad=n_pad)\n\n char_row = ['char' + str(i + 1) for i in range(n_pad_2)] + \\\n ['char-' + str(i + 1) for i in range(n_pad_2)] + ['char']\n type_row = ['type' + str(i + 1) for i in range(n_pad_2)] + \\\n ['type-' + str(i + 1) for i in range(n_pad_2)] + ['type']\n\n x_char = df_pad[char_row].as_matrix()\n x_type = df_pad[type_row].as_matrix()\n y = df_pad['target'].astype(int).as_matrix()\n\n return x_char, x_type, y"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntrain Keras model for words beginning alongside with character label encoder character type label encoder and character type label encoder and model weight.", "response": "def train_model(best_processed_path, weight_path='../weight/model_weight.h5', verbose=2):\n \"\"\"\n Given path to processed BEST dataset,\n train CNN model for words beginning alongside with\n character label encoder and character type label encoder\n\n Input\n =====\n best_processed_path: str, path to processed BEST dataset\n weight_path: str, path to weight path file\n verbose: int, verbost option for training Keras model\n\n Output\n ======\n model: keras model, keras model for tokenize prediction\n \"\"\"\n\n x_train_char, x_train_type, y_train = prepare_feature(best_processed_path, option='train')\n x_test_char, x_test_type, y_test = prepare_feature(best_processed_path, option='test')\n\n validation_set = False\n if os.path.isdir(os.path.join(best_processed_path, 'val')):\n validation_set = True\n x_val_char, x_val_type, y_val = prepare_feature(best_processed_path, option='val')\n\n if not os.path.isdir(os.path.dirname(weight_path)):\n os.makedirs(os.path.dirname(weight_path)) # make directory if weight does not exist\n\n callbacks_list = [\n ReduceLROnPlateau(),\n ModelCheckpoint(\n weight_path,\n save_best_only=True,\n save_weights_only=True,\n monitor='val_loss',\n mode='min',\n verbose=1\n )\n ]\n\n # train model\n model = get_convo_nn2()\n train_params = [(10, 256), (3, 512), (3, 2048), (3, 4096), (3, 8192)]\n for (epochs, batch_size) in train_params:\n print(\"train with {} epochs and {} batch size\".format(epochs, batch_size))\n if validation_set:\n model.fit([x_train_char, x_train_type], y_train,\n epochs=epochs, batch_size=batch_size,\n verbose=verbose,\n callbacks=callbacks_list,\n validation_data=([x_val_char, x_val_type], y_val))\n else:\n model.fit([x_train_char, x_train_type], y_train,\n epochs=epochs, batch_size=batch_size,\n verbose=verbose,\n callbacks=callbacks_list)\n return model"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nevaluate model on splitted 10 percent testing set", "response": "def evaluate(best_processed_path, model):\n \"\"\"\n Evaluate model on splitted 10 percent testing set\n \"\"\"\n x_test_char, x_test_type, y_test = prepare_feature(best_processed_path, option='test')\n\n y_predict = model.predict([x_test_char, x_test_type])\n y_predict = (y_predict.ravel() > 0.5).astype(int)\n\n f1score = f1_score(y_test, y_predict)\n precision = precision_score(y_test, y_predict)\n recall = recall_score(y_test, y_predict)\n\n return f1score, precision, recall"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tokenize(text, custom_dict=None):\n global TOKENIZER\n if not TOKENIZER:\n TOKENIZER = DeepcutTokenizer()\n return TOKENIZER.tokenize(text, custom_dict=custom_dict)", "response": "Tokenize given Thai text string"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _document_frequency(X):\n if sp.isspmatrix_csr(X):\n return np.bincount(X.indices, minlength=X.shape[1])\n return np.diff(sp.csc_matrix(X, copy=False).indptr)", "response": "Count the number of non - zero values for each feature in sparse X."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking stop words list for stop words.", "response": "def _check_stop_list(stop):\n \"\"\"\n Check stop words list\n ref: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L87-L95\n \"\"\"\n if stop == \"thai\":\n return THAI_STOP_WORDS\n elif isinstance(stop, six.string_types):\n raise ValueError(\"not a built-in stop list: %s\" % stop)\n elif stop is None:\n return None\n # assume it's a collection\n return frozenset(stop)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nturning tokens into a list of tokens of n - grams of the given n - grams.", "response": "def _word_ngrams(self, tokens):\n \"\"\"\n Turn tokens into a tokens of n-grams\n\n ref: https://github.com/scikit-learn/scikit-learn/blob/ef5cb84a/sklearn/feature_extraction/text.py#L124-L153\n \"\"\"\n # handle stop words\n if self.stop_words is not None:\n tokens = [w for w in tokens if w not in self.stop_words]\n\n # handle token n-grams\n min_n, max_n = self.ngram_range\n if max_n != 1:\n original_tokens = tokens\n if min_n == 1:\n # no need to do any slicing for unigrams\n # just iterate through the original tokens\n tokens = list(original_tokens)\n min_n += 1\n else:\n tokens = []\n\n n_original_tokens = len(original_tokens)\n\n # bind method outside of loop to reduce overhead\n tokens_append = tokens.append\n space_join = \" \".join\n\n for n in range(min_n,\n min(max_n + 1, n_original_tokens + 1)):\n for i in range(n_original_tokens - n + 1):\n tokens_append(space_join(original_tokens[i: i + n]))\n\n return tokens"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntransform given list of raw_documents to document - term matrix in sparse CSR format", "response": "def fit_tranform(self, raw_documents):\n \"\"\"\n Transform given list of raw_documents to document-term matrix in\n sparse CSR format (see scipy)\n \"\"\"\n X = self.transform(raw_documents, new_document=True)\n return X"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_feature_array(text, n_pad=21):\n n = len(text)\n n_pad_2 = int((n_pad - 1)/2)\n text_pad = [' '] * n_pad_2 + [t for t in text] + [' '] * n_pad_2\n x_char, x_type = [], []\n for i in range(n_pad_2, n_pad_2 + n):\n char_list = text_pad[i + 1: i + n_pad_2 + 1] + \\\n list(reversed(text_pad[i - n_pad_2: i])) + \\\n [text_pad[i]]\n char_map = [CHARS_MAP.get(c, 80) for c in char_list]\n char_type = [CHAR_TYPES_MAP.get(CHAR_TYPE_FLATTEN.get(c, 'o'), 4)\n for c in char_list]\n x_char.append(char_map)\n x_type.append(char_type)\n x_char = np.array(x_char).astype(float)\n x_type = np.array(x_type).astype(float)\n return x_char, x_type", "response": "Create feature array of character and surrounding characters"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_n_gram_df(df, n_pad):\n n_pad_2 = int((n_pad - 1)/2)\n for i in range(n_pad_2):\n df['char-{}'.format(i+1)] = df['char'].shift(i + 1)\n df['type-{}'.format(i+1)] = df['type'].shift(i + 1)\n df['char{}'.format(i+1)] = df['char'].shift(-i - 1)\n df['type{}'.format(i+1)] = df['type'].shift(-i - 1)\n return df[n_pad_2: -n_pad_2]", "response": "Given input dataframe create feature dataframe of shifted characters"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfetch the course enrollment data for the given enterprise customer UUID.", "response": "def _fetch_course_enrollment_data(self, enterprise_customer_uuid):\n \"\"\"\n Return enterprise customer UUID/user_id/course_run_id triples which represent CourseEnrollment records\n which do not have a matching EnterpriseCourseEnrollment record.\n\n The query used below looks for CourseEnrollment records that are associated with enterprise\n learners where the enrollment data is after the creation of the link between the learner\n and the enterprise. It also excludes learners with edx.org email addresses in order to\n filter out test users.\n \"\"\"\n query = '''\n SELECT\n au.id as user_id,\n ecu.enterprise_customer_id as enterprise_customer_uuid,\n sce.course_id as course_run_id\n FROM student_courseenrollment sce\n JOIN auth_user au\n ON au.id = sce.user_id\n JOIN enterprise_enterprisecustomeruser ecu\n ON ecu.user_id = au.id\n LEFT JOIN enterprise_enterprisecourseenrollment ece\n ON ece.enterprise_customer_user_id = ecu.id\n AND ece.course_id = sce.course_id\n WHERE\n ece.id IS NULL\n AND ecu.created <= sce.created\n AND au.email NOT LIKE '%@edx.org'\n {enterprise_customer_filter}\n ORDER BY sce.created;\n '''\n\n with connection.cursor() as cursor:\n if enterprise_customer_uuid:\n cursor.execute(\n query.format(enterprise_customer_filter='AND ecu.enterprise_customer_id = %s'),\n [enterprise_customer_uuid]\n )\n else:\n cursor.execute(\n query.format(enterprise_customer_filter='')\n )\n\n return self._dictfetchall(cursor)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn all rows from a cursor as a list of dicts.", "response": "def _dictfetchall(self, cursor):\n \"\"\" Return all rows from a cursor as a dict. \"\"\"\n columns = [col[0] for col in cursor.description]\n return [\n dict(zip(columns, row))\n for row in cursor.fetchall()\n ]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses a received datetime into a timezone - aware Python datetime object.", "response": "def parse_lms_api_datetime(datetime_string, datetime_format=LMS_API_DATETIME_FORMAT):\n \"\"\"\n Parse a received datetime into a timezone-aware, Python datetime object.\n\n Arguments:\n datetime_string: A string to be parsed.\n datetime_format: A datetime format string to be used for parsing\n\n \"\"\"\n if isinstance(datetime_string, datetime.datetime):\n date_time = datetime_string\n else:\n try:\n date_time = datetime.datetime.strptime(datetime_string, datetime_format)\n except ValueError:\n date_time = datetime.datetime.strptime(datetime_string, LMS_API_DATETIME_FORMAT_WITHOUT_TIMEZONE)\n\n # If the datetime format didn't include a timezone, then set to UTC.\n # Note that if we're using the default LMS_API_DATETIME_FORMAT, it ends in 'Z',\n # which denotes UTC for ISO-8661.\n if date_time.tzinfo is None:\n date_time = date_time.replace(tzinfo=timezone.utc)\n return date_time"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconnect to the REST API.", "response": "def connect(self):\n \"\"\"\n Connect to the REST API, authenticating with a JWT for the current user.\n \"\"\"\n if JwtBuilder is None:\n raise NotConnectedToOpenEdX(\"This package must be installed in an OpenEdX environment.\")\n\n now = int(time())\n jwt = JwtBuilder.create_jwt_for_user(self.user)\n self.client = EdxRestApiClient(\n self.API_BASE_URL, append_slash=self.APPEND_SLASH, jwt=jwt,\n )\n self.expires_at = now + self.expires_in"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nuse this method decorator to ensure the JWT token is refreshed when needed.", "response": "def refresh_token(func):\n \"\"\"\n Use this method decorator to ensure the JWT token is refreshed when needed.\n \"\"\"\n @wraps(func)\n def inner(self, *args, **kwargs):\n \"\"\"\n Before calling the wrapped function, we check if the JWT token is expired, and if so, re-connect.\n \"\"\"\n if self.token_expired():\n self.connect()\n return func(self, *args, **kwargs)\n return inner"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nredirect to embargo error page if the given user is blocked.", "response": "def redirect_if_blocked(course_run_ids, user=None, ip_address=None, url=None):\n \"\"\"\n Return redirect to embargo error page if the given user is blocked.\n \"\"\"\n for course_run_id in course_run_ids:\n redirect_url = embargo_api.redirect_if_blocked(\n CourseKey.from_string(course_run_id),\n user=user,\n ip_address=ip_address,\n url=url\n )\n if redirect_url:\n return redirect_url"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nqueries the Enrollment API for the course details of the given course_id.", "response": "def get_course_details(self, course_id):\n \"\"\"\n Query the Enrollment API for the course details of the given course_id.\n\n Args:\n course_id (str): The string value of the course's unique identifier\n\n Returns:\n dict: A dictionary containing details about the course, in an enrollment context (allowed modes, etc.)\n \"\"\"\n try:\n return self.client.course(course_id).get()\n except (SlumberBaseException, ConnectionError, Timeout) as exc:\n LOGGER.exception(\n 'Failed to retrieve course enrollment details for course [%s] due to: [%s]',\n course_id, str(exc)\n )\n return {}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _sort_course_modes(self, modes):\n def slug_weight(mode):\n \"\"\"\n Assign a weight to the course mode dictionary based on the position of its slug in the sorting list.\n \"\"\"\n sorting_slugs = COURSE_MODE_SORT_ORDER\n sorting_slugs_size = len(sorting_slugs)\n if mode['slug'] in sorting_slugs:\n return sorting_slugs_size - sorting_slugs.index(mode['slug'])\n return 0\n # Sort slug weights in descending order\n return sorted(modes, key=slug_weight, reverse=True)", "response": "Sort the course mode dictionaries by slug according to the COURSE_MODE_SORT_ORDER constant."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_course_modes(self, course_id):\n details = self.get_course_details(course_id)\n modes = details.get('course_modes', [])\n return self._sort_course_modes([mode for mode in modes if mode['slug'] not in EXCLUDED_COURSE_MODES])", "response": "Query the Enrollment API for the specific course modes that are available for the given course_id."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning True if the given course mode is available for the given course run.", "response": "def has_course_mode(self, course_run_id, mode):\n \"\"\"\n Query the Enrollment API to see whether a course run has a given course mode available.\n\n Arguments:\n course_run_id (str): The string value of the course run's unique identifier\n\n Returns:\n bool: Whether the course run has the given mode avaialble for enrollment.\n\n \"\"\"\n course_modes = self.get_course_modes(course_run_id)\n return any(course_mode for course_mode in course_modes if course_mode['slug'] == mode)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef enroll_user_in_course(self, username, course_id, mode, cohort=None):\n return self.client.enrollment.post(\n {\n 'user': username,\n 'course_details': {'course_id': course_id},\n 'mode': mode,\n 'cohort': cohort,\n }\n )", "response": "Enroll a user in the specified course"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncalls the enrollment API to unenroll the user in the course specified by course_id. Args: username (str): The username by which the user goes on the OpenEdx platform course_id (str): The string value of the course's unique identifier Returns: bool: Whether the unenrollment succeeded", "response": "def unenroll_user_from_course(self, username, course_id):\n \"\"\"\n Call the enrollment API to unenroll the user in the course specified by course_id.\n Args:\n username (str): The username by which the user goes on the OpenEdx platform\n course_id (str): The string value of the course's unique identifier\n Returns:\n bool: Whether the unenrollment succeeded\n \"\"\"\n enrollment = self.get_course_enrollment(username, course_id)\n if enrollment and enrollment['is_active']:\n response = self.client.enrollment.post({\n 'user': username,\n 'course_details': {'course_id': course_id},\n 'is_active': False,\n 'mode': enrollment['mode']\n })\n return not response['is_active']\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_course_enrollment(self, username, course_id):\n endpoint = getattr(\n self.client.enrollment,\n '{username},{course_id}'.format(username=username, course_id=course_id)\n )\n try:\n result = endpoint.get()\n except HttpNotFoundError:\n # This enrollment data endpoint returns a 404 if either the username or course_id specified isn't valid\n LOGGER.error(\n 'Course enrollment details not found for invalid username or course; username=[%s], course=[%s]',\n username,\n course_id\n )\n return None\n # This enrollment data endpoint returns an empty string if the username and course_id is valid, but there's\n # no matching enrollment found\n if not result:\n LOGGER.info('Failed to find course enrollment details for user [%s] and course [%s]', username, course_id)\n return None\n\n return result", "response": "Query the course enrollment API to get information about a single user s course."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning True if the user is enrolled in a course run False otherwise.", "response": "def is_enrolled(self, username, course_run_id):\n \"\"\"\n Query the enrollment API and determine if a learner is enrolled in a course run.\n\n Args:\n username (str): The username by which the user goes on the OpenEdX platform\n course_run_id (str): The string value of the course's unique identifier\n\n Returns:\n bool: Indicating whether the user is enrolled in the course run. Returns False under any errors.\n\n \"\"\"\n enrollment = self.get_course_enrollment(username, course_run_id)\n return enrollment is not None and enrollment.get('is_active', False)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the value of the result field_name in the result field of the user identified by param_name.", "response": "def _get_results(self, identity_provider, param_name, param_value, result_field_name):\n \"\"\"\n Calls the third party auth api endpoint to get the mapping between usernames and remote ids.\n \"\"\"\n try:\n kwargs = {param_name: param_value}\n returned = self.client.providers(identity_provider).users.get(**kwargs)\n results = returned.get('results', [])\n except HttpNotFoundError:\n LOGGER.error(\n 'username not found for third party provider={provider}, {querystring_param}={id}'.format(\n provider=identity_provider,\n querystring_param=param_name,\n id=param_value\n )\n )\n results = []\n\n for row in results:\n if row.get(param_name) == param_value:\n return row.get(result_field_name)\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves the grade for the given user for the given course.", "response": "def get_course_grade(self, course_id, username):\n \"\"\"\n Retrieve the grade for the given username for the given course_id.\n\n Args:\n * ``course_id`` (str): The string value of the course's unique identifier\n * ``username`` (str): The username ID identifying the user for which to retrieve the grade.\n\n Raises:\n\n HttpNotFoundError if no grade found for the given user+course.\n\n Returns:\n\n a dict containing:\n\n * ``username``: A string representation of a user's username passed in the request.\n * ``course_key``: A string representation of a Course ID.\n * ``passed``: Boolean representing whether the course has been passed according the course's grading policy.\n * ``percent``: A float representing the overall grade for the course\n * ``letter_grade``: A letter grade as defined in grading_policy (e.g. 'A' 'B' 'C' for 6.002x) or None\n\n \"\"\"\n results = self.client.courses(course_id).get(username=username)\n for row in results:\n if row.get('username') == username:\n return row\n\n raise HttpNotFoundError('No grade record found for course={}, username={}'.format(course_id, username))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nretrieves the certificate for the given username for the given course_id.", "response": "def get_course_certificate(self, course_id, username):\n \"\"\"\n Retrieve the certificate for the given username for the given course_id.\n\n Args:\n * ``course_id`` (str): The string value of the course's unique identifier\n * ``username`` (str): The username ID identifying the user for which to retrieve the certificate\n\n Raises:\n\n HttpNotFoundError if no certificate found for the given user+course.\n\n Returns:\n\n a dict containing:\n\n * ``username``: A string representation of an user's username passed in the request.\n * ``course_id``: A string representation of a Course ID.\n * ``certificate_type``: A string representation of the certificate type.\n * ``created_date`: Datetime the certificate was created (tz-aware).\n * ``status``: A string representation of the certificate status.\n * ``is_passing``: True if the certificate has a passing status, False if not.\n * ``download_url``: A string representation of the certificate url.\n * ``grade``: A string representation of a float for the user's course grade.\n\n \"\"\"\n return self.client.certificates(username).courses(course_id).get()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a Course Discovery API client setup with authentication for the specified user.", "response": "def course_discovery_api_client(user, catalog_url):\n \"\"\"\n Return a Course Discovery API client setup with authentication for the specified user.\n \"\"\"\n if JwtBuilder is None:\n raise NotConnectedToOpenEdX(\n _(\"To get a Catalog API client, this package must be \"\n \"installed in an Open edX environment.\")\n )\n\n jwt = JwtBuilder.create_jwt_for_user(user)\n return EdxRestApiClient(catalog_url, jwt=jwt)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntraverse a paginated API response and extracts and concatenates results returned by API.", "response": "def traverse_pagination(response, endpoint, content_filter_query, query_params):\n \"\"\"\n Traverse a paginated API response and extracts and concatenates \"results\" returned by API.\n\n Arguments:\n response (dict): API response object.\n endpoint (Slumber.Resource): API endpoint object.\n content_filter_query (dict): query parameters used to filter catalog results.\n query_params (dict): query parameters used to paginate results.\n\n Returns:\n list: all the results returned by the API.\n \"\"\"\n results = response.get('results', [])\n\n page = 1\n while response.get('next'):\n page += 1\n response = endpoint().post(content_filter_query, **dict(query_params, page=page))\n results += response.get('results', [])\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the results from the course - discovery search - all endpoint.", "response": "def get_catalog_results(self, content_filter_query, query_params=None, traverse_pagination=False):\n \"\"\"\n Return results from the discovery service's search/all endpoint.\n\n Arguments:\n content_filter_query (dict): query parameters used to filter catalog results.\n query_params (dict): query parameters used to paginate results.\n traverse_pagination (bool): True to return all results, False to return the paginated response.\n Defaults to False.\n\n Returns:\n dict: Paginated response or all the records.\n \"\"\"\n query_params = query_params or {}\n\n try:\n endpoint = getattr(self.client, self.SEARCH_ALL_ENDPOINT)\n response = endpoint().post(data=content_filter_query, **query_params)\n if traverse_pagination:\n response['results'] = self.traverse_pagination(response, endpoint, content_filter_query, query_params)\n response['next'] = response['previous'] = None\n except Exception as ex: # pylint: disable=broad-except\n LOGGER.exception(\n 'Attempted to call course-discovery search/all/ endpoint with the following parameters: '\n 'content_filter_query: %s, query_params: %s, traverse_pagination: %s. '\n 'Failed to retrieve data from the catalog API. content -- [%s]',\n content_filter_query,\n query_params,\n traverse_pagination,\n getattr(ex, 'content', '')\n )\n # We need to bubble up failures when we encounter them instead of masking them!\n raise ex\n\n return response"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets course catalog details.", "response": "def get_catalog(self, catalog_id):\n \"\"\"\n Return specified course catalog.\n\n Returns:\n dict: catalog details if it is available for the user.\n\n \"\"\"\n return self._load_data(\n self.CATALOGS_ENDPOINT,\n default=[],\n resource_id=catalog_id\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn paginated response for all catalog courses.", "response": "def get_paginated_catalog_courses(self, catalog_id, querystring=None):\n \"\"\"\n Return paginated response for all catalog courses.\n\n Returns:\n dict: API response with links to next and previous pages.\n\n \"\"\"\n return self._load_data(\n self.CATALOGS_COURSES_ENDPOINT.format(catalog_id),\n default=[],\n querystring=querystring,\n traverse_pagination=False,\n many=False,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_paginated_catalogs(self, querystring=None):\n return self._load_data(\n self.CATALOGS_ENDPOINT,\n default=[],\n querystring=querystring,\n traverse_pagination=False,\n many=False\n )", "response": "Returns a paginated list of course catalogs including name and ID."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_catalog_courses(self, catalog_id):\n return self._load_data(\n self.CATALOGS_COURSES_ENDPOINT.format(catalog_id),\n default=[]\n )", "response": "Returns the courses included in a single course catalog by ID."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_course_and_course_run(self, course_run_id):\n # Parse the course ID from the course run ID.\n course_id = parse_course_key(course_run_id)\n # Retrieve the course metadata from the catalog service.\n course = self.get_course_details(course_id)\n\n course_run = None\n if course:\n # Find the specified course run.\n course_run = None\n course_runs = [course_run for course_run in course['course_runs'] if course_run['key'] == course_run_id]\n if course_runs:\n course_run = course_runs[0]\n\n return course, course_run", "response": "Returns the course and course run metadata for the given course run ID."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the course details for a single course by id.", "response": "def get_course_details(self, course_id):\n \"\"\"\n Return the details of a single course by id - not a course run id.\n\n Args:\n course_id (str): The unique id for the course in question.\n\n Returns:\n dict: Details of the course in question.\n\n \"\"\"\n return self._load_data(\n self.COURSES_ENDPOINT,\n resource_id=course_id,\n many=False\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a program by title or None if not found.", "response": "def get_program_by_title(self, program_title):\n \"\"\"\n Return single program by name, or None if not found.\n\n Arguments:\n program_title(string): Program title as seen by students and in Course Catalog Admin\n\n Returns:\n dict: Program data provided by Course Catalog API\n\n \"\"\"\n all_programs = self._load_data(self.PROGRAMS_ENDPOINT, default=[])\n matching_programs = [program for program in all_programs if program.get('title') == program_title]\n if len(matching_programs) > 1:\n raise MultipleProgramMatchError(len(matching_programs))\n elif len(matching_programs) == 1:\n return matching_programs[0]\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a dictionary containing the program data for a given program UUID", "response": "def get_program_by_uuid(self, program_uuid):\n \"\"\"\n Return single program by UUID, or None if not found.\n\n Arguments:\n program_uuid(string): Program UUID in string form\n\n Returns:\n dict: Program data provided by Course Catalog API\n\n \"\"\"\n return self._load_data(\n self.PROGRAMS_ENDPOINT,\n resource_id=program_uuid,\n default=None\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting a list of the course keys contained in the program.", "response": "def get_program_course_keys(self, program_uuid):\n \"\"\"\n Get a list of the course IDs (not course run IDs) contained in the program.\n\n Arguments:\n program_uuid (str): Program UUID in string form\n\n Returns:\n list(str): List of course keys in string form that are included in the program\n\n \"\"\"\n program_details = self.get_program_by_uuid(program_uuid)\n if not program_details:\n return []\n return [course['key'] for course in program_details.get('courses', [])]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_program_type_by_slug(self, slug):\n return self._load_data(\n self.PROGRAM_TYPES_ENDPOINT,\n resource_id=slug,\n default=None,\n )", "response": "Get a program type by its slug."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nfind common course modes for a set of course runs. This function essentially returns an intersection of types of seats available for each course run. Arguments: course_run_ids(Iterable[str]): Target Course run IDs. Returns: set: course modes found in all given course runs Examples: # run1 has prof and audit, run 2 has the same get_common_course_modes(['course-v1:run1', 'course-v1:run2']) {'prof', 'audit'} # run1 has prof and audit, run 2 has only prof get_common_course_modes(['course-v1:run1', 'course-v1:run2']) {'prof'} # run1 has prof and audit, run 2 honor get_common_course_modes(['course-v1:run1', 'course-v1:run2']) {} # run1 has nothing, run2 has prof get_common_course_modes(['course-v1:run1', 'course-v1:run2']) {} # run1 has prof and audit, run 2 prof, run3 has audit get_common_course_modes(['course-v1:run1', 'course-v1:run2', 'course-v1:run3']) {} # run1 has nothing, run 2 prof, run3 has prof get_common_course_modes(['course-v1:run1', 'course-v1:run2', 'course-v1:run3']) {}", "response": "def get_common_course_modes(self, course_run_ids):\n \"\"\"\n Find common course modes for a set of course runs.\n\n This function essentially returns an intersection of types of seats available\n for each course run.\n\n Arguments:\n course_run_ids(Iterable[str]): Target Course run IDs.\n\n Returns:\n set: course modes found in all given course runs\n\n Examples:\n # run1 has prof and audit, run 2 has the same\n get_common_course_modes(['course-v1:run1', 'course-v1:run2'])\n {'prof', 'audit'}\n\n # run1 has prof and audit, run 2 has only prof\n get_common_course_modes(['course-v1:run1', 'course-v1:run2'])\n {'prof'}\n\n # run1 has prof and audit, run 2 honor\n get_common_course_modes(['course-v1:run1', 'course-v1:run2'])\n {}\n\n # run1 has nothing, run2 has prof\n get_common_course_modes(['course-v1:run1', 'course-v1:run2'])\n {}\n\n # run1 has prof and audit, run 2 prof, run3 has audit\n get_common_course_modes(['course-v1:run1', 'course-v1:run2', 'course-v1:run3'])\n {}\n\n # run1 has nothing, run 2 prof, run3 has prof\n get_common_course_modes(['course-v1:run1', 'course-v1:run2', 'course-v1:run3'])\n {}\n\n \"\"\"\n available_course_modes = None\n for course_run_id in course_run_ids:\n course_run = self.get_course_run(course_run_id) or {}\n course_run_modes = {seat.get('type') for seat in course_run.get('seats', [])}\n\n if available_course_modes is None:\n available_course_modes = course_run_modes\n else:\n available_course_modes &= course_run_modes\n\n if not available_course_modes:\n return available_course_modes\n\n return available_course_modes"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_course_in_catalog(self, catalog_id, course_id):\n try:\n # Determine if we have a course run ID, rather than a plain course ID\n course_run_id = str(CourseKey.from_string(course_id))\n except InvalidKeyError:\n course_run_id = None\n\n endpoint = self.client.catalogs(catalog_id).contains\n\n if course_run_id:\n resp = endpoint.get(course_run_id=course_run_id)\n else:\n resp = endpoint.get(course_id=course_id)\n\n return resp.get('courses', {}).get(course_id, False)", "response": "Determines if the given course or course run ID is contained in the given catalog."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading data from API client.", "response": "def _load_data(self, resource, default=DEFAULT_VALUE_SAFEGUARD, **kwargs):\n \"\"\"\n Load data from API client.\n\n Arguments:\n resource(string): type of resource to load\n default(any): value to return if API query returned empty result. Sensible values: [], {}, None etc.\n\n Returns:\n dict: Deserialized response from Course Catalog API\n\n \"\"\"\n default_val = default if default != self.DEFAULT_VALUE_SAFEGUARD else {}\n try:\n return get_edx_api_data(\n api_config=CatalogIntegration.current(),\n resource=resource,\n api=self.client,\n **kwargs\n ) or default_val\n except (SlumberBaseException, ConnectionError, Timeout) as exc:\n LOGGER.exception(\n 'Failed to load data from resource [%s] with kwargs [%s] due to: [%s]',\n resource, kwargs, str(exc)\n )\n return default_val"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns all content metadata for the EnterpriseCustomer.", "response": "def get_content_metadata(self, enterprise_customer):\n \"\"\"\n Return all content metadata contained in the catalogs associated with the EnterpriseCustomer.\n\n Arguments:\n enterprise_customer (EnterpriseCustomer): The EnterpriseCustomer to return content metadata for.\n\n Returns:\n list: List of dicts containing content metadata.\n \"\"\"\n content_metadata = OrderedDict()\n\n # TODO: This if block can be removed when we get rid of discovery service-based catalogs.\n if enterprise_customer.catalog:\n response = self._load_data(\n self.ENTERPRISE_CUSTOMER_ENDPOINT,\n detail_resource='courses',\n resource_id=str(enterprise_customer.uuid),\n traverse_pagination=True,\n )\n for course in response['results']:\n for course_run in course['course_runs']:\n course_run['content_type'] = 'courserun' # Make this look like a search endpoint result.\n content_metadata[course_run['key']] = course_run\n\n for enterprise_customer_catalog in enterprise_customer.enterprise_customer_catalogs.all():\n response = self._load_data(\n self.ENTERPRISE_CUSTOMER_CATALOGS_ENDPOINT,\n resource_id=str(enterprise_customer_catalog.uuid),\n traverse_pagination=True,\n querystring={'page_size': 1000},\n )\n\n for item in response['results']:\n content_id = utils.get_content_metadata_item_id(item)\n content_metadata[content_id] = item\n\n return content_metadata.values()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading a response from one of the Enterprise endpoints.", "response": "def _load_data(\n self,\n resource,\n detail_resource=None,\n resource_id=None,\n querystring=None,\n traverse_pagination=False,\n default=DEFAULT_VALUE_SAFEGUARD,\n ):\n \"\"\"\n Loads a response from a call to one of the Enterprise endpoints.\n\n :param resource: The endpoint resource name.\n :param detail_resource: The sub-resource to append to the path.\n :param resource_id: The resource ID for the specific detail to get from the endpoint.\n :param querystring: Optional query string parameters.\n :param traverse_pagination: Whether to traverse pagination or return paginated response.\n :param default: The default value to return in case of no response content.\n :return: Data returned by the API.\n \"\"\"\n default_val = default if default != self.DEFAULT_VALUE_SAFEGUARD else {}\n querystring = querystring if querystring else {}\n\n cache_key = utils.get_cache_key(\n resource=resource,\n querystring=querystring,\n traverse_pagination=traverse_pagination,\n resource_id=resource_id\n )\n response = cache.get(cache_key)\n if not response:\n # Response is not cached, so make a call.\n endpoint = getattr(self.client, resource)(resource_id)\n endpoint = getattr(endpoint, detail_resource) if detail_resource else endpoint\n response = endpoint.get(**querystring)\n if traverse_pagination:\n results = utils.traverse_pagination(response, endpoint)\n response = {\n 'count': len(results),\n 'next': 'None',\n 'previous': 'None',\n 'results': results,\n }\n if response:\n # Now that we've got a response, cache it.\n cache.set(cache_key, response, settings.ENTERPRISE_API_CACHE_TIMEOUT)\n return response or default_val"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef transmit(self, payload, **kwargs):\n items_to_create, items_to_update, items_to_delete, transmission_map = self._partition_items(payload)\n self._transmit_delete(items_to_delete)\n self._transmit_create(items_to_create)\n self._transmit_update(items_to_update, transmission_map)", "response": "Transmit content metadata items to the integrated channel."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _partition_items(self, channel_metadata_item_map):\n items_to_create = {}\n items_to_update = {}\n items_to_delete = {}\n transmission_map = {}\n export_content_ids = channel_metadata_item_map.keys()\n\n # Get the items that were previously transmitted to the integrated channel.\n # If we are not transmitting something that was previously transmitted,\n # we need to delete it from the integrated channel.\n for transmission in self._get_transmissions():\n transmission_map[transmission.content_id] = transmission\n if transmission.content_id not in export_content_ids:\n items_to_delete[transmission.content_id] = transmission.channel_metadata\n\n # Compare what is currently being transmitted to what was transmitted\n # previously, identifying items that need to be created or updated.\n for item in channel_metadata_item_map.values():\n content_id = item.content_id\n channel_metadata = item.channel_metadata\n transmitted_item = transmission_map.get(content_id, None)\n if transmitted_item is not None:\n if diff(channel_metadata, transmitted_item.channel_metadata):\n items_to_update[content_id] = channel_metadata\n else:\n items_to_create[content_id] = channel_metadata\n\n LOGGER.info(\n 'Preparing to transmit creation of [%s] content metadata items with plugin configuration [%s]: [%s]',\n len(items_to_create),\n self.enterprise_configuration,\n items_to_create.keys(),\n )\n LOGGER.info(\n 'Preparing to transmit update of [%s] content metadata items with plugin configuration [%s]: [%s]',\n len(items_to_update),\n self.enterprise_configuration,\n items_to_update.keys(),\n )\n LOGGER.info(\n 'Preparing to transmit deletion of [%s] content metadata items with plugin configuration [%s]: [%s]',\n len(items_to_delete),\n self.enterprise_configuration,\n items_to_delete.keys(),\n )\n\n return items_to_create, items_to_update, items_to_delete, transmission_map", "response": "Partition the items in the given channel metadata item map into two lists."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _serialize_items(self, channel_metadata_items):\n return json.dumps(\n self._prepare_items_for_transmission(channel_metadata_items),\n sort_keys=True\n ).encode('utf-8')", "response": "Serialize content metadata items for a create transmission to the integrated channel."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _transmit_create(self, channel_metadata_item_map):\n for chunk in chunks(channel_metadata_item_map, self.enterprise_configuration.transmission_chunk_size):\n serialized_chunk = self._serialize_items(list(chunk.values()))\n try:\n self.client.create_content_metadata(serialized_chunk)\n except ClientError as exc:\n LOGGER.error(\n 'Failed to update [%s] content metadata items for integrated channel [%s] [%s]',\n len(chunk),\n self.enterprise_configuration.enterprise_customer.name,\n self.enterprise_configuration.channel_code,\n )\n LOGGER.error(exc)\n else:\n self._create_transmissions(chunk)", "response": "Transmit create content metadata items to integrated channel."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates content metadata items for integrated channel.", "response": "def _transmit_update(self, channel_metadata_item_map, transmission_map):\n \"\"\"\n Transmit content metadata update to integrated channel.\n \"\"\"\n for chunk in chunks(channel_metadata_item_map, self.enterprise_configuration.transmission_chunk_size):\n serialized_chunk = self._serialize_items(list(chunk.values()))\n try:\n self.client.update_content_metadata(serialized_chunk)\n except ClientError as exc:\n LOGGER.error(\n 'Failed to update [%s] content metadata items for integrated channel [%s] [%s]',\n len(chunk),\n self.enterprise_configuration.enterprise_customer.name,\n self.enterprise_configuration.channel_code,\n )\n LOGGER.error(exc)\n else:\n self._update_transmissions(chunk, transmission_map)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _transmit_delete(self, channel_metadata_item_map):\n for chunk in chunks(channel_metadata_item_map, self.enterprise_configuration.transmission_chunk_size):\n serialized_chunk = self._serialize_items(list(chunk.values()))\n try:\n self.client.delete_content_metadata(serialized_chunk)\n except ClientError as exc:\n LOGGER.error(\n 'Failed to delete [%s] content metadata items for integrated channel [%s] [%s]',\n len(chunk),\n self.enterprise_configuration.enterprise_customer.name,\n self.enterprise_configuration.channel_code,\n )\n LOGGER.error(exc)\n else:\n self._delete_transmissions(chunk.keys())", "response": "Transmit delete to integrated channel."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the Transmissions for previously - transmitted content metadata items.", "response": "def _get_transmissions(self):\n \"\"\"\n Return the ContentMetadataItemTransmision models for previously\n transmitted content metadata items.\n \"\"\"\n # pylint: disable=invalid-name\n ContentMetadataItemTransmission = apps.get_model(\n 'integrated_channel',\n 'ContentMetadataItemTransmission'\n )\n return ContentMetadataItemTransmission.objects.filter(\n enterprise_customer=self.enterprise_configuration.enterprise_customer,\n integrated_channel_code=self.enterprise_configuration.channel_code()\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _create_transmissions(self, content_metadata_item_map):\n # pylint: disable=invalid-name\n ContentMetadataItemTransmission = apps.get_model(\n 'integrated_channel',\n 'ContentMetadataItemTransmission'\n )\n transmissions = []\n for content_id, channel_metadata in content_metadata_item_map.items():\n transmissions.append(\n ContentMetadataItemTransmission(\n enterprise_customer=self.enterprise_configuration.enterprise_customer,\n integrated_channel_code=self.enterprise_configuration.channel_code(),\n content_id=content_id,\n channel_metadata=channel_metadata\n )\n )\n ContentMetadataItemTransmission.objects.bulk_create(transmissions)", "response": "Create the Transmissions models for the given content metadata items."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _update_transmissions(self, content_metadata_item_map, transmission_map):\n for content_id, channel_metadata in content_metadata_item_map.items():\n transmission = transmission_map[content_id]\n transmission.channel_metadata = channel_metadata\n transmission.save()", "response": "Update the Transmissions models for the given content metadata items."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndelete Transmissions associated with the given content metadata items.", "response": "def _delete_transmissions(self, content_metadata_item_ids):\n \"\"\"\n Delete ContentMetadataItemTransmision models associated with the given content metadata items.\n \"\"\"\n # pylint: disable=invalid-name\n ContentMetadataItemTransmission = apps.get_model(\n 'integrated_channel',\n 'ContentMetadataItemTransmission'\n )\n ContentMetadataItemTransmission.objects.filter(\n enterprise_customer=self.enterprise_configuration.enterprise_customer,\n integrated_channel_code=self.enterprise_configuration.channel_code(),\n content_id__in=content_metadata_item_ids\n ).delete()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deprecated(extra):\n def decorator(func):\n \"\"\"\n Return a decorated function that emits a deprecation warning on use.\n \"\"\"\n @wraps(func)\n def wrapper(*args, **kwargs):\n \"\"\"\n Wrap the function.\n \"\"\"\n message = 'You called the deprecated function `{function}`. {extra}'.format(\n function=func.__name__,\n extra=extra\n )\n frame = inspect.currentframe().f_back\n warnings.warn_explicit(\n message,\n category=DeprecationWarning,\n filename=inspect.getfile(frame.f_code),\n lineno=frame.f_lineno\n )\n return func(*args, **kwargs)\n return wrapper\n return decorator", "response": "A decorator that marks a method as deprecated."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nignores any emitted warnings from a function. :param warning: The category of warning to ignore.", "response": "def ignore_warning(warning):\n \"\"\"\n Ignore any emitted warnings from a function.\n\n :param warning: The category of warning to ignore.\n \"\"\"\n def decorator(func):\n \"\"\"\n Return a decorated function whose emitted warnings are ignored.\n \"\"\"\n @wraps(func)\n def wrapper(*args, **kwargs):\n \"\"\"\n Wrap the function.\n \"\"\"\n warnings.simplefilter('ignore', warning)\n return func(*args, **kwargs)\n return wrapper\n return decorator"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef enterprise_login_required(view):\n @wraps(view)\n def wrapper(request, *args, **kwargs):\n \"\"\"\n Wrap the decorator.\n \"\"\"\n if 'enterprise_uuid' not in kwargs:\n raise Http404\n\n enterprise_uuid = kwargs['enterprise_uuid']\n enterprise_customer = get_enterprise_customer_or_404(enterprise_uuid)\n\n # Now verify if the user is logged in. If user is not logged in then\n # send the user to the login screen to sign in with an\n # Enterprise-linked IdP and the pipeline will get them back here.\n if not request.user.is_authenticated:\n parsed_current_url = urlparse(request.get_full_path())\n parsed_query_string = parse_qs(parsed_current_url.query)\n parsed_query_string.update({\n 'tpa_hint': enterprise_customer.identity_provider,\n FRESH_LOGIN_PARAMETER: 'yes'\n })\n next_url = '{current_path}?{query_string}'.format(\n current_path=quote(parsed_current_url.path),\n query_string=urlencode(parsed_query_string, doseq=True)\n )\n return redirect(\n '{login_url}?{params}'.format(\n login_url='/login',\n params=urlencode(\n {'next': next_url}\n )\n )\n )\n\n # Otherwise, they can proceed to the original view.\n return view(request, *args, **kwargs)\n\n return wrapper", "response": "Decorator for allowing authenticated user with valid enterprise UUID. This decorator will allow authenticated user with valid enterprise UUID."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nviews decorator which terminates stale TPA sessions. This decorator forces the user to obtain a new session the first time they access the decorated view. This prevents TPA-authenticated users from hijacking the session of another user who may have been previously logged in using the same browser window. This decorator should be used in conjunction with the enterprise_login_required decorator. Usage:: @enterprise_login_required @force_fresh_session() def my_view(request, enterprise_uuid): # Some functionality ... OR class MyView(View): ... @method_decorator(enterprise_login_required) @method_decorator(force_fresh_session) def get(self, request, enterprise_uuid): # Some functionality ...", "response": "def force_fresh_session(view):\n \"\"\"\n View decorator which terminates stale TPA sessions.\n\n This decorator forces the user to obtain a new session\n the first time they access the decorated view. This prevents\n TPA-authenticated users from hijacking the session of another\n user who may have been previously logged in using the same\n browser window.\n\n This decorator should be used in conjunction with the\n enterprise_login_required decorator.\n\n Usage::\n @enterprise_login_required\n @force_fresh_session()\n def my_view(request, enterprise_uuid):\n # Some functionality ...\n\n OR\n\n class MyView(View):\n ...\n @method_decorator(enterprise_login_required)\n @method_decorator(force_fresh_session)\n def get(self, request, enterprise_uuid):\n # Some functionality ...\n \"\"\"\n @wraps(view)\n def wrapper(request, *args, **kwargs):\n \"\"\"\n Wrap the function.\n \"\"\"\n if not request.GET.get(FRESH_LOGIN_PARAMETER):\n # The enterprise_login_required decorator promises to set the fresh login URL\n # parameter for this URL when it was the agent that initiated the login process;\n # if that parameter isn't set, we can safely assume that the session is \"stale\";\n # that isn't necessarily an issue, though. Redirect the user to\n # log out and then come back here - the enterprise_login_required decorator will\n # then take effect prior to us arriving back here again.\n enterprise_customer = get_enterprise_customer_or_404(kwargs.get('enterprise_uuid'))\n provider_id = enterprise_customer.identity_provider or ''\n sso_provider = get_identity_provider(provider_id)\n if sso_provider:\n # Parse the current request full path, quote just the path portion,\n # then reconstruct the full path string.\n # The path and query portions should be the only non-empty strings here.\n scheme, netloc, path, params, query, fragment = urlparse(request.get_full_path())\n redirect_url = urlunparse((scheme, netloc, quote(path), params, query, fragment))\n\n return redirect(\n '{logout_url}?{params}'.format(\n logout_url='/logout',\n params=urlencode(\n {'redirect_url': redirect_url}\n )\n )\n )\n return view(request, *args, **kwargs)\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nverifies that the username is a valid user and that the user has an associated EnterpriseCustomerUser.", "response": "def validate_username(self, value):\n \"\"\"\n Verify that the username has a matching user, and that the user has an associated EnterpriseCustomerUser.\n \"\"\"\n try:\n user = User.objects.get(username=value)\n except User.DoesNotExist:\n raise serializers.ValidationError(\"User does not exist\")\n\n try:\n enterprise_customer_user = models.EnterpriseCustomerUser.objects.get(user_id=user.pk)\n except models.EnterpriseCustomerUser.DoesNotExist:\n raise serializers.ValidationError(\"User has no EnterpriseCustomerUser\")\n\n self.enterprise_customer_user = enterprise_customer_user\n return value"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save(self): # pylint: disable=arguments-differ\n course_id = self.validated_data['course_id']\n\n __, created = models.EnterpriseCourseEnrollment.objects.get_or_create(\n enterprise_customer_user=self.enterprise_customer_user,\n course_id=course_id,\n )\n if created:\n track_enrollment('rest-api-enrollment', self.enterprise_customer_user.user_id, course_id)", "response": "Save the model with the found EnterpriseCustomerUser."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nserialize the EnterpriseCustomerCatalog object into a dict.", "response": "def to_representation(self, instance):\n \"\"\"\n Serialize the EnterpriseCustomerCatalog object.\n\n Arguments:\n instance (EnterpriseCustomerCatalog): The EnterpriseCustomerCatalog to serialize.\n\n Returns:\n dict: The EnterpriseCustomerCatalog converted to a dict.\n \"\"\"\n request = self.context['request']\n enterprise_customer = instance.enterprise_customer\n\n representation = super(EnterpriseCustomerCatalogDetailSerializer, self).to_representation(instance)\n\n # Retrieve the EnterpriseCustomerCatalog search results from the discovery service.\n paginated_content = instance.get_paginated_content(request.GET)\n count = paginated_content['count']\n search_results = paginated_content['results']\n\n for item in search_results:\n content_type = item['content_type']\n marketing_url = item.get('marketing_url')\n if marketing_url:\n item['marketing_url'] = utils.update_query_parameters(\n marketing_url, utils.get_enterprise_utm_context(enterprise_customer)\n )\n # Add the Enterprise enrollment URL to each content item returned from the discovery service.\n if content_type == 'course':\n item['enrollment_url'] = instance.get_course_enrollment_url(item['key'])\n if content_type == 'courserun':\n item['enrollment_url'] = instance.get_course_run_enrollment_url(item['key'])\n if content_type == 'program':\n item['enrollment_url'] = instance.get_program_enrollment_url(item['uuid'])\n\n # Build pagination URLs\n previous_url = None\n next_url = None\n page = int(request.GET.get('page', '1'))\n request_uri = request.build_absolute_uri()\n if paginated_content['previous']:\n previous_url = utils.update_query_parameters(request_uri, {'page': page - 1})\n if paginated_content['next']:\n next_url = utils.update_query_parameters(request_uri, {'page': page + 1})\n\n representation['count'] = count\n representation['previous'] = previous_url\n representation['next'] = next_url\n representation['results'] = search_results\n\n return representation"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the enterprise related django groups that this user is a part of.", "response": "def get_groups(self, obj):\n \"\"\"\n Return the enterprise related django groups that this user is a part of.\n \"\"\"\n if obj.user:\n return [group.name for group in obj.user.groups.filter(name__in=ENTERPRISE_PERMISSION_GROUPS)]\n return []"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef validate_username(self, value):\n try:\n self.user = User.objects.get(username=value)\n except User.DoesNotExist:\n raise serializers.ValidationError(\"User does not exist\")\n\n return value", "response": "Verify that the username is a valid user name and set the user attribute to the user object."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the updated course data dictionary.", "response": "def to_representation(self, instance):\n \"\"\"\n Return the updated course data dictionary.\n\n Arguments:\n instance (dict): The course data.\n\n Returns:\n dict: The updated course data.\n \"\"\"\n updated_course = copy.deepcopy(instance)\n enterprise_customer_catalog = self.context['enterprise_customer_catalog']\n updated_course['enrollment_url'] = enterprise_customer_catalog.get_course_enrollment_url(\n updated_course['key']\n )\n for course_run in updated_course['course_runs']:\n course_run['enrollment_url'] = enterprise_customer_catalog.get_course_run_enrollment_url(\n course_run['key']\n )\n return updated_course"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef to_representation(self, instance):\n updated_course_run = copy.deepcopy(instance)\n enterprise_customer_catalog = self.context['enterprise_customer_catalog']\n updated_course_run['enrollment_url'] = enterprise_customer_catalog.get_course_run_enrollment_url(\n updated_course_run['key']\n )\n return updated_course_run", "response": "Returns the updated course run data dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef to_representation(self, instance):\n updated_program = copy.deepcopy(instance)\n enterprise_customer_catalog = self.context['enterprise_customer_catalog']\n updated_program['enrollment_url'] = enterprise_customer_catalog.get_program_enrollment_url(\n updated_program['uuid']\n )\n for course in updated_program['courses']:\n course['enrollment_url'] = enterprise_customer_catalog.get_course_enrollment_url(course['key'])\n for course_run in course['course_runs']:\n course_run['enrollment_url'] = enterprise_customer_catalog.get_course_run_enrollment_url(\n course_run['key']\n )\n return updated_program", "response": "Returns the updated program data dictionary."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_internal_value(self, data):\n\n if not isinstance(data, list):\n message = self.error_messages['not_a_list'].format(\n input_type=type(data).__name__\n )\n raise serializers.ValidationError({\n api_settings.NON_FIELD_ERRORS_KEY: [message]\n })\n\n ret = []\n\n for item in data:\n try:\n validated = self.child.run_validation(item)\n except serializers.ValidationError as exc:\n ret.append(exc.detail)\n else:\n ret.append(validated)\n\n return ret", "response": "This method converts the data into a list of internal data structures."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef to_representation(self, data):\n return [\n self.child.to_representation(item) if 'detail' in item else item for item in data\n ]", "response": "Convert the data structure into a list of items."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating an enterprise customer user.", "response": "def create(self, validated_data):\n \"\"\"\n Perform the enrollment for existing enterprise customer users, or create the pending objects for new users.\n \"\"\"\n enterprise_customer = self.context.get('enterprise_customer')\n lms_user = validated_data.get('lms_user_id')\n tpa_user = validated_data.get('tpa_user_id')\n user_email = validated_data.get('user_email')\n course_run_id = validated_data.get('course_run_id')\n course_mode = validated_data.get('course_mode')\n cohort = validated_data.get('cohort')\n email_students = validated_data.get('email_students')\n is_active = validated_data.get('is_active')\n\n enterprise_customer_user = lms_user or tpa_user or user_email\n\n if isinstance(enterprise_customer_user, models.EnterpriseCustomerUser):\n validated_data['enterprise_customer_user'] = enterprise_customer_user\n try:\n if is_active:\n enterprise_customer_user.enroll(course_run_id, course_mode, cohort=cohort)\n else:\n enterprise_customer_user.unenroll(course_run_id)\n except (CourseEnrollmentDowngradeError, CourseEnrollmentPermissionError, HttpClientError) as exc:\n validated_data['detail'] = str(exc)\n return validated_data\n\n if is_active:\n track_enrollment('enterprise-customer-enrollment-api', enterprise_customer_user.user_id, course_run_id)\n else:\n if is_active:\n enterprise_customer_user = enterprise_customer.enroll_user_pending_registration(\n user_email,\n course_mode,\n course_run_id,\n cohort=cohort\n )\n else:\n enterprise_customer.clear_pending_registration(user_email, course_run_id)\n\n if email_students:\n enterprise_customer.notify_enrolled_learners(\n self.context.get('request_user'),\n course_run_id,\n [enterprise_customer_user]\n )\n\n validated_data['detail'] = 'success'\n\n return validated_data"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nvalidates the lms_user_id and returns the EnterpriseCustomerUser object if it exists.", "response": "def validate_lms_user_id(self, value):\n \"\"\"\n Validates the lms_user_id, if is given, to see if there is an existing EnterpriseCustomerUser for it.\n \"\"\"\n enterprise_customer = self.context.get('enterprise_customer')\n\n try:\n # Ensure the given user is associated with the enterprise.\n return models.EnterpriseCustomerUser.objects.get(\n user_id=value,\n enterprise_customer=enterprise_customer\n )\n except models.EnterpriseCustomerUser.DoesNotExist:\n pass\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef validate_tpa_user_id(self, value):\n enterprise_customer = self.context.get('enterprise_customer')\n\n try:\n tpa_client = ThirdPartyAuthApiClient()\n username = tpa_client.get_username_from_remote_id(\n enterprise_customer.identity_provider, value\n )\n user = User.objects.get(username=username)\n return models.EnterpriseCustomerUser.objects.get(\n user_id=user.id,\n enterprise_customer=enterprise_customer\n )\n except (models.EnterpriseCustomerUser.DoesNotExist, User.DoesNotExist):\n pass\n\n return None", "response": "Validates the tpa_user_id and returns the EnterpriseCustomerUser object if it exists."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_user_email(self, value):\n enterprise_customer = self.context.get('enterprise_customer')\n\n try:\n user = User.objects.get(email=value)\n return models.EnterpriseCustomerUser.objects.get(\n user_id=user.id,\n enterprise_customer=enterprise_customer\n )\n except (models.EnterpriseCustomerUser.DoesNotExist, User.DoesNotExist):\n pass\n\n return value", "response": "Validates the user_email and returns the user_id if it exists."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef validate_course_run_id(self, value):\n enterprise_customer = self.context.get('enterprise_customer')\n\n if not enterprise_customer.catalog_contains_course(value):\n raise serializers.ValidationError(\n 'The course run id {course_run_id} is not in the catalog '\n 'for Enterprise Customer {enterprise_customer}'.format(\n course_run_id=value,\n enterprise_customer=enterprise_customer.name,\n )\n )\n\n return value", "response": "Validates that the course run id is part of the Enterprise Customer s catalog."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate that at least one of the user identifier fields has been passed in.", "response": "def validate(self, data): # pylint: disable=arguments-differ\n \"\"\"\n Validate that at least one of the user identifier fields has been passed in.\n \"\"\"\n lms_user_id = data.get('lms_user_id')\n tpa_user_id = data.get('tpa_user_id')\n user_email = data.get('user_email')\n if not lms_user_id and not tpa_user_id and not user_email:\n raise serializers.ValidationError(\n 'At least one of the following fields must be specified and map to an EnterpriseCustomerUser: '\n 'lms_user_id, tpa_user_id, user_email'\n )\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a response object containing pagination links in course catalog data and returns a response object containing pagination links in course catalog data and pagination links in course catalog data and response object containing pagination links in course catalog data and response object containing pagination links in course catalog data.", "response": "def get_paginated_response(data, request):\n \"\"\"\n Update pagination links in course catalog data and return DRF Response.\n\n Arguments:\n data (dict): Dictionary containing catalog courses.\n request (HttpRequest): Current request object.\n\n Returns:\n (Response): DRF response object containing pagination links.\n \"\"\"\n url = urlparse(request.build_absolute_uri())._replace(query=None).geturl()\n\n next_page = None\n previous_page = None\n\n if data['next']:\n next_page = \"{base_url}?{query_parameters}\".format(\n base_url=url,\n query_parameters=urlparse(data['next']).query,\n )\n next_page = next_page.rstrip('?')\n if data['previous']:\n previous_page = \"{base_url}?{query_parameters}\".format(\n base_url=url,\n query_parameters=urlparse(data['previous'] or \"\").query,\n )\n previous_page = previous_page.rstrip('?')\n\n return Response(OrderedDict([\n ('count', data['count']),\n ('next', next_page),\n ('previous', previous_page),\n ('results', data['results'])\n ]))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_switch(apps, schema_editor):\n Switch = apps.get_model('waffle', 'Switch')\n Switch.objects.update_or_create(name=ENTERPRISE_ROLE_BASED_ACCESS_CONTROL_SWITCH, defaults={'active': False})", "response": "Create the role_based_access_control switch if it does not exist."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete_switch(apps, schema_editor):\n Switch = apps.get_model('waffle', 'Switch')\n Switch.objects.filter(name=ENTERPRISE_ROLE_BASED_ACCESS_CONTROL_SWITCH).delete()", "response": "Delete the role_based_access_control switch."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_switch(apps, schema_editor):\n Switch = apps.get_model('waffle', 'Switch')\n Switch.objects.get_or_create(name='SAP_USE_ENTERPRISE_ENROLLMENT_PAGE', defaults={'active': False})", "response": "Create and activate the SAP_USE_ENTERPRISE_ENROLLMENT_PAGE switch if it does not already exist."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef transmit(self, payload, **kwargs):\n kwargs['app_label'] = 'sap_success_factors'\n kwargs['model_name'] = 'SapSuccessFactorsLearnerDataTransmissionAudit'\n kwargs['remote_user_id'] = 'sapsf_user_id'\n super(SapSuccessFactorsLearnerTransmitter, self).transmit(payload, **kwargs)", "response": "Send a completion status call to SAP SuccessFactors using the client."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef handle_transmission_error(self, learner_data, request_exception):\n try:\n sys_msg = request_exception.response.content\n except AttributeError:\n pass\n else:\n if 'user account is inactive' in sys_msg:\n ecu = EnterpriseCustomerUser.objects.get(\n enterprise_enrollments__id=learner_data.enterprise_course_enrollment_id)\n ecu.active = False\n ecu.save()\n LOGGER.warning(\n 'User %s with ID %s and email %s is a former employee of %s '\n 'and has been marked inactive in SAPSF. Now marking inactive internally.',\n ecu.username, ecu.user_id, ecu.user_email, ecu.enterprise_customer\n )\n return\n super(SapSuccessFactorsLearnerTransmitter, self).handle_transmission_error(learner_data, request_exception)", "response": "Handle the case where the user is a former employee of the current user and the user is marked as inactive."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\noverrides the default allow_request method to update the throttling rate for the current service user.", "response": "def allow_request(self, request, view):\n \"\"\"\n Modify throttling for service users.\n\n Updates throttling rate if the request is coming from the service user, and\n defaults to UserRateThrottle's configured setting otherwise.\n\n Updated throttling rate comes from `DEFAULT_THROTTLE_RATES` key in `REST_FRAMEWORK`\n setting. service user throttling is specified in `DEFAULT_THROTTLE_RATES` by `service_user` key\n\n Example Setting:\n ```\n REST_FRAMEWORK = {\n ...\n 'DEFAULT_THROTTLE_RATES': {\n ...\n 'service_user': '50/day'\n }\n }\n ```\n \"\"\"\n service_users = get_service_usernames()\n\n # User service user throttling rates for service user.\n if request.user.username in service_users:\n self.update_throttle_scope()\n\n return super(ServiceUserThrottle, self).allow_request(request, view)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef update_throttle_scope(self):\n self.scope = SERVICE_USER_SCOPE\n self.rate = self.get_rate()\n self.num_requests, self.duration = self.parse_rate(self.rate)", "response": "Update throttle scope so that service user throttle rates are applied."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the final price of the course mode.", "response": "def get_course_final_price(self, mode, currency='$', enterprise_catalog_uuid=None):\n \"\"\"\n Get course mode's SKU discounted price after applying any entitlement available for this user.\n\n Returns:\n str: Discounted price of the course mode.\n\n \"\"\"\n try:\n price_details = self.client.baskets.calculate.get(\n sku=[mode['sku']],\n username=self.user.username,\n catalog=enterprise_catalog_uuid,\n )\n except (SlumberBaseException, ConnectionError, Timeout) as exc:\n LOGGER.exception('Failed to get price details for sku %s due to: %s', mode['sku'], str(exc))\n price_details = {}\n price = price_details.get('total_incl_tax', mode['min_price'])\n if price != mode['min_price']:\n return format_price(price, currency)\n return mode['original_price']"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_course(self, course, enterprise_customer, enterprise_context):\n course['course_runs'] = self.update_course_runs(\n course_runs=course.get('course_runs') or [],\n enterprise_customer=enterprise_customer,\n enterprise_context=enterprise_context,\n )\n\n # Update marketing urls in course metadata to include enterprise related info (i.e. our global context).\n marketing_url = course.get('marketing_url')\n if marketing_url:\n query_parameters = dict(enterprise_context, **utils.get_enterprise_utm_context(enterprise_customer))\n course.update({'marketing_url': utils.update_query_parameters(marketing_url, query_parameters)})\n\n # Finally, add context to the course as a whole.\n course.update(enterprise_context)\n return course", "response": "Updates the course metadata of the given course and returns updated course."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_course_runs(self, course_runs, enterprise_customer, enterprise_context):\n updated_course_runs = []\n for course_run in course_runs:\n track_selection_url = utils.get_course_track_selection_url(\n course_run=course_run,\n query_parameters=dict(enterprise_context, **utils.get_enterprise_utm_context(enterprise_customer)),\n )\n\n enrollment_url = enterprise_customer.get_course_run_enrollment_url(course_run.get('key'))\n\n course_run.update({\n 'enrollment_url': enrollment_url,\n 'track_selection_url': track_selection_url,\n })\n\n # Update marketing urls in course metadata to include enterprise related info.\n marketing_url = course_run.get('marketing_url')\n if marketing_url:\n query_parameters = dict(enterprise_context, **utils.get_enterprise_utm_context(enterprise_customer))\n course_run.update({'marketing_url': utils.update_query_parameters(marketing_url, query_parameters)})\n\n # Add updated course run to the list.\n updated_course_runs.append(course_run)\n return updated_course_runs", "response": "Updates the course runs in the metadata and returns the updated course runs."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef export(self):\n # Fetch the consenting enrollment data, including the enterprise_customer_user.\n # Order by the course_id, to avoid fetching course API data more than we have to.\n enrollment_queryset = EnterpriseCourseEnrollment.objects.select_related(\n 'enterprise_customer_user'\n ).filter(\n enterprise_customer_user__enterprise_customer=self.enterprise_customer,\n enterprise_customer_user__active=True,\n ).order_by('course_id')\n\n # Fetch course details from the Course API, and cache between calls.\n course_details = None\n for enterprise_enrollment in enrollment_queryset:\n\n course_id = enterprise_enrollment.course_id\n\n # Fetch course details from Courses API\n # pylint: disable=unsubscriptable-object\n if course_details is None or course_details['course_id'] != course_id:\n if self.course_api is None:\n self.course_api = CourseApiClient()\n course_details = self.course_api.get_course_details(course_id)\n\n if course_details is None:\n # Course not found, so we have nothing to report.\n LOGGER.error(\"No course run details found for enrollment [%d]: [%s]\",\n enterprise_enrollment.pk, course_id)\n continue\n\n consent = DataSharingConsent.objects.proxied_get(\n username=enterprise_enrollment.enterprise_customer_user.username,\n course_id=enterprise_enrollment.course_id,\n enterprise_customer=enterprise_enrollment.enterprise_customer_user.enterprise_customer\n )\n\n if not consent.granted or enterprise_enrollment.audit_reporting_disabled:\n continue\n\n # For instructor-paced courses, let the certificate determine course completion\n if course_details.get('pacing') == 'instructor':\n completed_date, grade, is_passing = self._collect_certificate_data(enterprise_enrollment)\n\n # For self-paced courses, check the Grades API\n else:\n completed_date, grade, is_passing = self._collect_grades_data(enterprise_enrollment, course_details)\n\n records = self.get_learner_data_records(\n enterprise_enrollment=enterprise_enrollment,\n completed_date=completed_date,\n grade=grade,\n is_passing=is_passing,\n )\n if records:\n # There are some cases where we won't receive a record from the above\n # method; right now, that should only happen if we have an Enterprise-linked\n # user for the integrated channel, and transmission of that user's\n # data requires an upstream user identifier that we don't have (due to a\n # failure of SSO or similar). In such a case, `get_learner_data_record`\n # would return None, and we'd simply skip yielding it here.\n for record in records:\n yield record", "response": "Export the data sharing consent for the current user."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngenerates a learner data transmission audit with fields properly filled in.", "response": "def get_learner_data_records(self, enterprise_enrollment, completed_date=None, grade=None, is_passing=False):\n \"\"\"\n Generate a learner data transmission audit with fields properly filled in.\n \"\"\"\n # pylint: disable=invalid-name\n LearnerDataTransmissionAudit = apps.get_model('integrated_channel', 'LearnerDataTransmissionAudit')\n completed_timestamp = None\n course_completed = False\n if completed_date is not None:\n completed_timestamp = parse_datetime_to_epoch_millis(completed_date)\n course_completed = is_passing\n\n return [\n LearnerDataTransmissionAudit(\n enterprise_course_enrollment_id=enterprise_enrollment.id,\n course_id=enterprise_enrollment.course_id,\n course_completed=course_completed,\n completed_timestamp=completed_timestamp,\n grade=grade,\n )\n ]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _collect_certificate_data(self, enterprise_enrollment):\n\n if self.certificates_api is None:\n self.certificates_api = CertificatesApiClient(self.user)\n\n course_id = enterprise_enrollment.course_id\n username = enterprise_enrollment.enterprise_customer_user.user.username\n\n try:\n certificate = self.certificates_api.get_course_certificate(course_id, username)\n completed_date = certificate.get('created_date')\n if completed_date:\n completed_date = parse_datetime(completed_date)\n else:\n completed_date = timezone.now()\n\n # For consistency with _collect_grades_data, we only care about Pass/Fail grades. This could change.\n is_passing = certificate.get('is_passing')\n grade = self.grade_passing if is_passing else self.grade_failing\n\n except HttpNotFoundError:\n completed_date = None\n grade = self.grade_incomplete\n is_passing = False\n\n return completed_date, grade, is_passing", "response": "Collect the learner completion data from the course certificate."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncollect the learner completion data from the Grades API.", "response": "def _collect_grades_data(self, enterprise_enrollment, course_details):\n \"\"\"\n Collect the learner completion data from the Grades API.\n\n Used for self-paced courses.\n\n Args:\n enterprise_enrollment (EnterpriseCourseEnrollment): the enterprise enrollment record for which we need to\n collect completion/grade data\n course_details (dict): the course details for the course in the enterprise enrollment record.\n\n Returns:\n completed_date: Date the course was completed, this is None if course has not been completed.\n grade: Current grade in the course.\n is_passing: Boolean indicating if the grade is a passing grade or not.\n \"\"\"\n if self.grades_api is None:\n self.grades_api = GradesApiClient(self.user)\n\n course_id = enterprise_enrollment.course_id\n username = enterprise_enrollment.enterprise_customer_user.user.username\n\n try:\n grades_data = self.grades_api.get_course_grade(course_id, username)\n\n except HttpNotFoundError as error:\n # Grade not found, so we have nothing to report.\n if hasattr(error, 'content'):\n response_content = json.loads(error.content)\n if response_content.get('error_code', '') == 'user_not_enrolled':\n # This means the user has an enterprise enrollment record but is not enrolled in the course yet\n LOGGER.info(\n \"User [%s] not enrolled in course [%s], enterprise enrollment [%d]\",\n username,\n course_id,\n enterprise_enrollment.pk\n )\n return None, None, None\n\n LOGGER.error(\"No grades data found for [%d]: [%s], [%s]\", enterprise_enrollment.pk, course_id, username)\n return None, None, None\n\n # Prepare to process the course end date and pass/fail grade\n course_end_date = course_details.get('end')\n if course_end_date is not None:\n course_end_date = parse_datetime(course_end_date)\n now = timezone.now()\n is_passing = grades_data.get('passed')\n\n # We can consider a course complete if:\n # * the course's end date has passed\n if course_end_date is not None and course_end_date < now:\n completed_date = course_end_date\n grade = self.grade_passing if is_passing else self.grade_failing\n\n # * Or, the learner has a passing grade (as of now)\n elif is_passing:\n completed_date = now\n grade = self.grade_passing\n\n # Otherwise, the course is still in progress\n else:\n completed_date = None\n grade = self.grade_incomplete\n\n return completed_date, grade, is_passing"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget enterprise user id from user object.", "response": "def get_enterprise_user_id(self, obj):\n \"\"\"\n Get enterprise user id from user object.\n\n Arguments:\n obj (User): Django User object\n\n Returns:\n (int): Primary Key identifier for enterprise user object.\n \"\"\"\n # An enterprise learner can not belong to multiple enterprise customer at the same time\n # but if such scenario occurs we will pick the first.\n enterprise_learner = EnterpriseCustomerUser.objects.filter(user_id=obj.id).first()\n\n return enterprise_learner and enterprise_learner.id"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_enterprise_sso_uid(self, obj):\n # An enterprise learner can not belong to multiple enterprise customer at the same time\n # but if such scenario occurs we will pick the first.\n enterprise_learner = EnterpriseCustomerUser.objects.filter(user_id=obj.id).first()\n\n return enterprise_learner and enterprise_learner.get_remote_id()", "response": "Get enterprise SSO UID."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_course_duration(self, obj):\n duration = obj.end - obj.start if obj.start and obj.end else None\n if duration:\n return strfdelta(duration, '{W} weeks {D} days.')\n return ''", "response": "Get course s duration as a timedelta."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef transmit(self, payload, **kwargs):\n items_to_create, items_to_update, items_to_delete, transmission_map = self._partition_items(payload)\n self._prepare_items_for_delete(items_to_delete)\n prepared_items = {}\n prepared_items.update(items_to_create)\n prepared_items.update(items_to_update)\n prepared_items.update(items_to_delete)\n\n skip_metadata_transmission = False\n\n for chunk in chunks(prepared_items, self.enterprise_configuration.transmission_chunk_size):\n chunked_items = list(chunk.values())\n if skip_metadata_transmission:\n # Remove the failed items from the create/update/delete dictionaries,\n # so ContentMetadataItemTransmission objects are not synchronized for\n # these items below.\n self._remove_failed_items(chunked_items, items_to_create, items_to_update, items_to_delete)\n else:\n try:\n self.client.update_content_metadata(self._serialize_items(chunked_items))\n except ClientError as exc:\n LOGGER.error(\n 'Failed to update [%s] content metadata items for integrated channel [%s] [%s]',\n len(chunked_items),\n self.enterprise_configuration.enterprise_customer.name,\n self.enterprise_configuration.channel_code,\n )\n LOGGER.error(exc)\n\n # Remove the failed items from the create/update/delete dictionaries,\n # so ContentMetadataItemTransmission objects are not synchronized for\n # these items below.\n self._remove_failed_items(chunked_items, items_to_create, items_to_update, items_to_delete)\n\n # SAP servers throttle incoming traffic, If a request fails than the subsequent would fail too,\n # So, no need to keep trying and failing. We should stop here and retry later.\n skip_metadata_transmission = True\n\n self._create_transmissions(items_to_create)\n self._update_transmissions(items_to_update, transmission_map)\n self._delete_transmissions(items_to_delete.keys())", "response": "Transmit the items to the integrated channel."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nremove failed items from the items_to_create items_to_update items_to_delete.", "response": "def _remove_failed_items(self, failed_items, items_to_create, items_to_update, items_to_delete):\n \"\"\"\n Remove content metadata items from the `items_to_create`, `items_to_update`, `items_to_delete` dicts.\n\n Arguments:\n failed_items (list): Failed Items to be removed.\n items_to_create (dict): dict containing the items created successfully.\n items_to_update (dict): dict containing the items updated successfully.\n items_to_delete (dict): dict containing the items deleted successfully.\n \"\"\"\n for item in failed_items:\n content_metadata_id = item['courseID']\n items_to_create.pop(content_metadata_id, None)\n items_to_update.pop(content_metadata_id, None)\n items_to_delete.pop(content_metadata_id, None)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_arguments(*args, **options): # pylint: disable=unused-argument\n days = options.get('days', 1)\n enterprise_customer_uuid = options.get('enterprise_customer_uuid')\n enterprise_customer = None\n\n if enterprise_customer_uuid:\n try:\n # pylint: disable=no-member\n enterprise_customer = EnterpriseCustomer.objects.get(uuid=enterprise_customer_uuid)\n except EnterpriseCustomer.DoesNotExist:\n raise CommandError('Enterprise customer with uuid \"{enterprise_customer_uuid}\" does not exist.'.format(\n enterprise_customer_uuid=enterprise_customer_uuid\n ))\n\n return days, enterprise_customer", "response": "Parse and validate arguments for send_course_enrollments command."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend xAPI statements to the configured LRSs.", "response": "def handle(self, *args, **options):\n \"\"\"\n Send xAPI statements.\n \"\"\"\n if not CourseEnrollment:\n raise NotConnectedToOpenEdX(\"This package must be installed in an OpenEdX environment.\")\n\n days, enterprise_customer = self.parse_arguments(*args, **options)\n\n if enterprise_customer:\n try:\n lrs_configuration = XAPILRSConfiguration.objects.get(\n active=True,\n enterprise_customer=enterprise_customer\n )\n except XAPILRSConfiguration.DoesNotExist:\n raise CommandError('No xAPI Configuration found for \"{enterprise_customer}\"'.format(\n enterprise_customer=enterprise_customer.name\n ))\n\n # Send xAPI analytics data to the configured LRS\n self.send_xapi_statements(lrs_configuration, days)\n else:\n for lrs_configuration in XAPILRSConfiguration.objects.filter(active=True):\n self.send_xapi_statements(lrs_configuration, days)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nsending xAPI statements to the LRS.", "response": "def send_xapi_statements(self, lrs_configuration, days):\n \"\"\"\n Send xAPI analytics data of the enterprise learners to the given LRS.\n\n Arguments:\n lrs_configuration (XAPILRSConfiguration): Configuration object containing LRS configurations\n of the LRS where to send xAPI learner analytics.\n days (int): Include course enrollment of this number of days.\n \"\"\"\n for course_enrollment in self.get_course_enrollments(lrs_configuration.enterprise_customer, days):\n try:\n send_course_enrollment_statement(lrs_configuration, course_enrollment)\n except ClientError:\n LOGGER.exception(\n 'Client error while sending course enrollment to xAPI for'\n ' enterprise customer {enterprise_customer}.'.format(\n enterprise_customer=lrs_configuration.enterprise_customer.name\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a queryset of all the course enrollments for all learners of given enterprise customer.", "response": "def get_course_enrollments(self, enterprise_customer, days):\n \"\"\"\n Get course enrollments for all the learners of given enterprise customer.\n\n Arguments:\n enterprise_customer (EnterpriseCustomer): Include Course enrollments for learners\n of this enterprise customer.\n days (int): Include course enrollment of this number of days.\n\n Returns:\n (list): A list of CourseEnrollment objects.\n \"\"\"\n return CourseEnrollment.objects.filter(\n created__gt=datetime.datetime.now() - datetime.timedelta(days=days)\n ).filter(\n user_id__in=enterprise_customer.enterprise_customer_users.values_list('user_id', flat=True)\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef course_modal(context, course=None):\n if course:\n context.update({\n 'course_image_uri': course.get('course_image_uri', ''),\n 'course_title': course.get('course_title', ''),\n 'course_level_type': course.get('course_level_type', ''),\n 'course_short_description': course.get('course_short_description', ''),\n 'course_effort': course.get('course_effort', ''),\n 'course_full_description': course.get('course_full_description', ''),\n 'expected_learning_items': course.get('expected_learning_items', []),\n 'staff': course.get('staff', []),\n 'premium_modes': course.get('premium_modes', []),\n })\n return context", "response": "This is a template tag that returns course information to display in a modal."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef link_to_modal(link_text, index, autoescape=True): # pylint: disable=unused-argument\n link = (\n '{link_text}'\n ).format(\n index=index,\n link_text=link_text,\n )\n return mark_safe(link)", "response": "Returns a Django template filter that returns an anchor with attributes useful for course modal selection."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef populate_data_sharing_consent(apps, schema_editor):\n DataSharingConsent = apps.get_model('consent', 'DataSharingConsent')\n EnterpriseCourseEnrollment = apps.get_model('enterprise', 'EnterpriseCourseEnrollment')\n User = apps.get_model('auth', 'User')\n for enrollment in EnterpriseCourseEnrollment.objects.all():\n user = User.objects.get(pk=enrollment.enterprise_customer_user.user_id)\n data_sharing_consent, __ = DataSharingConsent.objects.get_or_create(\n username=user.username,\n enterprise_customer=enrollment.enterprise_customer_user.enterprise_customer,\n course_id=enrollment.course_id,\n )\n if enrollment.consent_granted is not None:\n data_sharing_consent.granted = enrollment.consent_granted\n else:\n # Check UDSCA instead.\n consent_state = enrollment.enterprise_customer_user.data_sharing_consent.first()\n if consent_state is not None:\n data_sharing_consent.granted = consent_state.state in ['enabled', 'external']\n else:\n data_sharing_consent.granted = False\n data_sharing_consent.save()", "response": "Populates the DataSharingConsent model with the enterprise application s consent data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_course_completion(self, user_id, payload): # pylint: disable=unused-argument\n return self._post(\n urljoin(\n self.enterprise_configuration.degreed_base_url,\n self.global_degreed_config.completion_status_api_path\n ),\n payload,\n self.COMPLETION_PROVIDER_SCOPE\n )", "response": "Send a completion status payload to the Degreed Completion Status endpoint."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef delete_course_completion(self, user_id, payload): # pylint: disable=unused-argument\n return self._delete(\n urljoin(\n self.enterprise_configuration.degreed_base_url,\n self.global_degreed_config.completion_status_api_path\n ),\n payload,\n self.COMPLETION_PROVIDER_SCOPE\n )", "response": "Delete a course completion status"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _sync_content_metadata(self, serialized_data, http_method):\n try:\n status_code, response_body = getattr(self, '_' + http_method)(\n urljoin(self.enterprise_configuration.degreed_base_url, self.global_degreed_config.course_api_path),\n serialized_data,\n self.CONTENT_PROVIDER_SCOPE\n )\n except requests.exceptions.RequestException as exc:\n raise ClientError(\n 'DegreedAPIClient request failed: {error} {message}'.format(\n error=exc.__class__.__name__,\n message=str(exc)\n )\n )\n\n if status_code >= 400:\n raise ClientError(\n 'DegreedAPIClient request failed with status {status_code}: {message}'.format(\n status_code=status_code,\n message=response_body\n )\n )", "response": "Synchronize content metadata using the Degreed course content API."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a POST request to a Degreed endpoint.", "response": "def _post(self, url, data, scope):\n \"\"\"\n Make a POST request using the session object to a Degreed endpoint.\n\n Args:\n url (str): The url to send a POST request to.\n data (str): The json encoded payload to POST.\n scope (str): Must be one of the scopes Degreed expects:\n - `CONTENT_PROVIDER_SCOPE`\n - `COMPLETION_PROVIDER_SCOPE`\n \"\"\"\n self._create_session(scope)\n response = self.session.post(url, data=data)\n return response.status_code, response.text"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _delete(self, url, data, scope):\n self._create_session(scope)\n response = self.session.delete(url, data=data)\n return response.status_code, response.text", "response": "Make a DELETE request to a Degreed endpoint."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _create_session(self, scope):\n now = datetime.datetime.utcnow()\n if self.session is None or self.expires_at is None or now >= self.expires_at:\n # Create a new session with a valid token\n if self.session:\n self.session.close()\n oauth_access_token, expires_at = self._get_oauth_access_token(\n self.enterprise_configuration.key,\n self.enterprise_configuration.secret,\n self.enterprise_configuration.degreed_user_id,\n self.enterprise_configuration.degreed_user_password,\n scope\n )\n session = requests.Session()\n session.timeout = self.SESSION_TIMEOUT\n session.headers['Authorization'] = 'Bearer {}'.format(oauth_access_token)\n session.headers['content-type'] = 'application/json'\n self.session = session\n self.expires_at = expires_at", "response": "Create a new session object for use in connecting with Degreed\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nretrieving an OAuth 2. 0 access token using the client credentials grant.", "response": "def _get_oauth_access_token(self, client_id, client_secret, user_id, user_password, scope):\n \"\"\" Retrieves OAuth 2.0 access token using the client credentials grant.\n\n Args:\n client_id (str): API client ID\n client_secret (str): API client secret\n user_id (str): Degreed company ID\n user_password (str): Degreed user password\n scope (str): Must be one of the scopes Degreed expects:\n - `CONTENT_PROVIDER_SCOPE`\n - `COMPLETION_PROVIDER_SCOPE`\n\n Returns:\n tuple: Tuple containing access token string and expiration datetime.\n Raises:\n HTTPError: If we received a failure response code from Degreed.\n RequestException: If an unexpected response format was received that we could not parse.\n \"\"\"\n response = requests.post(\n urljoin(self.enterprise_configuration.degreed_base_url, self.global_degreed_config.oauth_api_path),\n data={\n 'grant_type': 'password',\n 'username': user_id,\n 'password': user_password,\n 'scope': scope,\n },\n auth=(client_id, client_secret),\n headers={'Content-Type': 'application/x-www-form-urlencoded'}\n )\n\n response.raise_for_status()\n data = response.json()\n try:\n expires_at = data['expires_in'] + int(time.time())\n return data['access_token'], datetime.datetime.utcfromtimestamp(expires_at)\n except KeyError:\n raise requests.RequestException(response=response)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nensures that the wrapped API client s response brings us valid data.", "response": "def ensure_data_exists(self, request, data, error_message=None):\n \"\"\"\n Ensure that the wrapped API client's response brings us valid data. If not, raise an error and log it.\n \"\"\"\n if not data:\n error_message = (\n error_message or \"Unable to fetch API response from endpoint '{}'.\".format(request.get_full_path())\n )\n LOGGER.error(error_message)\n raise NotFound(error_message)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn whether or not the specified content is available to the EnterpriseCustomer.", "response": "def contains_content_items(self, request, pk, course_run_ids, program_uuids):\n \"\"\"\n Return whether or not the specified content is available to the EnterpriseCustomer.\n\n Multiple course_run_ids and/or program_uuids query parameters can be sent to this view to check\n for their existence in the EnterpriseCustomerCatalogs associated with this EnterpriseCustomer.\n At least one course run key or program UUID value must be included in the request.\n \"\"\"\n enterprise_customer = self.get_object()\n\n # Maintain plus characters in course key.\n course_run_ids = [unquote(quote_plus(course_run_id)) for course_run_id in course_run_ids]\n\n contains_content_items = False\n for catalog in enterprise_customer.enterprise_customer_catalogs.all():\n contains_course_runs = not course_run_ids or catalog.contains_courses(course_run_ids)\n contains_program_uuids = not program_uuids or catalog.contains_programs(program_uuids)\n if contains_course_runs and contains_program_uuids:\n contains_content_items = True\n break\n\n return Response({'contains_content_items': contains_content_items})"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef courses(self, request, pk=None): # pylint: disable=invalid-name,unused-argument\n enterprise_customer = self.get_object()\n self.check_object_permissions(request, enterprise_customer)\n self.ensure_data_exists(\n request,\n enterprise_customer.catalog,\n error_message=\"No catalog is associated with Enterprise {enterprise_name} from endpoint '{path}'.\".format(\n enterprise_name=enterprise_customer.name,\n path=request.get_full_path()\n )\n )\n\n # We have handled potential error cases and are now ready to call out to the Catalog API.\n catalog_api = CourseCatalogApiClient(request.user, enterprise_customer.site)\n courses = catalog_api.get_paginated_catalog_courses(enterprise_customer.catalog, request.GET)\n\n # An empty response means that there was a problem fetching data from Catalog API, since\n # a Catalog with no courses has a non empty response indicating that there are no courses.\n self.ensure_data_exists(\n request,\n courses,\n error_message=(\n \"Unable to fetch API response for catalog courses for \"\n \"Enterprise {enterprise_name} from endpoint '{path}'.\".format(\n enterprise_name=enterprise_customer.name,\n path=request.get_full_path()\n )\n )\n )\n\n serializer = serializers.EnterpriseCatalogCoursesReadOnlySerializer(courses)\n\n # Add enterprise related context for the courses.\n serializer.update_enterprise_courses(enterprise_customer, catalog_id=enterprise_customer.catalog)\n return get_paginated_response(serializer.data, request)", "response": "Returns a list of courses that are linked to this enterprise."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a course enrollment for an EnterpriseCustomerUser.", "response": "def course_enrollments(self, request, pk):\n \"\"\"\n Creates a course enrollment for an EnterpriseCustomerUser.\n \"\"\"\n enterprise_customer = self.get_object()\n serializer = serializers.EnterpriseCustomerCourseEnrollmentsSerializer(\n data=request.data,\n many=True,\n context={\n 'enterprise_customer': enterprise_customer,\n 'request_user': request.user,\n }\n )\n if serializer.is_valid():\n serializer.save()\n return Response(serializer.data, status=HTTP_200_OK)\n\n return Response(serializer.errors, status=HTTP_400_BAD_REQUEST)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the list of enterprise customers the user has a specified group permission access to.", "response": "def with_access_to(self, request, *args, **kwargs): # pylint: disable=invalid-name,unused-argument\n \"\"\"\n Returns the list of enterprise customers the user has a specified group permission access to.\n \"\"\"\n self.queryset = self.queryset.order_by('name')\n enterprise_id = self.request.query_params.get('enterprise_id', None)\n enterprise_slug = self.request.query_params.get('enterprise_slug', None)\n enterprise_name = self.request.query_params.get('search', None)\n\n if enterprise_id is not None:\n self.queryset = self.queryset.filter(uuid=enterprise_id)\n elif enterprise_slug is not None:\n self.queryset = self.queryset.filter(slug=enterprise_slug)\n elif enterprise_name is not None:\n self.queryset = self.queryset.filter(name__icontains=enterprise_name)\n return self.list(request, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef entitlements(self, request, pk=None): # pylint: disable=invalid-name,unused-argument\n enterprise_customer_user = self.get_object()\n instance = {\"entitlements\": enterprise_customer_user.entitlements}\n serializer = serializers.EnterpriseCustomerUserEntitlementSerializer(instance, context={'request': request})\n return Response(serializer.data)", "response": "Returns the list of entitlements available to this learner."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns whether or not the EnterpriseCustomerCatalog contains the specified content items.", "response": "def contains_content_items(self, request, pk, course_run_ids, program_uuids):\n \"\"\"\n Return whether or not the EnterpriseCustomerCatalog contains the specified content.\n\n Multiple course_run_ids and/or program_uuids query parameters can be sent to this view to check\n for their existence in the EnterpriseCustomerCatalog. At least one course run key\n or program UUID value must be included in the request.\n \"\"\"\n enterprise_customer_catalog = self.get_object()\n\n # Maintain plus characters in course key.\n course_run_ids = [unquote(quote_plus(course_run_id)) for course_run_id in course_run_ids]\n\n contains_content_items = True\n if course_run_ids:\n contains_content_items = enterprise_customer_catalog.contains_courses(course_run_ids)\n if program_uuids:\n contains_content_items = (\n contains_content_items and\n enterprise_customer_catalog.contains_programs(program_uuids)\n )\n\n return Response({'contains_content_items': contains_content_items})"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the course metadata for the specified course.", "response": "def course_detail(self, request, pk, course_key): # pylint: disable=invalid-name,unused-argument\n \"\"\"\n Return the metadata for the specified course.\n\n The course needs to be included in the specified EnterpriseCustomerCatalog\n in order for metadata to be returned from this endpoint.\n \"\"\"\n enterprise_customer_catalog = self.get_object()\n course = enterprise_customer_catalog.get_course(course_key)\n if not course:\n raise Http404\n\n context = self.get_serializer_context()\n context['enterprise_customer_catalog'] = enterprise_customer_catalog\n serializer = serializers.CourseDetailSerializer(course, context=context)\n return Response(serializer.data)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef course_run_detail(self, request, pk, course_id): # pylint: disable=invalid-name,unused-argument\n enterprise_customer_catalog = self.get_object()\n course_run = enterprise_customer_catalog.get_course_run(course_id)\n if not course_run:\n raise Http404\n\n context = self.get_serializer_context()\n context['enterprise_customer_catalog'] = enterprise_customer_catalog\n serializer = serializers.CourseRunDetailSerializer(course_run, context=context)\n return Response(serializer.data)", "response": "Return the course run metadata for the specified course run."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the metadata for the specified program.", "response": "def program_detail(self, request, pk, program_uuid): # pylint: disable=invalid-name,unused-argument\n \"\"\"\n Return the metadata for the specified program.\n\n The program needs to be included in the specified EnterpriseCustomerCatalog\n in order for metadata to be returned from this endpoint.\n \"\"\"\n enterprise_customer_catalog = self.get_object()\n program = enterprise_customer_catalog.get_program(program_uuid)\n if not program:\n raise Http404\n\n context = self.get_serializer_context()\n context['enterprise_customer_catalog'] = enterprise_customer_catalog\n serializer = serializers.ProgramDetailSerializer(program, context=context)\n return Response(serializer.data)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef list(self, request):\n catalog_api = CourseCatalogApiClient(request.user)\n catalogs = catalog_api.get_paginated_catalogs(request.GET)\n self.ensure_data_exists(request, catalogs)\n serializer = serializers.ResponsePaginationSerializer(catalogs)\n return get_paginated_response(serializer.data, request)", "response": "This view returns a list of all course catalogs."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef retrieve(self, request, pk=None): # pylint: disable=invalid-name\n catalog_api = CourseCatalogApiClient(request.user)\n catalog = catalog_api.get_catalog(pk)\n self.ensure_data_exists(\n request,\n catalog,\n error_message=(\n \"Unable to fetch API response for given catalog from endpoint '/catalog/{pk}/'. \"\n \"The resource you are looking for does not exist.\".format(pk=pk)\n )\n )\n serializer = self.serializer_class(catalog)\n return Response(serializer.data)", "response": "This method is used to retrieve the details of a given course catalog."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a list of courses contained within this catalog.", "response": "def courses(self, request, enterprise_customer, pk=None): # pylint: disable=invalid-name\n \"\"\"\n Retrieve the list of courses contained within this catalog.\n\n Only courses with active course runs are returned. A course run is considered active if it is currently\n open for enrollment, or will open in the future.\n \"\"\"\n catalog_api = CourseCatalogApiClient(request.user, enterprise_customer.site)\n courses = catalog_api.get_paginated_catalog_courses(pk, request.GET)\n\n # If the API returned an empty response, that means pagination has ended.\n # An empty response can also mean that there was a problem fetching data from catalog API.\n self.ensure_data_exists(\n request,\n courses,\n error_message=(\n \"Unable to fetch API response for catalog courses from endpoint '{endpoint}'. \"\n \"The resource you are looking for does not exist.\".format(endpoint=request.get_full_path())\n )\n )\n serializer = serializers.EnterpriseCatalogCoursesReadOnlySerializer(courses)\n\n # Add enterprise related context for the courses.\n serializer.update_enterprise_courses(enterprise_customer, catalog_id=pk)\n return get_paginated_response(serializer.data, request)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the required query parameters from the request.", "response": "def get_required_query_params(self, request):\n \"\"\"\n Gets ``email``, ``enterprise_name``, and ``number_of_codes``,\n which are the relevant parameters for this API endpoint.\n\n :param request: The request to this endpoint.\n :return: The ``email``, ``enterprise_name``, and ``number_of_codes`` from the request.\n \"\"\"\n email = get_request_value(request, self.REQUIRED_PARAM_EMAIL, '')\n enterprise_name = get_request_value(request, self.REQUIRED_PARAM_ENTERPRISE_NAME, '')\n number_of_codes = get_request_value(request, self.OPTIONAL_PARAM_NUMBER_OF_CODES, '')\n if not (email and enterprise_name):\n raise CodesAPIRequestError(\n self.get_missing_params_message([\n (self.REQUIRED_PARAM_EMAIL, bool(email)),\n (self.REQUIRED_PARAM_ENTERPRISE_NAME, bool(enterprise_name)),\n ])\n )\n return email, enterprise_name, number_of_codes"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_missing_params_message(self, parameter_state):\n params = ', '.join(name for name, present in parameter_state if not present)\n return self.MISSING_REQUIRED_PARAMS_MSG.format(params)", "response": "Return a user - friendly message indicating a missing parameter for the API endpoint."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef post(self, request):\n try:\n email, enterprise_name, number_of_codes = self.get_required_query_params(request)\n except CodesAPIRequestError as invalid_request:\n return Response({'error': str(invalid_request)}, status=HTTP_400_BAD_REQUEST)\n\n subject_line = _('Code Management - Request for Codes by {token_enterprise_name}').format(\n token_enterprise_name=enterprise_name\n )\n msg_with_codes = _('{token_email} from {token_enterprise_name} has requested {token_number_codes} additional '\n 'codes. Please reach out to them.').format(\n token_email=email,\n token_enterprise_name=enterprise_name,\n token_number_codes=number_of_codes)\n msg_without_codes = _('{token_email} from {token_enterprise_name} has requested additional codes.'\n ' Please reach out to them.').format(\n token_email=email,\n token_enterprise_name=enterprise_name)\n app_config = apps.get_app_config(\"enterprise\")\n from_email_address = app_config.customer_success_email\n cs_email = app_config.customer_success_email\n data = {\n self.REQUIRED_PARAM_EMAIL: email,\n self.REQUIRED_PARAM_ENTERPRISE_NAME: enterprise_name,\n self.OPTIONAL_PARAM_NUMBER_OF_CODES: number_of_codes,\n }\n try:\n mail.send_mail(\n subject_line,\n msg_with_codes if number_of_codes else msg_without_codes,\n from_email_address,\n [cs_email],\n fail_silently=False\n )\n return Response(data, status=HTTP_200_OK)\n except SMTPException:\n error_message = _(\n '[Enterprise API] Failure in sending e-mail to {token_cs_email} for {token_email}'\n ' from {token_enterprise_name}'\n ).format(\n token_cs_email=cs_email,\n token_email=email,\n token_enterprise_name=enterprise_name\n )\n LOGGER.error(error_message)\n return Response(\n {'error': str('Request codes email could not be sent')},\n status=HTTP_500_INTERNAL_SERVER_ERROR\n )", "response": "This is the HTTP POST endpoint for the code management API."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntransform the title of the content item.", "response": "def transform_title(self, content_metadata_item):\n \"\"\"\n Return the title of the content item.\n \"\"\"\n title_with_locales = []\n\n for locale in self.enterprise_configuration.get_locales():\n title_with_locales.append({\n 'locale': locale,\n 'value': content_metadata_item.get('title', '')\n })\n\n return title_with_locales"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef transform_description(self, content_metadata_item):\n description_with_locales = []\n\n for locale in self.enterprise_configuration.get_locales():\n description_with_locales.append({\n 'locale': locale,\n 'value': (\n content_metadata_item.get('full_description') or\n content_metadata_item.get('short_description') or\n content_metadata_item.get('title', '')\n )\n })\n\n return description_with_locales", "response": "Transform the description of the content item."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transform_image(self, content_metadata_item):\n image_url = ''\n if content_metadata_item['content_type'] in ['course', 'program']:\n image_url = content_metadata_item.get('card_image_url')\n elif content_metadata_item['content_type'] == 'courserun':\n image_url = content_metadata_item.get('image_url')\n\n return image_url", "response": "Transform the image of the content item."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef transform_launch_points(self, content_metadata_item):\n return [{\n 'providerID': self.enterprise_configuration.provider_id,\n 'launchURL': content_metadata_item['enrollment_url'],\n 'contentTitle': content_metadata_item['title'],\n 'contentID': self.get_content_id(content_metadata_item),\n 'launchType': 3, # This tells SAPSF to launch the course in a new browser window.\n 'mobileEnabled': True, # Always return True per ENT-1401\n 'mobileLaunchURL': content_metadata_item['enrollment_url'],\n }]", "response": "Transform the launch points of a content item into a list of dictionaries."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef transform_courserun_title(self, content_metadata_item):\n title = content_metadata_item.get('title') or ''\n course_run_start = content_metadata_item.get('start')\n\n if course_run_start:\n if course_available_for_enrollment(content_metadata_item):\n title += ' ({starts}: {:%B %Y})'.format(\n parse_lms_api_datetime(course_run_start),\n starts=_('Starts')\n )\n else:\n title += ' ({:%B %Y} - {enrollment_closed})'.format(\n parse_lms_api_datetime(course_run_start),\n enrollment_closed=_('Enrollment Closed')\n )\n\n title_with_locales = []\n content_metadata_language_code = transform_language_code(content_metadata_item.get('content_language', ''))\n for locale in self.enterprise_configuration.get_locales(default_locale=content_metadata_language_code):\n title_with_locales.append({\n 'locale': locale,\n 'value': title\n })\n\n return title_with_locales", "response": "Transform the courserun title into a human readable title."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntransform the courserun description into a list of dictionaries.", "response": "def transform_courserun_description(self, content_metadata_item):\n \"\"\"\n Return the description of the courserun content item.\n \"\"\"\n description_with_locales = []\n content_metadata_language_code = transform_language_code(content_metadata_item.get('content_language', ''))\n for locale in self.enterprise_configuration.get_locales(default_locale=content_metadata_language_code):\n description_with_locales.append({\n 'locale': locale,\n 'value': (\n content_metadata_item['full_description'] or\n content_metadata_item['short_description'] or\n content_metadata_item['title'] or\n ''\n )\n })\n\n return description_with_locales"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef transform_courserun_schedule(self, content_metadata_item):\n start = content_metadata_item.get('start') or UNIX_MIN_DATE_STRING\n end = content_metadata_item.get('end') or UNIX_MAX_DATE_STRING\n return [{\n 'startDate': parse_datetime_to_epoch_millis(start),\n 'endDate': parse_datetime_to_epoch_millis(end),\n 'active': current_time_is_in_interval(start, end)\n }]", "response": "Transform the courseun content item into a schedule."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_content_id(self, content_metadata_item):\n content_id = content_metadata_item.get('key', '')\n if content_metadata_item['content_type'] == 'program':\n content_id = content_metadata_item.get('uuid', '')\n return content_id", "response": "Return the id for the given content_metadata_item"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert an ISO - 8601 datetime string to a Unix epoch timestamp in some magnitude.", "response": "def parse_datetime_to_epoch(datestamp, magnitude=1.0):\n \"\"\"\n Convert an ISO-8601 datetime string to a Unix epoch timestamp in some magnitude.\n\n By default, returns seconds.\n \"\"\"\n parsed_datetime = parse_lms_api_datetime(datestamp)\n time_since_epoch = parsed_datetime - UNIX_EPOCH\n return int(time_since_epoch.total_seconds() * magnitude)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef current_time_is_in_interval(start, end):\n interval_start = parse_lms_api_datetime(start or UNIX_MIN_DATE_STRING)\n interval_end = parse_lms_api_datetime(end or UNIX_MAX_DATE_STRING)\n return interval_start <= timezone.now() <= interval_end", "response": "Determines whether the current time is in the interval [ start end )."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef chunks(dictionary, chunk_size):\n iterable = iter(dictionary)\n for __ in range(0, len(dictionary), chunk_size):\n yield {key: dictionary[key] for key in islice(iterable, chunk_size)}", "response": "Yield successive n - sized chunks from dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a datetime.timedelta object or a regular number to a custom-formatted string. This function works like the strftime() method works for datetime.datetime objects. The fmt argument allows custom formatting to be specified. Fields can include seconds, minutes, hours, days, and weeks. Each field is optional. Arguments: tdelta (datetime.timedelta, int): time delta object containing the duration or an integer to go with the input_type. fmt (str): Expected format of the time delta. place holders can only be one of the following. 1. D to extract days from time delta 2. H to extract hours from time delta 3. M to extract months from time delta 4. S to extract seconds from timedelta input_type (str): The input_type argument allows tdelta to be a regular number instead of the default, which is a datetime.timedelta object. Valid input_type strings: 1. 's', 'seconds', 2. 'm', 'minutes', 3. 'h', 'hours', 4. 'd', 'days', 5. 'w', 'weeks' Returns: (str): timedelta object interpolated into a string following the given format. Examples: '{D:02}d {H:02}h {M:02}m {S:02}s' --> '05d 08h 04m 02s' (default) '{W}w {D}d {H}:{M:02}:{S:02}' --> '4w 5d 8:04:02' '{D:2}d {H:2}:{M:02}:{S:02}' --> ' 5d 8:04:02' '{H}h {S}s' --> '72h 800s'", "response": "def strfdelta(tdelta, fmt='{D:02}d {H:02}h {M:02}m {S:02}s', input_type='timedelta'):\n \"\"\"\n Convert a datetime.timedelta object or a regular number to a custom-formatted string.\n\n This function works like the strftime() method works for datetime.datetime\n objects.\n\n The fmt argument allows custom formatting to be specified. Fields can\n include seconds, minutes, hours, days, and weeks. Each field is optional.\n\n Arguments:\n tdelta (datetime.timedelta, int): time delta object containing the duration or an integer\n to go with the input_type.\n fmt (str): Expected format of the time delta. place holders can only be one of the following.\n 1. D to extract days from time delta\n 2. H to extract hours from time delta\n 3. M to extract months from time delta\n 4. S to extract seconds from timedelta\n input_type (str): The input_type argument allows tdelta to be a regular number instead of the\n default, which is a datetime.timedelta object.\n Valid input_type strings:\n 1. 's', 'seconds',\n 2. 'm', 'minutes',\n 3. 'h', 'hours',\n 4. 'd', 'days',\n 5. 'w', 'weeks'\n Returns:\n (str): timedelta object interpolated into a string following the given format.\n\n Examples:\n '{D:02}d {H:02}h {M:02}m {S:02}s' --> '05d 08h 04m 02s' (default)\n '{W}w {D}d {H}:{M:02}:{S:02}' --> '4w 5d 8:04:02'\n '{D:2}d {H:2}:{M:02}:{S:02}' --> ' 5d 8:04:02'\n '{H}h {S}s' --> '72h 800s'\n \"\"\"\n # Convert tdelta to integer seconds.\n if input_type == 'timedelta':\n remainder = int(tdelta.total_seconds())\n elif input_type in ['s', 'seconds']:\n remainder = int(tdelta)\n elif input_type in ['m', 'minutes']:\n remainder = int(tdelta) * 60\n elif input_type in ['h', 'hours']:\n remainder = int(tdelta) * 3600\n elif input_type in ['d', 'days']:\n remainder = int(tdelta) * 86400\n elif input_type in ['w', 'weeks']:\n remainder = int(tdelta) * 604800\n else:\n raise ValueError(\n 'input_type is not valid. Valid input_type strings are: \"timedelta\", \"s\", \"m\", \"h\", \"d\", \"w\"'\n )\n\n f = Formatter()\n desired_fields = [field_tuple[1] for field_tuple in f.parse(fmt)]\n possible_fields = ('W', 'D', 'H', 'M', 'S')\n constants = {'W': 604800, 'D': 86400, 'H': 3600, 'M': 60, 'S': 1}\n values = {}\n\n for field in possible_fields:\n if field in desired_fields and field in constants:\n values[field], remainder = divmod(remainder, constants[field])\n\n return f.format(fmt, **values)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the transformed version of the course description.", "response": "def transform_description(self, content_metadata_item):\n \"\"\"\n Return the transformed version of the course description.\n\n We choose one value out of the course's full description, short description, and title\n depending on availability and length limits.\n \"\"\"\n full_description = content_metadata_item.get('full_description') or ''\n if 0 < len(full_description) <= self.LONG_STRING_LIMIT: # pylint: disable=len-as-condition\n return full_description\n return content_metadata_item.get('short_description') or content_metadata_item.get('title') or ''"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the path of the enterprise customer logo image file.", "response": "def logo_path(instance, filename):\n \"\"\"\n Delete the file if it already exist and returns the enterprise customer logo image path.\n\n Arguments:\n instance (:class:`.EnterpriseCustomerBrandingConfiguration`): EnterpriseCustomerBrandingConfiguration object\n filename (str): file to upload\n\n Returns:\n path: path of image file e.g. enterprise/branding//_logo..lower()\n\n \"\"\"\n extension = os.path.splitext(filename)[1].lower()\n instance_id = str(instance.id)\n fullname = os.path.join(\"enterprise/branding/\", instance_id, instance_id + \"_logo\" + extension)\n if default_storage.exists(fullname):\n default_storage.delete(fullname)\n return fullname"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_link_by_email(self, user_email):\n try:\n user = User.objects.get(email=user_email)\n try:\n return self.get(user_id=user.id)\n except EnterpriseCustomerUser.DoesNotExist:\n pass\n except User.DoesNotExist:\n pass\n\n try:\n return PendingEnterpriseCustomerUser.objects.get(user_email=user_email)\n except PendingEnterpriseCustomerUser.DoesNotExist:\n pass\n\n return None", "response": "Get a link by email."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef link_user(self, enterprise_customer, user_email):\n try:\n existing_user = User.objects.get(email=user_email)\n self.get_or_create(enterprise_customer=enterprise_customer, user_id=existing_user.id)\n except User.DoesNotExist:\n PendingEnterpriseCustomerUser.objects.get_or_create(enterprise_customer=enterprise_customer,\n user_email=user_email)", "response": "Link user email to Enterprise Customer."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the enterprise customer uuid linked to the user.", "response": "def enterprise_customer_uuid(self):\n \"\"\"Get the enterprise customer uuid linked to the user.\"\"\"\n try:\n enterprise_user = EnterpriseCustomerUser.objects.get(user_id=self.user.id)\n except ObjectDoesNotExist:\n LOGGER.warning(\n 'User {} has a {} assignment but is not linked to an enterprise!'.format(\n self.__class__,\n self.user.id\n ))\n return None\n except MultipleObjectsReturned:\n LOGGER.warning(\n 'User {} is linked to multiple enterprises, which is not yet supported!'.format(self.user.id)\n )\n return None\n\n return str(enterprise_user.enterprise_customer.uuid)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_data_sharing_consent(username, enterprise_customer_uuid, course_id=None, program_uuid=None):\n EnterpriseCustomer = apps.get_model('enterprise', 'EnterpriseCustomer') # pylint: disable=invalid-name\n try:\n if course_id:\n return get_course_data_sharing_consent(username, course_id, enterprise_customer_uuid)\n return get_program_data_sharing_consent(username, program_uuid, enterprise_customer_uuid)\n except EnterpriseCustomer.DoesNotExist:\n return None", "response": "Get the data sharing consent object associated with a certain user."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the data sharing consent object associated with a certain user of a course.", "response": "def get_course_data_sharing_consent(username, course_id, enterprise_customer_uuid):\n \"\"\"\n Get the data sharing consent object associated with a certain user of a customer for a course.\n\n :param username: The user that grants consent.\n :param course_id: The course for which consent is granted.\n :param enterprise_customer_uuid: The consent requester.\n :return: The data sharing consent object\n \"\"\"\n # Prevent circular imports.\n DataSharingConsent = apps.get_model('consent', 'DataSharingConsent') # pylint: disable=invalid-name\n return DataSharingConsent.objects.proxied_get(\n username=username,\n course_id=course_id,\n enterprise_customer__uuid=enterprise_customer_uuid\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_program_data_sharing_consent(username, program_uuid, enterprise_customer_uuid):\n enterprise_customer = get_enterprise_customer(enterprise_customer_uuid)\n discovery_client = CourseCatalogApiServiceClient(enterprise_customer.site)\n course_ids = discovery_client.get_program_course_keys(program_uuid)\n child_consents = (\n get_data_sharing_consent(username, enterprise_customer_uuid, course_id=individual_course_id)\n for individual_course_id in course_ids\n )\n return ProxyDataSharingConsent.from_children(program_uuid, *child_consents)", "response": "Get the data sharing consent object associated with a certain user of a program."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef send_course_enrollment_statement(lrs_configuration, course_enrollment):\n user_details = LearnerInfoSerializer(course_enrollment.user)\n course_details = CourseInfoSerializer(course_enrollment.course)\n\n statement = LearnerCourseEnrollmentStatement(\n course_enrollment.user,\n course_enrollment.course,\n user_details.data,\n course_details.data,\n )\n EnterpriseXAPIClient(lrs_configuration).save_statement(statement)", "response": "Send a LearnerCourseEnrollmentStatement to the Enterprise XAPI."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsends a LearnerCourseCompletionStatement to the Enterprise XAPI.", "response": "def send_course_completion_statement(lrs_configuration, user, course_overview, course_grade):\n \"\"\"\n Send xAPI statement for course completion.\n\n Arguments:\n lrs_configuration (XAPILRSConfiguration): XAPILRSConfiguration instance where to send statements.\n user (User): Django User object.\n course_overview (CourseOverview): Course over view object containing course details.\n course_grade (CourseGrade): course grade object.\n \"\"\"\n user_details = LearnerInfoSerializer(user)\n course_details = CourseInfoSerializer(course_overview)\n\n statement = LearnerCourseCompletionStatement(\n user,\n course_overview,\n user_details.data,\n course_details.data,\n course_grade,\n )\n EnterpriseXAPIClient(lrs_configuration).save_statement(statement)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the exported and transformed content metadata as a dictionary.", "response": "def export(self):\n \"\"\"\n Return the exported and transformed content metadata as a dictionary.\n \"\"\"\n content_metadata_export = {}\n content_metadata_items = self.enterprise_api.get_content_metadata(self.enterprise_customer)\n LOGGER.info('Retrieved content metadata for enterprise [%s]', self.enterprise_customer.name)\n for item in content_metadata_items:\n transformed = self._transform_item(item)\n LOGGER.info(\n 'Exporting content metadata item with plugin configuration [%s]: [%s]',\n self.enterprise_configuration,\n json.dumps(transformed, indent=4),\n )\n content_metadata_item_export = ContentMetadataItemExport(item, transformed)\n content_metadata_export[content_metadata_item_export.content_id] = content_metadata_item_export\n return OrderedDict(sorted(content_metadata_export.items()))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntransforms the provided content metadata item to the schema expected by the integrated channel.", "response": "def _transform_item(self, content_metadata_item):\n \"\"\"\n Transform the provided content metadata item to the schema expected by the integrated channel.\n \"\"\"\n content_metadata_type = content_metadata_item['content_type']\n transformed_item = {}\n for integrated_channel_schema_key, edx_data_schema_key in self.DATA_TRANSFORM_MAPPING.items():\n # Look for transformer functions defined on subclasses.\n # Favor content type-specific functions.\n transformer = (\n getattr(\n self,\n 'transform_{content_type}_{edx_data_schema_key}'.format(\n content_type=content_metadata_type,\n edx_data_schema_key=edx_data_schema_key\n ),\n None\n )\n or\n getattr(\n self,\n 'transform_{edx_data_schema_key}'.format(\n edx_data_schema_key=edx_data_schema_key\n ),\n None\n )\n )\n if transformer:\n transformed_item[integrated_channel_schema_key] = transformer(content_metadata_item)\n else:\n # The concrete subclass does not define an override for the given field,\n # so just use the data key to index the content metadata item dictionary.\n try:\n transformed_item[integrated_channel_schema_key] = content_metadata_item[edx_data_schema_key]\n except KeyError:\n # There may be a problem with the DATA_TRANSFORM_MAPPING on\n # the concrete subclass or the concrete subclass does not implement\n # the appropriate field tranformer function.\n LOGGER.exception(\n 'Failed to transform content metadata item field [%s] for [%s]: [%s]',\n edx_data_schema_key,\n self.enterprise_customer.name,\n content_metadata_item,\n )\n\n return transformed_item"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the consent record relevant to the request at hand.", "response": "def get_consent_record(self, request):\n \"\"\"\n Get the consent record relevant to the request at hand.\n \"\"\"\n username, course_id, program_uuid, enterprise_customer_uuid = self.get_required_query_params(request)\n return get_data_sharing_consent(\n username,\n enterprise_customer_uuid,\n course_id=course_id,\n program_uuid=program_uuid\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the required query parameters from the request.", "response": "def get_required_query_params(self, request):\n \"\"\"\n Gets ``username``, ``course_id``, and ``enterprise_customer_uuid``,\n which are the relevant query parameters for this API endpoint.\n\n :param request: The request to this endpoint.\n :return: The ``username``, ``course_id``, and ``enterprise_customer_uuid`` from the request.\n \"\"\"\n username = get_request_value(request, self.REQUIRED_PARAM_USERNAME, '')\n course_id = get_request_value(request, self.REQUIRED_PARAM_COURSE_ID, '')\n program_uuid = get_request_value(request, self.REQUIRED_PARAM_PROGRAM_UUID, '')\n enterprise_customer_uuid = get_request_value(request, self.REQUIRED_PARAM_ENTERPRISE_CUSTOMER)\n if not (username and (course_id or program_uuid) and enterprise_customer_uuid):\n raise ConsentAPIRequestError(\n self.get_missing_params_message([\n (\"'username'\", bool(username)),\n (\"'enterprise_customer_uuid'\", bool(enterprise_customer_uuid)),\n (\"one of 'course_id' or 'program_uuid'\", bool(course_id or program_uuid)),\n ])\n )\n return username, course_id, program_uuid, enterprise_customer_uuid"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_no_record_response(self, request):\n username, course_id, program_uuid, enterprise_customer_uuid = self.get_required_query_params(request)\n data = {\n self.REQUIRED_PARAM_USERNAME: username,\n self.REQUIRED_PARAM_ENTERPRISE_CUSTOMER: enterprise_customer_uuid,\n self.CONSENT_EXISTS: False,\n self.CONSENT_GRANTED: False,\n self.CONSENT_REQUIRED: False,\n }\n if course_id:\n data[self.REQUIRED_PARAM_COURSE_ID] = course_id\n\n if program_uuid:\n data[self.REQUIRED_PARAM_PROGRAM_UUID] = program_uuid\n\n return Response(data, status=HTTP_200_OK)", "response": "Returns an HTTPResponse that can be used when there s no related EnterpriseCustomer."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the consent record for the current user.", "response": "def get(self, request):\n \"\"\"\n GET /consent/api/v1/data_sharing_consent?username=bob&course_id=id&enterprise_customer_uuid=uuid\n *username*\n The edX username from whom to get consent.\n *course_id*\n The course for which consent is granted.\n *enterprise_customer_uuid*\n The UUID of the enterprise customer that requires consent.\n \"\"\"\n try:\n consent_record = self.get_consent_record(request)\n if consent_record is None:\n return self.get_no_record_response(request)\n except ConsentAPIRequestError as invalid_request:\n return Response({'error': str(invalid_request)}, status=HTTP_400_BAD_REQUEST)\n\n return Response(consent_record.serialize(), status=HTTP_200_OK)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a JSON object that represents a new data sharing consent record for the given enterprise customer.", "response": "def post(self, request):\n \"\"\"\n POST /consent/api/v1/data_sharing_consent\n\n Requires a JSON object of the following format:\n >>> {\n >>> \"username\": \"bob\",\n >>> \"course_id\": \"course-v1:edX+DemoX+Demo_Course\",\n >>> \"enterprise_customer_uuid\": \"enterprise-uuid-goes-right-here\"\n >>> }\n\n Keys:\n *username*\n The edX username from whom to get consent.\n *course_id*\n The course for which consent is granted.\n *enterprise_customer_uuid*\n The UUID of the enterprise customer that requires consent.\n \"\"\"\n try:\n consent_record = self.get_consent_record(request)\n if consent_record is None:\n return self.get_no_record_response(request)\n if consent_record.consent_required():\n # If and only if the given EnterpriseCustomer requires data sharing consent\n # for the given course, then, since we've received a POST request, set the\n # consent state for the EC/user/course combo.\n consent_record.granted = True\n\n # Models don't have return values when saving, but ProxyDataSharingConsent\n # objects do - they should return either a model instance, or another instance\n # of ProxyDataSharingConsent if representing a multi-course consent record.\n consent_record = consent_record.save() or consent_record\n\n except ConsentAPIRequestError as invalid_request:\n return Response({'error': str(invalid_request)}, status=HTTP_400_BAD_REQUEST)\n\n return Response(consent_record.serialize())"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nhandle the DELETE request for a single - course consent record.", "response": "def delete(self, request):\n \"\"\"\n DELETE /consent/api/v1/data_sharing_consent\n\n Requires a JSON object of the following format:\n >>> {\n >>> \"username\": \"bob\",\n >>> \"course_id\": \"course-v1:edX+DemoX+Demo_Course\",\n >>> \"enterprise_customer_uuid\": \"enterprise-uuid-goes-right-here\"\n >>> }\n\n Keys:\n *username*\n The edX username from whom to get consent.\n *course_id*\n The course for which consent is granted.\n *enterprise_customer_uuid*\n The UUID of the enterprise customer that requires consent.\n \"\"\"\n try:\n consent_record = self.get_consent_record(request)\n if consent_record is None:\n return self.get_no_record_response(request)\n\n # We're fine with proactively refusing consent, even when there's no actual\n # requirement for consent yet.\n consent_record.granted = False\n\n # Models don't have return values when saving, but ProxyDataSharingConsent\n # objects do - they should return either a model instance, or another instance\n # of ProxyDataSharingConsent if representing a multi-course consent record.\n consent_record = consent_record.save() or consent_record\n\n except ConsentAPIRequestError as invalid_request:\n return Response({'error': str(invalid_request)}, status=HTTP_400_BAD_REQUEST)\n\n return Response(consent_record.serialize())"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ready(self):\n from enterprise.signals import handle_user_post_save\n from django.db.models.signals import pre_migrate, post_save\n\n post_save.connect(handle_user_post_save, sender=self.auth_user_model, dispatch_uid=USER_POST_SAVE_DISPATCH_UID)\n pre_migrate.connect(self._disconnect_user_post_save_for_migrations)", "response": "Connect the user post - save signals to the user model."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _disconnect_user_post_save_for_migrations(self, sender, **kwargs): # pylint: disable=unused-argument\n from django.db.models.signals import post_save\n post_save.disconnect(sender=self.auth_user_model, dispatch_uid=USER_POST_SAVE_DISPATCH_UID)", "response": "Disconnects user post save signal for migrations."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_actor(self, username, email):\n return Agent(\n name=username,\n mbox='mailto:{email}'.format(email=email),\n )", "response": "Get actor for the statement."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_object(self, name, description):\n return Activity(\n id=X_API_ACTIVITY_COURSE,\n definition=ActivityDefinition(\n name=LanguageMap({'en-US': (name or '').encode(\"ascii\", \"ignore\").decode('ascii')}),\n description=LanguageMap({'en-US': (description or '').encode(\"ascii\", \"ignore\").decode('ascii')}),\n ),\n )", "response": "Get object for the statement."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_csv(file_stream, expected_columns=None):\n reader = unicodecsv.DictReader(file_stream, encoding=\"utf-8\")\n\n if expected_columns and set(expected_columns) - set(reader.fieldnames):\n raise ValidationError(ValidationMessages.MISSING_EXPECTED_COLUMNS.format(\n expected_columns=\", \".join(expected_columns), actual_columns=\", \".join(reader.fieldnames)\n ))\n\n # \"yield from reader\" would be nicer, but we're on python2.7 yet.\n for row in reader:\n yield row", "response": "Parse a CSV file and return a stream of dictionaries representing each row."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate that the user email to be linked to the Enterprise Customer record.", "response": "def validate_email_to_link(email, raw_email=None, message_template=None, ignore_existing=False):\n \"\"\"\n Validate email to be linked to Enterprise Customer.\n\n Performs two checks:\n * Checks that email is valid\n * Checks that it is not already linked to any Enterprise Customer\n\n Arguments:\n email (str): user email to link\n raw_email (str): raw value as it was passed by user - used in error message.\n message_template (str): Validation error template string.\n ignore_existing (bool): If True to skip the check for an existing Enterprise Customer\n\n Raises:\n ValidationError: if email is invalid or already linked to Enterprise Customer.\n\n Returns:\n bool: Whether or not there is an existing record with the same email address.\n \"\"\"\n raw_email = raw_email if raw_email is not None else email\n message_template = message_template if message_template is not None else ValidationMessages.INVALID_EMAIL\n try:\n validate_email(email)\n except ValidationError:\n raise ValidationError(message_template.format(argument=raw_email))\n\n existing_record = EnterpriseCustomerUser.objects.get_link_by_email(email)\n if existing_record and not ignore_existing:\n raise ValidationError(ValidationMessages.USER_ALREADY_REGISTERED.format(\n email=email, ec_name=existing_record.enterprise_customer.name\n ))\n return existing_record or False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_course_runs_from_program(program):\n course_runs = set()\n for course in program.get(\"courses\", []):\n for run in course.get(\"course_runs\", []):\n if \"key\" in run and run[\"key\"]:\n course_runs.add(run[\"key\"])\n\n return course_runs", "response": "Returns a set of all course runs in a given program"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_earliest_start_date_from_program(program):\n start_dates = []\n for course in program.get('courses', []):\n for run in course.get('course_runs', []):\n if run.get('start'):\n start_dates.append(parse_lms_api_datetime(run['start']))\n if not start_dates:\n return None\n return min(start_dates)", "response": "Get the earliest start date from a program"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a paginated list of the objects in the object_list.", "response": "def paginated_list(object_list, page, page_size=25):\n \"\"\"\n Returns paginated list.\n\n Arguments:\n object_list (QuerySet): A list of records to be paginated.\n page (int): Current page number.\n page_size (int): Number of records displayed in each paginated set.\n show_all (bool): Whether to show all records.\n\n Adopted from django/contrib/admin/templatetags/admin_list.py\n https://github.com/django/django/blob/1.11.1/django/contrib/admin/templatetags/admin_list.py#L50\n \"\"\"\n paginator = CustomPaginator(object_list, page_size)\n try:\n object_list = paginator.page(page)\n except PageNotAnInteger:\n object_list = paginator.page(1)\n except EmptyPage:\n object_list = paginator.page(paginator.num_pages)\n\n page_range = []\n page_num = object_list.number\n\n # If there are 10 or fewer pages, display links to every page.\n # Otherwise, do some fancy\n if paginator.num_pages <= 10:\n page_range = range(paginator.num_pages)\n else:\n # Insert \"smart\" pagination links, so that there are always ON_ENDS\n # links at either end of the list of pages, and there are always\n # ON_EACH_SIDE links at either end of the \"current page\" link.\n if page_num > (PAGES_ON_EACH_SIDE + PAGES_ON_ENDS + 1):\n page_range.extend(range(1, PAGES_ON_ENDS + 1))\n page_range.append(DOT)\n page_range.extend(range(page_num - PAGES_ON_EACH_SIDE, page_num + 1))\n else:\n page_range.extend(range(1, page_num + 1))\n if page_num < (paginator.num_pages - PAGES_ON_EACH_SIDE - PAGES_ON_ENDS):\n page_range.extend(range(page_num + 1, page_num + PAGES_ON_EACH_SIDE + 1))\n page_range.append(DOT)\n page_range.extend(range(paginator.num_pages + 1 - PAGES_ON_ENDS, paginator.num_pages + 1))\n else:\n page_range.extend(range(page_num + 1, paginator.num_pages + 1))\n\n # Override page range to implement custom smart links.\n object_list.paginator.page_range = page_range\n\n return object_list"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef clean_email_or_username(self):\n email_or_username = self.cleaned_data[self.Fields.EMAIL_OR_USERNAME].strip()\n\n if not email_or_username:\n # The field is blank; we just return the existing blank value.\n return email_or_username\n\n email = email_or_username__to__email(email_or_username)\n bulk_entry = len(split_usernames_and_emails(email)) > 1\n if bulk_entry:\n for email in split_usernames_and_emails(email):\n validate_email_to_link(\n email,\n None,\n ValidationMessages.INVALID_EMAIL_OR_USERNAME,\n ignore_existing=True\n )\n email = email_or_username\n else:\n validate_email_to_link(\n email,\n email_or_username,\n ValidationMessages.INVALID_EMAIL_OR_USERNAME,\n ignore_existing=True\n )\n\n return email", "response": "Clean email form field\n\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nverifying course ID and retrieve course details.", "response": "def clean_course(self):\n \"\"\"\n Verify course ID and retrieve course details.\n \"\"\"\n course_id = self.cleaned_data[self.Fields.COURSE].strip()\n if not course_id:\n return None\n try:\n client = EnrollmentApiClient()\n return client.get_course_details(course_id)\n except (HttpClientError, HttpServerError):\n raise ValidationError(ValidationMessages.INVALID_COURSE_ID.format(course_id=course_id))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef clean_program(self):\n program_id = self.cleaned_data[self.Fields.PROGRAM].strip()\n if not program_id:\n return None\n\n try:\n client = CourseCatalogApiClient(self._user, self._enterprise_customer.site)\n program = client.get_program_by_uuid(program_id) or client.get_program_by_title(program_id)\n except MultipleProgramMatchError as exc:\n raise ValidationError(ValidationMessages.MULTIPLE_PROGRAM_MATCH.format(program_count=exc.programs_matched))\n except (HttpClientError, HttpServerError):\n raise ValidationError(ValidationMessages.INVALID_PROGRAM_ID.format(program_id=program_id))\n\n if not program:\n raise ValidationError(ValidationMessages.INVALID_PROGRAM_ID.format(program_id=program_id))\n\n if program['status'] != ProgramStatuses.ACTIVE:\n raise ValidationError(\n ValidationMessages.PROGRAM_IS_INACTIVE.format(program_id=program_id, status=program['status'])\n )\n\n return program", "response": "Try obtaining program from CourseCatalog API and return it."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncleaning the notify_on_enrollment field.", "response": "def clean_notify(self):\n \"\"\"\n Clean the notify_on_enrollment field.\n \"\"\"\n return self.cleaned_data.get(self.Fields.NOTIFY, self.NotificationTypes.DEFAULT)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncleans fields that depend on each other. In this case, the form can be used to link single user or bulk link multiple users. These are mutually exclusive modes, so this method checks that only one field is passed.", "response": "def clean(self):\n \"\"\"\n Clean fields that depend on each other.\n\n In this case, the form can be used to link single user or bulk link multiple users. These are mutually\n exclusive modes, so this method checks that only one field is passed.\n \"\"\"\n cleaned_data = super(ManageLearnersForm, self).clean()\n\n # Here we take values from `data` (and not `cleaned_data`) as we need raw values - field clean methods\n # might \"invalidate\" the value and set it to None, while all we care here is if it was provided at all or not\n email_or_username = self.data.get(self.Fields.EMAIL_OR_USERNAME, None)\n bulk_upload_csv = self.files.get(self.Fields.BULK_UPLOAD, None)\n\n if not email_or_username and not bulk_upload_csv:\n raise ValidationError(ValidationMessages.NO_FIELDS_SPECIFIED)\n\n if email_or_username and bulk_upload_csv:\n raise ValidationError(ValidationMessages.BOTH_FIELDS_SPECIFIED)\n\n if email_or_username:\n mode = self.Modes.MODE_SINGULAR\n else:\n mode = self.Modes.MODE_BULK\n\n cleaned_data[self.Fields.MODE] = mode\n cleaned_data[self.Fields.NOTIFY] = self.clean_notify()\n\n self._validate_course()\n self._validate_program()\n\n if self.data.get(self.Fields.PROGRAM, None) and self.data.get(self.Fields.COURSE, None):\n raise ValidationError(ValidationMessages.COURSE_AND_PROGRAM_ERROR)\n\n return cleaned_data"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _validate_course(self):\n # Verify that the selected mode is valid for the given course .\n course_details = self.cleaned_data.get(self.Fields.COURSE)\n if course_details:\n course_mode = self.cleaned_data.get(self.Fields.COURSE_MODE)\n if not course_mode:\n raise ValidationError(ValidationMessages.COURSE_WITHOUT_COURSE_MODE)\n valid_course_modes = course_details[\"course_modes\"]\n if all(course_mode != mode[\"slug\"] for mode in valid_course_modes):\n error = ValidationError(ValidationMessages.COURSE_MODE_INVALID_FOR_COURSE.format(\n course_mode=course_mode,\n course_id=course_details[\"course_id\"],\n ))\n raise ValidationError({self.Fields.COURSE_MODE: error})", "response": "Validate that the selected mode is valid for the given course."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _validate_program(self):\n program = self.cleaned_data.get(self.Fields.PROGRAM)\n if not program:\n return\n\n course_runs = get_course_runs_from_program(program)\n try:\n client = CourseCatalogApiClient(self._user, self._enterprise_customer.site)\n available_modes = client.get_common_course_modes(course_runs)\n course_mode = self.cleaned_data.get(self.Fields.COURSE_MODE)\n except (HttpClientError, HttpServerError):\n raise ValidationError(\n ValidationMessages.FAILED_TO_OBTAIN_COURSE_MODES.format(program_title=program.get(\"title\"))\n )\n\n if not course_mode:\n raise ValidationError(ValidationMessages.COURSE_WITHOUT_COURSE_MODE)\n if course_mode not in available_modes:\n raise ValidationError(ValidationMessages.COURSE_MODE_NOT_AVAILABLE.format(\n mode=course_mode, program_title=program.get(\"title\"), modes=\", \".join(available_modes)\n ))", "response": "Validate that the selected mode is available for the program and all courses in the program."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nretrieving a list of catalog ID and name pairs.", "response": "def get_catalog_options(self):\n \"\"\"\n Retrieve a list of catalog ID and name pairs.\n\n Once retrieved, these name pairs can be used directly as a value\n for the `choices` argument to a ChoiceField.\n \"\"\"\n # TODO: We will remove the discovery service catalog implementation\n # once we have fully migrated customer's to EnterpriseCustomerCatalogs.\n # For now, this code will prevent an admin from creating a new\n # EnterpriseCustomer with a discovery service catalog. They will have to first\n # save the EnterpriseCustomer admin form and then edit the EnterpriseCustomer\n # to add a discovery service catalog.\n if hasattr(self.instance, 'site'):\n catalog_api = CourseCatalogApiClient(self.user, self.instance.site)\n else:\n catalog_api = CourseCatalogApiClient(self.user)\n catalogs = catalog_api.get_all_catalogs()\n # order catalogs by name.\n catalogs = sorted(catalogs, key=lambda catalog: catalog.get('name', '').lower())\n\n return BLANK_CHOICE_DASH + [\n (catalog['id'], catalog['name'],)\n for catalog in catalogs\n ]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef clean(self):\n cleaned_data = super(EnterpriseCustomerAdminForm, self).clean()\n if 'catalog' in cleaned_data and not cleaned_data['catalog']:\n cleaned_data['catalog'] = None\n return cleaned_data", "response": "Clean the data prior to database entry."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef clean(self):\n super(EnterpriseCustomerIdentityProviderAdminForm, self).clean()\n\n provider_id = self.cleaned_data.get('provider_id', None)\n enterprise_customer = self.cleaned_data.get('enterprise_customer', None)\n\n if provider_id is None or enterprise_customer is None:\n # field validation for either provider_id or enterprise_customer has already raised\n # a validation error.\n return\n\n identity_provider = utils.get_identity_provider(provider_id)\n if not identity_provider:\n # This should not happen, as identity providers displayed in drop down are fetched dynamically.\n message = _(\n \"The specified Identity Provider does not exist. For more \"\n \"information, contact a system administrator.\",\n )\n # Log message for debugging\n logger.exception(message)\n\n raise ValidationError(message)\n\n if identity_provider and identity_provider.site != enterprise_customer.site:\n raise ValidationError(\n _(\n \"The site for the selected identity provider \"\n \"({identity_provider_site}) does not match the site for \"\n \"this enterprise customer ({enterprise_customer_site}). \"\n \"To correct this problem, select a site that has a domain \"\n \"of '{identity_provider_site}', or update the identity \"\n \"provider to '{enterprise_customer_site}'.\"\n ).format(\n enterprise_customer_site=enterprise_customer.site,\n identity_provider_site=identity_provider.site,\n ),\n )", "response": "Validate that the selected site for this enterprise customer matches with the selected identity provider s site."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nchecking that all selected catalogs for reporting are tied to the selected enterprise.", "response": "def clean(self):\n \"\"\"\n Override of clean method to perform additional validation\n \"\"\"\n cleaned_data = super(EnterpriseCustomerReportingConfigAdminForm, self).clean()\n report_customer = cleaned_data.get('enterprise_customer')\n\n # Check that any selected catalogs are tied to the selected enterprise.\n invalid_catalogs = [\n '{} ({})'.format(catalog.title, catalog.uuid)\n for catalog in cleaned_data.get('enterprise_customer_catalogs')\n if catalog.enterprise_customer != report_customer\n ]\n\n if invalid_catalogs:\n message = _(\n 'These catalogs for reporting do not match enterprise'\n 'customer {enterprise_customer}: {invalid_catalogs}',\n ).format(\n enterprise_customer=report_customer,\n invalid_catalogs=invalid_catalogs,\n )\n self.add_error('enterprise_customer_catalogs', message)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef clean_channel_worker_username(self):\n channel_worker_username = self.cleaned_data['channel_worker_username'].strip()\n\n try:\n User.objects.get(username=channel_worker_username)\n except User.DoesNotExist:\n raise ValidationError(\n ValidationMessages.INVALID_CHANNEL_WORKER.format(\n channel_worker_username=channel_worker_username\n )\n )\n\n return channel_worker_username", "response": "Clean enterprise channel worker user username form field\n\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef verify_edx_resources():\n required_methods = {\n 'ProgramDataExtender': ProgramDataExtender,\n }\n\n for method in required_methods:\n if required_methods[method] is None:\n raise NotConnectedToOpenEdX(\n _(\"The following method from the Open edX platform is necessary for this view but isn't available.\")\n + \"\\nUnavailable: {method}\".format(method=method)\n )", "response": "Ensures that all required resources to render the view are present."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_global_context(request, enterprise_customer):\n platform_name = get_configuration_value(\"PLATFORM_NAME\", settings.PLATFORM_NAME)\n # pylint: disable=no-member\n return {\n 'enterprise_customer': enterprise_customer,\n 'LMS_SEGMENT_KEY': settings.LMS_SEGMENT_KEY,\n 'LANGUAGE_CODE': get_language_from_request(request),\n 'tagline': get_configuration_value(\"ENTERPRISE_TAGLINE\", settings.ENTERPRISE_TAGLINE),\n 'platform_description': get_configuration_value(\n \"PLATFORM_DESCRIPTION\",\n settings.PLATFORM_DESCRIPTION,\n ),\n 'LMS_ROOT_URL': settings.LMS_ROOT_URL,\n 'platform_name': platform_name,\n 'header_logo_alt_text': _('{platform_name} home page').format(platform_name=platform_name),\n 'welcome_text': constants.WELCOME_TEXT.format(platform_name=platform_name),\n 'enterprise_welcome_text': constants.ENTERPRISE_WELCOME_TEXT.format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n strong_start='',\n strong_end='',\n line_break='
',\n privacy_policy_link_start=\"\".format(\n pp_url=get_configuration_value('PRIVACY', 'https://www.edx.org/edx-privacy-policy', type='url'),\n ),\n privacy_policy_link_end=\"\",\n ),\n }", "response": "Get the set of variables that are needed by default across views."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the localized converted price as string", "response": "def get_price_text(price, request):\n \"\"\"\n Return the localized converted price as string (ex. '$150 USD').\n\n If the local_currency switch is enabled and the users location has been determined this will convert the\n given price based on conversion rate from the Catalog service and return a localized string\n \"\"\"\n if waffle.switch_is_active('local_currency') and get_localized_price_text:\n return get_localized_price_text(price, request)\n\n return format_price(price)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nrendering a page with specified error_code and log_message.", "response": "def render_page_with_error_code_message(request, context_data, error_code, log_message):\n \"\"\"\n Return a 404 page with specified error_code after logging error and adding message to django messages.\n \"\"\"\n LOGGER.error(log_message)\n messages.add_generic_error_message_with_code(request, error_code)\n return render(\n request,\n ENTERPRISE_GENERAL_ERROR_PAGE,\n context=context_data,\n status=404,\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning whether the input course or program exist.", "response": "def course_or_program_exist(self, course_id, program_uuid):\n \"\"\"\n Return whether the input course or program exist.\n \"\"\"\n course_exists = course_id and CourseApiClient().get_course_details(course_id)\n program_exists = program_uuid and CourseCatalogApiServiceClient().program_exists(program_uuid)\n return course_exists or program_exists"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the default set of variables that will be populated by default.", "response": "def get_default_context(self, enterprise_customer, platform_name):\n \"\"\"\n Get the set of variables that will populate the template by default.\n \"\"\"\n context_data = {\n 'page_title': _('Data sharing consent required'),\n 'consent_message_header': _('Consent to share your data'),\n 'requested_permissions_header': _(\n 'Per the {start_link}Data Sharing Policy{end_link}, '\n '{bold_start}{enterprise_customer_name}{bold_end} would like to know about:'\n ).format(\n enterprise_customer_name=enterprise_customer.name,\n bold_start='',\n bold_end='',\n start_link='',\n end_link='',\n ),\n 'agreement_text': _(\n 'I agree to allow {platform_name} to share data about my enrollment, completion and performance in all '\n '{platform_name} courses and programs where my enrollment is sponsored by {enterprise_customer_name}.'\n ).format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n ),\n 'continue_text': _('Yes, continue'),\n 'abort_text': _('No, take me back.'),\n 'policy_dropdown_header': _('Data Sharing Policy'),\n 'sharable_items_header': _(\n 'Enrollment, completion, and performance data that may be shared with {enterprise_customer_name} '\n '(or its designee) for these courses and programs are limited to the following:'\n ).format(\n enterprise_customer_name=enterprise_customer.name\n ),\n 'sharable_items': [\n _(\n 'My email address for my {platform_name} account, '\n 'and the date when I created my {platform_name} account'\n ).format(\n platform_name=platform_name\n ),\n _(\n 'My {platform_name} ID, and if I log in via single sign-on, '\n 'my {enterprise_customer_name} SSO user-ID'\n ).format(\n platform_name=platform_name,\n enterprise_customer_name=enterprise_customer.name,\n ),\n _('My {platform_name} username').format(platform_name=platform_name),\n _('My country or region of residence'),\n _(\n 'What courses and/or programs I\\'ve enrolled in or unenrolled from, what track I '\n 'enrolled in (audit or verified) and the date when I enrolled in each course or program'\n ),\n _(\n 'Information about each course or program I\\'ve enrolled in, '\n 'including its duration and level of effort required'\n ),\n _(\n 'Whether I completed specific parts of each course or program (for example, whether '\n 'I watched a given video or completed a given homework assignment)'\n ),\n _(\n 'My overall percentage completion of each course or program on a periodic basis, '\n 'including the total time spent in each course or program and the date when I last '\n 'logged in to each course or program'\n ),\n _('My performance in each course or program'),\n _('My final grade in each course or program, and the date when I completed each course or program'),\n _('Whether I received a certificate in each course or program'),\n ],\n 'sharable_items_footer': _(\n 'My permission applies only to data from courses or programs that are sponsored by '\n '{enterprise_customer_name}, and not to data from any {platform_name} courses or programs that '\n 'I take on my own. I understand that I may withdraw my permission only by fully unenrolling '\n 'from any courses or programs that are sponsored by {enterprise_customer_name}.'\n ).format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n ),\n 'sharable_items_note_header': _('Please note'),\n 'sharable_items_notes': [\n _('If you decline to consent, that fact may be shared with {enterprise_customer_name}.').format(\n enterprise_customer_name=enterprise_customer.name\n ),\n ],\n 'confirmation_modal_header': _('Are you aware...'),\n 'confirmation_modal_affirm_decline_text': _('I decline'),\n 'confirmation_modal_abort_decline_text': _('View the data sharing policy'),\n 'policy_link_template': _('View the {start_link}data sharing policy{end_link}.').format(\n start_link='',\n end_link='',\n ),\n 'policy_return_link_text': _('Return to Top'),\n }\n return context_data"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_context_from_db(self, consent_page, platform_name, item, context):\n enterprise_customer = consent_page.enterprise_customer\n course_title = context.get('course_title', None)\n course_start_date = context.get('course_start_date', None)\n context_data = {\n 'text_override_available': True,\n 'page_title': consent_page.page_title,\n 'left_sidebar_text': consent_page.left_sidebar_text.format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n item=item,\n course_title=course_title,\n course_start_date=course_start_date,\n ),\n 'top_paragraph': consent_page.top_paragraph.format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n item=item,\n course_title=course_title,\n course_start_date=course_start_date,\n ),\n 'agreement_text': consent_page.agreement_text.format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n item=item,\n course_title=course_title,\n course_start_date=course_start_date,\n ),\n 'continue_text': consent_page.continue_text,\n 'abort_text': consent_page.abort_text,\n 'policy_dropdown_header': consent_page.policy_dropdown_header,\n 'policy_paragraph': consent_page.policy_paragraph.format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n item=item,\n course_title=course_title,\n course_start_date=course_start_date,\n ),\n 'confirmation_modal_header': consent_page.confirmation_modal_header.format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n item=item,\n course_title=course_title,\n course_start_date=course_start_date,\n ),\n 'confirmation_alert_prompt': consent_page.confirmation_modal_text.format(\n enterprise_customer_name=enterprise_customer.name,\n platform_name=platform_name,\n item=item,\n course_title=course_title,\n course_start_date=course_start_date,\n ),\n 'confirmation_modal_affirm_decline_text': consent_page.modal_affirm_decline_text,\n 'confirmation_modal_abort_decline_text': consent_page.modal_abort_decline_text,\n }\n return context_data", "response": "Get the context from the DB."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_course_or_program_context(self, enterprise_customer, course_id=None, program_uuid=None):\n context_data = {}\n if course_id:\n context_data.update({'course_id': course_id, 'course_specific': True})\n if not self.preview_mode:\n try:\n catalog_api_client = CourseCatalogApiServiceClient(enterprise_customer.site)\n except ImproperlyConfigured:\n raise Http404\n\n course_run_details = catalog_api_client.get_course_run(course_id)\n course_start_date = ''\n if course_run_details['start']:\n course_start_date = parse(course_run_details['start']).strftime('%B %d, %Y')\n\n context_data.update({\n 'course_title': course_run_details['title'],\n 'course_start_date': course_start_date,\n })\n else:\n context_data.update({\n 'course_title': 'Demo Course',\n 'course_start_date': datetime.datetime.now().strftime('%B %d, %Y'),\n })\n else:\n context_data.update({\n 'program_uuid': program_uuid,\n 'program_specific': True,\n })\n return context_data", "response": "Returns a dict having course or program specific keys for data sharing consent page."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get(self, request):\n enterprise_customer_uuid = request.GET.get('enterprise_customer_uuid')\n success_url = request.GET.get('next')\n failure_url = request.GET.get('failure_url')\n course_id = request.GET.get('course_id', '')\n program_uuid = request.GET.get('program_uuid', '')\n self.preview_mode = bool(request.GET.get('preview_mode', False))\n\n # Get enterprise_customer to start in case we need to render a custom 404 page\n # Then go through other business logic to determine (and potentially overwrite) the enterprise customer\n enterprise_customer = get_enterprise_customer_or_404(enterprise_customer_uuid)\n context_data = get_global_context(request, enterprise_customer)\n\n if not self.preview_mode:\n if not self.course_or_program_exist(course_id, program_uuid):\n error_code = 'ENTGDS000'\n log_message = (\n 'Neither the course with course_id: {course_id} '\n 'or program with {program_uuid} exist for '\n 'enterprise customer {enterprise_customer_uuid}'\n 'Error code {error_code} presented to user {userid}'.format(\n course_id=course_id,\n program_uuid=program_uuid,\n error_code=error_code,\n userid=request.user.id,\n enterprise_customer_uuid=enterprise_customer_uuid,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n try:\n consent_record = get_data_sharing_consent(\n request.user.username,\n enterprise_customer_uuid,\n program_uuid=program_uuid,\n course_id=course_id\n )\n except NotConnectedToOpenEdX as error:\n error_code = 'ENTGDS001'\n log_message = (\n 'The was a problem with getting the consent record of user {userid} with '\n 'uuid {enterprise_customer_uuid}. get_data_sharing_consent threw '\n 'the following NotConnectedToOpenEdX error: {error}'\n 'for course_id {course_id}.'\n 'Error code {error_code} presented to user'.format(\n userid=request.user.id,\n enterprise_customer_uuid=enterprise_customer_uuid,\n error=error,\n error_code=error_code,\n course_id=course_id,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n try:\n consent_required = consent_record.consent_required()\n except AttributeError:\n consent_required = None\n\n if consent_record is None or not consent_required:\n error_code = 'ENTGDS002'\n log_message = (\n 'The was a problem with the consent record of user {userid} with '\n 'enterprise_customer_uuid {enterprise_customer_uuid}. consent_record has a value '\n 'of {consent_record} and consent_record.consent_required() a '\n 'value of {consent_required} for course_id {course_id}. '\n 'Error code {error_code} presented to user'.format(\n userid=request.user.id,\n enterprise_customer_uuid=enterprise_customer_uuid,\n consent_record=consent_record,\n consent_required=consent_required,\n error_code=error_code,\n course_id=course_id,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n else:\n enterprise_customer = consent_record.enterprise_customer\n elif not request.user.is_staff:\n raise PermissionDenied()\n\n # Retrieve context data again now that enterprise_customer logic has been run\n context_data = get_global_context(request, enterprise_customer)\n\n if not (enterprise_customer_uuid and success_url and failure_url):\n error_code = 'ENTGDS003'\n log_message = (\n 'Error: one or more of the following values was falsy: '\n 'enterprise_customer_uuid: {enterprise_customer_uuid}, '\n 'success_url: {success_url}, '\n 'failure_url: {failure_url} for course id {course_id}'\n 'The following error code was reported to user {userid}: {error_code}'.format(\n userid=request.user.id,\n enterprise_customer_uuid=enterprise_customer_uuid,\n success_url=success_url,\n failure_url=failure_url,\n error_code=error_code,\n course_id=course_id,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n try:\n updated_context_dict = self.get_course_or_program_context(\n enterprise_customer,\n course_id=course_id,\n program_uuid=program_uuid\n )\n context_data.update(updated_context_dict)\n except Http404:\n error_code = 'ENTGDS004'\n log_message = (\n 'CourseCatalogApiServiceClient is improperly configured. '\n 'Returned error code {error_code} to user {userid} '\n 'and enterprise_customer {enterprise_customer} '\n 'for course_id {course_id}'.format(\n error_code=error_code,\n userid=request.user.id,\n enterprise_customer=enterprise_customer.uuid,\n course_id=course_id,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n item = 'course' if course_id else 'program'\n # Translators: bold_start and bold_end are HTML tags for specifying enterprise name in bold text.\n context_data.update({\n 'consent_request_prompt': _(\n 'To access this {item}, you must first consent to share your learning achievements '\n 'with {bold_start}{enterprise_customer_name}{bold_end}.'\n ).format(\n enterprise_customer_name=enterprise_customer.name,\n bold_start='',\n bold_end='',\n item=item,\n ),\n 'confirmation_alert_prompt': _(\n 'In order to start this {item} and use your discount, {bold_start}you must{bold_end} consent '\n 'to share your {item} data with {enterprise_customer_name}.'\n ).format(\n enterprise_customer_name=enterprise_customer.name,\n bold_start='',\n bold_end='',\n item=item,\n ),\n 'redirect_url': success_url,\n 'failure_url': failure_url,\n 'defer_creation': request.GET.get('defer_creation') is not None,\n 'requested_permissions': [\n _('your enrollment in this {item}').format(item=item),\n _('your learning progress'),\n _('course completion'),\n ],\n 'policy_link_template': '',\n })\n platform_name = context_data['platform_name']\n published_only = False if self.preview_mode else True\n enterprise_consent_page = enterprise_customer.get_data_sharing_consent_text_overrides(\n published_only=published_only\n )\n if enterprise_consent_page:\n context_data.update(self.get_context_from_db(enterprise_consent_page, platform_name, item, context_data))\n else:\n context_data.update(self.get_default_context(enterprise_customer, platform_name))\n\n return render(request, 'enterprise/grant_data_sharing_permissions.html', context=context_data)", "response": "Render a form to collect user input about data sharing consent."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef post(self, request):\n enterprise_uuid = request.POST.get('enterprise_customer_uuid')\n success_url = request.POST.get('redirect_url')\n failure_url = request.POST.get('failure_url')\n course_id = request.POST.get('course_id', '')\n program_uuid = request.POST.get('program_uuid', '')\n\n enterprise_customer = get_enterprise_customer_or_404(enterprise_uuid)\n context_data = get_global_context(request, enterprise_customer)\n\n if not (enterprise_uuid and success_url and failure_url):\n error_code = 'ENTGDS005'\n log_message = (\n 'Error: one or more of the following values was falsy: '\n 'enterprise_uuid: {enterprise_uuid}, '\n 'success_url: {success_url}, '\n 'failure_url: {failure_url} for course_id {course_id}. '\n 'The following error code was reported to the user {userid}: {error_code}'.format(\n userid=request.user.id,\n enterprise_uuid=enterprise_uuid,\n success_url=success_url,\n failure_url=failure_url,\n error_code=error_code,\n course_id=course_id,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n if not self.course_or_program_exist(course_id, program_uuid):\n error_code = 'ENTGDS006'\n log_message = (\n 'Neither the course with course_id: {course_id} '\n 'or program with {program_uuid} exist for '\n 'enterprise customer {enterprise_uuid}'\n 'Error code {error_code} presented to user {userid}'.format(\n course_id=course_id,\n program_uuid=program_uuid,\n error_code=error_code,\n userid=request.user.id,\n enterprise_uuid=enterprise_uuid,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n consent_record = get_data_sharing_consent(\n request.user.username,\n enterprise_uuid,\n program_uuid=program_uuid,\n course_id=course_id\n )\n if consent_record is None:\n error_code = 'ENTGDS007'\n log_message = (\n 'The was a problem with the consent record of user {userid} with '\n 'enterprise_uuid {enterprise_uuid}. consent_record has a value '\n 'of {consent_record} and a '\n 'value for course_id {course_id}. '\n 'Error code {error_code} presented to user'.format(\n userid=request.user.id,\n enterprise_uuid=enterprise_uuid,\n consent_record=consent_record,\n error_code=error_code,\n course_id=course_id,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n defer_creation = request.POST.get('defer_creation')\n consent_provided = bool(request.POST.get('data_sharing_consent', False))\n if defer_creation is None and consent_record.consent_required():\n if course_id:\n enterprise_customer_user, __ = EnterpriseCustomerUser.objects.get_or_create(\n enterprise_customer=consent_record.enterprise_customer,\n user_id=request.user.id\n )\n enterprise_customer_user.update_session(request)\n __, created = EnterpriseCourseEnrollment.objects.get_or_create(\n enterprise_customer_user=enterprise_customer_user,\n course_id=course_id,\n )\n if created:\n track_enrollment('data-consent-page-enrollment', request.user.id, course_id, request.path)\n\n consent_record.granted = consent_provided\n consent_record.save()\n\n return redirect(success_url if consent_provided else failure_url)", "response": "Process the POST request."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, request, enterprise_uuid, course_id):\n enrollment_course_mode = request.GET.get('course_mode')\n enterprise_catalog_uuid = request.GET.get('catalog')\n\n # Redirect the learner to LMS dashboard in case no course mode is\n # provided as query parameter `course_mode`\n if not enrollment_course_mode:\n return redirect(LMS_DASHBOARD_URL)\n\n enrollment_api_client = EnrollmentApiClient()\n course_modes = enrollment_api_client.get_course_modes(course_id)\n\n # Verify that the request user belongs to the enterprise against the\n # provided `enterprise_uuid`.\n enterprise_customer = get_enterprise_customer_or_404(enterprise_uuid)\n enterprise_customer_user = get_enterprise_customer_user(request.user.id, enterprise_customer.uuid)\n\n if not course_modes:\n context_data = get_global_context(request, enterprise_customer)\n error_code = 'ENTHCE000'\n log_message = (\n 'No course_modes for course_id {course_id} for enterprise_catalog_uuid '\n '{enterprise_catalog_uuid}.'\n 'The following error was presented to '\n 'user {userid}: {error_code}'.format(\n userid=request.user.id,\n enterprise_catalog_uuid=enterprise_catalog_uuid,\n course_id=course_id,\n error_code=error_code\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n selected_course_mode = None\n for course_mode in course_modes:\n if course_mode['slug'] == enrollment_course_mode:\n selected_course_mode = course_mode\n break\n\n if not selected_course_mode:\n return redirect(LMS_DASHBOARD_URL)\n\n # Create the Enterprise backend database records for this course\n # enrollment\n __, created = EnterpriseCourseEnrollment.objects.get_or_create(\n enterprise_customer_user=enterprise_customer_user,\n course_id=course_id,\n )\n if created:\n track_enrollment('course-landing-page-enrollment', request.user.id, course_id, request.get_full_path())\n\n DataSharingConsent.objects.update_or_create(\n username=enterprise_customer_user.username,\n course_id=course_id,\n enterprise_customer=enterprise_customer_user.enterprise_customer,\n defaults={\n 'granted': True\n },\n )\n\n audit_modes = getattr(settings, 'ENTERPRISE_COURSE_ENROLLMENT_AUDIT_MODES', ['audit', 'honor'])\n if selected_course_mode['slug'] in audit_modes:\n # In case of Audit course modes enroll the learner directly through\n # enrollment API client and redirect the learner to dashboard.\n enrollment_api_client.enroll_user_in_course(\n request.user.username, course_id, selected_course_mode['slug']\n )\n\n return redirect(LMS_COURSEWARE_URL.format(course_id=course_id))\n\n # redirect the enterprise learner to the ecommerce flow in LMS\n # Note: LMS start flow automatically detects the paid mode\n premium_flow = LMS_START_PREMIUM_COURSE_FLOW_URL.format(course_id=course_id)\n if enterprise_catalog_uuid:\n premium_flow += '?catalog={catalog_uuid}'.format(\n catalog_uuid=enterprise_catalog_uuid\n )\n\n return redirect(premium_flow)", "response": "This function handles the get action for the given enterprise learner in the provided course."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the final discounted price on each premium mode.", "response": "def set_final_prices(self, modes, request):\n \"\"\"\n Set the final discounted price on each premium mode.\n \"\"\"\n result = []\n for mode in modes:\n if mode['premium']:\n mode['final_price'] = EcommerceApiClient(request.user).get_course_final_price(\n mode=mode,\n enterprise_catalog_uuid=request.GET.get(\n 'catalog'\n ) if request.method == 'GET' else None,\n )\n result.append(mode)\n return result"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_available_course_modes(self, request, course_run_id, enterprise_catalog):\n modes = EnrollmentApiClient().get_course_modes(course_run_id)\n if not modes:\n LOGGER.warning('Unable to get course modes for course run id {course_run_id}.'.format(\n course_run_id=course_run_id\n ))\n messages.add_generic_info_message_for_error(request)\n\n if enterprise_catalog:\n # filter and order course modes according to the enterprise catalog\n modes = [mode for mode in modes if mode['slug'] in enterprise_catalog.enabled_course_modes]\n modes.sort(key=lambda course_mode: enterprise_catalog.enabled_course_modes.index(course_mode['slug']))\n if not modes:\n LOGGER.info(\n 'No matching course modes found for course run {course_run_id} in '\n 'EnterpriseCustomerCatalog [{enterprise_catalog_uuid}]'.format(\n course_run_id=course_run_id,\n enterprise_catalog_uuid=enterprise_catalog,\n )\n )\n messages.add_generic_info_message_for_error(request)\n\n return modes", "response": "Get the available course modes for the given course run."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nretrieves the base details for the specified enterprise customer and course run.", "response": "def get_base_details(self, request, enterprise_uuid, course_run_id):\n \"\"\"\n Retrieve fundamental details used by both POST and GET versions of this view.\n\n Specifically, take an EnterpriseCustomer UUID and a course run ID, and transform those\n into an actual EnterpriseCustomer, a set of details about the course, and a list\n of the available course modes for that course run.\n \"\"\"\n enterprise_customer = get_enterprise_customer_or_404(enterprise_uuid)\n\n # If the catalog query parameter was provided, we need to scope\n # this request to the specified EnterpriseCustomerCatalog.\n enterprise_catalog_uuid = request.GET.get('catalog')\n enterprise_catalog = None\n if enterprise_catalog_uuid:\n try:\n enterprise_catalog_uuid = UUID(enterprise_catalog_uuid)\n enterprise_catalog = enterprise_customer.enterprise_customer_catalogs.get(\n uuid=enterprise_catalog_uuid\n )\n except (ValueError, EnterpriseCustomerCatalog.DoesNotExist):\n LOGGER.warning(\n 'EnterpriseCustomerCatalog [{enterprise_catalog_uuid}] does not exist'.format(\n enterprise_catalog_uuid=enterprise_catalog_uuid,\n )\n )\n messages.add_generic_info_message_for_error(request)\n\n course = None\n course_run = None\n course_modes = []\n if enterprise_catalog:\n course, course_run = enterprise_catalog.get_course_and_course_run(course_run_id)\n else:\n try:\n course, course_run = CourseCatalogApiServiceClient(\n enterprise_customer.site\n ).get_course_and_course_run(course_run_id)\n except ImproperlyConfigured:\n LOGGER.warning('CourseCatalogApiServiceClient is improperly configured.')\n messages.add_generic_info_message_for_error(request)\n return enterprise_customer, course, course_run, course_modes\n\n if not course or not course_run:\n course_id = course['key'] if course else \"Not Found\"\n course_title = course['title'] if course else \"Not Found\"\n course_run_title = course_run['title'] if course_run else \"Not Found\"\n enterprise_catalog_title = enterprise_catalog.title if enterprise_catalog else \"Not Found\"\n # The specified course either does not exist in the specified\n # EnterpriseCustomerCatalog, or does not exist at all in the\n # discovery service.\n LOGGER.warning(\n 'Failed to fetch details for course \"{course_title}\" [{course_id}] '\n 'or course run \"{course_run_title}\" [{course_run_id}] '\n 'for enterprise \"{enterprise_name}\" [{enterprise_uuid}] '\n 'with catalog \"{enterprise_catalog_title}\" [{enterprise_catalog_uuid}]'.format(\n course_title=course_title,\n course_id=course_id,\n course_run_title=course_run_title,\n course_run_id=course_run_id,\n enterprise_name=enterprise_customer.name,\n enterprise_uuid=enterprise_customer.uuid,\n enterprise_catalog_title=enterprise_catalog_title,\n enterprise_catalog_uuid=enterprise_catalog_uuid,\n )\n )\n messages.add_generic_info_message_for_error(request)\n return enterprise_customer, course, course_run, course_modes\n\n if enterprise_catalog_uuid and not enterprise_catalog:\n # A catalog query parameter was given, but the specified\n # EnterpriseCustomerCatalog does not exist, so just return and\n # display the generic error message.\n return enterprise_customer, course, course_run, course_modes\n\n modes = self.get_available_course_modes(request, course_run_id, enterprise_catalog)\n audit_modes = getattr(\n settings,\n 'ENTERPRISE_COURSE_ENROLLMENT_AUDIT_MODES',\n ['audit', 'honor']\n )\n\n for mode in modes:\n if mode['min_price']:\n price_text = get_price_text(mode['min_price'], request)\n else:\n price_text = _('FREE')\n if mode['slug'] in audit_modes:\n description = _('Not eligible for a certificate.')\n else:\n description = _('Earn a verified certificate!')\n course_modes.append({\n 'mode': mode['slug'],\n 'min_price': mode['min_price'],\n 'sku': mode['sku'],\n 'title': mode['name'],\n 'original_price': price_text,\n 'final_price': price_text,\n 'description': description,\n 'premium': mode['slug'] not in audit_modes\n })\n\n return enterprise_customer, course, course_run, course_modes"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_enterprise_course_enrollment_page(\n self,\n request,\n enterprise_customer,\n course,\n course_run,\n course_modes,\n enterprise_course_enrollment,\n data_sharing_consent\n ):\n \"\"\"\n Render enterprise-specific course track selection page.\n \"\"\"\n context_data = get_global_context(request, enterprise_customer)\n enterprise_catalog_uuid = request.GET.get(\n 'catalog'\n ) if request.method == 'GET' else None\n html_template_for_rendering = ENTERPRISE_GENERAL_ERROR_PAGE\n if course and course_run:\n course_enrollable = True\n course_start_date = ''\n course_in_future = False\n organization_name = ''\n organization_logo = ''\n expected_learning_items = course['expected_learning_items']\n # Parse organization name and logo.\n if course['owners']:\n # The owners key contains the organizations associated with the course.\n # We pick the first one in the list here to meet UX requirements.\n organization = course['owners'][0]\n organization_name = organization['name']\n organization_logo = organization['logo_image_url']\n\n course_title = course_run['title']\n course_short_description = course_run['short_description'] or ''\n course_full_description = clean_html_for_template_rendering(course_run['full_description'] or '')\n course_pacing = self.PACING_FORMAT.get(course_run['pacing_type'], '')\n if course_run['start']:\n course_start_date = parse(course_run['start']).strftime('%B %d, %Y')\n now = datetime.datetime.now(pytz.UTC)\n course_in_future = parse(course_run['start']) > now\n\n course_level_type = course_run.get('level_type', '')\n staff = course_run['staff']\n # Format the course effort string using the min/max effort fields for the course run.\n course_effort = ungettext_min_max(\n '{} hour per week',\n '{} hours per week',\n '{}-{} hours per week',\n course_run['min_effort'] or None,\n course_run['max_effort'] or None,\n ) or ''\n\n # Parse course run image.\n course_run_image = course_run['image'] or {}\n course_image_uri = course_run_image.get('src', '')\n\n # Retrieve the enterprise-discounted price from ecommerce.\n course_modes = self.set_final_prices(course_modes, request)\n premium_modes = [mode for mode in course_modes if mode['premium']]\n\n # Filter audit course modes.\n course_modes = filter_audit_course_modes(enterprise_customer, course_modes)\n\n # Allows automatic assignment to a cohort upon enrollment.\n cohort = request.GET.get('cohort')\n # Add a message to the message display queue if the learner\n # has gone through the data sharing consent flow and declined\n # to give data sharing consent.\n if enterprise_course_enrollment and not data_sharing_consent.granted:\n messages.add_consent_declined_message(request, enterprise_customer, course_run.get('title', ''))\n\n if not is_course_run_enrollable(course_run):\n messages.add_unenrollable_item_message(request, 'course')\n course_enrollable = False\n context_data.update({\n 'course_enrollable': course_enrollable,\n 'course_title': course_title,\n 'course_short_description': course_short_description,\n 'course_pacing': course_pacing,\n 'course_start_date': course_start_date,\n 'course_in_future': course_in_future,\n 'course_image_uri': course_image_uri,\n 'course_modes': course_modes,\n 'course_effort': course_effort,\n 'course_full_description': course_full_description,\n 'cohort': cohort,\n 'organization_logo': organization_logo,\n 'organization_name': organization_name,\n 'course_level_type': course_level_type,\n 'premium_modes': premium_modes,\n 'expected_learning_items': expected_learning_items,\n 'catalog': enterprise_catalog_uuid,\n 'staff': staff,\n 'discount_text': _('Discount provided by {strong_start}{enterprise_customer_name}{strong_end}').format(\n enterprise_customer_name=enterprise_customer.name,\n strong_start='',\n strong_end='',\n ),\n 'hide_course_original_price': enterprise_customer.hide_course_original_price\n })\n html_template_for_rendering = 'enterprise/enterprise_course_enrollment_page.html'\n\n context_data.update({\n 'page_title': _('Confirm your course'),\n 'confirmation_text': _('Confirm your course'),\n 'starts_at_text': _('Starts'),\n 'view_course_details_text': _('View Course Details'),\n 'select_mode_text': _('Please select one:'),\n 'price_text': _('Price'),\n 'continue_link_text': _('Continue'),\n 'level_text': _('Level'),\n 'effort_text': _('Effort'),\n 'close_modal_button_text': _('Close'),\n 'expected_learning_items_text': _(\"What you'll learn\"),\n 'course_full_description_text': _('About This Course'),\n 'staff_text': _('Course Staff'),\n })\n return render(request, html_template_for_rendering, context=context_data)", "response": "Render enterprise - specific course enrollment page."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef post(self, request, enterprise_uuid, course_id):\n enterprise_customer, course, course_run, course_modes = self.get_base_details(\n request, enterprise_uuid, course_id\n )\n\n # Create a link between the user and the enterprise customer if it does not already exist.\n enterprise_customer_user, __ = EnterpriseCustomerUser.objects.get_or_create(\n enterprise_customer=enterprise_customer,\n user_id=request.user.id\n )\n enterprise_customer_user.update_session(request)\n\n data_sharing_consent = DataSharingConsent.objects.proxied_get(\n username=enterprise_customer_user.username,\n course_id=course_id,\n enterprise_customer=enterprise_customer\n )\n\n try:\n enterprise_course_enrollment = EnterpriseCourseEnrollment.objects.get(\n enterprise_customer_user__enterprise_customer=enterprise_customer,\n enterprise_customer_user__user_id=request.user.id,\n course_id=course_id\n )\n except EnterpriseCourseEnrollment.DoesNotExist:\n enterprise_course_enrollment = None\n\n enterprise_catalog_uuid = request.POST.get('catalog')\n selected_course_mode_name = request.POST.get('course_mode')\n cohort_name = request.POST.get('cohort')\n\n selected_course_mode = None\n for course_mode in course_modes:\n if course_mode['mode'] == selected_course_mode_name:\n selected_course_mode = course_mode\n break\n\n if not selected_course_mode:\n return self.get_enterprise_course_enrollment_page(\n request,\n enterprise_customer,\n course,\n course_run,\n course_modes,\n enterprise_course_enrollment,\n data_sharing_consent\n )\n\n user_consent_needed = get_data_sharing_consent(\n enterprise_customer_user.username,\n enterprise_customer.uuid,\n course_id=course_id\n ).consent_required()\n if not selected_course_mode.get('premium') and not user_consent_needed:\n # For the audit course modes (audit, honor), where DSC is not\n # required, enroll the learner directly through enrollment API\n # client and redirect the learner to LMS courseware page.\n if not enterprise_course_enrollment:\n # Create the Enterprise backend database records for this course enrollment.\n enterprise_course_enrollment = EnterpriseCourseEnrollment.objects.create(\n enterprise_customer_user=enterprise_customer_user,\n course_id=course_id,\n )\n track_enrollment('course-landing-page-enrollment', request.user.id, course_id, request.get_full_path())\n\n client = EnrollmentApiClient()\n client.enroll_user_in_course(\n request.user.username,\n course_id,\n selected_course_mode_name,\n cohort=cohort_name\n )\n\n return redirect(LMS_COURSEWARE_URL.format(course_id=course_id))\n\n if user_consent_needed:\n # For the audit course modes (audit, honor) or for the premium\n # course modes (Verified, Prof Ed) where DSC is required, redirect\n # the learner to course specific DSC with enterprise UUID from\n # there the learner will be directed to the ecommerce flow after\n # providing DSC.\n query_string_params = {\n 'course_mode': selected_course_mode_name,\n }\n if enterprise_catalog_uuid:\n query_string_params.update({'catalog': enterprise_catalog_uuid})\n\n next_url = '{handle_consent_enrollment_url}?{query_string}'.format(\n handle_consent_enrollment_url=reverse(\n 'enterprise_handle_consent_enrollment', args=[enterprise_customer.uuid, course_id]\n ),\n query_string=urlencode(query_string_params)\n )\n\n failure_url = reverse('enterprise_course_run_enrollment_page', args=[enterprise_customer.uuid, course_id])\n if request.META['QUERY_STRING']:\n # Preserve all querystring parameters in the request to build\n # failure url, so that learner views the same enterprise course\n # enrollment page (after redirect) as for the first time.\n # Since this is a POST view so use `request.META` to get\n # querystring instead of `request.GET`.\n # https://docs.djangoproject.com/en/1.11/ref/request-response/#django.http.HttpRequest.META\n failure_url = '{course_enrollment_url}?{query_string}'.format(\n course_enrollment_url=reverse(\n 'enterprise_course_run_enrollment_page', args=[enterprise_customer.uuid, course_id]\n ),\n query_string=request.META['QUERY_STRING']\n )\n\n return redirect(\n '{grant_data_sharing_url}?{params}'.format(\n grant_data_sharing_url=reverse('grant_data_sharing_permissions'),\n params=urlencode(\n {\n 'next': next_url,\n 'failure_url': failure_url,\n 'enterprise_customer_uuid': enterprise_customer.uuid,\n 'course_id': course_id,\n }\n )\n )\n )\n\n # For the premium course modes (Verified, Prof Ed) where DSC is\n # not required, redirect the enterprise learner to the ecommerce\n # flow in LMS.\n # Note: LMS start flow automatically detects the paid mode\n premium_flow = LMS_START_PREMIUM_COURSE_FLOW_URL.format(course_id=course_id)\n if enterprise_catalog_uuid:\n premium_flow += '?catalog={catalog_uuid}'.format(\n catalog_uuid=enterprise_catalog_uuid\n )\n\n return redirect(premium_flow)", "response": "Process a submitted track selection form for the enterprise."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the details of a specific course.", "response": "def get(self, request, enterprise_uuid, course_id):\n \"\"\"\n Show course track selection page for the enterprise.\n\n Based on `enterprise_uuid` in URL, the view will decide which\n enterprise customer's course enrollment page is to use.\n\n Unauthenticated learners will be redirected to enterprise-linked SSO.\n\n A 404 will be raised if any of the following conditions are met:\n * No enterprise customer uuid kwarg `enterprise_uuid` in request.\n * No enterprise customer found against the enterprise customer\n uuid `enterprise_uuid` in the request kwargs.\n * No course is found in database against the provided `course_id`.\n\n \"\"\"\n # Check to see if access to the course run is restricted for this user.\n embargo_url = EmbargoApiClient.redirect_if_blocked([course_id], request.user, get_ip(request), request.path)\n if embargo_url:\n return redirect(embargo_url)\n\n enterprise_customer, course, course_run, modes = self.get_base_details(\n request, enterprise_uuid, course_id\n )\n enterprise_customer_user = get_enterprise_customer_user(request.user.id, enterprise_uuid)\n data_sharing_consent = DataSharingConsent.objects.proxied_get(\n username=enterprise_customer_user.username,\n course_id=course_id,\n enterprise_customer=enterprise_customer\n )\n\n enrollment_client = EnrollmentApiClient()\n enrolled_course = enrollment_client.get_course_enrollment(request.user.username, course_id)\n try:\n enterprise_course_enrollment = EnterpriseCourseEnrollment.objects.get(\n enterprise_customer_user__enterprise_customer=enterprise_customer,\n enterprise_customer_user__user_id=request.user.id,\n course_id=course_id\n )\n except EnterpriseCourseEnrollment.DoesNotExist:\n enterprise_course_enrollment = None\n\n if enrolled_course and enterprise_course_enrollment:\n # The user is already enrolled in the course through the Enterprise Customer, so redirect to the course\n # info page.\n return redirect(LMS_COURSEWARE_URL.format(course_id=course_id))\n\n return self.get_enterprise_course_enrollment_page(\n request,\n enterprise_customer,\n course,\n course_run,\n modes,\n enterprise_course_enrollment,\n data_sharing_consent,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef extend_course(course, enterprise_customer, request):\n course_run_id = course['course_runs'][0]['key']\n try:\n catalog_api_client = CourseCatalogApiServiceClient(enterprise_customer.site)\n except ImproperlyConfigured:\n error_code = 'ENTPEV000'\n LOGGER.error(\n 'CourseCatalogApiServiceClient is improperly configured. '\n 'Returned error code {error_code} to user {userid} '\n 'and enterprise_customer {enterprise_customer} '\n 'for course_run_id {course_run_id}'.format(\n error_code=error_code,\n userid=request.user.id,\n enterprise_customer=enterprise_customer.uuid,\n course_run_id=course_run_id,\n )\n )\n messages.add_generic_error_message_with_code(request, error_code)\n return ({}, error_code)\n\n course_details, course_run_details = catalog_api_client.get_course_and_course_run(course_run_id)\n if not course_details or not course_run_details:\n error_code = 'ENTPEV001'\n LOGGER.error(\n 'User {userid} of enterprise customer {enterprise_customer} encountered an error.'\n 'No course_details or course_run_details found for '\n 'course_run_id {course_run_id}. '\n 'The following error code reported to the user: {error_code}'.format(\n userid=request.user.id,\n enterprise_customer=enterprise_customer.uuid,\n course_run_id=course_run_id,\n error_code=error_code,\n )\n )\n messages.add_generic_error_message_with_code(request, error_code)\n return ({}, error_code)\n\n weeks_to_complete = course_run_details['weeks_to_complete']\n course_run_image = course_run_details['image'] or {}\n course.update({\n 'course_image_uri': course_run_image.get('src', ''),\n 'course_title': course_run_details['title'],\n 'course_level_type': course_run_details.get('level_type', ''),\n 'course_short_description': course_run_details['short_description'] or '',\n 'course_full_description': clean_html_for_template_rendering(course_run_details['full_description'] or ''),\n 'expected_learning_items': course_details.get('expected_learning_items', []),\n 'staff': course_run_details.get('staff', []),\n 'course_effort': ungettext_min_max(\n '{} hour per week',\n '{} hours per week',\n '{}-{} hours per week',\n course_run_details['min_effort'] or None,\n course_run_details['max_effort'] or None,\n ) or '',\n 'weeks_to_complete': ungettext(\n '{} week',\n '{} weeks',\n weeks_to_complete\n ).format(weeks_to_complete) if weeks_to_complete else '',\n })\n return course, None", "response": "Extend a course with more details needed for the program landing page."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_program_details(self, request, program_uuid, enterprise_customer):\n try:\n course_catalog_api_client = CourseCatalogApiServiceClient(enterprise_customer.site)\n except ImproperlyConfigured:\n error_code = 'ENTPEV002'\n LOGGER.error(\n 'CourseCatalogApiServiceClient is improperly configured. '\n 'Returned error code {error_code} to user {userid} '\n 'and enterprise_customer {enterprise_customer} '\n 'for program {program_uuid}'.format(\n error_code=error_code,\n userid=request.user.id,\n enterprise_customer=enterprise_customer.uuid,\n program_uuid=program_uuid,\n )\n )\n messages.add_generic_error_message_with_code(request, error_code)\n return ({}, error_code)\n\n program_details = course_catalog_api_client.get_program_by_uuid(program_uuid)\n if program_details is None:\n error_code = 'ENTPEV003'\n LOGGER.error(\n 'User {userid} of enterprise customer {enterprise_customer} encountered an error. '\n 'program_details is None for program_uuid {program_uuid}. '\n 'Returned error code {error_code} to user'.format(\n userid=request.user.id,\n enterprise_customer=enterprise_customer.uuid,\n program_uuid=program_uuid,\n error_code=error_code,\n )\n )\n messages.add_generic_error_message_with_code(request, error_code)\n return ({}, error_code)\n\n program_type = course_catalog_api_client.get_program_type_by_slug(slugify(program_details['type']))\n if program_type is None:\n error_code = 'ENTPEV004'\n LOGGER.error(\n 'User {userid} of enterprise customer {enterprise_customer} encountered an error. '\n 'program_type is None for program_details of program_uuid {program_uuid}. '\n 'Returned error code {error_code} to user'.format(\n userid=request.user.id,\n enterprise_customer=enterprise_customer.uuid,\n program_uuid=program_uuid,\n error_code=error_code,\n )\n )\n messages.add_generic_error_message_with_code(request, error_code)\n return ({}, error_code)\n\n # Extend our program details with context we'll need for display or for deciding redirects.\n program_details = ProgramDataExtender(program_details, request.user).extend()\n\n # TODO: Upstream this additional context to the platform's `ProgramDataExtender` so we can avoid this here.\n program_details['enrolled_in_program'] = False\n enrollment_count = 0\n for extended_course in program_details['courses']:\n # We need to extend our course data further for modals and other displays.\n extended_data, error_code = ProgramEnrollmentView.extend_course(\n extended_course,\n enterprise_customer,\n request\n )\n\n if error_code:\n return ({}, error_code)\n\n extended_course.update(extended_data)\n # We're enrolled in the program if we have certificate-eligible enrollment in even 1 of its courses.\n extended_course_run = extended_course['course_runs'][0]\n if extended_course_run['is_enrolled'] and extended_course_run['upgrade_url'] is None:\n program_details['enrolled_in_program'] = True\n enrollment_count += 1\n\n # We're certificate eligible for the program if we have certificate-eligible enrollment in all of its courses.\n program_details['certificate_eligible_for_program'] = (enrollment_count == len(program_details['courses']))\n program_details['type_details'] = program_type\n return program_details, None", "response": "Get fundamental details for a program."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrenders the Enterprise - specific program enrollment page.", "response": "def get_enterprise_program_enrollment_page(self, request, enterprise_customer, program_details):\n \"\"\"\n Render Enterprise-specific program enrollment page.\n \"\"\"\n # Safely make the assumption that we can use the first authoring organization.\n organizations = program_details['authoring_organizations']\n organization = organizations[0] if organizations else {}\n platform_name = get_configuration_value('PLATFORM_NAME', settings.PLATFORM_NAME)\n program_title = program_details['title']\n program_type_details = program_details['type_details']\n program_type = program_type_details['name']\n\n # Make any modifications for singular/plural-dependent text.\n program_courses = program_details['courses']\n course_count = len(program_courses)\n course_count_text = ungettext(\n '{count} Course',\n '{count} Courses',\n course_count,\n ).format(count=course_count)\n effort_info_text = ungettext_min_max(\n '{} hour per week, per course',\n '{} hours per week, per course',\n _('{}-{} hours per week, per course'),\n program_details.get('min_hours_effort_per_week'),\n program_details.get('max_hours_effort_per_week'),\n )\n length_info_text = ungettext_min_max(\n '{} week per course',\n '{} weeks per course',\n _('{}-{} weeks per course'),\n program_details.get('weeks_to_complete_min'),\n program_details.get('weeks_to_complete_max'),\n )\n\n # Update some enrollment-related text requirements.\n if program_details['enrolled_in_program']:\n purchase_action = _('Purchase all unenrolled courses')\n item = _('enrollment')\n else:\n purchase_action = _('Pursue the program')\n item = _('program enrollment')\n\n # Add any DSC warning messages.\n program_data_sharing_consent = get_data_sharing_consent(\n request.user.username,\n enterprise_customer.uuid,\n program_uuid=program_details['uuid'],\n )\n if program_data_sharing_consent.exists and not program_data_sharing_consent.granted:\n messages.add_consent_declined_message(request, enterprise_customer, program_title)\n\n discount_data = program_details.get('discount_data', {})\n one_click_purchase_eligibility = program_details.get('is_learner_eligible_for_one_click_purchase', False)\n # The following messages shouldn't both appear at the same time, and we prefer the eligibility message.\n if not one_click_purchase_eligibility:\n messages.add_unenrollable_item_message(request, 'program')\n elif discount_data.get('total_incl_tax_excl_discounts') is None:\n messages.add_missing_price_information_message(request, program_title)\n\n context_data = get_global_context(request, enterprise_customer)\n context_data.update({\n 'enrolled_in_course_and_paid_text': _('enrolled'),\n 'enrolled_in_course_and_unpaid_text': _('already enrolled, must pay for certificate'),\n 'expected_learning_items_text': _(\"What you'll learn\"),\n 'expected_learning_items_show_count': 2,\n 'corporate_endorsements_text': _('Real Career Impact'),\n 'corporate_endorsements_show_count': 1,\n 'see_more_text': _('See More'),\n 'see_less_text': _('See Less'),\n 'confirm_button_text': _('Confirm Program'),\n 'summary_header': _('Program Summary'),\n 'price_text': _('Price'),\n 'length_text': _('Length'),\n 'effort_text': _('Effort'),\n 'level_text': _('Level'),\n 'course_full_description_text': _('About This Course'),\n 'staff_text': _('Course Staff'),\n 'close_modal_button_text': _('Close'),\n 'program_not_eligible_for_one_click_purchase_text': _('Program not eligible for one-click purchase.'),\n 'program_type_description_header': _('What is an {platform_name} {program_type}?').format(\n platform_name=platform_name,\n program_type=program_type,\n ),\n 'platform_description_header': _('What is {platform_name}?').format(\n platform_name=platform_name\n ),\n 'organization_name': organization.get('name'),\n 'organization_logo': organization.get('logo_image_url'),\n 'organization_text': _('Presented by {organization}').format(organization=organization.get('name')),\n 'page_title': _('Confirm your {item}').format(item=item),\n 'program_type_logo': program_type_details['logo_image'].get('medium', {}).get('url', ''),\n 'program_type': program_type,\n 'program_type_description': get_program_type_description(program_type),\n 'program_title': program_title,\n 'program_subtitle': program_details['subtitle'],\n 'program_overview': program_details['overview'],\n 'program_price': get_price_text(discount_data.get('total_incl_tax_excl_discounts', 0), request),\n 'program_discounted_price': get_price_text(discount_data.get('total_incl_tax', 0), request),\n 'is_discounted': discount_data.get('is_discounted', False),\n 'courses': program_courses,\n 'item_bullet_points': [\n _('Credit- and Certificate-eligible'),\n _('Self-paced; courses can be taken in any order'),\n ],\n 'purchase_text': _('{purchase_action} for').format(purchase_action=purchase_action),\n 'expected_learning_items': program_details['expected_learning_items'],\n 'corporate_endorsements': program_details['corporate_endorsements'],\n 'course_count_text': course_count_text,\n 'length_info_text': length_info_text,\n 'effort_info_text': effort_info_text,\n 'is_learner_eligible_for_one_click_purchase': one_click_purchase_eligibility,\n })\n return render(request, 'enterprise/enterprise_program_enrollment_page.html', context=context_data)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the Enterprise s Program and its associated Entry Point.", "response": "def get(self, request, enterprise_uuid, program_uuid):\n \"\"\"\n Show Program Landing page for the Enterprise's Program.\n\n Render the Enterprise's Program Enrollment page for a specific program.\n The Enterprise and Program are both selected by their respective UUIDs.\n\n Unauthenticated learners will be redirected to enterprise-linked SSO.\n\n A 404 will be raised if any of the following conditions are met:\n * No enterprise customer UUID query parameter ``enterprise_uuid`` found in request.\n * No enterprise customer found against the enterprise customer\n uuid ``enterprise_uuid`` in the request kwargs.\n * No Program can be found given ``program_uuid`` either at all or associated with\n the Enterprise..\n \"\"\"\n verify_edx_resources()\n\n enterprise_customer = get_enterprise_customer_or_404(enterprise_uuid)\n context_data = get_global_context(request, enterprise_customer)\n program_details, error_code = self.get_program_details(request, program_uuid, enterprise_customer)\n if error_code:\n return render(\n request,\n ENTERPRISE_GENERAL_ERROR_PAGE,\n context=context_data,\n status=404,\n )\n if program_details['certificate_eligible_for_program']:\n # The user is already enrolled in the program, so redirect to the program's dashboard.\n return redirect(LMS_PROGRAMS_DASHBOARD_URL.format(uuid=program_uuid))\n\n # Check to see if access to any of the course runs in the program are restricted for this user.\n course_run_ids = []\n for course in program_details['courses']:\n for course_run in course['course_runs']:\n course_run_ids.append(course_run['key'])\n embargo_url = EmbargoApiClient.redirect_if_blocked(course_run_ids, request.user, get_ip(request), request.path)\n if embargo_url:\n return redirect(embargo_url)\n\n return self.get_enterprise_program_enrollment_page(request, enterprise_customer, program_details)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprocess a submitted track selection form for the enterprise.", "response": "def post(self, request, enterprise_uuid, program_uuid):\n \"\"\"\n Process a submitted track selection form for the enterprise.\n \"\"\"\n verify_edx_resources()\n\n # Create a link between the user and the enterprise customer if it does not already exist.\n enterprise_customer = get_enterprise_customer_or_404(enterprise_uuid)\n with transaction.atomic():\n enterprise_customer_user, __ = EnterpriseCustomerUser.objects.get_or_create(\n enterprise_customer=enterprise_customer,\n user_id=request.user.id\n )\n enterprise_customer_user.update_session(request)\n\n context_data = get_global_context(request, enterprise_customer)\n program_details, error_code = self.get_program_details(request, program_uuid, enterprise_customer)\n if error_code:\n return render(\n request,\n ENTERPRISE_GENERAL_ERROR_PAGE,\n context=context_data,\n status=404,\n )\n if program_details['certificate_eligible_for_program']:\n # The user is already enrolled in the program, so redirect to the program's dashboard.\n return redirect(LMS_PROGRAMS_DASHBOARD_URL.format(uuid=program_uuid))\n\n basket_page = '{basket_url}?{params}'.format(\n basket_url=BASKET_URL,\n params=urlencode(\n [tuple(['sku', sku]) for sku in program_details['skus']] +\n [tuple(['bundle', program_uuid])]\n )\n )\n if get_data_sharing_consent(\n enterprise_customer_user.username,\n enterprise_customer.uuid,\n program_uuid=program_uuid,\n ).consent_required():\n return redirect(\n '{grant_data_sharing_url}?{params}'.format(\n grant_data_sharing_url=reverse('grant_data_sharing_permissions'),\n params=urlencode(\n {\n 'next': basket_page,\n 'failure_url': reverse(\n 'enterprise_program_enrollment_page',\n args=[enterprise_customer.uuid, program_uuid]\n ),\n 'enterprise_customer_uuid': enterprise_customer.uuid,\n 'program_uuid': program_uuid,\n }\n )\n )\n )\n\n return redirect(basket_page)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the base variables for any view to route to.", "response": "def get_path_variables(**kwargs):\n \"\"\"\n Get the base variables for any view to route to.\n\n Currently gets:\n - `enterprise_uuid` - the UUID of the enterprise customer.\n - `course_run_id` - the ID of the course, if applicable.\n - `program_uuid` - the UUID of the program, if applicable.\n \"\"\"\n enterprise_customer_uuid = kwargs.get('enterprise_uuid', '')\n course_run_id = kwargs.get('course_id', '')\n course_key = kwargs.get('course_key', '')\n program_uuid = kwargs.get('program_uuid', '')\n\n return enterprise_customer_uuid, course_run_id, course_key, program_uuid"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the course run ID for a user", "response": "def get_course_run_id(user, enterprise_customer, course_key):\n \"\"\"\n User is requesting a course, we need to translate that into the current course run.\n\n :param user:\n :param enterprise_customer:\n :param course_key:\n :return: course_run_id\n \"\"\"\n try:\n course = CourseCatalogApiServiceClient(enterprise_customer.site).get_course_details(course_key)\n except ImproperlyConfigured:\n raise Http404\n\n users_all_enrolled_courses = EnrollmentApiClient().get_enrolled_courses(user.username)\n users_active_course_runs = get_active_course_runs(\n course,\n users_all_enrolled_courses\n ) if users_all_enrolled_courses else []\n course_run = get_current_course_run(course, users_active_course_runs)\n if course_run:\n course_run_id = course_run['key']\n return course_run_id\n else:\n raise Http404"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef eligible_for_direct_audit_enrollment(self, request, enterprise_customer, resource_id, course_key=None):\n course_identifier = course_key if course_key else resource_id\n\n # Return it in one big statement to utilize short-circuiting behavior. Avoid the API call if possible.\n return request.GET.get('audit') and \\\n request.path == self.COURSE_ENROLLMENT_VIEW_URL.format(enterprise_customer.uuid, course_identifier) and \\\n enterprise_customer.catalog_contains_course(resource_id) and \\\n EnrollmentApiClient().has_course_mode(resource_id, 'audit')", "response": "Returns whether a request is eligible for direct audit enrollment for a particular enterprise customer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef redirect(self, request, *args, **kwargs):\n enterprise_customer_uuid, course_run_id, course_key, program_uuid = RouterView.get_path_variables(**kwargs)\n resource_id = course_key or course_run_id or program_uuid\n # Replace enterprise UUID and resource ID with '{}', to easily match with a path in RouterView.VIEWS. Example:\n # /enterprise/fake-uuid/course/course-v1:cool+course+2017/enroll/ -> /enterprise/{}/course/{}/enroll/\n path = re.sub('{}|{}'.format(enterprise_customer_uuid, re.escape(resource_id)), '{}', request.path)\n\n # Remove course_key from kwargs if it exists because delegate views are not expecting it.\n kwargs.pop('course_key', None)\n\n return self.VIEWS[path].as_view()(request, *args, **kwargs)", "response": "Redirects to the appropriate view depending on where the user came from."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nruns some custom GET logic for Enterprise workflows before routing the user through existing views. In particular, before routing to existing views: - If the requested resource is a course, find the current course run for that course, and make that course run the requested resource instead. - Look to see whether a request is eligible for direct audit enrollment, and if so, directly enroll the user.", "response": "def get(self, request, *args, **kwargs):\n \"\"\"\n Run some custom GET logic for Enterprise workflows before routing the user through existing views.\n\n In particular, before routing to existing views:\n - If the requested resource is a course, find the current course run for that course,\n and make that course run the requested resource instead.\n - Look to see whether a request is eligible for direct audit enrollment, and if so, directly enroll the user.\n \"\"\"\n enterprise_customer_uuid, course_run_id, course_key, program_uuid = RouterView.get_path_variables(**kwargs)\n enterprise_customer = get_enterprise_customer_or_404(enterprise_customer_uuid)\n if course_key:\n try:\n course_run_id = RouterView.get_course_run_id(request.user, enterprise_customer, course_key)\n except Http404:\n context_data = get_global_context(request, enterprise_customer)\n error_code = 'ENTRV000'\n log_message = (\n 'Could not find course run with id {course_run_id} '\n 'for course key {course_key} and program_uuid {program_uuid} '\n 'for enterprise_customer_uuid {enterprise_customer_uuid} '\n 'Returned error code {error_code} to user {userid}'.format(\n course_key=course_key,\n course_run_id=course_run_id,\n enterprise_customer_uuid=enterprise_customer_uuid,\n error_code=error_code,\n userid=request.user.id,\n program_uuid=program_uuid,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n kwargs['course_id'] = course_run_id\n\n # Ensure that the link is saved to the database prior to making some call in a downstream view\n # which may need to know that the user belongs to an enterprise customer.\n with transaction.atomic():\n enterprise_customer_user, __ = EnterpriseCustomerUser.objects.get_or_create(\n enterprise_customer=enterprise_customer,\n user_id=request.user.id\n )\n enterprise_customer_user.update_session(request)\n\n # Directly enroll in audit mode if the request in question has full direct audit enrollment eligibility.\n resource_id = course_run_id or program_uuid\n if self.eligible_for_direct_audit_enrollment(request, enterprise_customer, resource_id, course_key):\n try:\n enterprise_customer_user.enroll(resource_id, 'audit', cohort=request.GET.get('cohort', None))\n track_enrollment('direct-audit-enrollment', request.user.id, resource_id, request.get_full_path())\n except (CourseEnrollmentDowngradeError, CourseEnrollmentPermissionError):\n pass\n # The courseware view logic will check for DSC requirements, and route to the DSC page if necessary.\n return redirect(LMS_COURSEWARE_URL.format(course_id=resource_id))\n\n return self.redirect(request, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrun some custom POST logic for Enterprise workflows before routing the user through existing views.", "response": "def post(self, request, *args, **kwargs):\n \"\"\"\n Run some custom POST logic for Enterprise workflows before routing the user through existing views.\n \"\"\"\n # pylint: disable=unused-variable\n enterprise_customer_uuid, course_run_id, course_key, program_uuid = RouterView.get_path_variables(**kwargs)\n enterprise_customer = get_enterprise_customer_or_404(enterprise_customer_uuid)\n\n if course_key:\n context_data = get_global_context(request, enterprise_customer)\n try:\n kwargs['course_id'] = RouterView.get_course_run_id(request.user, enterprise_customer, course_key)\n except Http404:\n error_code = 'ENTRV001'\n log_message = (\n 'Could not find course run with id {course_run_id} '\n 'for course key {course_key} and '\n 'for enterprise_customer_uuid {enterprise_customer_uuid} '\n 'and program {program_uuid}. '\n 'Returned error code {error_code} to user {userid}'.format(\n course_key=course_key,\n course_run_id=course_run_id,\n enterprise_customer_uuid=enterprise_customer_uuid,\n error_code=error_code,\n userid=request.user.id,\n program_uuid=program_uuid,\n )\n )\n return render_page_with_error_code_message(request, context_data, error_code, log_message)\n\n return self.redirect(request, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntasking to send content metadata to each linked integrated channel. Arguments: username (str): The username of the User to be used for making API requests to retrieve content metadata. channel_code (str): Capitalized identifier for the integrated channel. channel_pk (str): Primary key for identifying integrated channel.", "response": "def transmit_content_metadata(username, channel_code, channel_pk):\n \"\"\"\n Task to send content metadata to each linked integrated channel.\n\n Arguments:\n username (str): The username of the User to be used for making API requests to retrieve content metadata.\n channel_code (str): Capitalized identifier for the integrated channel.\n channel_pk (str): Primary key for identifying integrated channel.\n\n \"\"\"\n start = time.time()\n api_user = User.objects.get(username=username)\n integrated_channel = INTEGRATED_CHANNEL_CHOICES[channel_code].objects.get(pk=channel_pk)\n LOGGER.info('Transmitting content metadata to integrated channel using configuration: [%s]', integrated_channel)\n try:\n integrated_channel.transmit_content_metadata(api_user)\n except Exception: # pylint: disable=broad-except\n LOGGER.exception(\n 'Transmission of content metadata failed for user [%s] and for integrated '\n 'channel with code [%s] and id [%s].', username, channel_code, channel_pk\n )\n duration = time.time() - start\n LOGGER.info(\n 'Content metadata transmission task for integrated channel configuration [%s] took [%s] seconds',\n integrated_channel,\n duration\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntasks to send learner data to each linked integrated channel. Arguments: username (str): The username of the User to be used for making API requests for learner data. channel_code (str): Capitalized identifier for the integrated channel channel_pk (str): Primary key for identifying integrated channel", "response": "def transmit_learner_data(username, channel_code, channel_pk):\n \"\"\"\n Task to send learner data to each linked integrated channel.\n\n Arguments:\n username (str): The username of the User to be used for making API requests for learner data.\n channel_code (str): Capitalized identifier for the integrated channel\n channel_pk (str): Primary key for identifying integrated channel\n\n \"\"\"\n start = time.time()\n api_user = User.objects.get(username=username)\n integrated_channel = INTEGRATED_CHANNEL_CHOICES[channel_code].objects.get(pk=channel_pk)\n LOGGER.info('Processing learners for integrated channel using configuration: [%s]', integrated_channel)\n\n # Note: learner data transmission code paths don't raise any uncaught exception, so we don't need a broad\n # try-except block here.\n integrated_channel.transmit_learner_data(api_user)\n\n duration = time.time() - start\n LOGGER.info(\n 'Learner data transmission task for integrated channel configuration [%s] took [%s] seconds',\n integrated_channel,\n duration\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntasks to unlink inactive learners of provided integrated channel. Arguments: channel_code (str): Capitalized identifier for the integrated channel channel_pk (str): Primary key for identifying integrated channel", "response": "def unlink_inactive_learners(channel_code, channel_pk):\n \"\"\"\n Task to unlink inactive learners of provided integrated channel.\n\n Arguments:\n channel_code (str): Capitalized identifier for the integrated channel\n channel_pk (str): Primary key for identifying integrated channel\n\n \"\"\"\n start = time.time()\n integrated_channel = INTEGRATED_CHANNEL_CHOICES[channel_code].objects.get(pk=channel_pk)\n LOGGER.info('Processing learners to unlink inactive users using configuration: [%s]', integrated_channel)\n\n # Note: learner data transmission code paths don't raise any uncaught exception, so we don't need a broad\n # try-except block here.\n integrated_channel.unlink_inactive_learners()\n\n duration = time.time() - start\n LOGGER.info(\n 'Unlink inactive learners task for integrated channel configuration [%s] took [%s] seconds',\n integrated_channel,\n duration\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef handle_user_post_save(sender, **kwargs): # pylint: disable=unused-argument\n created = kwargs.get(\"created\", False)\n user_instance = kwargs.get(\"instance\", None)\n\n if user_instance is None:\n return # should never happen, but better safe than 500 error\n\n try:\n pending_ecu = PendingEnterpriseCustomerUser.objects.get(user_email=user_instance.email)\n except PendingEnterpriseCustomerUser.DoesNotExist:\n return # nothing to do in this case\n\n if not created:\n # existing user changed his email to match one of pending link records - try linking him to EC\n try:\n existing_record = EnterpriseCustomerUser.objects.get(user_id=user_instance.id)\n message_template = \"User {user} have changed email to match pending Enterprise Customer link, \" \\\n \"but was already linked to Enterprise Customer {enterprise_customer} - \" \\\n \"deleting pending link record\"\n logger.info(message_template.format(\n user=user_instance, enterprise_customer=existing_record.enterprise_customer\n ))\n pending_ecu.delete()\n return\n except EnterpriseCustomerUser.DoesNotExist:\n pass # everything ok - current user is not linked to other ECs\n\n enterprise_customer_user = EnterpriseCustomerUser.objects.create(\n enterprise_customer=pending_ecu.enterprise_customer,\n user_id=user_instance.id\n )\n pending_enrollments = list(pending_ecu.pendingenrollment_set.all())\n if pending_enrollments:\n def _complete_user_enrollment(): # pylint: disable=missing-docstring\n for enrollment in pending_enrollments:\n # EnterpriseCustomers may enroll users in courses before the users themselves\n # actually exist in the system; in such a case, the enrollment for each such\n # course is finalized when the user registers with the OpenEdX platform.\n enterprise_customer_user.enroll(\n enrollment.course_id, enrollment.course_mode, cohort=enrollment.cohort_name)\n track_enrollment('pending-admin-enrollment', user_instance.id, enrollment.course_id)\n pending_ecu.delete()\n transaction.on_commit(_complete_user_enrollment)\n else:\n pending_ecu.delete()", "response": "Handle User model changes - checks if the user already linked to an existing user and upgrades it to actual link."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef default_content_filter(sender, instance, **kwargs): # pylint: disable=unused-argument\n if kwargs['created'] and not instance.content_filter:\n instance.content_filter = get_default_catalog_content_filter()\n instance.save()", "response": "Set default value for EnterpriseCustomerCatalog. content_filter"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nassign an enterprise learner role to EnterpriseCustomerUser when a new record is created.", "response": "def assign_enterprise_learner_role(sender, instance, **kwargs): # pylint: disable=unused-argument\n \"\"\"\n Assign an enterprise learner role to EnterpriseCustomerUser whenever a new record is created.\n \"\"\"\n if kwargs['created'] and instance.user:\n enterprise_learner_role, __ = SystemWideEnterpriseRole.objects.get_or_create(name=ENTERPRISE_LEARNER_ROLE)\n SystemWideEnterpriseUserRoleAssignment.objects.get_or_create(\n user=instance.user,\n role=enterprise_learner_role\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndelete the enterprise learner role assignment record when deleting an EnterpriseCustomerUser record.", "response": "def delete_enterprise_learner_role_assignment(sender, instance, **kwargs): # pylint: disable=unused-argument\n \"\"\"\n Delete the associated enterprise learner role assignment record when deleting an EnterpriseCustomerUser record.\n \"\"\"\n if instance.user:\n enterprise_learner_role, __ = SystemWideEnterpriseRole.objects.get_or_create(name=ENTERPRISE_LEARNER_ROLE)\n try:\n SystemWideEnterpriseUserRoleAssignment.objects.get(\n user=instance.user,\n role=enterprise_learner_role\n ).delete()\n except SystemWideEnterpriseUserRoleAssignment.DoesNotExist:\n # Do nothing if no role assignment is present for the enterprise customer user.\n pass"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef enterprise_customer_required(view):\n @wraps(view)\n def wrapper(request, *args, **kwargs):\n \"\"\"\n Checks for an enterprise customer associated with the user, calls the view function\n if one exists, raises PermissionDenied if not.\n \"\"\"\n user = request.user\n enterprise_customer = get_enterprise_customer_for_user(user)\n if enterprise_customer:\n args = args + (enterprise_customer,)\n return view(request, *args, **kwargs)\n else:\n raise PermissionDenied(\n 'User {username} is not associated with an EnterpriseCustomer.'.format(\n username=user.username\n )\n )\n return wrapper", "response": "Decorator that ensures that the user making the API request is associated with an EnterpriseCustomer."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef require_at_least_one_query_parameter(*query_parameter_names):\n def outer_wrapper(view):\n \"\"\" Allow the passing of parameters to require_at_least_one_query_parameter. \"\"\"\n @wraps(view)\n def wrapper(request, *args, **kwargs):\n \"\"\"\n Checks for the existence of the specified query parameters, raises a\n ValidationError if none of them were included in the request.\n \"\"\"\n requirement_satisfied = False\n for query_parameter_name in query_parameter_names:\n query_parameter_values = request.query_params.getlist(query_parameter_name)\n kwargs[query_parameter_name] = query_parameter_values\n if query_parameter_values:\n requirement_satisfied = True\n if not requirement_satisfied:\n raise ValidationError(\n detail='You must provide at least one of the following query parameters: {params}.'.format(\n params=', '.join(query_parameter_names)\n )\n )\n return view(request, *args, **kwargs)\n return wrapper\n return outer_wrapper", "response": "Decorator that ensures that at least one of the specified query parameters are included in the request."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntransmits the courseware data for the EnterpriseCustomer to the active integration channels.", "response": "def handle(self, *args, **options):\n \"\"\"\n Transmit the courseware data for the EnterpriseCustomer(s) to the active integration channels.\n \"\"\"\n username = options['catalog_user']\n\n # Before we do a whole bunch of database queries, make sure that the user we were passed exists.\n try:\n User.objects.get(username=username)\n except User.DoesNotExist:\n raise CommandError('A user with the username {} was not found.'.format(username))\n\n channels = self.get_integrated_channels(options)\n\n for channel in channels:\n channel_code = channel.channel_code()\n channel_pk = channel.pk\n transmit_content_metadata.delay(username, channel_code, channel_pk)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_roles(apps, schema_editor):\n SystemWideEnterpriseRole = apps.get_model('enterprise', 'SystemWideEnterpriseRole')\n SystemWideEnterpriseRole.objects.update_or_create(name=ENTERPRISE_ADMIN_ROLE)\n SystemWideEnterpriseRole.objects.update_or_create(name=ENTERPRISE_LEARNER_ROLE)", "response": "Create the enterprise roles if they do not already exist."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndeletes the enterprise roles.", "response": "def delete_roles(apps, schema_editor):\n \"\"\"Delete the enterprise roles.\"\"\"\n SystemWideEnterpriseRole = apps.get_model('enterprise', 'SystemWideEnterpriseRole')\n SystemWideEnterpriseRole.objects.filter(\n name__in=[ENTERPRISE_ADMIN_ROLE, ENTERPRISE_LEARNER_ROLE]\n ).delete()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a batched queryset of enterprise admin users from the given indexes.", "response": "def _get_enterprise_admin_users_batch(self, start, end):\n \"\"\"\n Returns a batched queryset of User objects.\n \"\"\"\n LOGGER.info('Fetching new batch of enterprise admin users from indexes: %s to %s', start, end)\n return User.objects.filter(groups__name=ENTERPRISE_DATA_API_ACCESS_GROUP, is_staff=False)[start:end]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_enterprise_operator_users_batch(self, start, end):\n LOGGER.info('Fetching new batch of enterprise operator users from indexes: %s to %s', start, end)\n return User.objects.filter(groups__name=ENTERPRISE_DATA_API_ACCESS_GROUP, is_staff=True)[start:end]", "response": "Returns a batched queryset of enterprise operator users from the given indexes."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a batched queryset of EnterpriseCustomerUser objects.", "response": "def _get_enterprise_customer_users_batch(self, start, end):\n \"\"\"\n Returns a batched queryset of EnterpriseCustomerUser objects.\n \"\"\"\n LOGGER.info('Fetching new batch of enterprise customer users from indexes: %s to %s', start, end)\n return User.objects.filter(pk__in=self._get_enterprise_customer_user_ids())[start:end]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a batched queryset of User objects from the given indexes.", "response": "def _get_enterprise_enrollment_api_admin_users_batch(self, start, end): # pylint: disable=invalid-name\n \"\"\"\n Returns a batched queryset of User objects.\n \"\"\"\n LOGGER.info('Fetching new batch of enterprise enrollment admin users from indexes: %s to %s', start, end)\n return User.objects.filter(groups__name=ENTERPRISE_ENROLLMENT_API_ACCESS_GROUP, is_staff=False)[start:end]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_enterprise_catalog_admin_users_batch(self, start, end):\n Application = apps.get_model(OAUTH2_PROVIDER_APPLICATION_MODEL) # pylint: disable=invalid-name\n LOGGER.info('Fetching new batch of enterprise catalog admin users from indexes: %s to %s', start, end)\n catalog_admin_user_ids = Application.objects.filter(\n user_id__in=self._get_enterprise_customer_user_ids()\n ).exclude(name=EDX_ORG_NAME).values('user_id')\n return User.objects.filter(pk__in=catalog_admin_user_ids)[start:end]", "response": "Returns a batched queryset of User objects."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _assign_enterprise_role_to_users(self, _get_batch_method, options, is_feature_role=False):\n role_name = options['role']\n batch_limit = options['batch_limit']\n batch_sleep = options['batch_sleep']\n batch_offset = options['batch_offset']\n\n current_batch_index = batch_offset\n\n users_batch = _get_batch_method(\n batch_offset,\n batch_offset + batch_limit\n )\n\n role_class = SystemWideEnterpriseRole\n role_assignment_class = SystemWideEnterpriseUserRoleAssignment\n\n if is_feature_role:\n role_class = EnterpriseFeatureRole\n role_assignment_class = EnterpriseFeatureUserRoleAssignment\n\n enterprise_role = role_class.objects.get(name=role_name)\n while users_batch.count() > 0:\n for index, user in enumerate(users_batch):\n LOGGER.info(\n 'Processing user with index %s and id %s',\n current_batch_index + index, user.id\n )\n role_assignment_class.objects.get_or_create(\n user=user,\n role=enterprise_role\n )\n\n sleep(batch_sleep)\n current_batch_index += len(users_batch)\n users_batch = _get_batch_method(\n current_batch_index,\n current_batch_index + batch_limit\n )", "response": "Assign enterprise role to users."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef handle(self, *args, **options):\n LOGGER.info('Starting assigning enterprise roles to users!')\n\n role = options['role']\n if role == ENTERPRISE_ADMIN_ROLE:\n # Assign admin role to non-staff users with enterprise data api access.\n self._assign_enterprise_role_to_users(self._get_enterprise_admin_users_batch, options)\n elif role == ENTERPRISE_OPERATOR_ROLE:\n # Assign operator role to staff users with enterprise data api access.\n self._assign_enterprise_role_to_users(self._get_enterprise_operator_users_batch, options)\n elif role == ENTERPRISE_LEARNER_ROLE:\n # Assign enterprise learner role to enterprise customer users.\n self._assign_enterprise_role_to_users(self._get_enterprise_customer_users_batch, options)\n elif role == ENTERPRISE_ENROLLMENT_API_ADMIN_ROLE:\n # Assign enterprise enrollment api admin to non-staff users with enterprise data api access.\n self._assign_enterprise_role_to_users(self._get_enterprise_enrollment_api_admin_users_batch, options, True)\n elif role == ENTERPRISE_CATALOG_ADMIN_ROLE:\n # Assign enterprise catalog admin role to users with having credentials in catalog.\n self._assign_enterprise_role_to_users(self._get_enterprise_catalog_admin_users_batch, options, True)\n else:\n raise CommandError('Please provide a valid role name. Supported roles are {admin} and {learner}'.format(\n admin=ENTERPRISE_ADMIN_ROLE,\n learner=ENTERPRISE_LEARNER_ROLE\n ))\n\n LOGGER.info('Successfully finished assigning enterprise roles to users!')", "response": "Assign the roles to the users."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends a completion status call to Degreed using the client.", "response": "def transmit(self, payload, **kwargs):\n \"\"\"\n Send a completion status call to Degreed using the client.\n\n Args:\n payload: The learner completion data payload to send to Degreed\n \"\"\"\n kwargs['app_label'] = 'degreed'\n kwargs['model_name'] = 'DegreedLearnerDataTransmissionAudit'\n kwargs['remote_user_id'] = 'degreed_user_email'\n super(DegreedLearnerTransmitter, self).transmit(payload, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the EnterpriseCustomer associated with a running pipeline.", "response": "def get_enterprise_customer_for_running_pipeline(request, pipeline): # pylint: disable=invalid-name\n \"\"\"\n Get the EnterpriseCustomer associated with a running pipeline.\n \"\"\"\n sso_provider_id = request.GET.get('tpa_hint')\n if pipeline:\n sso_provider_id = Registry.get_from_pipeline(pipeline).provider_id\n return get_enterprise_customer_for_sso(sso_provider_id)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef handle_enterprise_logistration(backend, user, **kwargs):\n request = backend.strategy.request\n enterprise_customer = get_enterprise_customer_for_running_pipeline(\n request,\n {\n 'backend': backend.name,\n 'kwargs': kwargs\n }\n )\n if enterprise_customer is None:\n # This pipeline element is not being activated as a part of an Enterprise logistration\n return\n\n # proceed with the creation of a link between the user and the enterprise customer, then exit.\n enterprise_customer_user, _ = EnterpriseCustomerUser.objects.update_or_create(\n enterprise_customer=enterprise_customer,\n user_id=user.id\n )\n enterprise_customer_user.update_session(request)", "response": "This function handles the Enterprise logistration of a user in the process of logging into the Enterprise Customer."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_user_from_social_auth(tpa_provider, tpa_username):\n user_social_auth = UserSocialAuth.objects.select_related('user').filter(\n user__username=tpa_username, provider=tpa_provider.backend_name\n ).first()\n\n return user_social_auth.user if user_social_auth else None", "response": "Find the user from the LMS model UserSocialAuth."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_oauth_access_token(url_base, client_id, client_secret, company_id, user_id, user_type):\n SAPSuccessFactorsGlobalConfiguration = apps.get_model( # pylint: disable=invalid-name\n 'sap_success_factors',\n 'SAPSuccessFactorsGlobalConfiguration'\n )\n global_sap_config = SAPSuccessFactorsGlobalConfiguration.current()\n url = url_base + global_sap_config.oauth_api_path\n\n response = requests.post(\n url,\n json={\n 'grant_type': 'client_credentials',\n 'scope': {\n 'userId': user_id,\n 'companyId': company_id,\n 'userType': user_type,\n 'resourceType': 'learning_public_api',\n }\n },\n auth=(client_id, client_secret),\n headers={'content-type': 'application/json'}\n )\n\n response.raise_for_status()\n data = response.json()\n try:\n return data['access_token'], datetime.datetime.utcfromtimestamp(data['expires_in'] + int(time.time()))\n except KeyError:\n raise requests.RequestException(response=response)", "response": "Retrieves an OAuth 2. 0 access token using the client credentials grant."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _create_session(self):\n session = requests.Session()\n session.timeout = self.SESSION_TIMEOUT\n\n oauth_access_token, expires_at = SAPSuccessFactorsAPIClient.get_oauth_access_token(\n self.enterprise_configuration.sapsf_base_url,\n self.enterprise_configuration.key,\n self.enterprise_configuration.secret,\n self.enterprise_configuration.sapsf_company_id,\n self.enterprise_configuration.sapsf_user_id,\n self.enterprise_configuration.user_type\n )\n\n session.headers['Authorization'] = 'Bearer {}'.format(oauth_access_token)\n session.headers['content-type'] = 'application/json'\n self.session = session\n self.expires_at = expires_at", "response": "Create a new session object for use in connecting with SAP SuccessFactors"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_course_completion(self, user_id, payload):\n url = self.enterprise_configuration.sapsf_base_url + self.global_sap_config.completion_status_api_path\n return self._call_post_with_user_override(user_id, url, payload)", "response": "Send a completion status payload to the SuccessFactors OCN Completion Status endpoint"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _call_post_with_user_override(self, sap_user_id, url, payload):\n SAPSuccessFactorsEnterpriseCustomerConfiguration = apps.get_model( # pylint: disable=invalid-name\n 'sap_success_factors',\n 'SAPSuccessFactorsEnterpriseCustomerConfiguration'\n )\n oauth_access_token, _ = SAPSuccessFactorsAPIClient.get_oauth_access_token(\n self.enterprise_configuration.sapsf_base_url,\n self.enterprise_configuration.key,\n self.enterprise_configuration.secret,\n self.enterprise_configuration.sapsf_company_id,\n sap_user_id,\n SAPSuccessFactorsEnterpriseCustomerConfiguration.USER_TYPE_USER\n )\n\n response = requests.post(\n url,\n data=payload,\n headers={\n 'Authorization': 'Bearer {}'.format(oauth_access_token),\n 'content-type': 'application/json'\n }\n )\n\n return response.status_code, response.text", "response": "Make a post request with an auth token acquired for a specific user to a SuccessFactors endpoint."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _call_post_with_session(self, url, payload):\n now = datetime.datetime.utcnow()\n if now >= self.expires_at:\n # Create a new session with a valid token\n self.session.close()\n self._create_session()\n response = self.session.post(url, data=payload)\n return response.status_code, response.text", "response": "Make a post request using the session object to a SuccessFactors endpoint."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nmaking a GET request using the session object to a SuccessFactors endpoint for inactive learners. Example: sap_search_student_url: \"/learning/odatav4/searchStudent/v1/Students? $filter=criteria/isActive eq False&$select=studentID\" SAP API response: { u'@odata.metadataEtag': u'W/\"17090d86-20fa-49c8-8de0-de1d308c8b55\"', u'value': [ { u'studentID': u'admint6', }, { u'studentID': u'adminsap1', } ] } Returns: List of inactive learners [ { u'studentID': u'admint6' }, { u'studentID': u'adminsap1' } ]", "response": "def get_inactive_sap_learners(self):\n \"\"\"\n Make a GET request using the session object to a SuccessFactors endpoint for inactive learners.\n\n Example:\n sap_search_student_url: \"/learning/odatav4/searchStudent/v1/Students?\n $filter=criteria/isActive eq False&$select=studentID\"\n\n SAP API response: {\n u'@odata.metadataEtag': u'W/\"17090d86-20fa-49c8-8de0-de1d308c8b55\"',\n u'value': [\n {\n u'studentID': u'admint6',\n },\n {\n u'studentID': u'adminsap1',\n }\n ]\n }\n\n Returns: List of inactive learners\n [\n {\n u'studentID': u'admint6'\n },\n {\n u'studentID': u'adminsap1'\n }\n ]\n \"\"\"\n now = datetime.datetime.utcnow()\n if now >= self.expires_at:\n # Create a new session with a valid token\n self.session.close()\n self._create_session()\n\n sap_search_student_url = '{sapsf_base_url}/{search_students_path}?$filter={search_filter}'.format(\n sapsf_base_url=self.enterprise_configuration.sapsf_base_url.rstrip('/'),\n search_students_path=self.global_sap_config.search_student_api_path.rstrip('/'),\n search_filter='criteria/isActive eq False&$select=studentID',\n )\n all_inactive_learners = self._call_search_students_recursively(\n sap_search_student_url,\n all_inactive_learners=[],\n page_size=500,\n start_at=0\n )\n return all_inactive_learners"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _call_search_students_recursively(self, sap_search_student_url, all_inactive_learners, page_size, start_at):\n search_student_paginated_url = '{sap_search_student_url}&{pagination_criterion}'.format(\n sap_search_student_url=sap_search_student_url,\n pagination_criterion='$count=true&$top={page_size}&$skip={start_at}'.format(\n page_size=page_size,\n start_at=start_at,\n ),\n )\n try:\n response = self.session.get(search_student_paginated_url)\n sap_inactive_learners = response.json()\n except (ConnectionError, Timeout):\n LOGGER.warning(\n 'Unable to fetch inactive learners from SAP searchStudent API with url '\n '\"{%s}\".', search_student_paginated_url,\n )\n return None\n\n if 'error' in sap_inactive_learners:\n LOGGER.warning(\n 'SAP searchStudent API for customer %s and base url %s returned response with '\n 'error message \"%s\" and with error code \"%s\".',\n self.enterprise_configuration.enterprise_customer.name,\n self.enterprise_configuration.sapsf_base_url,\n sap_inactive_learners['error'].get('message'),\n sap_inactive_learners['error'].get('code'),\n )\n return None\n\n new_page_start_at = page_size + start_at\n all_inactive_learners += sap_inactive_learners['value']\n if sap_inactive_learners['@odata.count'] > new_page_start_at:\n return self._call_search_students_recursively(\n sap_search_student_url,\n all_inactive_learners,\n page_size=page_size,\n start_at=new_page_start_at,\n )\n\n return all_inactive_learners", "response": "Make recursive GET calls to traverse the paginated API response for search students."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef filter_queryset(self, request, queryset, view):\n if not request.user.is_staff:\n filter_kwargs = {view.USER_ID_FILTER: request.user.id}\n queryset = queryset.filter(**filter_kwargs)\n return queryset", "response": "Filter the queryset for the user s ID if non - staff."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef filter_queryset(self, request, queryset, view):\n if request.user.is_staff:\n email = request.query_params.get('email', None)\n username = request.query_params.get('username', None)\n query_parameters = {}\n\n if email:\n query_parameters.update(email=email)\n if username:\n query_parameters.update(username=username)\n if query_parameters:\n users = User.objects.filter(**query_parameters).values_list('id', flat=True)\n queryset = queryset.filter(user_id__in=users)\n else:\n queryset = queryset.filter(user_id=request.user.id)\n\n return queryset", "response": "Apply incoming filters only if user is staff."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef transmit(self, payload, **kwargs):\n IntegratedChannelLearnerDataTransmissionAudit = apps.get_model( # pylint: disable=invalid-name\n app_label=kwargs.get('app_label', 'integrated_channel'),\n model_name=kwargs.get('model_name', 'LearnerDataTransmissionAudit'),\n )\n # Since we have started sending courses to integrated channels instead of course runs,\n # we need to attempt to send transmissions with course keys and course run ids in order to\n # ensure that we account for whether courses or course runs exist in the integrated channel.\n # The exporters have been changed to return multiple transmission records to attempt,\n # one by course key and one by course run id.\n # If the transmission with the course key succeeds, the next one will get skipped.\n # If it fails, the one with the course run id will be attempted and (presumably) succeed.\n for learner_data in payload.export():\n serialized_payload = learner_data.serialize(enterprise_configuration=self.enterprise_configuration)\n LOGGER.debug('Attempting to transmit serialized payload: %s', serialized_payload)\n\n enterprise_enrollment_id = learner_data.enterprise_course_enrollment_id\n if learner_data.completed_timestamp is None:\n # The user has not completed the course, so we shouldn't send a completion status call\n LOGGER.info('Skipping in-progress enterprise enrollment {}'.format(enterprise_enrollment_id))\n continue\n\n previous_transmissions = IntegratedChannelLearnerDataTransmissionAudit.objects.filter(\n enterprise_course_enrollment_id=enterprise_enrollment_id,\n error_message=''\n )\n if previous_transmissions.exists():\n # We've already sent a completion status call for this enrollment\n LOGGER.info('Skipping previously sent enterprise enrollment {}'.format(enterprise_enrollment_id))\n continue\n\n try:\n code, body = self.client.create_course_completion(\n getattr(learner_data, kwargs.get('remote_user_id')),\n serialized_payload\n )\n LOGGER.info(\n 'Successfully sent completion status call for enterprise enrollment {}'.format(\n enterprise_enrollment_id,\n )\n )\n except RequestException as request_exception:\n code = 500\n body = str(request_exception)\n self.handle_transmission_error(learner_data, request_exception)\n\n learner_data.status = str(code)\n learner_data.error_message = body if code >= 400 else ''\n learner_data.save()", "response": "Transmit a learner completion data payload to the integrated channel using the client."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef handle_transmission_error(self, learner_data, request_exception):\n try:\n sys_msg = request_exception.response.content\n except AttributeError:\n sys_msg = 'Not available'\n LOGGER.error(\n (\n 'Failed to send completion status call for enterprise enrollment %s'\n 'with payload %s'\n '\\nError message: %s'\n '\\nSystem message: %s'\n ),\n learner_data.enterprise_course_enrollment_id,\n learner_data,\n str(request_exception),\n sys_msg\n )", "response": "Handle the case where the transmission fails."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_missing_price_information_message(request, item):\n messages.warning(\n request,\n _(\n '{strong_start}We could not gather price information for {em_start}{item}{em_end}.{strong_end} '\n '{span_start}If you continue to have these issues, please contact '\n '{link_start}{platform_name} support{link_end}.{span_end}'\n ).format(\n item=item,\n em_start='',\n em_end='',\n link_start=''.format(\n support_link=get_configuration_value('ENTERPRISE_SUPPORT_URL', settings.ENTERPRISE_SUPPORT_URL),\n ),\n platform_name=get_configuration_value('PLATFORM_NAME', settings.PLATFORM_NAME),\n link_end='',\n span_start='',\n span_end='',\n strong_start='',\n strong_end='',\n )\n )", "response": "Adds a message to the messages store indicating that we could not retrieve price information about an item."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_unenrollable_item_message(request, item):\n messages.info(\n request,\n _(\n '{strong_start}Something happened.{strong_end} '\n '{span_start}This {item} is not currently open to new learners. Please start over and select a different '\n '{item}.{span_end}'\n ).format(\n item=item,\n strong_start='',\n strong_end='',\n span_start='',\n span_end='',\n )\n )", "response": "Adds a message to the message store indicating that the item is unenrollable."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_generic_info_message_for_error(request):\n messages.info(\n request,\n _(\n '{strong_start}Something happened.{strong_end} '\n '{span_start}This course link is currently invalid. '\n 'Please reach out to your Administrator for assistance to this course.{span_end}'\n ).format(\n span_start='',\n span_end='',\n strong_start='',\n strong_end='',\n )\n )", "response": "Add generic info message to the request indicating that there was an issue processing request."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_generic_error_message_with_code(request, error_code):\n messages.error(\n request,\n _(\n '{strong_start}Something happened.{strong_end} '\n '{span_start}Please reach out to your learning administrator with '\n 'the following error code and they will be able to help you out.{span_end}'\n '{span_start}Error code: {error_code}{span_end}'\n ).format(\n error_code=error_code,\n strong_start='',\n strong_end='',\n span_start='',\n span_end='',\n )\n )", "response": "Add generic error message to the current request."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate that a particular image extension is valid.", "response": "def validate_image_extension(value):\n \"\"\"\n Validate that a particular image extension.\n \"\"\"\n config = get_app_config()\n ext = os.path.splitext(value.name)[1]\n if config and not ext.lower() in config.valid_image_extensions:\n raise ValidationError(_(\"Unsupported file extension.\"))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nvalidating that a particular image size is less than or equal to the configured max_image_size.", "response": "def validate_image_size(image):\n \"\"\"\n Validate that a particular image size.\n \"\"\"\n config = get_app_config()\n valid_max_image_size_in_bytes = config.valid_max_image_size * 1024\n if config and not image.size <= valid_max_image_size_in_bytes:\n raise ValidationError(\n _(\"The logo image file size must be less than or equal to %s KB.\") % config.valid_max_image_size)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_enterprise_customer_from_catalog_id(catalog_id):\n try:\n return str(EnterpriseCustomerCatalog.objects.get(pk=catalog_id).enterprise_customer.uuid)\n except EnterpriseCustomerCatalog.DoesNotExist:\n return None", "response": "Get the enterprise customer id given an enterprise customer catalog id."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef on_init(app): # pylint: disable=unused-argument\n docs_path = os.path.abspath(os.path.dirname(__file__))\n root_path = os.path.abspath(os.path.join(docs_path, '..'))\n apidoc_path = 'sphinx-apidoc'\n if hasattr(sys, 'real_prefix'): # Check to see if we are in a virtualenv\n # If we are, assemble the path manually\n bin_path = os.path.abspath(os.path.join(sys.prefix, 'bin'))\n apidoc_path = os.path.join(bin_path, apidoc_path)\n check_call([apidoc_path, '-o', docs_path, os.path.join(root_path, 'enterprise'),\n os.path.join(root_path, 'enterprise/migrations')])", "response": "Run sphinx - apidoc after Sphinx initialization."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef setup(app):\n event = 'builder-inited' if six.PY3 else b'builder-inited'\n app.connect(event, on_init)", "response": "Sphinx extension : run sphinx - apidoc."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef add_arguments(self, parser):\n parser.add_argument(\n '--enterprise_customer',\n dest='enterprise_customer',\n default=None,\n metavar='ENTERPRISE_CUSTOMER_UUID',\n help=_('Transmit data for only this EnterpriseCustomer. '\n 'Omit this option to transmit to all EnterpriseCustomers with active integrated channels.'),\n )\n parser.add_argument(\n '--channel',\n dest='channel',\n default='',\n metavar='INTEGRATED_CHANNEL',\n help=_('Transmit data to this IntegrateChannel. '\n 'Omit this option to transmit to all configured, active integrated channels.'),\n choices=INTEGRATED_CHANNEL_CHOICES.keys(),\n )", "response": "Adds the optional arguments to the parser."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a list of integrated channels for active customers filtered from the given options.", "response": "def get_integrated_channels(self, options):\n \"\"\"\n Generates a list of active integrated channels for active customers, filtered from the given options.\n\n Raises errors when invalid options are encountered.\n\n See ``add_arguments`` for the accepted options.\n \"\"\"\n channel_classes = self.get_channel_classes(options.get('channel'))\n filter_kwargs = {\n 'active': True,\n 'enterprise_customer__active': True,\n }\n enterprise_customer = self.get_enterprise_customer(options.get('enterprise_customer'))\n if enterprise_customer:\n filter_kwargs['enterprise_customer'] = enterprise_customer\n\n for channel_class in channel_classes:\n for integrated_channel in channel_class.objects.filter(**filter_kwargs):\n yield integrated_channel"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_enterprise_customer(uuid):\n if uuid is None:\n return None\n try:\n return EnterpriseCustomer.active_customers.get(uuid=uuid)\n except EnterpriseCustomer.DoesNotExist:\n raise CommandError(\n _('Enterprise customer {uuid} not found, or not active').format(uuid=uuid))", "response": "Returns the enterprise customer with the given uuid Raises CommandError if uuid is invalid."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nassemble a list of integrated channel classes to transmit to.", "response": "def get_channel_classes(channel_code):\n \"\"\"\n Assemble a list of integrated channel classes to transmit to.\n\n If a valid channel type was provided, use it.\n\n Otherwise, use all the available channel types.\n \"\"\"\n if channel_code:\n # Channel code is case-insensitive\n channel_code = channel_code.upper()\n\n if channel_code not in INTEGRATED_CHANNEL_CHOICES:\n raise CommandError(_('Invalid integrated channel: {channel}').format(channel=channel_code))\n\n channel_classes = [INTEGRATED_CHANNEL_CHOICES[channel_code]]\n else:\n channel_classes = INTEGRATED_CHANNEL_CHOICES.values()\n\n return channel_classes"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a Result object for the statement.", "response": "def get_result(self, course_grade):\n \"\"\"\n Get result for the statement.\n\n Arguments:\n course_grade (CourseGrade): Course grade.\n \"\"\"\n return Result(\n score=Score(\n scaled=course_grade.percent,\n raw=course_grade.percent * 100,\n min=MIN_SCORE,\n max=MAX_SCORE,\n ),\n success=course_grade.passed,\n completion=course_grade.passed\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_requirements(requirements_file):\n lines = open(requirements_file).readlines()\n dependencies = []\n dependency_links = []\n\n for line in lines:\n package = line.strip()\n if package.startswith('#'):\n # Skip pure comment lines\n continue\n\n if any(package.startswith(prefix) for prefix in VCS_PREFIXES):\n # VCS reference for dev purposes, expect a trailing comment\n # with the normal requirement\n package_link, __, package = package.rpartition('#')\n\n # Remove -e string\n package_link = re.sub(r'(.*)(?Phttps?.*$)', r'\\g', package_link)\n package = re.sub(r'(egg=)?(?P.*)==.*$', r'\\g', package)\n package_version = re.sub(r'.*[^=]==', '', line.strip())\n\n if package:\n dependency_links.append(\n '{package_link}#egg={package}-{package_version}'.format(\n package_link=package_link,\n package=package,\n package_version=package_version,\n )\n )\n else:\n # Ignore any trailing comment\n package, __, __ = package.partition('#')\n # Remove any whitespace and assume non-empty results are dependencies\n package = package.strip()\n\n if package:\n dependencies.append(package)\n return dependencies, dependency_links", "response": "Get the contents of a file listing the requirements"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef transmit_learner_data(self, user):\n exporter = self.get_learner_data_exporter(user)\n transmitter = self.get_learner_data_transmitter()\n transmitter.transmit(exporter)", "response": "Transmit the learner data records to the integrated channel."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ntransmit content metadata to integrated channel.", "response": "def transmit_content_metadata(self, user):\n \"\"\"\n Transmit content metadata to integrated channel.\n \"\"\"\n exporter = self.get_content_metadata_exporter(user)\n transmitter = self.get_content_metadata_transmitter()\n transmitter.transmit(exporter.export())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of DegreedLearnerDataTransmissionAudit objects with the given enrollment and course completion data.", "response": "def get_learner_data_records(\n self,\n enterprise_enrollment,\n completed_date=None,\n is_passing=False,\n **kwargs\n ): # pylint: disable=arguments-differ,unused-argument\n \"\"\"\n Return a DegreedLearnerDataTransmissionAudit with the given enrollment and course completion data.\n\n If completed_date is None, then course completion has not been met.\n\n If no remote ID can be found, return None.\n \"\"\"\n # Degreed expects completion dates of the form 'yyyy-mm-dd'.\n completed_timestamp = completed_date.strftime(\"%F\") if isinstance(completed_date, datetime) else None\n if enterprise_enrollment.enterprise_customer_user.get_remote_id() is not None:\n DegreedLearnerDataTransmissionAudit = apps.get_model( # pylint: disable=invalid-name\n 'degreed',\n 'DegreedLearnerDataTransmissionAudit'\n )\n # We return two records here, one with the course key and one with the course run id, to account for\n # uncertainty about the type of content (course vs. course run) that was sent to the integrated channel.\n return [\n DegreedLearnerDataTransmissionAudit(\n enterprise_course_enrollment_id=enterprise_enrollment.id,\n degreed_user_email=enterprise_enrollment.enterprise_customer_user.user_email,\n course_id=parse_course_key(enterprise_enrollment.course_id),\n course_completed=completed_date is not None and is_passing,\n completed_timestamp=completed_timestamp,\n ),\n DegreedLearnerDataTransmissionAudit(\n enterprise_course_enrollment_id=enterprise_enrollment.id,\n degreed_user_email=enterprise_enrollment.enterprise_customer_user.user_email,\n course_id=enterprise_enrollment.course_id,\n course_completed=completed_date is not None and is_passing,\n completed_timestamp=completed_timestamp,\n )\n ]\n else:\n LOGGER.debug(\n 'No learner data was sent for user [%s] because a Degreed user ID could not be found.',\n enterprise_enrollment.enterprise_customer_user.username\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nrender the given template with the stock data.", "response": "def get(self, request, template_id, view_type):\n \"\"\"\n Render the given template with the stock data.\n \"\"\"\n template = get_object_or_404(EnrollmentNotificationEmailTemplate, pk=template_id)\n if view_type not in self.view_type_contexts:\n return HttpResponse(status=404)\n base_context = self.view_type_contexts[view_type].copy()\n base_context.update({'user_name': self.get_user_name(request)})\n return HttpResponse(template.render_html_template(base_context), content_type='text/html')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nbuilding common admin context.", "response": "def _build_admin_context(request, customer):\n \"\"\"\n Build common admin context.\n \"\"\"\n opts = customer._meta\n codename = get_permission_codename('change', opts)\n has_change_permission = request.user.has_perm('%s.%s' % (opts.app_label, codename))\n return {\n 'has_change_permission': has_change_permission,\n 'opts': opts\n }"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _build_context(self, request, enterprise_customer_uuid):\n enterprise_customer = EnterpriseCustomer.objects.get(uuid=enterprise_customer_uuid) # pylint: disable=no-member\n\n context = {\n self.ContextParameters.ENTERPRISE_CUSTOMER: enterprise_customer,\n }\n context.update(admin.site.each_context(request))\n context.update(self._build_admin_context(request, enterprise_customer))\n return context", "response": "Build common context parts used by different handlers in this view."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get(self, request, enterprise_customer_uuid):\n context = self._build_context(request, enterprise_customer_uuid)\n transmit_courses_metadata_form = TransmitEnterpriseCoursesForm()\n context.update({self.ContextParameters.TRANSMIT_COURSES_METADATA_FORM: transmit_courses_metadata_form})\n\n return render(request, self.template, context)", "response": "Handles GET request - render Transmit courses metadata form."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhandling POST request - handle form submissions. Arguments: request (django.http.request.HttpRequest): Request instance enterprise_customer_uuid (str): Enterprise Customer UUID", "response": "def post(self, request, enterprise_customer_uuid):\n \"\"\"\n Handle POST request - handle form submissions.\n\n Arguments:\n request (django.http.request.HttpRequest): Request instance\n enterprise_customer_uuid (str): Enterprise Customer UUID\n \"\"\"\n transmit_courses_metadata_form = TransmitEnterpriseCoursesForm(request.POST)\n\n # check that form data is well-formed\n if transmit_courses_metadata_form.is_valid():\n channel_worker_username = transmit_courses_metadata_form.cleaned_data['channel_worker_username']\n\n # call `transmit_content_metadata` management command to trigger\n # transmission of enterprise courses metadata\n call_command(\n 'transmit_content_metadata',\n '--catalog_user', channel_worker_username,\n enterprise_customer=enterprise_customer_uuid\n )\n\n # Redirect to GET\n return HttpResponseRedirect('')\n\n context = self._build_context(request, enterprise_customer_uuid)\n context.update({self.ContextParameters.TRANSMIT_COURSES_METADATA_FORM: transmit_courses_metadata_form})\n return render(request, self.template, context)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _build_context(self, request, customer_uuid):\n # TODO: pylint acts stupid - find a way around it without suppressing\n enterprise_customer = EnterpriseCustomer.objects.get(uuid=customer_uuid) # pylint: disable=no-member\n\n search_keyword = self.get_search_keyword(request)\n linked_learners = self.get_enterprise_customer_user_queryset(request, search_keyword, customer_uuid)\n pending_linked_learners = self.get_pending_users_queryset(search_keyword, customer_uuid)\n\n context = {\n self.ContextParameters.ENTERPRISE_CUSTOMER: enterprise_customer,\n self.ContextParameters.PENDING_LEARNERS: pending_linked_learners,\n self.ContextParameters.LEARNERS: linked_learners,\n self.ContextParameters.SEARCH_KEYWORD: search_keyword or '',\n self.ContextParameters.ENROLLMENT_URL: settings.LMS_ENROLLMENT_API_PATH,\n }\n context.update(admin.site.each_context(request))\n context.update(self._build_admin_context(request, enterprise_customer))\n return context", "response": "Build common context parts used by different handlers in this view."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a queryset of EnterpriseCustomerUser objects that can be used to view the list of users linked to a specific EnterpriseCustomer.", "response": "def get_enterprise_customer_user_queryset(self, request, search_keyword, customer_uuid, page_size=PAGE_SIZE):\n \"\"\"\n Get the list of EnterpriseCustomerUsers we want to render.\n\n Arguments:\n request (HttpRequest): HTTP Request instance.\n search_keyword (str): The keyword to search for in users' email addresses and usernames.\n customer_uuid (str): A unique identifier to filter down to only users linked to a\n particular EnterpriseCustomer.\n page_size (int): Number of learners displayed in each paginated set.\n \"\"\"\n page = request.GET.get('page', 1)\n learners = EnterpriseCustomerUser.objects.filter(enterprise_customer__uuid=customer_uuid)\n user_ids = learners.values_list('user_id', flat=True)\n matching_users = User.objects.filter(pk__in=user_ids)\n if search_keyword is not None:\n matching_users = matching_users.filter(\n Q(email__icontains=search_keyword) | Q(username__icontains=search_keyword)\n )\n matching_user_ids = matching_users.values_list('pk', flat=True)\n learners = learners.filter(user_id__in=matching_user_ids)\n return paginated_list(learners, page, page_size)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_pending_users_queryset(self, search_keyword, customer_uuid):\n queryset = PendingEnterpriseCustomerUser.objects.filter(\n enterprise_customer__uuid=customer_uuid\n )\n\n if search_keyword is not None:\n queryset = queryset.filter(user_email__icontains=search_keyword)\n\n return queryset", "response": "Returns a queryset of all pending users that have a given keyword in the customer s email addresses."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nhandling a single user by email or username.", "response": "def _handle_singular(cls, enterprise_customer, manage_learners_form):\n \"\"\"\n Link single user by email or username.\n\n Arguments:\n enterprise_customer (EnterpriseCustomer): learners will be linked to this Enterprise Customer instance\n manage_learners_form (ManageLearnersForm): bound ManageLearners form instance\n \"\"\"\n form_field_value = manage_learners_form.cleaned_data[ManageLearnersForm.Fields.EMAIL_OR_USERNAME]\n email = email_or_username__to__email(form_field_value)\n try:\n validate_email_to_link(email, form_field_value, ValidationMessages.INVALID_EMAIL_OR_USERNAME, True)\n except ValidationError as exc:\n manage_learners_form.add_error(ManageLearnersForm.Fields.EMAIL_OR_USERNAME, exc)\n else:\n EnterpriseCustomerUser.objects.link_user(enterprise_customer, email)\n return [email]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nhandle bulk upload of a user into a single object.", "response": "def _handle_bulk_upload(cls, enterprise_customer, manage_learners_form, request, email_list=None):\n \"\"\"\n Bulk link users by email.\n\n Arguments:\n enterprise_customer (EnterpriseCustomer): learners will be linked to this Enterprise Customer instance\n manage_learners_form (ManageLearnersForm): bound ManageLearners form instance\n request (django.http.request.HttpRequest): HTTP Request instance\n email_list (iterable): A list of pre-processed email addresses to handle using the form\n \"\"\"\n errors = []\n emails = set()\n already_linked_emails = []\n duplicate_emails = []\n csv_file = manage_learners_form.cleaned_data[ManageLearnersForm.Fields.BULK_UPLOAD]\n if email_list:\n parsed_csv = [{ManageLearnersForm.CsvColumns.EMAIL: email} for email in email_list]\n else:\n parsed_csv = parse_csv(csv_file, expected_columns={ManageLearnersForm.CsvColumns.EMAIL})\n\n try:\n for index, row in enumerate(parsed_csv):\n email = row[ManageLearnersForm.CsvColumns.EMAIL]\n try:\n already_linked = validate_email_to_link(email, ignore_existing=True)\n except ValidationError as exc:\n message = _(\"Error at line {line}: {message}\\n\").format(line=index + 1, message=exc)\n errors.append(message)\n else:\n if already_linked:\n already_linked_emails.append((email, already_linked.enterprise_customer))\n elif email in emails:\n duplicate_emails.append(email)\n else:\n emails.add(email)\n except ValidationError as exc:\n errors.append(exc)\n\n if errors:\n manage_learners_form.add_error(\n ManageLearnersForm.Fields.GENERAL_ERRORS, ValidationMessages.BULK_LINK_FAILED\n )\n for error in errors:\n manage_learners_form.add_error(ManageLearnersForm.Fields.BULK_UPLOAD, error)\n return\n\n # There were no errors. Now do the actual linking:\n for email in emails:\n EnterpriseCustomerUser.objects.link_user(enterprise_customer, email)\n\n # Report what happened:\n count = len(emails)\n messages.success(request, ungettext(\n \"{count} new learner was added to {enterprise_customer_name}.\",\n \"{count} new learners were added to {enterprise_customer_name}.\",\n count\n ).format(count=count, enterprise_customer_name=enterprise_customer.name))\n this_customer_linked_emails = [\n email for email, customer in already_linked_emails if customer == enterprise_customer\n ]\n other_customer_linked_emails = [\n email for email, __ in already_linked_emails if email not in this_customer_linked_emails\n ]\n if this_customer_linked_emails:\n messages.warning(\n request,\n _(\n \"The following learners were already associated with this Enterprise \"\n \"Customer: {list_of_emails}\"\n ).format(\n list_of_emails=\", \".join(this_customer_linked_emails)\n )\n )\n if other_customer_linked_emails:\n messages.warning(\n request,\n _(\n \"The following learners are already associated with \"\n \"another Enterprise Customer. These learners were not \"\n \"added to {enterprise_customer_name}: {list_of_emails}\"\n ).format(\n enterprise_customer_name=enterprise_customer.name,\n list_of_emails=\", \".join(other_customer_linked_emails),\n )\n )\n if duplicate_emails:\n messages.warning(\n request,\n _(\n \"The following duplicate email addresses were not added: \"\n \"{list_of_emails}\"\n ).format(\n list_of_emails=\", \".join(duplicate_emails)\n )\n )\n # Build a list of all the emails that we can act on further; that is,\n # emails that we either linked to this customer, or that were linked already.\n all_processable_emails = list(emails) + this_customer_linked_emails\n\n return all_processable_emails"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nenrolling a single user in any number of courses using a particular course mode.", "response": "def enroll_user(cls, enterprise_customer, user, course_mode, *course_ids):\n \"\"\"\n Enroll a single user in any number of courses using a particular course mode.\n\n Args:\n enterprise_customer: The EnterpriseCustomer which is sponsoring the enrollment\n user: The user who needs to be enrolled in the course\n course_mode: The mode with which the enrollment should be created\n *course_ids: An iterable containing any number of course IDs to eventually enroll the user in.\n\n Returns:\n Boolean: Whether or not enrollment succeeded for all courses specified\n \"\"\"\n enterprise_customer_user, __ = EnterpriseCustomerUser.objects.get_or_create(\n enterprise_customer=enterprise_customer,\n user_id=user.id\n )\n enrollment_client = EnrollmentApiClient()\n succeeded = True\n for course_id in course_ids:\n try:\n enrollment_client.enroll_user_in_course(user.username, course_id, course_mode)\n except HttpClientError as exc:\n # Check if user is already enrolled then we should ignore exception\n if cls.is_user_enrolled(user, course_id, course_mode):\n succeeded = True\n else:\n succeeded = False\n default_message = 'No error message provided'\n try:\n error_message = json.loads(exc.content.decode()).get('message', default_message)\n except ValueError:\n error_message = default_message\n logging.error(\n 'Error while enrolling user %(user)s: %(message)s',\n dict(user=user.username, message=error_message)\n )\n if succeeded:\n __, created = EnterpriseCourseEnrollment.objects.get_or_create(\n enterprise_customer_user=enterprise_customer_user,\n course_id=course_id\n )\n if created:\n track_enrollment('admin-enrollment', user.id, course_id)\n return succeeded"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef is_user_enrolled(cls, user, course_id, course_mode):\n enrollment_client = EnrollmentApiClient()\n try:\n enrollments = enrollment_client.get_course_enrollment(user.username, course_id)\n if enrollments and course_mode == enrollments.get('mode'):\n return True\n except HttpClientError as exc:\n logging.error(\n 'Error while checking enrollment status of user %(user)s: %(message)s',\n dict(user=user.username, message=str(exc))\n )\n except KeyError as exc:\n logging.warning(\n 'Error while parsing enrollment data of user %(user)s: %(message)s',\n dict(user=user.username, message=str(exc))\n )\n return False", "response": "Checks if a user is enrolled in a given course run track."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_users_by_email(cls, emails):\n users = User.objects.filter(email__in=emails)\n present_emails = users.values_list('email', flat=True)\n missing_emails = list(set(emails) - set(present_emails))\n return users, missing_emails", "response": "Accept a list of emails and separate them into users that don t exist on OpenEdX and users who don t exist on OpenEdX and users who don t exist on OpenEdX and users who don t exist on OpenEdX and users who don t exist on OpenEdX and users who don t exist on OpenEdX and users who don t exist on OpenEdX and users who don t exist on OpenEdX"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef enroll_users_in_program(cls, enterprise_customer, program_details, course_mode, emails, cohort=None):\n existing_users, unregistered_emails = cls.get_users_by_email(emails)\n course_ids = get_course_runs_from_program(program_details)\n\n successes = []\n pending = []\n failures = []\n\n for user in existing_users:\n succeeded = cls.enroll_user(enterprise_customer, user, course_mode, *course_ids)\n if succeeded:\n successes.append(user)\n else:\n failures.append(user)\n\n for email in unregistered_emails:\n pending_user = enterprise_customer.enroll_user_pending_registration(\n email,\n course_mode,\n *course_ids,\n cohort=cohort\n )\n pending.append(pending_user)\n\n return successes, pending, failures", "response": "Enroll existing users in all courses in a program and create pending enrollments for nonexisting users."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef enroll_users_in_course(cls, enterprise_customer, course_id, course_mode, emails):\n existing_users, unregistered_emails = cls.get_users_by_email(emails)\n\n successes = []\n pending = []\n failures = []\n\n for user in existing_users:\n succeeded = cls.enroll_user(enterprise_customer, user, course_mode, course_id)\n if succeeded:\n successes.append(user)\n else:\n failures.append(user)\n\n for email in unregistered_emails:\n pending_user = enterprise_customer.enroll_user_pending_registration(\n email,\n course_mode,\n course_id\n )\n pending.append(pending_user)\n\n return successes, pending, failures", "response": "Enroll existing users in a course and create pending enrollments for nonexisting users."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nnotifies learners about a specific program in which they have been enrolled.", "response": "def notify_program_learners(cls, enterprise_customer, program_details, users):\n \"\"\"\n Notify learners about a program in which they've been enrolled.\n\n Args:\n enterprise_customer: The EnterpriseCustomer being linked to\n program_details: Details about the specific program the learners were enrolled in\n users: An iterable of the users or pending users who were enrolled\n \"\"\"\n program_name = program_details.get('title')\n program_branding = program_details.get('type')\n program_uuid = program_details.get('uuid')\n\n lms_root_url = get_configuration_value_for_site(\n enterprise_customer.site,\n 'LMS_ROOT_URL',\n settings.LMS_ROOT_URL\n )\n program_path = urlquote(\n '/dashboard/programs/{program_uuid}/?tpa_hint={tpa_hint}'.format(\n program_uuid=program_uuid,\n tpa_hint=enterprise_customer.identity_provider,\n )\n )\n destination_url = '{site}/{login_or_register}?next={program_path}'.format(\n site=lms_root_url,\n login_or_register='{login_or_register}',\n program_path=program_path\n )\n program_type = 'program'\n program_start = get_earliest_start_date_from_program(program_details)\n\n with mail.get_connection() as email_conn:\n for user in users:\n login_or_register = 'register' if isinstance(user, PendingEnterpriseCustomerUser) else 'login'\n destination_url = destination_url.format(login_or_register=login_or_register)\n send_email_notification_message(\n user=user,\n enrolled_in={\n 'name': program_name,\n 'url': destination_url,\n 'type': program_type,\n 'start': program_start,\n 'branding': program_branding,\n },\n enterprise_customer=enterprise_customer,\n email_connection=email_conn\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a message for the users who were enrolled in a course or program.", "response": "def get_success_enrollment_message(cls, users, enrolled_in):\n \"\"\"\n Create message for the users who were enrolled in a course or program.\n\n Args:\n users: An iterable of users who were successfully enrolled\n enrolled_in (str): A string identifier for the course or program the users were enrolled in\n\n Returns:\n tuple: A 2-tuple containing a message type and message text\n \"\"\"\n enrolled_count = len(users)\n return (\n 'success',\n ungettext(\n '{enrolled_count} learner was enrolled in {enrolled_in}.',\n '{enrolled_count} learners were enrolled in {enrolled_in}.',\n enrolled_count,\n ).format(\n enrolled_count=enrolled_count,\n enrolled_in=enrolled_in,\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a message for the users who have not been enrolled in the given course or program.", "response": "def get_failed_enrollment_message(cls, users, enrolled_in):\n \"\"\"\n Create message for the users who were not able to be enrolled in a course or program.\n\n Args:\n users: An iterable of users who were not successfully enrolled\n enrolled_in (str): A string identifier for the course or program with which enrollment was attempted\n\n Returns:\n tuple: A 2-tuple containing a message type and message text\n \"\"\"\n failed_emails = [user.email for user in users]\n return (\n 'error',\n _(\n 'The following learners could not be enrolled in {enrolled_in}: {user_list}'\n ).format(\n enrolled_in=enrolled_in,\n user_list=', '.join(failed_emails),\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_pending_enrollment_message(cls, pending_users, enrolled_in):\n pending_emails = [pending_user.user_email for pending_user in pending_users]\n return (\n 'warning',\n _(\n \"The following learners do not have an account on \"\n \"{platform_name}. They have not been enrolled in \"\n \"{enrolled_in}. When these learners create an account, they will \"\n \"be enrolled automatically: {pending_email_list}\"\n ).format(\n platform_name=settings.PLATFORM_NAME,\n enrolled_in=enrolled_in,\n pending_email_list=', '.join(pending_emails),\n )\n )", "response": "Create message for the pending users who have not been enrolled in a course or program."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _enroll_users(\n cls,\n request,\n enterprise_customer,\n emails,\n mode,\n course_id=None,\n program_details=None,\n notify=True\n ):\n \"\"\"\n Enroll the users with the given email addresses to the courses specified, either specifically or by program.\n\n Args:\n cls (type): The EnterpriseCustomerManageLearnersView class itself\n request: The HTTP request the enrollment is being created by\n enterprise_customer: The instance of EnterpriseCustomer whose attached users we're enrolling\n emails: An iterable of strings containing email addresses to enroll in a course\n mode: The enrollment mode the users will be enrolled in the course with\n course_id: The ID of the course in which we want to enroll\n program_details: Details about a program in which we want to enroll\n notify: Whether to notify (by email) the users that have been enrolled\n \"\"\"\n pending_messages = []\n\n if course_id:\n succeeded, pending, failed = cls.enroll_users_in_course(\n enterprise_customer=enterprise_customer,\n course_id=course_id,\n course_mode=mode,\n emails=emails,\n )\n all_successes = succeeded + pending\n if notify:\n enterprise_customer.notify_enrolled_learners(\n catalog_api_user=request.user,\n course_id=course_id,\n users=all_successes,\n )\n if succeeded:\n pending_messages.append(cls.get_success_enrollment_message(succeeded, course_id))\n if failed:\n pending_messages.append(cls.get_failed_enrollment_message(failed, course_id))\n if pending:\n pending_messages.append(cls.get_pending_enrollment_message(pending, course_id))\n\n if program_details:\n succeeded, pending, failed = cls.enroll_users_in_program(\n enterprise_customer=enterprise_customer,\n program_details=program_details,\n course_mode=mode,\n emails=emails,\n )\n all_successes = succeeded + pending\n if notify:\n cls.notify_program_learners(\n enterprise_customer=enterprise_customer,\n program_details=program_details,\n users=all_successes\n )\n program_identifier = program_details.get('title', program_details.get('uuid', _('the program')))\n if succeeded:\n pending_messages.append(cls.get_success_enrollment_message(succeeded, program_identifier))\n if failed:\n pending_messages.append(cls.get_failed_enrollment_message(failed, program_identifier))\n if pending:\n pending_messages.append(cls.get_pending_enrollment_message(pending, program_identifier))\n\n cls.send_messages(request, pending_messages)", "response": "Enrolls the users in the specified courses."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nhandle GET requests. Returns the appropriate response object.", "response": "def get(self, request, customer_uuid):\n \"\"\"\n Handle GET request - render linked learners list and \"Link learner\" form.\n\n Arguments:\n request (django.http.request.HttpRequest): Request instance\n customer_uuid (str): Enterprise Customer UUID\n\n Returns:\n django.http.response.HttpResponse: HttpResponse\n \"\"\"\n context = self._build_context(request, customer_uuid)\n manage_learners_form = ManageLearnersForm(\n user=request.user,\n enterprise_customer=context[self.ContextParameters.ENTERPRISE_CUSTOMER]\n )\n context.update({self.ContextParameters.MANAGE_LEARNERS_FORM: manage_learners_form})\n\n return render(request, self.template, context)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef post(self, request, customer_uuid):\n enterprise_customer = EnterpriseCustomer.objects.get(uuid=customer_uuid) # pylint: disable=no-member\n manage_learners_form = ManageLearnersForm(\n request.POST,\n request.FILES,\n user=request.user,\n enterprise_customer=enterprise_customer\n )\n\n # initial form validation - check that form data is well-formed\n if manage_learners_form.is_valid():\n email_field_as_bulk_input = split_usernames_and_emails(\n manage_learners_form.cleaned_data[ManageLearnersForm.Fields.EMAIL_OR_USERNAME]\n )\n is_bulk_entry = len(email_field_as_bulk_input) > 1\n # The form is valid. Call the appropriate helper depending on the mode:\n mode = manage_learners_form.cleaned_data[ManageLearnersForm.Fields.MODE]\n if mode == ManageLearnersForm.Modes.MODE_SINGULAR and not is_bulk_entry:\n linked_learners = self._handle_singular(enterprise_customer, manage_learners_form)\n elif mode == ManageLearnersForm.Modes.MODE_SINGULAR:\n linked_learners = self._handle_bulk_upload(\n enterprise_customer,\n manage_learners_form,\n request,\n email_list=email_field_as_bulk_input\n )\n else:\n linked_learners = self._handle_bulk_upload(enterprise_customer, manage_learners_form, request)\n\n # _handle_form might add form errors, so we check if it is still valid\n if manage_learners_form.is_valid():\n course_details = manage_learners_form.cleaned_data.get(ManageLearnersForm.Fields.COURSE)\n program_details = manage_learners_form.cleaned_data.get(ManageLearnersForm.Fields.PROGRAM)\n\n notification_type = manage_learners_form.cleaned_data.get(ManageLearnersForm.Fields.NOTIFY)\n notify = notification_type == ManageLearnersForm.NotificationTypes.BY_EMAIL\n\n course_id = None\n if course_details:\n course_id = course_details['course_id']\n\n if course_id or program_details:\n course_mode = manage_learners_form.cleaned_data[ManageLearnersForm.Fields.COURSE_MODE]\n self._enroll_users(\n request=request,\n enterprise_customer=enterprise_customer,\n emails=linked_learners,\n mode=course_mode,\n course_id=course_id,\n program_details=program_details,\n notify=notify,\n )\n\n # Redirect to GET if everything went smooth.\n manage_learners_url = reverse(\"admin:\" + UrlNames.MANAGE_LEARNERS, args=(customer_uuid,))\n search_keyword = self.get_search_keyword(request)\n if search_keyword:\n manage_learners_url = manage_learners_url + \"?q=\" + search_keyword\n return HttpResponseRedirect(manage_learners_url)\n\n # if something went wrong - display bound form on the page\n context = self._build_context(request, customer_uuid)\n context.update({self.ContextParameters.MANAGE_LEARNERS_FORM: manage_learners_form})\n return render(request, self.template, context)", "response": "Handle POST request - handle form submissions.\n\n Arguments:\n request (django.http.request.HttpRequest): Request instance\n customer_uuid (str): Enterprise Customer UUID\n\n Returns:\n django.http.response.HttpResponse: HttpResponse"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete(self, request, customer_uuid):\n # TODO: pylint acts stupid - find a way around it without suppressing\n enterprise_customer = EnterpriseCustomer.objects.get(uuid=customer_uuid) # pylint: disable=no-member\n email_to_unlink = request.GET[\"unlink_email\"]\n try:\n EnterpriseCustomerUser.objects.unlink_user(\n enterprise_customer=enterprise_customer, user_email=email_to_unlink\n )\n except (EnterpriseCustomerUser.DoesNotExist, PendingEnterpriseCustomerUser.DoesNotExist):\n message = _(\"Email {email} is not associated with Enterprise \"\n \"Customer {ec_name}\").format(\n email=email_to_unlink, ec_name=enterprise_customer.name)\n return HttpResponse(message, content_type=\"application/json\", status=404)\n\n return HttpResponse(\n json.dumps({}),\n content_type=\"application/json\"\n )", "response": "Handle DELETE request - handle unlinking learner."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nperforming the query and returns a single object matching the given keyword arguments.", "response": "def proxied_get(self, *args, **kwargs):\n \"\"\"\n Perform the query and returns a single object matching the given keyword arguments.\n\n This customizes the queryset to return an instance of ``ProxyDataSharingConsent`` when\n the searched-for ``DataSharingConsent`` instance does not exist.\n \"\"\"\n original_kwargs = kwargs.copy()\n if 'course_id' in kwargs:\n try:\n # Check if we have a course ID or a course run ID\n course_run_key = str(CourseKey.from_string(kwargs['course_id']))\n except InvalidKeyError:\n # The ID we have is for a course instead of a course run; fall through\n # to the second check.\n pass\n else:\n try:\n # Try to get the record for the course run specifically\n return self.get(*args, **kwargs)\n except DataSharingConsent.DoesNotExist:\n # A record for the course run didn't exist, so modify the query\n # parameters to look for just a course record on the second pass.\n kwargs['course_id'] = parse_course_key(course_run_key)\n\n try:\n return self.get(*args, **kwargs)\n except DataSharingConsent.DoesNotExist:\n return ProxyDataSharingConsent(**original_kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_children(cls, program_uuid, *children):\n if not children or any(child is None for child in children):\n return None\n granted = all((child.granted for child in children))\n exists = any((child.exists for child in children))\n usernames = set([child.username for child in children])\n enterprises = set([child.enterprise_customer for child in children])\n if not len(usernames) == len(enterprises) == 1:\n raise InvalidProxyConsent(\n 'Children used to create a bulk proxy consent object must '\n 'share a single common username and EnterpriseCustomer.'\n )\n username = children[0].username\n enterprise_customer = children[0].enterprise_customer\n return cls(\n enterprise_customer=enterprise_customer,\n username=username,\n program_uuid=program_uuid,\n exists=exists,\n granted=granted,\n child_consents=children\n )", "response": "Build a ProxyDataSharingConsent object from a list of ConsentRecords."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef commit(self):\n if self._child_consents:\n consents = []\n\n for consent in self._child_consents:\n consent.granted = self.granted\n consents.append(consent.save() or consent)\n\n return ProxyDataSharingConsent.from_children(self.program_uuid, *consents)\n\n consent, _ = DataSharingConsent.objects.update_or_create(\n enterprise_customer=self.enterprise_customer,\n username=self.username,\n course_id=self.course_id,\n defaults={\n 'granted': self.granted\n }\n )\n self._exists = consent.exists\n return consent", "response": "Commits a real DataSharingConsent object to the database."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending xAPI statements to the LRS for the given number of days.", "response": "def send_xapi_statements(self, lrs_configuration, days):\n \"\"\"\n Send xAPI analytics data of the enterprise learners to the given LRS.\n\n Arguments:\n lrs_configuration (XAPILRSConfiguration): Configuration object containing LRS configurations\n of the LRS where to send xAPI learner analytics.\n days (int): Include course enrollment of this number of days.\n \"\"\"\n persistent_course_grades = self.get_course_completions(lrs_configuration.enterprise_customer, days)\n users = self.prefetch_users(persistent_course_grades)\n course_overviews = self.prefetch_courses(persistent_course_grades)\n\n for persistent_course_grade in persistent_course_grades:\n try:\n user = users.get(persistent_course_grade.user_id)\n course_overview = course_overviews.get(persistent_course_grade.course_id)\n course_grade = CourseGradeFactory().read(user, course_key=persistent_course_grade.course_id)\n send_course_completion_statement(lrs_configuration, user, course_overview, course_grade)\n except ClientError:\n LOGGER.exception(\n 'Client error while sending course completion to xAPI for'\n ' enterprise customer {enterprise_customer}.'.format(\n enterprise_customer=lrs_configuration.enterprise_customer.name\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_course_completions(self, enterprise_customer, days):\n return PersistentCourseGrade.objects.filter(\n passed_timestamp__gt=datetime.datetime.now() - datetime.timedelta(days=days)\n ).filter(\n user_id__in=enterprise_customer.enterprise_customer_users.values_list('user_id', flat=True)\n )", "response": "Returns a list of PersistentCourseGrade objects for all the learners of given enterprise customer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef prefetch_users(persistent_course_grades):\n users = User.objects.filter(\n id__in=[grade.user_id for grade in persistent_course_grades]\n )\n return {\n user.id: user for user in users\n }", "response": "Prefetches users from the list of user_ids present in the persistent_course_grades."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates the enterprise roles if they do not already exist.", "response": "def create_roles(apps, schema_editor):\n \"\"\"Create the enterprise roles if they do not already exist.\"\"\"\n EnterpriseFeatureRole = apps.get_model('enterprise', 'EnterpriseFeatureRole')\n EnterpriseFeatureRole.objects.update_or_create(name=ENTERPRISE_CATALOG_ADMIN_ROLE)\n EnterpriseFeatureRole.objects.update_or_create(name=ENTERPRISE_DASHBOARD_ADMIN_ROLE)\n EnterpriseFeatureRole.objects.update_or_create(name=ENTERPRISE_ENROLLMENT_API_ADMIN_ROLE)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef delete_roles(apps, schema_editor):\n EnterpriseFeatureRole = apps.get_model('enterprise', 'EnterpriseFeatureRole')\n EnterpriseFeatureRole.objects.filter(\n name__in=[ENTERPRISE_CATALOG_ADMIN_ROLE, ENTERPRISE_DASHBOARD_ADMIN_ROLE, ENTERPRISE_ENROLLMENT_API_ADMIN_ROLE]\n ).delete()", "response": "Delete the enterprise roles."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_identity_provider(provider_id):\n try:\n from third_party_auth.provider import Registry # pylint: disable=redefined-outer-name\n except ImportError as exception:\n LOGGER.warning(\"Could not import Registry from third_party_auth.provider\")\n LOGGER.warning(exception)\n Registry = None # pylint: disable=redefined-outer-name\n\n try:\n return Registry and Registry.get(provider_id)\n except ValueError:\n return None", "response": "Get Identity Provider with given id."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_idp_choices():\n try:\n from third_party_auth.provider import Registry # pylint: disable=redefined-outer-name\n except ImportError as exception:\n LOGGER.warning(\"Could not import Registry from third_party_auth.provider\")\n LOGGER.warning(exception)\n Registry = None # pylint: disable=redefined-outer-name\n\n first = [(\"\", \"-\" * 7)]\n if Registry:\n return first + [(idp.provider_id, idp.name) for idp in Registry.enabled()]\n return None", "response": "Get a list of identity providers choices for enterprise customer."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the template of catalog admin url.", "response": "def get_catalog_admin_url_template(mode='change'):\n \"\"\"\n Get template of catalog admin url.\n\n URL template will contain a placeholder '{catalog_id}' for catalog id.\n Arguments:\n mode e.g. change/add.\n\n Returns:\n A string containing template for catalog url.\n\n Example:\n >>> get_catalog_admin_url_template('change')\n \"http://localhost:18381/admin/catalogs/catalog/{catalog_id}/change/\"\n\n \"\"\"\n api_base_url = getattr(settings, \"COURSE_CATALOG_API_URL\", \"\")\n\n # Extract FQDN (Fully Qualified Domain Name) from API URL.\n match = re.match(r\"^(?P(?:https?://)?[^/]+)\", api_base_url)\n\n if not match:\n return \"\"\n\n # Return matched FQDN from catalog api url appended with catalog admin path\n if mode == 'change':\n return match.group(\"fqdn\").rstrip(\"/\") + \"/admin/catalogs/catalog/{catalog_id}/change/\"\n elif mode == 'add':\n return match.group(\"fqdn\").rstrip(\"/\") + \"/admin/catalogs/catalog/add/\""} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbuilding a message body for a user notification.", "response": "def build_notification_message(template_context, template_configuration=None):\n \"\"\"\n Create HTML and plaintext message bodies for a notification.\n\n We receive a context with data we can use to render, as well as an optional site\n template configration - if we don't get a template configuration, we'll use the\n standard, built-in template.\n\n Arguments:\n template_context (dict): A set of data to render\n template_configuration: A database-backed object with templates\n stored that can be used to render a notification.\n\n \"\"\"\n if (\n template_configuration is not None and\n template_configuration.html_template and\n template_configuration.plaintext_template\n ):\n plain_msg, html_msg = template_configuration.render_all_templates(template_context)\n else:\n plain_msg = render_to_string(\n 'enterprise/emails/user_notification.txt',\n template_context\n )\n html_msg = render_to_string(\n 'enterprise/emails/user_notification.html',\n template_context\n )\n\n return plain_msg, html_msg"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_notification_subject_line(course_name, template_configuration=None):\n stock_subject_template = _('You\\'ve been enrolled in {course_name}!')\n default_subject_template = getattr(\n settings,\n 'ENTERPRISE_ENROLLMENT_EMAIL_DEFAULT_SUBJECT_LINE',\n stock_subject_template,\n )\n if template_configuration is not None and template_configuration.subject_line:\n final_subject_template = template_configuration.subject_line\n else:\n final_subject_template = default_subject_template\n\n try:\n return final_subject_template.format(course_name=course_name)\n except KeyError:\n pass\n\n try:\n return default_subject_template.format(course_name=course_name)\n except KeyError:\n return stock_subject_template.format(course_name=course_name)", "response": "Returns a string that can be used to display a subject line for a notification email."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef send_email_notification_message(user, enrolled_in, enterprise_customer, email_connection=None):\n if hasattr(user, 'first_name') and hasattr(user, 'username'):\n # PendingEnterpriseCustomerUsers don't have usernames or real names. We should\n # template slightly differently to make sure weird stuff doesn't happen.\n user_name = user.first_name\n if not user_name:\n user_name = user.username\n else:\n user_name = None\n\n # Users have an `email` attribute; PendingEnterpriseCustomerUsers have `user_email`.\n if hasattr(user, 'email'):\n user_email = user.email\n elif hasattr(user, 'user_email'):\n user_email = user.user_email\n else:\n raise TypeError(_('`user` must have one of either `email` or `user_email`.'))\n\n msg_context = {\n 'user_name': user_name,\n 'enrolled_in': enrolled_in,\n 'organization_name': enterprise_customer.name,\n }\n try:\n enterprise_template_config = enterprise_customer.enterprise_enrollment_template\n except (ObjectDoesNotExist, AttributeError):\n enterprise_template_config = None\n\n plain_msg, html_msg = build_notification_message(msg_context, enterprise_template_config)\n\n subject_line = get_notification_subject_line(enrolled_in['name'], enterprise_template_config)\n\n from_email_address = get_configuration_value_for_site(\n enterprise_customer.site,\n 'DEFAULT_FROM_EMAIL',\n default=settings.DEFAULT_FROM_EMAIL\n )\n\n return mail.send_mail(\n subject_line,\n plain_msg,\n from_email_address,\n [user_email],\n html_message=html_msg,\n connection=email_connection\n )", "response": "Sends an email notification message to the user that the user has enrolled in."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_enterprise_customer(uuid):\n EnterpriseCustomer = apps.get_model('enterprise', 'EnterpriseCustomer') # pylint: disable=invalid-name\n try:\n return EnterpriseCustomer.objects.get(uuid=uuid) # pylint: disable=no-member\n except EnterpriseCustomer.DoesNotExist:\n return None", "response": "Get the EnterpriseCustomer instance associated with uuid."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_enterprise_customer_for_user(auth_user):\n EnterpriseCustomerUser = apps.get_model('enterprise', 'EnterpriseCustomerUser') # pylint: disable=invalid-name\n try:\n return EnterpriseCustomerUser.objects.get(user_id=auth_user.id).enterprise_customer # pylint: disable=no-member\n except EnterpriseCustomerUser.DoesNotExist:\n return None", "response": "Returns the enterprise customer instance for given user."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_enterprise_customer_user(user_id, enterprise_uuid):\n EnterpriseCustomerUser = apps.get_model('enterprise', 'EnterpriseCustomerUser') # pylint: disable=invalid-name\n try:\n return EnterpriseCustomerUser.objects.get( # pylint: disable=no-member\n enterprise_customer__uuid=enterprise_uuid,\n user_id=user_id\n )\n except EnterpriseCustomerUser.DoesNotExist:\n return None", "response": "Returns the object for EnterpriseCustomerUser."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the URL for the given course run.", "response": "def get_course_track_selection_url(course_run, query_parameters):\n \"\"\"\n Return track selection url for the given course.\n\n Arguments:\n course_run (dict): A dictionary containing course run metadata.\n query_parameters (dict): A dictionary containing query parameters to be added to course selection url.\n\n Raises:\n (KeyError): Raised when course run dict does not have 'key' key.\n\n Returns:\n (str): Course track selection url.\n\n \"\"\"\n try:\n course_root = reverse('course_modes_choose', kwargs={'course_id': course_run['key']})\n except KeyError:\n LOGGER.exception(\n \"KeyError while parsing course run data.\\nCourse Run: \\n[%s]\", course_run,\n )\n raise\n\n url = '{}{}'.format(\n settings.LMS_ROOT_URL,\n course_root\n )\n course_run_url = update_query_parameters(url, query_parameters)\n\n return course_run_url"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef update_query_parameters(url, query_parameters):\n scheme, netloc, path, query_string, fragment = urlsplit(url)\n url_params = parse_qs(query_string)\n\n # Update url query parameters\n url_params.update(query_parameters)\n\n return urlunsplit(\n (scheme, netloc, path, urlencode(sorted(url_params.items()), doseq=True), fragment),\n )", "response": "Updates the query parameters of the url with the given dictionary."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfiltering audit course modes out if the enterprise customer has not enabled the Enable audit enrollment flag.", "response": "def filter_audit_course_modes(enterprise_customer, course_modes):\n \"\"\"\n Filter audit course modes out if the enterprise customer has not enabled the 'Enable audit enrollment' flag.\n\n Arguments:\n enterprise_customer: The EnterpriseCustomer that the enrollment was created using.\n course_modes: iterable with dictionaries containing a required 'mode' key\n\n \"\"\"\n audit_modes = getattr(settings, 'ENTERPRISE_COURSE_ENROLLMENT_AUDIT_MODES', ['audit'])\n if not enterprise_customer.enable_audit_enrollment:\n return [course_mode for course_mode in course_modes if course_mode['mode'] not in audit_modes]\n return course_modes"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the EnterpriseCustomer given an EnterpriseCustomer UUID or raises a 404.", "response": "def get_enterprise_customer_or_404(enterprise_uuid):\n \"\"\"\n Given an EnterpriseCustomer UUID, return the corresponding EnterpriseCustomer or raise a 404.\n\n Arguments:\n enterprise_uuid (str): The UUID (in string form) of the EnterpriseCustomer to fetch.\n\n Returns:\n (EnterpriseCustomer): The EnterpriseCustomer given the UUID.\n\n \"\"\"\n EnterpriseCustomer = apps.get_model('enterprise', 'EnterpriseCustomer') # pylint: disable=invalid-name\n try:\n enterprise_uuid = UUID(enterprise_uuid)\n return EnterpriseCustomer.objects.get(uuid=enterprise_uuid) # pylint: disable=no-member\n except (TypeError, ValueError, EnterpriseCustomer.DoesNotExist):\n LOGGER.error('Unable to find enterprise customer for UUID: [%s]', enterprise_uuid)\n raise Http404"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_cache_key(**kwargs):\n key = '__'.join(['{}:{}'.format(item, value) for item, value in iteritems(kwargs)])\n\n return hashlib.md5(key.encode('utf-8')).hexdigest()", "response": "Returns a MD5 encoded cache key for given arguments."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef traverse_pagination(response, endpoint):\n results = response.get('results', [])\n\n next_page = response.get('next')\n while next_page:\n querystring = parse_qs(urlparse(next_page).query, keep_blank_values=True)\n response = endpoint.get(**querystring)\n results += response.get('results', [])\n next_page = response.get('next')\n\n return results", "response": "Traverse a paginated API response."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ungettext_min_max(singular, plural, range_text, min_val, max_val):\n if min_val is None and max_val is None:\n return None\n if min_val == max_val or min_val is None or max_val is None:\n # pylint: disable=translation-of-non-string\n return ungettext(singular, plural, min_val or max_val).format(min_val or max_val)\n return range_text.format(min_val, max_val)", "response": "Return grammatically correct translated text based off of a minimum and maximum value."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef format_price(price, currency='$'):\n if int(price) == price:\n return '{}{}'.format(currency, int(price))\n return '{}{:0.2f}'.format(currency, price)", "response": "Format the price to have the appropriate currency and digits."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the value for a key in the site configuration or settings file.", "response": "def get_configuration_value_for_site(site, key, default=None):\n \"\"\"\n Get the site configuration value for a key, unless a site configuration does not exist for that site.\n\n Useful for testing when no Site Configuration exists in edx-enterprise or if a site in LMS doesn't have\n a configuration tied to it.\n\n :param site: A Site model object\n :param key: The name of the value to retrieve\n :param default: The default response if there's no key in site config or settings\n :return: The value located at that key in the site configuration or settings file.\n \"\"\"\n if hasattr(site, 'configuration'):\n return site.configuration.get_value(key, default)\n return default"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_configuration_value(val_name, default=None, **kwargs):\n if kwargs.get('type') == 'url':\n return get_url(val_name) or default if callable(get_url) else default\n return configuration_helpers.get_value(val_name, default, **kwargs) if configuration_helpers else default", "response": "Get a configuration value or fall back to default if it doesn t exist."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_request_value(request, key, default=None):\n if request.method in ['GET', 'DELETE']:\n return request.query_params.get(key, request.data.get(key, default))\n return request.data.get(key, request.query_params.get(key, default))", "response": "Get the value from a request."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntracks an event for some parts of the Enterprise workflows.", "response": "def track_event(user_id, event_name, properties):\n \"\"\"\n Emit a track event to segment (and forwarded to GA) for some parts of the Enterprise workflows.\n \"\"\"\n # Only call the endpoint if the import was successful.\n if segment:\n segment.track(user_id, event_name, properties)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nemitting a track event for enterprise course enrollment.", "response": "def track_enrollment(pathway, user_id, course_run_id, url_path=None):\n \"\"\"\n Emit a track event for enterprise course enrollment.\n \"\"\"\n track_event(user_id, 'edx.bi.user.enterprise.onboarding', {\n 'pathway': pathway,\n 'url_path': url_path,\n 'course_run_id': course_run_id,\n })"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_course_run_enrollable(course_run):\n now = datetime.datetime.now(pytz.UTC)\n end = parse_datetime_handle_invalid(course_run.get('end'))\n enrollment_start = parse_datetime_handle_invalid(course_run.get('enrollment_start'))\n enrollment_end = parse_datetime_handle_invalid(course_run.get('enrollment_end'))\n return (not end or end > now) and \\\n (not enrollment_start or enrollment_start < now) and \\\n (not enrollment_end or enrollment_end > now)", "response": "Return true if the course run is enrollable false otherwise."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_course_run_upgradeable(course_run):\n now = datetime.datetime.now(pytz.UTC)\n for seat in course_run.get('seats', []):\n if seat.get('type') == 'verified':\n upgrade_deadline = parse_datetime_handle_invalid(seat.get('upgrade_deadline'))\n return not upgrade_deadline or upgrade_deadline > now\n return False", "response": "Return true if the course run has a verified seat with an unexpired upgrade deadline false otherwise."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_closest_course_run(course_runs):\n if len(course_runs) == 1:\n return course_runs[0]\n\n now = datetime.datetime.now(pytz.UTC)\n # course runs with no start date should be considered last.\n never = now - datetime.timedelta(days=3650)\n return min(course_runs, key=lambda x: abs(get_course_run_start(x, never) - now))", "response": "Return the closest course run to now."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the course_runs of the given course which have the active enrolment.", "response": "def get_active_course_runs(course, users_all_enrolled_courses):\n \"\"\"\n Return active course runs (user is enrolled in) of the given course.\n\n This function will return the course_runs of 'course' which have\n active enrollment by looking into 'users_all_enrolled_courses'\n \"\"\"\n # User's all course_run ids in which he has enrolled.\n enrolled_course_run_ids = [\n enrolled_course_run['course_details']['course_id'] for enrolled_course_run in users_all_enrolled_courses\n if enrolled_course_run['is_active'] and enrolled_course_run.get('course_details')\n ]\n return [course_run for course_run in course['course_runs'] if course_run['key'] in enrolled_course_run_ids]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_current_course_run(course, users_active_course_runs):\n current_course_run = None\n filtered_course_runs = []\n all_course_runs = course['course_runs']\n\n if users_active_course_runs:\n current_course_run = get_closest_course_run(users_active_course_runs)\n else:\n for course_run in all_course_runs:\n if is_course_run_enrollable(course_run) and is_course_run_upgradeable(course_run):\n filtered_course_runs.append(course_run)\n\n if not filtered_course_runs:\n # Consider all runs if there were not any enrollable/upgradeable ones.\n filtered_course_runs = all_course_runs\n\n if filtered_course_runs:\n current_course_run = get_closest_course_run(filtered_course_runs)\n return current_course_run", "response": "Returns the current course run on the following conditions."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef strip_html_tags(text, allowed_tags=None):\n if text is None:\n return\n if allowed_tags is None:\n allowed_tags = ALLOWED_TAGS\n return bleach.clean(text, tags=allowed_tags, attributes=['id', 'class', 'style', 'href', 'title'], strip=True)", "response": "Strip all html tags from a string except those tags provided in allowed_tags parameter."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_course_key(course_identifier):\n try:\n course_run_key = CourseKey.from_string(course_identifier)\n except InvalidKeyError:\n # Assume we already have a course key.\n return course_identifier\n\n return quote_plus(' '.join([course_run_key.org, course_run_key.course]))", "response": "Parse a course key given either a course run ID or a course key."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef create_roles(apps, schema_editor):\n SystemWideEnterpriseRole = apps.get_model('enterprise', 'SystemWideEnterpriseRole')\n SystemWideEnterpriseRole.objects.update_or_create(name=ENTERPRISE_OPERATOR_ROLE)", "response": "Create the enterprise roles if they do not already exist."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_roles(apps, schema_editor):\n SystemWideEnterpriseRole = apps.get_model('enterprise', 'SystemWideEnterpriseRole')\n SystemWideEnterpriseRole.objects.filter(\n name__in=[ENTERPRISE_OPERATOR_ROLE]\n ).delete()", "response": "Delete the enterprise roles."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a new instance of RemoteLRS.", "response": "def lrs(self):\n \"\"\"\n LRS client instance to be used for sending statements.\n \"\"\"\n return RemoteLRS(\n version=self.lrs_configuration.version,\n endpoint=self.lrs_configuration.endpoint,\n auth=self.lrs_configuration.authorization_header,\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef save_statement(self, statement):\n response = self.lrs.save_statement(statement)\n\n if not response:\n raise ClientError('EnterpriseXAPIClient request failed.')", "response": "Save an enterprise statement to the LRS."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_learner_data_records(self, enterprise_enrollment, completed_date=None, grade=None, is_passing=False):\n completed_timestamp = None\n course_completed = False\n if completed_date is not None:\n completed_timestamp = parse_datetime_to_epoch_millis(completed_date)\n course_completed = is_passing\n\n sapsf_user_id = enterprise_enrollment.enterprise_customer_user.get_remote_id()\n\n if sapsf_user_id is not None:\n SapSuccessFactorsLearnerDataTransmissionAudit = apps.get_model( # pylint: disable=invalid-name\n 'sap_success_factors',\n 'SapSuccessFactorsLearnerDataTransmissionAudit'\n )\n # We return two records here, one with the course key and one with the course run id, to account for\n # uncertainty about the type of content (course vs. course run) that was sent to the integrated channel.\n return [\n SapSuccessFactorsLearnerDataTransmissionAudit(\n enterprise_course_enrollment_id=enterprise_enrollment.id,\n sapsf_user_id=sapsf_user_id,\n course_id=parse_course_key(enterprise_enrollment.course_id),\n course_completed=course_completed,\n completed_timestamp=completed_timestamp,\n grade=grade,\n ),\n SapSuccessFactorsLearnerDataTransmissionAudit(\n enterprise_course_enrollment_id=enterprise_enrollment.id,\n sapsf_user_id=sapsf_user_id,\n course_id=enterprise_enrollment.course_id,\n course_completed=course_completed,\n completed_timestamp=completed_timestamp,\n grade=grade,\n ),\n ]\n else:\n LOGGER.debug(\n 'No learner data was sent for user [%s] because an SAP SuccessFactors user ID could not be found.',\n enterprise_enrollment.enterprise_customer_user.username\n )", "response": "Returns a list of SapSuccessFactorsLearnerDataTransmissionAudit records with the given enrollment and course completion data."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\niterate over each learner and unlink inactive SAP channel learners. This method iterates over each enterprise learner and unlink learner from the enterprise if the learner is marked inactive in the related integrated channel.", "response": "def unlink_learners(self):\n \"\"\"\n Iterate over each learner and unlink inactive SAP channel learners.\n\n This method iterates over each enterprise learner and unlink learner\n from the enterprise if the learner is marked inactive in the related\n integrated channel.\n \"\"\"\n sap_inactive_learners = self.client.get_inactive_sap_learners()\n enterprise_customer = self.enterprise_configuration.enterprise_customer\n if not sap_inactive_learners:\n LOGGER.info(\n 'Enterprise customer {%s} has no SAPSF inactive learners',\n enterprise_customer.name\n )\n return\n\n provider_id = enterprise_customer.identity_provider\n tpa_provider = get_identity_provider(provider_id)\n if not tpa_provider:\n LOGGER.info(\n 'Enterprise customer {%s} has no associated identity provider',\n enterprise_customer.name\n )\n return None\n\n for sap_inactive_learner in sap_inactive_learners:\n social_auth_user = get_user_from_social_auth(tpa_provider, sap_inactive_learner['studentID'])\n if not social_auth_user:\n continue\n\n try:\n # Unlink user email from related Enterprise Customer\n EnterpriseCustomerUser.objects.unlink_user(\n enterprise_customer=enterprise_customer,\n user_email=social_auth_user.email,\n )\n except (EnterpriseCustomerUser.DoesNotExist, PendingEnterpriseCustomerUser.DoesNotExist):\n LOGGER.info(\n 'Learner with email {%s} is not associated with Enterprise Customer {%s}',\n social_auth_user.email,\n enterprise_customer.name\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef has_implicit_access_to_dashboard(user, obj): # pylint: disable=unused-argument\n request = get_request_or_stub()\n decoded_jwt = get_decoded_jwt_from_request(request)\n return request_user_has_implicit_access_via_jwt(decoded_jwt, ENTERPRISE_DASHBOARD_ADMIN_ROLE)", "response": "Check that if request user has implicit access to dashboard admin role."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef has_implicit_access_to_catalog(user, obj): # pylint: disable=unused-argument\n request = get_request_or_stub()\n decoded_jwt = get_decoded_jwt_from_request(request)\n return request_user_has_implicit_access_via_jwt(decoded_jwt, ENTERPRISE_CATALOG_ADMIN_ROLE, obj)", "response": "Check that if request user has implicit access to catalog admin role."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef has_implicit_access_to_enrollment_api(user, obj): # pylint: disable=unused-argument\n request = get_request_or_stub()\n decoded_jwt = get_decoded_jwt_from_request(request)\n return request_user_has_implicit_access_via_jwt(decoded_jwt, ENTERPRISE_ENROLLMENT_API_ADMIN_ROLE, obj)", "response": "Check that if request user has implicit access to enrollment api."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntransforming ISO language code to the language name expected by SAPSF.", "response": "def transform_language_code(code):\n \"\"\"\n Transform ISO language code (e.g. en-us) to the language name expected by SAPSF.\n \"\"\"\n if code is None:\n return 'English'\n\n components = code.split('-', 2)\n language_code = components[0]\n try:\n country_code = components[1]\n except IndexError:\n country_code = '_'\n\n language_family = SUCCESSFACTORS_OCN_LANGUAGE_CODES.get(language_code)\n if not language_family:\n return 'English'\n\n return language_family.get(country_code, language_family['_'])"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ecommerce_coupon_url(self, instance):\n if not instance.entitlement_id:\n return \"N/A\"\n\n return format_html(\n 'View coupon \"{id}\" details',\n base_url=settings.ECOMMERCE_PUBLIC_URL_ROOT, id=instance.entitlement_id\n )", "response": "Return e - commerce coupon url."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndropping the historical sap_success_factors table named herein.", "response": "def dropHistoricalTable(apps, schema_editor):\n \"\"\"\n Drops the historical sap_success_factors table named herein.\n \"\"\"\n table_name = 'sap_success_factors_historicalsapsuccessfactorsenterprisecus80ad'\n if table_name in connection.introspection.table_names():\n migrations.DeleteModel(\n name=table_name,\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef handle(self, *args, **options):\n # Ensure that we were given an api_user name, and that User exists.\n api_username = options['api_user']\n try:\n User.objects.get(username=api_username)\n except User.DoesNotExist:\n raise CommandError(_('A user with the username {username} was not found.').format(username=api_username))\n\n # Transmit the learner data to each integrated channel\n for integrated_channel in self.get_integrated_channels(options):\n transmit_learner_data.delay(api_username, integrated_channel.channel_code(), integrated_channel.pk)", "response": "Transmit the learner data for the EnterpriseCustomer to the active integration channels."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an export csv action.", "response": "def export_as_csv_action(description=\"Export selected objects as CSV file\", fields=None, header=True):\n \"\"\"\n Return an export csv action.\n\n Arguments:\n description (string): action description\n fields ([string]): list of model fields to include\n header (bool): whether or not to output the column names as the first row\n \"\"\"\n # adapted from https://gist.github.com/mgerring/3645889\n def export_as_csv(modeladmin, request, queryset): # pylint: disable=unused-argument\n \"\"\"\n Export model fields to CSV.\n \"\"\"\n opts = modeladmin.model._meta\n\n if not fields:\n field_names = [field.name for field in opts.fields]\n else:\n field_names = fields\n\n response = HttpResponse(content_type=\"text/csv\")\n response[\"Content-Disposition\"] = \"attachment; filename={filename}.csv\".format(\n filename=str(opts).replace(\".\", \"_\")\n )\n\n writer = unicodecsv.writer(response, encoding=\"utf-8\")\n if header:\n writer.writerow(field_names)\n for obj in queryset:\n row = []\n for field_name in field_names:\n field = getattr(obj, field_name)\n if callable(field):\n value = field()\n else:\n value = field\n if value is None:\n row.append(\"[Not Set]\")\n elif not value and isinstance(value, string_types):\n row.append(\"[Empty]\")\n else:\n row.append(value)\n writer.writerow(row)\n return response\n\n export_as_csv.short_description = description\n return export_as_csv"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_clear_catalog_id_action(description=None):\n description = description or _(\"Unlink selected objects from existing course catalogs\")\n\n def clear_catalog_id(modeladmin, request, queryset): # pylint: disable=unused-argument\n \"\"\"\n Clear the catalog ID for a selected EnterpriseCustomer.\n \"\"\"\n queryset.update(catalog=None)\n clear_catalog_id.short_description = description\n return clear_catalog_id", "response": "Returns the action method to clear the catalog ID for a selected EnterpriseCustomer."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlogging-in to pybotvac account using provided email and password.", "response": "def _login(self, email, password):\n \"\"\"\n Login to pybotvac account using provided email and password.\n\n :param email: email for pybotvac account\n :param password: Password for pybotvac account\n :return:\n \"\"\"\n response = requests.post(urljoin(self.ENDPOINT, 'sessions'),\n json={'email': email,\n 'password': password,\n 'platform': 'ios',\n 'token': binascii.hexlify(os.urandom(64)).decode('utf8')},\n headers=self._headers)\n\n response.raise_for_status()\n access_token = response.json()['access_token']\n\n self._headers['Authorization'] = 'Token token=%s' % access_token"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrefresh the maps of the user s maps.", "response": "def refresh_maps(self):\n \"\"\"\n Get information about maps of the robots.\n\n :return:\n \"\"\"\n for robot in self.robots:\n resp2 = (\n requests.get(urljoin(self.ENDPOINT, 'users/me/robots/{}/maps'.format(robot.serial)),\n headers=self._headers))\n resp2.raise_for_status()\n self._maps.update({robot.serial: resp2.json()})"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nrefresh the list of robots connected to account.", "response": "def refresh_robots(self):\n \"\"\"\n Get information about robots connected to account.\n\n :return:\n \"\"\"\n resp = requests.get(urljoin(self.ENDPOINT, 'dashboard'),\n headers=self._headers)\n resp.raise_for_status()\n\n for robot in resp.json()['robots']:\n if robot['mac_address'] is None:\n continue # Ignore robots without mac-address\n\n try:\n self._robots.add(Robot(name=robot['name'],\n serial=robot['serial'],\n secret=robot['secret_key'],\n traits=robot['traits'],\n endpoint=robot['nucleo_url']))\n except requests.exceptions.HTTPError:\n print (\"Your '{}' robot is offline.\".format(robot['name']))\n continue\n\n self.refresh_persistent_maps()\n for robot in self._robots:\n robot.has_persistent_maps = robot.serial in self._persistent_maps"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_map_image(url, dest_path=None):\n image = requests.get(url, stream=True, timeout=10)\n\n if dest_path:\n image_url = url.rsplit('/', 2)[1] + '-' + url.rsplit('/', 1)[1]\n image_filename = image_url.split('?')[0]\n dest = os.path.join(dest_path, image_filename)\n image.raise_for_status()\n with open(dest, 'wb') as data:\n image.raw.decode_content = True\n shutil.copyfileobj(image.raw, data)\n\n return image.raw", "response": "Get a requested map from a robot."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef refresh_persistent_maps(self):\n for robot in self._robots:\n resp2 = (requests.get(urljoin(\n self.ENDPOINT,\n 'users/me/robots/{}/persistent_maps'.format(robot.serial)),\n headers=self._headers))\n resp2.raise_for_status()\n self._persistent_maps.update({robot.serial: resp2.json()})", "response": "Refreshes the persistent maps of the robots."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend a message to the robot with data from the json parameter.", "response": "def _message(self, json):\n \"\"\"\n Sends message to robot with data from parameter 'json'\n :param json: dict containing data to send\n :return: server response\n \"\"\"\n\n cert_path = os.path.join(os.path.dirname(__file__), 'cert', 'neatocloud.com.crt')\n response = requests.post(self._url,\n json=json,\n verify=cert_path,\n auth=Auth(self.serial, self.secret),\n headers=self._headers)\n response.raise_for_status()\n return response"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd the edge lengths to the graph and returns the resulting graph.", "response": "def add_edge_lengths(g):\n \"\"\"Add add the edge lengths as a :any:`DiGraph`\n for the graph.\n\n Uses the ``pos`` vertex property to get the location of each\n vertex. These are then used to calculate the length of an edge\n between two vertices.\n\n Parameters\n ----------\n g : :any:`networkx.DiGraph`, :class:`numpy.ndarray`, dict, \\\n ``None``, etc.\n Any object that networkx can turn into a\n :any:`DiGraph`\n\n Returns\n -------\n :class:`.QueueNetworkDiGraph`\n Returns the a graph with the ``edge_length`` edge property.\n\n Raises\n ------\n TypeError\n Raised when the parameter ``g`` is not of a type that can be\n made into a :any:`networkx.DiGraph`.\n\n \"\"\"\n g = _test_graph(g)\n g.new_edge_property('edge_length')\n\n for e in g.edges():\n latlon1 = g.vp(e[1], 'pos')\n latlon2 = g.vp(e[0], 'pos')\n g.set_ep(e, 'edge_length', np.round(_calculate_distance(latlon1, latlon2), 3))\n\n return g"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _prepare_graph(g, g_colors, q_cls, q_arg, adjust_graph):\n g = _test_graph(g)\n\n if adjust_graph:\n pos = nx.get_node_attributes(g, 'pos')\n ans = nx.to_dict_of_dicts(g)\n g = adjacency2graph(ans, adjust=2, is_directed=g.is_directed())\n g = QueueNetworkDiGraph(g)\n if len(pos) > 0:\n g.set_pos(pos)\n\n g.new_vertex_property('vertex_color')\n g.new_vertex_property('vertex_fill_color')\n g.new_vertex_property('vertex_pen_width')\n g.new_vertex_property('vertex_size')\n\n g.new_edge_property('edge_control_points')\n g.new_edge_property('edge_color')\n g.new_edge_property('edge_marker_size')\n g.new_edge_property('edge_pen_width')\n\n queues = _set_queues(g, q_cls, q_arg, 'cap' in g.vertex_properties())\n\n if 'pos' not in g.vertex_properties():\n g.set_pos()\n\n for k, e in enumerate(g.edges()):\n g.set_ep(e, 'edge_pen_width', 1.25)\n g.set_ep(e, 'edge_marker_size', 8)\n if e[0] == e[1]:\n g.set_ep(e, 'edge_color', queues[k].colors['edge_loop_color'])\n else:\n g.set_ep(e, 'edge_color', queues[k].colors['edge_color'])\n\n for v in g.nodes():\n g.set_vp(v, 'vertex_pen_width', 1)\n g.set_vp(v, 'vertex_size', 8)\n e = (v, v)\n if g.is_edge(e):\n g.set_vp(v, 'vertex_color', queues[g.edge_index[e]]._current_color(2))\n g.set_vp(v, 'vertex_fill_color', queues[g.edge_index[e]]._current_color())\n else:\n g.set_vp(v, 'vertex_color', g_colors['vertex_color'])\n g.set_vp(v, 'vertex_fill_color', g_colors['vertex_fill_color'])\n\n return g, queues", "response": "Prepares a node - level graph for use in the QueueNetwork class."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef desired_destination(self, network, edge):\n n = len(network.out_edges[edge[1]])\n if n <= 1:\n return network.out_edges[edge[1]][0]\n\n u = uniform()\n pr = network._route_probs[edge[1]]\n k = _choice(pr, u, n)\n\n # _choice returns an integer between 0 and n-1 where the\n # probability of k being selected is equal to pr[k].\n return network.out_edges[edge[1]][k]", "response": "Returns the agents next destination given their current\n location on the network."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef desired_destination(self, network, edge):\n adjacent_edges = network.out_edges[edge[1]]\n d = _argmin([network.edge2queue[d].number_queued() for d in adjacent_edges])\n return adjacent_edges[d]", "response": "Returns the agents next destination given their current\n location on the network."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _calculate_distance(latlon1, latlon2):\n lat1, lon1 = latlon1\n lat2, lon2 = latlon2\n dlon = lon2 - lon1\n dlat = lat2 - lat1\n R = 6371 # radius of the earth in kilometers\n a = np.sin(dlat / 2)**2 + np.cos(lat1) * np.cos(lat2) * (np.sin(dlon / 2))**2\n c = 2 * np.pi * R * np.arctan2(np.sqrt(a), np.sqrt(1 - a)) / 180\n return c", "response": "Calculates the distance between two points on earth."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef graph2dict(g, return_dict_of_dict=True):\n if not isinstance(g, nx.DiGraph):\n g = QueueNetworkDiGraph(g)\n\n dict_of_dicts = nx.to_dict_of_dicts(g)\n if return_dict_of_dict:\n return dict_of_dicts\n else:\n return {k: list(val.keys()) for k, val in dict_of_dicts.items()}", "response": "Takes a graph and returns an adjacency list of dicts."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _matrix2dict(matrix, etype=False):\n n = len(matrix)\n adj = {k: {} for k in range(n)}\n for k in range(n):\n for j in range(n):\n if matrix[k, j] != 0:\n adj[k][j] = {} if not etype else matrix[k, j]\n\n return adj", "response": "Takes an adjacency matrix and returns an adjacency list."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntake a dictionary based representation of an adjacency list and returns a dict of dicts based representation of the adjacency list .", "response": "def _dict2dict(adj_dict):\n \"\"\"Takes a dictionary based representation of an adjacency list\n and returns a dict of dicts based representation.\n \"\"\"\n item = adj_dict.popitem()\n adj_dict[item[0]] = item[1]\n if not isinstance(item[1], dict):\n new_dict = {}\n for key, value in adj_dict.items():\n new_dict[key] = {v: {} for v in value}\n\n adj_dict = new_dict\n return adj_dict"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntaking an adjacency list and returns a possibly modified adjacency list.", "response": "def _adjacency_adjust(adjacency, adjust, is_directed):\n \"\"\"Takes an adjacency list and returns a (possibly) modified\n adjacency list.\n \"\"\"\n\n for v, adj in adjacency.items():\n for properties in adj.values():\n if properties.get('edge_type') is None:\n properties['edge_type'] = 1\n\n if is_directed:\n if adjust == 2:\n null_nodes = set()\n\n for k, adj in adjacency.items():\n if len(adj) == 0:\n null_nodes.add(k)\n\n for k, adj in adjacency.items():\n for v in adj.keys():\n if v in null_nodes:\n adj[v]['edge_type'] = 0\n\n else:\n for k, adj in adjacency.items():\n if len(adj) == 0:\n adj[k] = {'edge_type': 0}\n\n return adjacency"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef adjacency2graph(adjacency, edge_type=None, adjust=1, **kwargs):\n\n if isinstance(adjacency, np.ndarray):\n adjacency = _matrix2dict(adjacency)\n elif isinstance(adjacency, dict):\n adjacency = _dict2dict(adjacency)\n else:\n msg = (\"If the adjacency parameter is supplied it must be a \"\n \"dict, or a numpy.ndarray.\")\n raise TypeError(msg)\n\n if edge_type is None:\n edge_type = {}\n else:\n if isinstance(edge_type, np.ndarray):\n edge_type = _matrix2dict(edge_type, etype=True)\n elif isinstance(edge_type, dict):\n edge_type = _dict2dict(edge_type)\n\n for u, ty in edge_type.items():\n for v, et in ty.items():\n adjacency[u][v]['edge_type'] = et\n\n g = nx.from_dict_of_dicts(adjacency, create_using=nx.DiGraph())\n adjacency = nx.to_dict_of_dicts(g)\n adjacency = _adjacency_adjust(adjacency, adjust, True)\n\n return nx.from_dict_of_dicts(adjacency, create_using=nx.DiGraph())", "response": "Takes an adjacency list dict or matrix and returns a directed graph."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns all edges with the specified edge type.", "response": "def get_edge_type(self, edge_type):\n \"\"\"Returns all edges with the specified edge type.\n\n Parameters\n ----------\n edge_type : int\n An integer specifying what type of edges to return.\n\n Returns\n -------\n out : list of 2-tuples\n A list of 2-tuples representing the edges in the graph\n with the specified edge type.\n\n Examples\n --------\n Lets get type 2 edges from the following graph\n\n >>> import queueing_tool as qt\n >>> adjacency = {\n ... 0: {1: {'edge_type': 2}},\n ... 1: {2: {'edge_type': 1},\n ... 3: {'edge_type': 4}},\n ... 2: {0: {'edge_type': 2}},\n ... 3: {3: {'edge_type': 0}}\n ... }\n >>> G = qt.QueueNetworkDiGraph(adjacency)\n >>> ans = G.get_edge_type(2)\n >>> ans.sort()\n >>> ans\n [(0, 1), (2, 0)]\n \"\"\"\n edges = []\n for e in self.edges():\n if self.adj[e[0]][e[1]].get('edge_type') == edge_type:\n edges.append(e)\n return edges"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef draw_graph(self, line_kwargs=None, scatter_kwargs=None, **kwargs):\n if not HAS_MATPLOTLIB:\n raise ImportError(\"Matplotlib is required to draw the graph.\")\n\n fig = plt.figure(figsize=kwargs.get('figsize', (7, 7)))\n ax = fig.gca()\n\n mpl_kwargs = {\n 'line_kwargs': line_kwargs,\n 'scatter_kwargs': scatter_kwargs,\n 'pos': kwargs.get('pos')\n }\n\n line_kwargs, scatter_kwargs = self.lines_scatter_args(**mpl_kwargs)\n\n edge_collection = LineCollection(**line_kwargs)\n ax.add_collection(edge_collection)\n ax.scatter(**scatter_kwargs)\n\n if hasattr(ax, 'set_facecolor'):\n ax.set_facecolor(kwargs.get('bgcolor', [1, 1, 1, 1]))\n else:\n ax.set_axis_bgcolor(kwargs.get('bgcolor', [1, 1, 1, 1]))\n\n ax.get_xaxis().set_visible(False)\n ax.get_yaxis().set_visible(False)\n\n if 'fname' in kwargs:\n # savefig needs a positional argument for some reason\n new_kwargs = {k: v for k, v in kwargs.items() if k in SAVEFIG_KWARGS}\n fig.savefig(kwargs['fname'], **new_kwargs)\n else:\n plt.ion()\n plt.show()", "response": "Draws the graph of the current state of the entry."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the keyword arguments used when plotting a line collection.", "response": "def lines_scatter_args(self, line_kwargs=None, scatter_kwargs=None, pos=None):\n \"\"\"Returns the arguments used when plotting.\n\n Takes any keyword arguments for\n :class:`~matplotlib.collections.LineCollection` and\n :meth:`~matplotlib.axes.Axes.scatter` and returns two\n dictionaries with all the defaults set.\n\n Parameters\n ----------\n line_kwargs : dict (optional, default: ``None``)\n Any keyword arguments accepted by\n :class:`~matplotlib.collections.LineCollection`.\n scatter_kwargs : dict (optional, default: ``None``)\n Any keyword arguments accepted by\n :meth:`~matplotlib.axes.Axes.scatter`.\n\n Returns\n -------\n tuple\n A 2-tuple of dicts. The first entry is the keyword\n arguments for\n :class:`~matplotlib.collections.LineCollection` and the\n second is the keyword args for\n :meth:`~matplotlib.axes.Axes.scatter`.\n\n Notes\n -----\n If a specific keyword argument is not passed then the defaults\n are used.\n \"\"\"\n if pos is not None:\n self.set_pos(pos)\n elif self.pos is None:\n self.set_pos()\n\n edge_pos = [0 for e in self.edges()]\n for e in self.edges():\n ei = self.edge_index[e]\n edge_pos[ei] = (self.pos[e[0]], self.pos[e[1]])\n\n line_collecton_kwargs = {\n 'segments': edge_pos,\n 'colors': self.edge_color,\n 'linewidths': (1,),\n 'antialiaseds': (1,),\n 'linestyle': 'solid',\n 'transOffset': None,\n 'cmap': plt.cm.ocean_r,\n 'pickradius': 5,\n 'zorder': 0,\n 'facecolors': None,\n 'norm': None,\n 'offsets': None,\n 'offset_position': 'screen',\n 'hatch': None,\n }\n scatter_kwargs_ = {\n 'x': self.pos[:, 0],\n 'y': self.pos[:, 1],\n 's': 50,\n 'c': self.vertex_fill_color,\n 'alpha': None,\n 'norm': None,\n 'vmin': None,\n 'vmax': None,\n 'marker': 'o',\n 'zorder': 2,\n 'cmap': plt.cm.ocean_r,\n 'linewidths': 1,\n 'edgecolors': self.vertex_color,\n 'facecolors': None,\n 'antialiaseds': None,\n 'offset_position': 'screen',\n 'hatch': None,\n }\n\n line_kwargs = {} if line_kwargs is None else line_kwargs\n scatter_kwargs = {} if scatter_kwargs is None else scatter_kwargs\n\n for key, value in line_kwargs.items():\n if key in line_collecton_kwargs:\n line_collecton_kwargs[key] = value\n\n for key, value in scatter_kwargs.items():\n if key in scatter_kwargs_:\n scatter_kwargs_[key] = value\n\n return line_collecton_kwargs, scatter_kwargs_"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nclear out the queue.", "response": "def clear(self):\n \"\"\"Clears out the queue. Removes all arrivals, departures, and\n queued agents from the :class:`.QueueServer`, resets\n ``num_arrivals``, ``num_departures``, ``num_system``, and the clock to\n zero. It also clears any stored ``data`` and the server is then\n set to inactive.\n \"\"\"\n self.data = {}\n self._num_arrivals = 0\n self._oArrivals = 0\n self.num_departures = 0\n self.num_system = 0\n self._num_total = 0\n self._current_t = 0\n self._time = infty\n self._next_ct = 0\n self._active = False\n self.queue = collections.deque()\n inftyAgent = InftyAgent()\n self._arrivals = [inftyAgent]\n self._departures = [inftyAgent]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a color for the current a .", "response": "def _current_color(self, which=0):\n \"\"\"Returns a color for the queue.\n\n Parameters\n ----------\n which : int (optional, default: ``0``)\n Specifies the type of color to return.\n\n Returns\n -------\n color : list\n Returns a RGBA color that is represented as a list with 4\n entries where each entry can be any floating point number\n between 0 and 1.\n\n * If ``which`` is 1 then it returns the color of the edge\n as if it were a self loop. This is specified in\n ``colors['edge_loop_color']``.\n * If ``which`` is 2 then it returns the color of the vertex\n pen color (defined as color/vertex_color in\n :meth:`.QueueNetworkDiGraph.graph_draw`). This is\n specified in ``colors['vertex_color']``.\n * If ``which`` is anything else, then it returns the a\n shade of the edge that is proportional to the number of\n agents in the system -- which includes those being\n servered and those waiting to be served. More agents\n correspond to darker edge colors. Uses\n ``colors['vertex_fill_color']`` if the queue sits on a\n loop, and ``colors['edge_color']`` otherwise.\n \"\"\"\n if which == 1:\n color = self.colors['edge_loop_color']\n\n elif which == 2:\n color = self.colors['vertex_color']\n\n else:\n div = self.coloring_sensitivity * self.num_servers + 1.\n tmp = 1. - min(self.num_system / div, 1)\n\n if self.edge[0] == self.edge[1]:\n color = [i * tmp for i in self.colors['vertex_fill_color']]\n color[3] = 1.0\n else:\n color = [i * tmp for i in self.colors['edge_color']]\n color[3] = 1 / 2.\n\n return color"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delay_service(self, t=None):\n if len(self._departures) > 1:\n agent = heappop(self._departures)\n\n if t is None:\n agent._time = self.service_f(agent._time)\n else:\n agent._time = t\n\n heappush(self._departures, agent)\n self._update_time()", "response": "Adds an extra service time to the next departing agent."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef fetch_data(self, return_header=False):\n\n qdata = []\n for d in self.data.values():\n qdata.extend(d)\n\n dat = np.zeros((len(qdata), 6))\n if len(qdata) > 0:\n dat[:, :5] = np.array(qdata)\n dat[:, 5] = self.edge[2]\n\n dType = [\n ('a', float),\n ('s', float),\n ('d', float),\n ('q', float),\n ('n', float),\n ('id', float)\n ]\n dat = np.array([tuple(d) for d in dat], dtype=dType)\n dat = np.sort(dat, order='a')\n dat = np.array([tuple(d) for d in dat])\n\n if return_header:\n return dat, 'arrival,service,departure,num_queued,num_total,q_id'\n\n return dat", "response": "Fetches the data from the queue."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef next_event(self):\n if self._departures[0]._time < self._arrivals[0]._time:\n new_depart = heappop(self._departures)\n self._current_t = new_depart._time\n self._num_total -= 1\n self.num_system -= 1\n self.num_departures += 1\n\n if self.collect_data and new_depart.agent_id in self.data:\n self.data[new_depart.agent_id][-1][2] = self._current_t\n\n if len(self.queue) > 0:\n agent = self.queue.popleft()\n if self.collect_data and agent.agent_id in self.data:\n self.data[agent.agent_id][-1][1] = self._current_t\n\n agent._time = self.service_f(self._current_t)\n agent.queue_action(self, 1)\n heappush(self._departures, agent)\n\n new_depart.queue_action(self, 2)\n self._update_time()\n return new_depart\n\n elif self._arrivals[0]._time < infty:\n arrival = heappop(self._arrivals)\n self._current_t = arrival._time\n\n if self._active:\n self._add_arrival()\n\n self.num_system += 1\n self._num_arrivals += 1\n\n if self.collect_data:\n b = 0 if self.num_system <= self.num_servers else 1\n if arrival.agent_id not in self.data:\n self.data[arrival.agent_id] = \\\n [[arrival._time, 0, 0, len(self.queue) + b, self.num_system]]\n else:\n self.data[arrival.agent_id]\\\n .append([arrival._time, 0, 0, len(self.queue) + b, self.num_system])\n\n arrival.queue_action(self, 0)\n\n if self.num_system <= self.num_servers:\n if self.collect_data:\n self.data[arrival.agent_id][-1][1] = arrival._time\n\n arrival._time = self.service_f(arrival._time)\n arrival.queue_action(self, 1)\n heappush(self._departures, arrival)\n else:\n self.queue.append(arrival)\n\n self._update_time()", "response": "Simulates the queue forward one event."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef next_event_description(self):\n if self._departures[0]._time < self._arrivals[0]._time:\n return 2\n elif self._arrivals[0]._time < infty:\n return 1\n else:\n return 0", "response": "Returns an integer representing whether the next event is an ancipline or an ancipline."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef set_num_servers(self, n):\n if not isinstance(n, numbers.Integral) and n is not infty:\n the_str = \"n must be an integer or infinity.\\n{0}\"\n raise TypeError(the_str.format(str(self)))\n elif n <= 0:\n the_str = \"n must be a positive integer or infinity.\\n{0}\"\n raise ValueError(the_str.format(str(self)))\n else:\n self.num_servers = n", "response": "Change the number of servers in the queue to n."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef simulate(self, n=1, t=None, nA=None, nD=None):\n if t is None and nD is None and nA is None:\n for dummy in range(n):\n self.next_event()\n elif t is not None:\n then = self._current_t + t\n while self._current_t < then and self._time < infty:\n self.next_event()\n elif nD is not None:\n num_departures = self.num_departures + nD\n while self.num_departures < num_departures and self._time < infty:\n self.next_event()\n elif nA is not None:\n num_arrivals = self._oArrivals + nA\n while self._oArrivals < num_arrivals and self._time < infty:\n self.next_event()", "response": "This method simulates the queue forward for a specific number of events or for a specific number of departures."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nuses to specify edge indices from different types of arguments.", "response": "def _get_queues(g, queues, edge, edge_type):\n \"\"\"Used to specify edge indices from different types of arguments.\"\"\"\n INT = numbers.Integral\n if isinstance(queues, INT):\n queues = [queues]\n\n elif queues is None:\n if edge is not None:\n if isinstance(edge, tuple):\n if isinstance(edge[0], INT) and isinstance(edge[1], INT):\n queues = [g.edge_index[edge]]\n elif isinstance(edge[0], collections.Iterable):\n if np.array([len(e) == 2 for e in edge]).all():\n queues = [g.edge_index[e] for e in edge]\n else:\n queues = [g.edge_index[edge]]\n elif edge_type is not None:\n if isinstance(edge_type, collections.Iterable):\n edge_type = set(edge_type)\n else:\n edge_type = set([edge_type])\n tmp = []\n for e in g.edges():\n if g.ep(e, 'edge_type') in edge_type:\n tmp.append(g.edge_index[e])\n\n queues = np.array(tmp, int)\n\n if queues is None:\n queues = range(g.number_of_edges())\n\n return queues"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef animate(self, out=None, t=None, line_kwargs=None,\n scatter_kwargs=None, **kwargs):\n \"\"\"Animates the network as it's simulating.\n\n The animations can be saved to disk or viewed in interactive\n mode. Closing the window ends the animation if viewed in\n interactive mode. This method calls\n :meth:`~matplotlib.axes.scatter`, and\n :class:`~matplotlib.collections.LineCollection`, and any\n keyword arguments they accept can be passed to them.\n\n Parameters\n ----------\n out : str (optional)\n The location where the frames for the images will be saved.\n If this parameter is not given, then the animation is shown\n in interactive mode.\n t : float (optional)\n The amount of simulation time to simulate forward. If\n given, and ``out`` is given, ``t`` is used instead of\n ``n``.\n line_kwargs : dict (optional, default: None)\n Any keyword arguments accepted by\n :class:`~matplotlib.collections.LineCollection`.\n scatter_kwargs : dict (optional, default: None)\n Any keyword arguments accepted by\n :meth:`~matplotlib.axes.Axes.scatter`.\n bgcolor : list (optional, keyword only)\n A list with 4 floats representing a RGBA color. The\n default is defined in ``self.colors['bgcolor']``.\n figsize : tuple (optional, keyword only, default: ``(7, 7)``)\n The width and height of the figure in inches.\n **kwargs :\n This method calls\n :class:`~matplotlib.animation.FuncAnimation` and\n optionally :meth:`.matplotlib.animation.FuncAnimation.save`.\n Any keyword that can be passed to these functions are\n passed via ``kwargs``.\n\n Notes\n -----\n There are several parameters automatically set and passed to\n matplotlib's :meth:`~matplotlib.axes.Axes.scatter`,\n :class:`~matplotlib.collections.LineCollection`, and\n :class:`~matplotlib.animation.FuncAnimation` by default.\n These include:\n\n * :class:`~matplotlib.animation.FuncAnimation`: Uses the\n defaults for that function. Saving the animation is done\n by passing the 'filename' keyword argument to this method.\n This method also accepts any keyword arguments accepted\n by :meth:`~matplotlib.animation.FuncAnimation.save`.\n * :class:`~matplotlib.collections.LineCollection`: The default\n arguments are taken from\n :meth:`.QueueNetworkDiGraph.lines_scatter_args`.\n * :meth:`~matplotlib.axes.Axes.scatter`: The default\n arguments are taken from\n :meth:`.QueueNetworkDiGraph.lines_scatter_args`.\n\n Raises\n ------\n QueueingToolError\n Will raise a :exc:`.QueueingToolError` if the\n ``QueueNetwork`` has not been initialized. Call\n :meth:`.initialize` before running.\n\n Examples\n --------\n This function works similarly to ``QueueNetwork's``\n :meth:`.draw` method.\n\n >>> import queueing_tool as qt\n >>> g = qt.generate_pagerank_graph(100, seed=13)\n >>> net = qt.QueueNetwork(g, seed=13)\n >>> net.initialize()\n >>> net.animate(figsize=(4, 4)) # doctest: +SKIP\n\n To stop the animation just close the window. If you want to\n write the animation to disk run something like the following:\n\n >>> kwargs = {\n ... 'filename': 'test.mp4',\n ... 'frames': 300,\n ... 'fps': 30,\n ... 'writer': 'mencoder',\n ... 'figsize': (4, 4),\n ... 'vertex_size': 15\n ... }\n >>> net.animate(**kwargs) # doctest: +SKIP\n \"\"\"\n\n if not self._initialized:\n msg = (\"Network has not been initialized. \"\n \"Call '.initialize()' first.\")\n raise QueueingToolError(msg)\n\n if not HAS_MATPLOTLIB:\n msg = \"Matplotlib is necessary to animate a simulation.\"\n raise ImportError(msg)\n\n self._update_all_colors()\n kwargs.setdefault('bgcolor', self.colors['bgcolor'])\n\n fig = plt.figure(figsize=kwargs.get('figsize', (7, 7)))\n ax = fig.gca()\n\n mpl_kwargs = {\n 'line_kwargs': line_kwargs,\n 'scatter_kwargs': scatter_kwargs,\n 'pos': kwargs.get('pos')\n }\n\n line_args, scat_args = self.g.lines_scatter_args(**mpl_kwargs)\n\n lines = LineCollection(**line_args)\n lines = ax.add_collection(lines)\n scatt = ax.scatter(**scat_args)\n\n t = np.infty if t is None else t\n now = self._t\n\n def update(frame_number):\n if t is not None:\n if self._t > now + t:\n return False\n self._simulate_next_event(slow=True)\n lines.set_color(line_args['colors'])\n scatt.set_edgecolors(scat_args['edgecolors'])\n scatt.set_facecolor(scat_args['c'])\n\n if hasattr(ax, 'set_facecolor'):\n ax.set_facecolor(kwargs['bgcolor'])\n else:\n ax.set_axis_bgcolor(kwargs['bgcolor'])\n\n ax.get_xaxis().set_visible(False)\n ax.get_yaxis().set_visible(False)\n\n animation_args = {\n 'fargs': None,\n 'event_source': None,\n 'init_func': None,\n 'frames': None,\n 'blit': False,\n 'interval': 10,\n 'repeat': None,\n 'func': update,\n 'repeat_delay': None,\n 'fig': fig,\n 'save_count': None,\n }\n for key, value in kwargs.items():\n if key in animation_args:\n animation_args[key] = value\n\n animation = FuncAnimation(**animation_args)\n if 'filename' not in kwargs:\n plt.ioff()\n plt.show()\n else:\n save_args = {\n 'filename': None,\n 'writer': None,\n 'fps': None,\n 'dpi': None,\n 'codec': None,\n 'bitrate': None,\n 'extra_args': None,\n 'metadata': None,\n 'extra_anim': None,\n 'savefig_kwargs': None\n }\n for key, value in kwargs.items():\n if key in save_args:\n save_args[key] = value\n\n animation.save(**save_args)", "response": "Animate the network as it s simulating."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nresets the queue to its initial state.", "response": "def clear(self):\n \"\"\"Resets the queue to its initial state.\n\n The attributes ``t``, ``num_events``, ``num_agents`` are set to\n zero, :meth:`.reset_colors` is called, and the\n :meth:`.QueueServer.clear` method is called for each queue in\n the network.\n\n Notes\n -----\n ``QueueNetwork`` must be re-initialized before any simulations\n can run.\n \"\"\"\n self._t = 0\n self.num_events = 0\n self.num_agents = np.zeros(self.nE, int)\n self._fancy_heap = PriorityQueue()\n self._prev_edge = None\n self._initialized = False\n self.reset_colors()\n for q in self.edge2queue:\n q.clear()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef clear_data(self, queues=None, edge=None, edge_type=None):\n queues = _get_queues(self.g, queues, edge, edge_type)\n\n for k in queues:\n self.edge2queue[k].data = {}", "response": "Clears data from all queues."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef copy(self):\n net = QueueNetwork(None)\n net.g = self.g.copy()\n net.max_agents = copy.deepcopy(self.max_agents)\n net.nV = copy.deepcopy(self.nV)\n net.nE = copy.deepcopy(self.nE)\n net.num_agents = copy.deepcopy(self.num_agents)\n net.num_events = copy.deepcopy(self.num_events)\n net._t = copy.deepcopy(self._t)\n net._initialized = copy.deepcopy(self._initialized)\n net._prev_edge = copy.deepcopy(self._prev_edge)\n net._blocking = copy.deepcopy(self._blocking)\n net.colors = copy.deepcopy(self.colors)\n net.out_edges = copy.deepcopy(self.out_edges)\n net.in_edges = copy.deepcopy(self.in_edges)\n net.edge2queue = copy.deepcopy(self.edge2queue)\n net._route_probs = copy.deepcopy(self._route_probs)\n\n if net._initialized:\n keys = [q._key() for q in net.edge2queue if q._time < np.infty]\n net._fancy_heap = PriorityQueue(keys, net.nE)\n\n return net", "response": "Returns a deep copy of itself."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndrawing the current state of the queue entry in the queue.", "response": "def draw(self, update_colors=True, line_kwargs=None,\n scatter_kwargs=None, **kwargs):\n \"\"\"Draws the network. The coloring of the network corresponds\n to the number of agents at each queue.\n\n Parameters\n ----------\n update_colors : ``bool`` (optional, default: ``True``).\n Specifies whether all the colors are updated.\n line_kwargs : dict (optional, default: None)\n Any keyword arguments accepted by\n :class:`~matplotlib.collections.LineCollection`\n scatter_kwargs : dict (optional, default: None)\n Any keyword arguments accepted by\n :meth:`~matplotlib.axes.Axes.scatter`.\n bgcolor : list (optional, keyword only)\n A list with 4 floats representing a RGBA color. The\n default is defined in ``self.colors['bgcolor']``.\n figsize : tuple (optional, keyword only, default: ``(7, 7)``)\n The width and height of the canvas in inches.\n **kwargs\n Any parameters to pass to\n :meth:`.QueueNetworkDiGraph.draw_graph`.\n\n Notes\n -----\n This method relies heavily on\n :meth:`.QueueNetworkDiGraph.draw_graph`. Also, there is a\n parameter that sets the background color of the canvas, which\n is the ``bgcolor`` parameter.\n\n Examples\n --------\n To draw the current state of the network, call:\n\n >>> import queueing_tool as qt\n >>> g = qt.generate_pagerank_graph(100, seed=13)\n >>> net = qt.QueueNetwork(g, seed=13)\n >>> net.initialize(100)\n >>> net.simulate(1200)\n >>> net.draw() # doctest: +SKIP\n\n If you specify a file name and location, the drawing will be\n saved to disk. For example, to save the drawing to the current\n working directory do the following:\n\n >>> net.draw(fname=\"state.png\", scatter_kwargs={'s': 40}) # doctest: +SKIP\n\n .. figure:: current_state1.png\n :align: center\n\n The shade of each edge depicts how many agents are located at\n the corresponding queue. The shade of each vertex is determined\n by the total number of inbound agents. Although loops are not\n visible by default, the vertex that corresponds to a loop shows\n how many agents are in that loop.\n\n There are several additional parameters that can be passed --\n all :meth:`.QueueNetworkDiGraph.draw_graph` parameters are\n valid. For example, to show the edges as dashed lines do the\n following.\n\n >>> net.draw(line_kwargs={'linestyle': 'dashed'}) # doctest: +SKIP\n \"\"\"\n if not HAS_MATPLOTLIB:\n raise ImportError(\"matplotlib is necessary to draw the network.\")\n\n if update_colors:\n self._update_all_colors()\n\n if 'bgcolor' not in kwargs:\n kwargs['bgcolor'] = self.colors['bgcolor']\n\n self.g.draw_graph(line_kwargs=line_kwargs,\n scatter_kwargs=scatter_kwargs, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_agent_data(self, queues=None, edge=None, edge_type=None, return_header=False):\n queues = _get_queues(self.g, queues, edge, edge_type)\n\n data = {}\n for qid in queues:\n for agent_id, dat in self.edge2queue[qid].data.items():\n datum = np.zeros((len(dat), 6))\n datum[:, :5] = np.array(dat)\n datum[:, 5] = qid\n if agent_id in data:\n data[agent_id] = np.vstack((data[agent_id], datum))\n else:\n data[agent_id] = datum\n\n dType = [\n ('a', float),\n ('s', float),\n ('d', float),\n ('q', float),\n ('n', float),\n ('id', float)\n ]\n for agent_id, dat in data.items():\n datum = np.array([tuple(d) for d in dat.tolist()], dtype=dType)\n datum = np.sort(datum, order='a')\n data[agent_id] = np.array([tuple(d) for d in datum])\n\n if return_header:\n return data, 'arrival,service,departure,num_queued,num_total,q_id'\n\n return data", "response": "Gets data from queues and organizes it by agent."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the data from all the queues.", "response": "def get_queue_data(self, queues=None, edge=None, edge_type=None, return_header=False):\n \"\"\"Gets data from all the queues.\n\n If none of the parameters are given then data from every\n :class:`.QueueServer` is retrieved.\n\n Parameters\n ----------\n queues : int or an *array_like* of int, (optional)\n The edge index (or an iterable of edge indices) identifying\n the :class:`QueueServer(s)<.QueueServer>` whose data will\n be retrieved.\n edge : 2-tuple of int or *array_like* (optional)\n Explicitly specify which queues to retrieve data from. Must\n be either:\n\n * A 2-tuple of the edge's source and target vertex\n indices, or\n * An iterable of 2-tuples of the edge's source and\n target vertex indices.\n\n edge_type : int or an iterable of int (optional)\n A integer, or a collection of integers identifying which\n edge types to retrieve data from.\n return_header : bool (optonal, default: False)\n Determines whether the column headers are returned.\n\n Returns\n -------\n out : :class:`~numpy.ndarray`\n * 1st: The arrival time of an agent.\n * 2nd: The service start time of an agent.\n * 3rd: The departure time of an agent.\n * 4th: The length of the queue upon the agents arrival.\n * 5th: The total number of :class:`Agents<.Agent>` in the\n :class:`.QueueServer`.\n * 6th: The :class:`QueueServer's<.QueueServer>` edge index.\n\n out : str (optional)\n A comma seperated string of the column headers. Returns\n ``'arrival,service,departure,num_queued,num_total,q_id'```\n\n Examples\n --------\n Data is not collected by default. Before simulating, by sure to\n turn it on (as well as initialize the network). The following\n returns data from queues with ``edge_type`` 1 or 3:\n\n >>> import queueing_tool as qt\n >>> g = qt.generate_pagerank_graph(100, seed=13)\n >>> net = qt.QueueNetwork(g, seed=13)\n >>> net.start_collecting_data()\n >>> net.initialize(10)\n >>> net.simulate(2000)\n >>> data = net.get_queue_data(edge_type=(1, 3))\n\n To get data from an edge connecting two vertices do the\n following:\n\n >>> data = net.get_queue_data(edge=(1, 50))\n\n To get data from several edges do the following:\n\n >>> data = net.get_queue_data(edge=[(1, 50), (10, 91), (99, 99)])\n\n You can specify the edge indices as well:\n\n >>> data = net.get_queue_data(queues=(20, 14, 0, 4))\n \"\"\"\n queues = _get_queues(self.g, queues, edge, edge_type)\n\n data = np.zeros((0, 6))\n for q in queues:\n dat = self.edge2queue[q].fetch_data()\n\n if len(dat) > 0:\n data = np.vstack((data, dat))\n\n if return_header:\n return data, 'arrival,service,departure,num_queued,num_total,q_id'\n\n return data"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef initialize(self, nActive=1, queues=None, edges=None, edge_type=None):\n if queues is None and edges is None and edge_type is None:\n if nActive >= 1 and isinstance(nActive, numbers.Integral):\n qs = [q.edge[2] for q in self.edge2queue if q.edge[3] != 0]\n n = min(nActive, len(qs))\n queues = np.random.choice(qs, size=n, replace=False)\n elif not isinstance(nActive, numbers.Integral):\n msg = \"If queues is None, then nActive must be an integer.\"\n raise TypeError(msg)\n else:\n msg = (\"If queues is None, then nActive must be a \"\n \"positive int.\")\n raise ValueError(msg)\n else:\n queues = _get_queues(self.g, queues, edges, edge_type)\n\n queues = [e for e in queues if self.edge2queue[e].edge[3] != 0]\n\n if len(queues) == 0:\n raise QueueingToolError(\"There were no queues to initialize.\")\n\n if len(queues) > self.max_agents:\n queues = queues[:self.max_agents]\n\n for ei in queues:\n self.edge2queue[ei].set_active()\n self.num_agents[ei] = self.edge2queue[ei]._num_total\n\n keys = [q._key() for q in self.edge2queue if q._time < np.infty]\n self._fancy_heap = PriorityQueue(keys, self.nE)\n self._initialized = True", "response": "Initializes the base class instance with the given parameters."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef next_event_description(self):\n if self._fancy_heap.size == 0:\n event_type = 'Nothing'\n edge_index = None\n else:\n s = [q._key() for q in self.edge2queue]\n s.sort()\n e = s[0][1]\n q = self.edge2queue[e]\n\n event_type = 'Arrival' if q.next_event_description() == 1 else 'Departure'\n edge_index = q.edge[2]\n return event_type, edge_index", "response": "Returns the next event description of the next event in the queue."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef reset_colors(self):\n for k, e in enumerate(self.g.edges()):\n self.g.set_ep(e, 'edge_color', self.edge2queue[k].colors['edge_color'])\n for v in self.g.nodes():\n self.g.set_vp(v, 'vertex_fill_color', self.colors['vertex_fill_color'])", "response": "Resets all edge and vertex colors to their default values."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_transitions(self, mat):\n if isinstance(mat, dict):\n for key, value in mat.items():\n probs = list(value.values())\n\n if key not in self.g.node:\n msg = \"One of the keys don't correspond to a vertex.\"\n raise ValueError(msg)\n elif len(self.out_edges[key]) > 0 and not np.isclose(sum(probs), 1):\n msg = \"Sum of transition probabilities at a vertex was not 1.\"\n raise ValueError(msg)\n elif (np.array(probs) < 0).any():\n msg = \"Some transition probabilities were negative.\"\n raise ValueError(msg)\n\n for k, e in enumerate(sorted(self.g.out_edges(key))):\n self._route_probs[key][k] = value.get(e[1], 0)\n\n elif isinstance(mat, np.ndarray):\n non_terminal = np.array([self.g.out_degree(v) > 0 for v in self.g.nodes()])\n if mat.shape != (self.nV, self.nV):\n msg = (\"Matrix is the wrong shape, should \"\n \"be {0} x {1}.\").format(self.nV, self.nV)\n raise ValueError(msg)\n elif not np.allclose(np.sum(mat[non_terminal, :], axis=1), 1):\n msg = \"Sum of transition probabilities at a vertex was not 1.\"\n raise ValueError(msg)\n elif (mat < 0).any():\n raise ValueError(\"Some transition probabilities were negative.\")\n\n for k in range(self.nV):\n for j, e in enumerate(sorted(self.g.out_edges(k))):\n self._route_probs[k][j] = mat[k, e[1]]\n else:\n raise TypeError(\"mat must be a numpy array or a dict.\")", "response": "Change the routing transitions probabilities for the given node."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndrawing the network highlighting active queues.", "response": "def show_active(self, **kwargs):\n \"\"\"Draws the network, highlighting active queues.\n\n The colored vertices represent vertices that have at least one\n queue on an in-edge that is active. Dark edges represent\n queues that are active, light edges represent queues that are\n inactive.\n\n Parameters\n ----------\n **kwargs\n Any additional parameters to pass to :meth:`.draw`, and\n :meth:`.QueueNetworkDiGraph.draw_graph`.\n\n Notes\n -----\n Active queues are :class:`QueueServers<.QueueServer>` that\n accept arrivals from outside the network. The colors are\n defined by the class attribute ``colors``. The relevant keys\n are ``vertex_active``, ``vertex_inactive``, ``edge_active``,\n and ``edge_inactive``.\n \"\"\"\n g = self.g\n for v in g.nodes():\n self.g.set_vp(v, 'vertex_color', [0, 0, 0, 0.9])\n is_active = False\n my_iter = g.in_edges(v) if g.is_directed() else g.out_edges(v)\n for e in my_iter:\n ei = g.edge_index[e]\n if self.edge2queue[ei]._active:\n is_active = True\n break\n if is_active:\n self.g.set_vp(v, 'vertex_fill_color', self.colors['vertex_active'])\n else:\n self.g.set_vp(v, 'vertex_fill_color', self.colors['vertex_inactive'])\n\n for e in g.edges():\n ei = g.edge_index[e]\n if self.edge2queue[ei]._active:\n self.g.set_ep(e, 'edge_color', self.colors['edge_active'])\n else:\n self.g.set_ep(e, 'edge_color', self.colors['edge_inactive'])\n\n self.draw(update_colors=False, **kwargs)\n self._update_all_colors()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndraw the network highlighting queues of a certain type.", "response": "def show_type(self, edge_type, **kwargs):\n \"\"\"Draws the network, highlighting queues of a certain type.\n\n The colored vertices represent self loops of type ``edge_type``.\n Dark edges represent queues of type ``edge_type``.\n\n Parameters\n ----------\n edge_type : int\n The type of vertices and edges to be shown.\n **kwargs\n Any additional parameters to pass to :meth:`.draw`, and\n :meth:`.QueueNetworkDiGraph.draw_graph`\n\n Notes\n -----\n The colors are defined by the class attribute ``colors``. The\n relevant colors are ``vertex_active``, ``vertex_inactive``,\n ``vertex_highlight``, ``edge_active``, and ``edge_inactive``.\n\n Examples\n --------\n The following code highlights all edges with edge type ``2``.\n If the edge is a loop then the vertex is highlighted as well.\n In this case all edges with edge type ``2`` happen to be loops.\n\n >>> import queueing_tool as qt\n >>> g = qt.generate_pagerank_graph(100, seed=13)\n >>> net = qt.QueueNetwork(g, seed=13)\n >>> fname = 'edge_type_2.png'\n >>> net.show_type(2, fname=fname) # doctest: +SKIP\n\n .. figure:: edge_type_2-1.png\n :align: center\n \"\"\"\n for v in self.g.nodes():\n e = (v, v)\n if self.g.is_edge(e) and self.g.ep(e, 'edge_type') == edge_type:\n ei = self.g.edge_index[e]\n self.g.set_vp(v, 'vertex_fill_color', self.colors['vertex_highlight'])\n self.g.set_vp(v, 'vertex_color', self.edge2queue[ei].colors['vertex_color'])\n else:\n self.g.set_vp(v, 'vertex_fill_color', self.colors['vertex_inactive'])\n self.g.set_vp(v, 'vertex_color', [0, 0, 0, 0.9])\n\n for e in self.g.edges():\n if self.g.ep(e, 'edge_type') == edge_type:\n self.g.set_ep(e, 'edge_color', self.colors['edge_active'])\n else:\n self.g.set_ep(e, 'edge_color', self.colors['edge_inactive'])\n\n self.draw(update_colors=False, **kwargs)\n self._update_all_colors()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef simulate(self, n=1, t=None):\n if not self._initialized:\n msg = (\"Network has not been initialized. \"\n \"Call '.initialize()' first.\")\n raise QueueingToolError(msg)\n if t is None:\n for dummy in range(n):\n self._simulate_next_event(slow=False)\n else:\n now = self._t\n while self._t < now + t:\n self._simulate_next_event(slow=False)", "response": "Simulates the network forward."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef start_collecting_data(self, queues=None, edge=None, edge_type=None):\n queues = _get_queues(self.g, queues, edge, edge_type)\n\n for k in queues:\n self.edge2queue[k].collect_data = True", "response": "Tells the queues to collect data on agents arrival service\n start and departure times."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntell the queues to stop collecting data on agents.", "response": "def stop_collecting_data(self, queues=None, edge=None, edge_type=None):\n \"\"\"Tells the queues to stop collecting data on agents.\n\n If none of the parameters are given then every\n :class:`.QueueServer` will stop collecting data.\n\n Parameters\n ----------\n queues : int, *array_like* (optional)\n The edge index (or an iterable of edge indices) identifying\n the :class:`QueueServer(s)<.QueueServer>` that will stop\n collecting data.\n edge : 2-tuple of int or *array_like* (optional)\n Explicitly specify which queues will stop collecting data.\n Must be either:\n\n * A 2-tuple of the edge's source and target vertex\n indices, or\n * An iterable of 2-tuples of the edge's source and\n target vertex indices.\n\n edge_type : int or an iterable of int (optional)\n A integer, or a collection of integers identifying which\n edge types will stop collecting data.\n \"\"\"\n queues = _get_queues(self.g, queues, edge, edge_type)\n\n for k in queues:\n self.edge2queue[k].collect_data = False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the routing probabilities for each edge in the graph.", "response": "def transitions(self, return_matrix=True):\n \"\"\"Returns the routing probabilities for each vertex in the\n graph.\n\n Parameters\n ----------\n return_matrix : bool (optional, the default is ``True``)\n Specifies whether an :class:`~numpy.ndarray` is returned.\n If ``False``, a dict is returned instead.\n\n Returns\n -------\n out : a dict or :class:`~numpy.ndarray`\n The transition probabilities for each vertex in the graph.\n If ``out`` is an :class:`~numpy.ndarray`, then\n ``out[v, u]`` returns the probability of a transition from\n vertex ``v`` to vertex ``u``. If ``out`` is a dict\n then ``out_edge[v][u]`` is the probability of moving from\n vertex ``v`` to the vertex ``u``.\n\n Examples\n --------\n Lets change the routing probabilities:\n\n >>> import queueing_tool as qt\n >>> import networkx as nx\n >>> g = nx.sedgewick_maze_graph()\n >>> net = qt.QueueNetwork(g)\n\n Below is an adjacency list for the graph ``g``.\n\n >>> ans = qt.graph2dict(g, False)\n >>> {k: sorted(v) for k, v in ans.items()}\n ... # doctest: +NORMALIZE_WHITESPACE\n {0: [2, 5, 7],\n 1: [7],\n 2: [0, 6],\n 3: [4, 5],\n 4: [3, 5, 6, 7],\n 5: [0, 3, 4],\n 6: [2, 4],\n 7: [0, 1, 4]}\n\n The default transition matrix is every out edge being equally\n likely:\n\n >>> net.transitions(False) # doctest: +ELLIPSIS\n ... # doctest: +NORMALIZE_WHITESPACE\n {0: {2: 0.333..., 5: 0.333..., 7: 0.333...},\n 1: {7: 1.0},\n 2: {0: 0.5, 6: 0.5},\n 3: {4: 0.5, 5: 0.5},\n 4: {3: 0.25, 5: 0.25, 6: 0.25, 7: 0.25},\n 5: {0: 0.333..., 3: 0.333..., 4: 0.333...},\n 6: {2: 0.5, 4: 0.5},\n 7: {0: 0.333..., 1: 0.333..., 4: 0.333...}}\n\n Now we will generate a random routing matrix:\n\n >>> mat = qt.generate_transition_matrix(g, seed=96)\n >>> net.set_transitions(mat)\n >>> net.transitions(False) # doctest: +ELLIPSIS\n ... # doctest: +NORMALIZE_WHITESPACE\n {0: {2: 0.112..., 5: 0.466..., 7: 0.420...},\n 1: {7: 1.0},\n 2: {0: 0.561..., 6: 0.438...},\n 3: {4: 0.545..., 5: 0.454...},\n 4: {3: 0.374..., 5: 0.381..., 6: 0.026..., 7: 0.217...},\n 5: {0: 0.265..., 3: 0.460..., 4: 0.274...},\n 6: {2: 0.673..., 4: 0.326...},\n 7: {0: 0.033..., 1: 0.336..., 4: 0.630...}}\n\n What this shows is the following: when an :class:`.Agent` is at\n vertex ``2`` they will transition to vertex ``0`` with\n probability ``0.561`` and route to vertex ``6`` probability\n ``0.438``, when at vertex ``6`` they will transition back to\n vertex ``2`` with probability ``0.673`` and route vertex ``4``\n probability ``0.326``, etc.\n \"\"\"\n if return_matrix:\n mat = np.zeros((self.nV, self.nV))\n for v in self.g.nodes():\n ind = [e[1] for e in sorted(self.g.out_edges(v))]\n mat[v, ind] = self._route_probs[v]\n else:\n mat = {\n k: {e[1]: p for e, p in zip(sorted(self.g.out_edges(k)), value)}\n for k, value in enumerate(self._route_probs)\n }\n\n return mat"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef queue_action(self, queue, *args, **kwargs):\n if isinstance(queue, ResourceQueue):\n if self._has_resource:\n self._has_resource = False\n self._had_resource = True\n else:\n if queue.num_servers > 0:\n queue.set_num_servers(queue.num_servers - 1)\n self._has_resource = True\n self._had_resource = False", "response": "Function that specifies the interaction with a specific resource queue. This method is called when the agent is departing from a specific resource."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nsimulate the queue forward one event.", "response": "def next_event(self):\n \"\"\"Simulates the queue forward one event.\n\n This method behaves identically to a :class:`.LossQueue` if the\n arriving/departing agent is anything other than a\n :class:`.ResourceAgent`. The differences are;\n\n Arriving:\n\n * If the :class:`.ResourceAgent` has a resource then it deletes\n the agent upon arrival and adds one to ``num_servers``.\n * If the :class:`.ResourceAgent` is arriving without a resource\n then nothing special happens.\n\n Departing:\n\n * If the :class:`.ResourceAgent` does not have a resource, then\n ``num_servers`` decreases by one and the agent then *has a\n resource*.\n\n Use :meth:`~QueueServer.simulate` for simulating instead.\n \"\"\"\n if isinstance(self._arrivals[0], ResourceAgent):\n if self._departures[0]._time < self._arrivals[0]._time:\n return super(ResourceQueue, self).next_event()\n elif self._arrivals[0]._time < infty:\n if self._arrivals[0]._has_resource:\n arrival = heappop(self._arrivals)\n self._current_t = arrival._time\n self._num_total -= 1\n self.set_num_servers(self.num_servers + 1)\n\n if self.collect_data:\n t = arrival._time\n if arrival.agent_id not in self.data:\n self.data[arrival.agent_id] = [[t, t, t, len(self.queue), self.num_system]]\n else:\n self.data[arrival.agent_id].append([t, t, t, len(self.queue), self.num_system])\n\n if self._arrivals[0]._time < self._departures[0]._time:\n self._time = self._arrivals[0]._time\n else:\n self._time = self._departures[0]._time\n\n elif self.num_system < self.num_servers:\n super(ResourceQueue, self).next_event()\n\n else:\n self.num_blocked += 1\n self._num_arrivals += 1\n self._num_total -= 1\n arrival = heappop(self._arrivals)\n self._current_t = arrival._time\n\n if self.collect_data:\n if arrival.agent_id not in self.data:\n self.data[arrival.agent_id] = [[arrival._time, 0, 0, len(self.queue), self.num_system]]\n else:\n self.data[arrival.agent_id].append([arrival._time, 0, 0, len(self.queue), self.num_system])\n\n if self._arrivals[0]._time < self._departures[0]._time:\n self._time = self._arrivals[0]._time\n else:\n self._time = self._departures[0]._time\n else:\n return super(ResourceQueue, self).next_event()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the number of elements in the set that s belongs to.", "response": "def size(self, s):\n \"\"\"Returns the number of elements in the set that ``s`` belongs to.\n\n Parameters\n ----------\n s : object\n An object\n\n Returns\n -------\n out : int\n The number of elements in the set that ``s`` belongs to.\n \"\"\"\n leader = self.find(s)\n return self._size[leader]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef find(self, s):\n pSet = [s]\n parent = self._leader[s]\n\n while parent != self._leader[parent]:\n pSet.append(parent)\n parent = self._leader[parent]\n\n if len(pSet) > 1:\n for a in pSet:\n self._leader[a] = parent\n\n return parent", "response": "Locates the leader of the set to which the element s belongs."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmerges the set that contains a with the set that contains b.", "response": "def union(self, a, b):\n \"\"\"Merges the set that contains ``a`` with the set that contains ``b``.\n\n Parameters\n ----------\n a, b : objects\n Two objects whose sets are to be merged.\n \"\"\"\n s1, s2 = self.find(a), self.find(b)\n if s1 != s2:\n r1, r2 = self._rank[s1], self._rank[s2]\n if r2 > r1:\n r1, r2 = r2, r1\n s1, s2 = s2, s1\n if r1 == r2:\n self._rank[s1] += 1\n\n self._leader[s2] = s1\n self._size[s1] += self._size[s2]\n self.nClusters -= 1"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ngenerate a random transition matrix for the given graph g.", "response": "def generate_transition_matrix(g, seed=None):\n \"\"\"Generates a random transition matrix for the graph ``g``.\n\n Parameters\n ----------\n g : :any:`networkx.DiGraph`, :class:`numpy.ndarray`, dict, etc.\n Any object that :any:`DiGraph` accepts.\n seed : int (optional)\n An integer used to initialize numpy's psuedo-random number\n generator.\n\n Returns\n -------\n mat : :class:`~numpy.ndarray`\n Returns a transition matrix where ``mat[i, j]`` is the\n probability of transitioning from vertex ``i`` to vertex ``j``.\n If there is no edge connecting vertex ``i`` to vertex ``j``\n then ``mat[i, j] = 0``.\n \"\"\"\n g = _test_graph(g)\n\n if isinstance(seed, numbers.Integral):\n np.random.seed(seed)\n\n nV = g.number_of_nodes()\n mat = np.zeros((nV, nV))\n\n for v in g.nodes():\n ind = [e[1] for e in sorted(g.out_edges(v))]\n deg = len(ind)\n if deg == 1:\n mat[v, ind] = 1\n elif deg > 1:\n probs = np.ceil(np.random.rand(deg) * 100) / 100.\n if np.isclose(np.sum(probs), 0):\n probs[np.random.randint(deg)] = 1\n\n mat[v, ind] = probs / np.sum(probs)\n\n return mat"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a random graph where the edges have different types.", "response": "def generate_random_graph(num_vertices=250, prob_loop=0.5, **kwargs):\n \"\"\"Creates a random graph where the edges have different types.\n\n This method calls :func:`.minimal_random_graph`, and then adds\n a loop to each vertex with ``prob_loop`` probability. It then\n calls :func:`.set_types_random` on the resulting graph.\n\n Parameters\n ----------\n num_vertices : int (optional, default: 250)\n The number of vertices in the graph.\n prob_loop : float (optional, default: 0.5)\n The probability that a loop gets added to a vertex.\n **kwargs :\n Any parameters to send to :func:`.minimal_random_graph` or\n :func:`.set_types_random`.\n\n Returns\n -------\n :class:`.QueueNetworkDiGraph`\n A graph with the position of the vertex set as a property.\n The position property is called ``pos``. Also, the ``edge_type``\n edge property is set for each edge.\n\n Examples\n --------\n The following generates a directed graph with 50 vertices where half\n the edges are type 1 and 1/4th are type 2 and 1/4th are type 3:\n\n >>> import queueing_tool as qt\n >>> pTypes = {1: 0.5, 2: 0.25, 3: 0.25}\n >>> g = qt.generate_random_graph(100, proportions=pTypes, seed=17)\n >>> non_loops = [e for e in g.edges() if e[0] != e[1]]\n >>> p1 = np.sum([g.ep(e, 'edge_type') == 1 for e in non_loops])\n >>> float(p1) / len(non_loops) # doctest: +ELLIPSIS\n 0.486...\n >>> p2 = np.sum([g.ep(e, 'edge_type') == 2 for e in non_loops])\n >>> float(p2) / len(non_loops) # doctest: +ELLIPSIS\n 0.249...\n >>> p3 = np.sum([g.ep(e, 'edge_type') == 3 for e in non_loops])\n >>> float(p3) / len(non_loops) # doctest: +ELLIPSIS\n 0.264...\n\n To make an undirected graph with 25 vertices where there are 4\n different edge types with random proportions:\n\n >>> p = np.random.rand(4)\n >>> p = p / sum(p)\n >>> p = {k + 1: p[k] for k in range(4)}\n >>> g = qt.generate_random_graph(num_vertices=25, is_directed=False, proportions=p)\n\n Note that none of the edge types in the above example are 0. It is\n recommended use edge type indices starting at 1, since 0 is\n typically used for terminal edges.\n \"\"\"\n g = minimal_random_graph(num_vertices, **kwargs)\n for v in g.nodes():\n e = (v, v)\n if not g.is_edge(e):\n if np.random.uniform() < prob_loop:\n g.add_edge(*e)\n g = set_types_random(g, **kwargs)\n return g"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef generate_pagerank_graph(num_vertices=250, **kwargs):\n g = minimal_random_graph(num_vertices, **kwargs)\n r = np.zeros(num_vertices)\n for k, pr in nx.pagerank(g).items():\n r[k] = pr\n g = set_types_rank(g, rank=r, **kwargs)\n return g", "response": "Generates a random graph where the vertex types are selected using their pagerank."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a minimal random graph with num_vertices vertices connected to each other.", "response": "def minimal_random_graph(num_vertices, seed=None, **kwargs):\n \"\"\"Creates a connected graph with random vertex locations.\n\n Parameters\n ----------\n num_vertices : int\n The number of vertices in the graph.\n seed : int (optional)\n An integer used to initialize numpy's psuedorandom number\n generators.\n **kwargs :\n Unused.\n\n Returns\n -------\n :class:`.QueueNetworkDiGraph`\n A graph with a ``pos`` vertex property for each vertex's\n position.\n\n Notes\n -----\n This function first places ``num_vertices`` points in the unit square\n randomly (using the uniform distribution). Then, for every vertex\n ``v``, all other vertices with Euclidean distance less or equal to\n ``r`` are connect by an edge --- where ``r`` is the smallest number\n such that the graph ends up connected.\n \"\"\"\n if isinstance(seed, numbers.Integral):\n np.random.seed(seed)\n\n points = np.random.random((num_vertices, 2)) * 10\n edges = []\n\n for k in range(num_vertices - 1):\n for j in range(k + 1, num_vertices):\n v = points[k] - points[j]\n edges.append((k, j, v[0]**2 + v[1]**2))\n\n mytype = [('n1', int), ('n2', int), ('distance', np.float)]\n edges = np.array(edges, dtype=mytype)\n edges = np.sort(edges, order='distance')\n unionF = UnionFind([k for k in range(num_vertices)])\n\n g = nx.Graph()\n\n for n1, n2, dummy in edges:\n unionF.union(n1, n2)\n g.add_edge(n1, n2)\n if unionF.nClusters == 1:\n break\n\n pos = {j: p for j, p in enumerate(points)}\n g = QueueNetworkDiGraph(g.to_directed())\n g.set_pos(pos)\n return g"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_types_random(g, proportions=None, loop_proportions=None, seed=None,\n **kwargs):\n \"\"\"Randomly sets ``edge_type`` (edge type) properties of the graph.\n\n This function randomly assigns each edge a type. The probability of\n an edge being a specific type is proscribed in the\n ``proportions``, ``loop_proportions`` variables.\n\n Parameters\n ----------\n g : :any:`networkx.DiGraph`, :class:`numpy.ndarray`, dict, etc.\n Any object that :any:`DiGraph` accepts.\n proportions : dict (optional, default: ``{k: 0.25 for k in range(1, 4)}``)\n A dictionary of edge types and proportions, where the keys are\n the types and the values are the proportion of non-loop edges\n that are expected to be of that type. The values can must sum\n to one.\n loop_proportions : dict (optional, default: ``{k: 0.25 for k in range(4)}``)\n A dictionary of edge types and proportions, where the keys are\n the types and the values are the proportion of loop edges\n that are expected to be of that type. The values can must sum\n to one.\n seed : int (optional)\n An integer used to initialize numpy's psuedorandom number\n generator.\n **kwargs :\n Unused.\n\n Returns\n -------\n :class:`.QueueNetworkDiGraph`\n Returns the a graph with an ``edge_type`` edge property.\n\n Raises\n ------\n TypeError\n Raised when the parameter ``g`` is not of a type that can be\n made into a :any:`networkx.DiGraph`.\n\n ValueError\n Raises a :exc:`~ValueError` if the ``pType`` values do not sum\n to one.\n\n Notes\n -----\n If ``pTypes`` is not explicitly specified in the arguments, then it\n defaults to four types in the graph (types 0, 1, 2, and 3). It sets\n non-loop edges to be either 1, 2, or 3 33\\% chance, and loops are\n types 0, 1, 2, 3 with 25\\% chance.\n \"\"\"\n g = _test_graph(g)\n\n if isinstance(seed, numbers.Integral):\n np.random.seed(seed)\n\n if proportions is None:\n proportions = {k: 1. / 3 for k in range(1, 4)}\n\n if loop_proportions is None:\n loop_proportions = {k: 1. / 4 for k in range(4)}\n\n edges = [e for e in g.edges() if e[0] != e[1]]\n loops = [e for e in g.edges() if e[0] == e[1]]\n props = list(proportions.values())\n lprops = list(loop_proportions.values())\n\n if not np.isclose(sum(props), 1.0):\n raise ValueError(\"proportions values must sum to one.\")\n if not np.isclose(sum(lprops), 1.0):\n raise ValueError(\"loop_proportions values must sum to one.\")\n\n eTypes = {}\n types = list(proportions.keys())\n values = np.random.choice(types, size=len(edges), replace=True, p=props)\n\n for k, e in enumerate(edges):\n eTypes[e] = values[k]\n\n types = list(loop_proportions.keys())\n values = np.random.choice(types, size=len(loops), replace=True, p=lprops)\n\n for k, e in enumerate(loops):\n eTypes[e] = values[k]\n\n g.new_edge_property('edge_type')\n for e in g.edges():\n g.set_ep(e, 'edge_type', eTypes[e])\n\n return g", "response": "This function sets the edge_type property of the base network with random values."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsetting the edge and types of a graph based on the given rank.", "response": "def set_types_rank(g, rank, pType2=0.1, pType3=0.1, seed=None, **kwargs):\n \"\"\"Creates a stylized graph. Sets edge and types using `pagerank`_.\n\n This function sets the edge types of a graph to be either 1, 2, or\n 3. It sets the vertices to type 2 by selecting the top\n ``pType2 * g.number_of_nodes()`` vertices given by the\n :func:`~networkx.pagerank` of the graph. A loop is added\n to all vertices identified this way (if one does not exist\n already). It then randomly sets vertices close to the type 2\n vertices as type 3, and adds loops to these vertices as well. These\n loops then have edge types the correspond to the vertices type. The\n rest of the edges are set to type 1.\n\n .. _pagerank: http://en.wikipedia.org/wiki/PageRank\n\n Parameters\n ----------\n g : :any:`networkx.DiGraph`, :class:`~numpy.ndarray`, dict, etc.\n Any object that :any:`DiGraph` accepts.\n rank : :class:`numpy.ndarray`\n An ordering of the vertices.\n pType2 : float (optional, default: 0.1)\n Specifies the proportion of vertices that will be of type 2.\n pType3 : float (optional, default: 0.1)\n Specifies the proportion of vertices that will be of type 3 and\n that are near pType2 vertices.\n seed : int (optional)\n An integer used to initialize numpy's psuedo-random number\n generator.\n **kwargs :\n Unused.\n\n Returns\n -------\n :class:`.QueueNetworkDiGraph`\n Returns the a graph with an ``edge_type`` edge property.\n\n Raises\n ------\n TypeError\n Raised when the parameter ``g`` is not of a type that can be\n made into a :any:`DiGraph`.\n \"\"\"\n g = _test_graph(g)\n\n if isinstance(seed, numbers.Integral):\n np.random.seed(seed)\n\n tmp = np.sort(np.array(rank))\n nDests = int(np.ceil(g.number_of_nodes() * pType2))\n dests = np.where(rank >= tmp[-nDests])[0]\n\n if 'pos' not in g.vertex_properties():\n g.set_pos()\n\n dest_pos = np.array([g.vp(v, 'pos') for v in dests])\n nFCQ = int(pType3 * g.number_of_nodes())\n min_g_dist = np.ones(nFCQ) * np.infty\n ind_g_dist = np.ones(nFCQ, int)\n\n r, theta = np.random.random(nFCQ) / 500., np.random.random(nFCQ) * 360.\n xy_pos = np.array([r * np.cos(theta), r * np.sin(theta)]).transpose()\n g_pos = xy_pos + dest_pos[np.array(np.mod(np.arange(nFCQ), nDests), int)]\n\n for v in g.nodes():\n if v not in dests:\n tmp = np.array([_calculate_distance(g.vp(v, 'pos'), g_pos[k, :]) for k in range(nFCQ)])\n min_g_dist = np.min((tmp, min_g_dist), 0)\n ind_g_dist[min_g_dist == tmp] = v\n\n ind_g_dist = np.unique(ind_g_dist)\n fcqs = set(ind_g_dist[:min(nFCQ, len(ind_g_dist))])\n dests = set(dests)\n g.new_vertex_property('loop_type')\n\n for v in g.nodes():\n if v in dests:\n g.set_vp(v, 'loop_type', 3)\n if not g.is_edge((v, v)):\n g.add_edge(v, v)\n elif v in fcqs:\n g.set_vp(v, 'loop_type', 2)\n if not g.is_edge((v, v)):\n g.add_edge(v, v)\n\n g.new_edge_property('edge_type')\n for e in g.edges():\n g.set_ep(e, 'edge_type', 1)\n\n for v in g.nodes():\n if g.vp(v, 'loop_type') in [2, 3]:\n e = (v, v)\n if g.vp(v, 'loop_type') == 2:\n g.set_ep(e, 'edge_type', 2)\n else:\n g.set_ep(e, 'edge_type', 3)\n\n return g"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nstrip comment markers at the front of a block of comment text.", "response": "def strip_comment_marker(text):\n \"\"\" Strip # markers at the front of a block of comment text.\n \"\"\"\n lines = []\n for line in text.splitlines():\n lines.append(line.lstrip('#'))\n text = textwrap.dedent('\\n'.join(lines))\n return text"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nyields all of the documentation for trait definitions on a class object.", "response": "def get_class_traits(klass):\n \"\"\" Yield all of the documentation for trait definitions on a class object.\n \"\"\"\n # FIXME: gracefully handle errors here or in the caller?\n source = inspect.getsource(klass)\n cb = CommentBlocker()\n cb.process_file(StringIO(source))\n mod_ast = compiler.parse(source)\n class_ast = mod_ast.node.nodes[0]\n for node in class_ast.code.nodes:\n # FIXME: handle other kinds of assignments?\n if isinstance(node, compiler.ast.Assign):\n name = node.nodes[0].name\n rhs = unparse(node.expr).strip()\n doc = strip_comment_marker(cb.search_for_comment(node.lineno, default=''))\n yield name, rhs, doc"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds lines to the block.", "response": "def add(self, string, start, end, line):\n \"\"\" Add lines to the block.\n \"\"\"\n if string.strip():\n # Only add if not entirely whitespace.\n self.start_lineno = min(self.start_lineno, start[0])\n self.end_lineno = max(self.end_lineno, end[0])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef process_file(self, file):\n if sys.version_info[0] >= 3:\n nxt = file.__next__\n else:\n nxt = file.next\n for token in tokenize.generate_tokens(nxt):\n self.process_token(*token)\n self.make_index()", "response": "Process a file object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef process_token(self, kind, string, start, end, line):\n if self.current_block.is_comment:\n if kind == tokenize.COMMENT:\n self.current_block.add(string, start, end, line)\n else:\n self.new_noncomment(start[0], end[0])\n else:\n if kind == tokenize.COMMENT:\n self.new_comment(string, start, end, line)\n else:\n self.current_block.add(string, start, end, line)", "response": "Process a single token."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a new noncomment block and add it to the list of blocks.", "response": "def new_noncomment(self, start_lineno, end_lineno):\n \"\"\" We are transitioning from a noncomment to a comment.\n \"\"\"\n block = NonComment(start_lineno, end_lineno)\n self.blocks.append(block)\n self.current_block = block"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef new_comment(self, string, start, end, line):\n prefix = line[:start[1]]\n if prefix.strip():\n # Oops! Trailing comment, not a comment block.\n self.current_block.add(string, start, end, line)\n else:\n # A comment block.\n block = Comment(start[0], end[0], string)\n self.blocks.append(block)\n self.current_block = block", "response": "Add a new comment."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef make_index(self):\n for prev, block in zip(self.blocks[:-1], self.blocks[1:]):\n if not block.is_comment:\n self.index[block.start_lineno] = prev", "response": "Make the index mapping lines of actual code to their associated\n prefix comments."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsearching for the comment block just before the given line number. Returns None if no such comment block is found.", "response": "def search_for_comment(self, lineno, default=None):\n \"\"\" Find the comment block just before the given line number.\n\n Returns None (or the specified default) if there is no such block.\n \"\"\"\n if not self.index:\n self.make_index()\n block = self.index.get(lineno, None)\n text = getattr(block, 'text', default)\n return text"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _generate_contents(self, tar):\n uci = self.render(files=False)\n # create a list with all the packages (and remove empty entries)\n packages = packages_pattern.split(uci)\n if '' in packages:\n packages.remove('')\n # create an UCI file for each configuration package used\n for package in packages:\n lines = package.split('\\n')\n package_name = lines[0]\n text_contents = '\\n'.join(lines[2:])\n self._add_file(tar=tar,\n name='{0}{1}'.format(config_path, package_name),\n contents=text_contents)", "response": "Generates the contents of the configuration files for the current locale."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload the config from a string or dict.", "response": "def _load(self, config):\n \"\"\"\n Loads config from string or dict\n \"\"\"\n if isinstance(config, six.string_types):\n try:\n config = json.loads(config)\n except ValueError:\n pass\n if not isinstance(config, dict):\n raise TypeError('config block must be an istance '\n 'of dict or a valid NetJSON string')\n return config"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _merge_config(self, config, templates):\n if not templates:\n return config\n # type check\n if not isinstance(templates, list):\n raise TypeError('templates argument must be an instance of list')\n # merge templates with main configuration\n result = {}\n config_list = templates + [config]\n for merging in config_list:\n result = merge_config(result, self._load(merging), self.list_identifiers)\n return result", "response": "Merges config with templates"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _render_files(self):\n output = ''\n # render files\n files = self.config.get('files', [])\n # add delimiter\n if files:\n output += '\\n{0}\\n\\n'.format(self.FILE_SECTION_DELIMITER)\n for f in files:\n mode = f.get('mode', DEFAULT_FILE_MODE)\n # add file to output\n file_output = '# path: {0}\\n'\\\n '# mode: {1}\\n\\n'\\\n '{2}\\n\\n'.format(f['path'], mode, f['contents'])\n output += file_output\n return output", "response": "Renders additional files specified in self. config. files."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef render(self, files=True):\n self.validate()\n # convert NetJSON config to intermediate data structure\n if self.intermediate_data is None:\n self.to_intermediate()\n # support multiple renderers\n renderers = getattr(self, 'renderers', None) or [self.renderer]\n # convert intermediate data structure to native configuration\n output = ''\n for renderer_class in renderers:\n renderer = renderer_class(self)\n output += renderer.render()\n # remove reference to renderer instance (not needed anymore)\n del renderer\n # are we required to include\n # additional files?\n if files:\n # render additional files\n files_output = self._render_files()\n if files_output:\n # max 2 new lines\n output += files_output.replace('\\n\\n\\n', '\\n\\n')\n # return the configuration\n return output", "response": "Renders the NetJSON config into a string with the corresponding configuration format."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a string formatted as NetJSON DeviceConfiguration", "response": "def json(self, validate=True, *args, **kwargs):\n \"\"\"\n returns a string formatted as **NetJSON DeviceConfiguration**;\n performs validation before returning output;\n\n ``*args`` and ``*kwargs`` will be passed to ``json.dumps``;\n\n :returns: string\n \"\"\"\n if validate:\n self.validate()\n # automatically adds NetJSON type\n config = deepcopy(self.config)\n config.update({'type': 'DeviceConfiguration'})\n return json.dumps(config, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate(self):\n tar_bytes = BytesIO()\n tar = tarfile.open(fileobj=tar_bytes, mode='w')\n self._generate_contents(tar)\n self._process_files(tar)\n tar.close()\n tar_bytes.seek(0) # set pointer to beginning of stream\n # `mtime` parameter of gzip file must be 0, otherwise any checksum operation\n # would return a different digest even when content is the same.\n # to achieve this we must use the python `gzip` library because the `tarfile`\n # library does not seem to offer the possibility to modify the gzip `mtime`.\n gzip_bytes = BytesIO()\n gz = gzip.GzipFile(fileobj=gzip_bytes, mode='wb', mtime=0)\n gz.write(tar_bytes.getvalue())\n gz.close()\n gzip_bytes.seek(0) # set pointer to beginning of stream\n return gzip_bytes", "response": "Generates a in - memory tar. gz archive containing the native router configuration."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write(self, name, path='./'):\n byte_object = self.generate()\n file_name = '{0}.tar.gz'.format(name)\n if not path.endswith('/'):\n path += '/'\n f = open('{0}{1}'.format(path, file_name), 'wb')\n f.write(byte_object.getvalue())\n f.close()", "response": "Like generate but writes to disk."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _process_files(self, tar):\n # insert additional files\n for file_item in self.config.get('files', []):\n path = file_item['path']\n # remove leading slashes from path\n if path.startswith('/'):\n path = path[1:]\n self._add_file(tar=tar,\n name=path,\n contents=file_item['contents'],\n mode=file_item.get('mode', DEFAULT_FILE_MODE))", "response": "Adds files specified in self. config. files to tarfile instance."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _add_file(self, tar, name, contents, mode=DEFAULT_FILE_MODE):\n byte_contents = BytesIO(contents.encode('utf8'))\n info = tarfile.TarInfo(name=name)\n info.size = len(contents)\n # mtime must be 0 or any checksum operation\n # will return a different digest even when content is the same\n info.mtime = 0\n info.type = tarfile.REGTYPE\n info.mode = int(mode, 8) # permissions converted to decimal notation\n tar.addfile(tarinfo=info, fileobj=byte_contents)", "response": "Adds a single file in tarfile instance."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts the NetJSON configuration dictionary to the intermediate data structure that will be used by the renderer class to generate the router configuration.", "response": "def to_intermediate(self):\n \"\"\"\n Converts the NetJSON configuration dictionary (self.config)\n to the intermediate data structure (self.intermediate_data) that will\n be then used by the renderer class to generate the router configuration\n \"\"\"\n self.validate()\n self.intermediate_data = OrderedDict()\n for converter_class in self.converters:\n # skip unnecessary loop cycles\n if not converter_class.should_run_forward(self.config):\n continue\n converter = converter_class(self)\n value = converter.to_intermediate()\n # maintain backward compatibility with backends\n # that are currently in development by GSoC students\n # TODO for >= 0.6.2: remove once all backends have upgraded\n if value and isinstance(value, (tuple, list)): # pragma: nocover\n value = OrderedDict(value)\n if value:\n self.intermediate_data = merge_config(self.intermediate_data,\n value,\n list_identifiers=['.name'])"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses a native configuration and converts it to a NetJSON configuration dictionary.", "response": "def parse(self, native):\n \"\"\"\n Parses a native configuration and converts\n it to a NetJSON configuration dictionary\n \"\"\"\n if not hasattr(self, 'parser') or not self.parser:\n raise NotImplementedError('Parser class not specified')\n parser = self.parser(native)\n self.intermediate_data = parser.intermediate_data\n del parser\n self.to_netjson()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert the intermediate data structure to the NetJSON configuration dictionary.", "response": "def to_netjson(self):\n \"\"\"\n Converts the intermediate data structure (self.intermediate_data)\n to the NetJSON configuration dictionary (self.config)\n \"\"\"\n self.__backup_intermediate_data()\n self.config = OrderedDict()\n for converter_class in self.converters:\n if not converter_class.should_run_backward(self.intermediate_data):\n continue\n converter = converter_class(self)\n value = converter.to_netjson()\n if value:\n self.config = merge_config(self.config,\n value,\n list_identifiers=self.list_identifiers)\n self.__restore_intermediate_data()\n self.validate()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nmerges config dict into template dict.", "response": "def merge_config(template, config, list_identifiers=None):\n \"\"\"\n Merges ``config`` on top of ``template``.\n\n Conflicting keys are handled in the following way:\n\n * simple values (eg: ``str``, ``int``, ``float``, ecc) in ``config`` will\n overwrite the ones in ``template``\n * values of type ``list`` in both ``config`` and ``template`` will be\n merged using to the ``merge_list`` function\n * values of type ``dict`` will be merged recursively\n\n :param template: template ``dict``\n :param config: config ``dict``\n :param list_identifiers: ``list`` or ``None``\n :returns: merged ``dict``\n \"\"\"\n result = template.copy()\n for key, value in config.items():\n if isinstance(value, dict):\n node = result.get(key, OrderedDict())\n result[key] = merge_config(node, value)\n elif isinstance(value, list) and isinstance(result.get(key), list):\n result[key] = merge_list(result[key], value, list_identifiers)\n else:\n result[key] = value\n return result"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef merge_list(list1, list2, identifiers=None):\n identifiers = identifiers or []\n dict_map = {'list1': OrderedDict(), 'list2': OrderedDict()}\n counter = 1\n for list_ in [list1, list2]:\n container = dict_map['list{0}'.format(counter)]\n for el in list_:\n # merge by internal python id by default\n key = id(el)\n # if el is a dict, merge by keys specified in ``identifiers``\n if isinstance(el, dict):\n for id_key in identifiers:\n if id_key in el:\n key = el[id_key]\n break\n container[key] = deepcopy(el)\n counter += 1\n merged = merge_config(dict_map['list1'], dict_map['list2'])\n return list(merged.values())", "response": "Merges two lists of lists into a single list."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef evaluate_vars(data, context=None):\n context = context or {}\n if isinstance(data, (dict, list)):\n if isinstance(data, dict):\n loop_items = data.items()\n elif isinstance(data, list):\n loop_items = enumerate(data)\n for key, value in loop_items:\n data[key] = evaluate_vars(value, context)\n elif isinstance(data, six.string_types):\n vars_found = var_pattern.findall(data)\n for var in vars_found:\n var = var.strip()\n # if found multiple variables, create a new regexp pattern for each\n # variable, otherwise different variables would get the same value\n # (see https://github.com/openwisp/netjsonconfig/issues/55)\n if len(vars_found) > 1:\n pattern = r'\\{\\{(\\s*%s\\s*)\\}\\}' % var\n # in case of single variables, use the precompiled\n # regexp pattern to save computation\n else:\n pattern = var_pattern\n if var in context:\n data = re.sub(pattern, context[var], data)\n return data", "response": "Evaluates variables in data structure containing variables in context."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a deepcopied version of the value for the given key.", "response": "def get_copy(dict_, key, default=None):\n \"\"\"\n Looks for a key in a dictionary, if found returns\n a deepcopied value, otherwise returns default value\n \"\"\"\n value = dict_.get(key, default)\n if value:\n return deepcopy(value)\n return value"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef type_cast(self, item, schema=None):\n if schema is None:\n schema = self._schema\n properties = schema['properties']\n for key, value in item.items():\n if key not in properties:\n continue\n try:\n json_type = properties[key]['type']\n except KeyError:\n json_type = None\n if json_type == 'integer' and not isinstance(value, int):\n value = int(value)\n elif json_type == 'boolean' and not isinstance(value, bool):\n value = value == '1'\n item[key] = value\n return item", "response": "Takes a dictionary item and applies type casting according to schema fragment\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef to_intermediate(self):\n result = OrderedDict()\n # copy netjson dictionary\n netjson = get_copy(self.netjson, self.netjson_key)\n if isinstance(netjson, list):\n # iterate over copied netjson data structure\n for index, block in enumerate(netjson):\n result = self.to_intermediate_loop(block, result, index + 1)\n else:\n result = self.to_intermediate_loop(netjson, result)\n # return result, expects dict\n return result", "response": "Converts the NetJSON configuration dictionary to the intermediate data structure."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the intermediate data structure to a NetJSON configuration dictionary.", "response": "def to_netjson(self, remove_block=True):\n \"\"\"\n Converts the intermediate data structure (``self.intermediate_datra``)\n to a NetJSON configuration dictionary (``self.config``)\n \"\"\"\n result = OrderedDict()\n # copy list\n intermediate_data = list(self.intermediate_data[self.intermediate_key])\n # iterate over copied intermediate data structure\n for index, block in enumerate(intermediate_data):\n if self.should_skip_block(block):\n continue\n # remove processed block from intermediate data\n # this makes processing remaining blocks easier\n # for some backends\n if remove_block:\n self.intermediate_data[self.intermediate_key].remove(block)\n # specific converter operations are delegated\n # to the ``to_netjson_loop`` method\n result = self.to_netjson_loop(block, result, index + 1)\n # return result, expects dict\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _add_unique_file(self, item):\n if item not in self.config['files']:\n self.config['files'].append(item)", "response": "Add a file to the config if not already present"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_install_context(self):\n config = self.config\n # layer2 VPN list\n l2vpn = []\n for vpn in self.config.get('openvpn', []):\n if vpn.get('dev_type') != 'tap':\n continue\n tap = vpn.copy()\n l2vpn.append(tap)\n # bridge list\n bridges = []\n for interface in self.config.get('interfaces', []):\n if interface['type'] != 'bridge':\n continue\n bridge = interface.copy()\n if bridge.get('addresses'):\n bridge['proto'] = interface['addresses'][0].get('proto')\n bridge['ip'] = interface['addresses'][0].get('address')\n bridges.append(bridge)\n # crontabs present?\n cron = False\n for _file in config.get('files', []):\n path = _file['path']\n if path.startswith('/crontabs') or path.startswith('crontabs'):\n cron = True\n break\n # return context\n return dict(hostname=config['general']['hostname'], # hostname is required\n l2vpn=l2vpn,\n bridges=bridges,\n radios=config.get('radios', []), # radios might be empty\n cron=cron)", "response": "returns the template context for install. sh and uninstall. sh"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate install. sh and adds it to included files AttributeNames", "response": "def _add_install(self, context):\n \"\"\"\n generates install.sh and adds it to included files\n \"\"\"\n contents = self._render_template('install.sh', context)\n self.config.setdefault('files', []) # file list might be empty\n # add install.sh to list of included files\n self._add_unique_file({\n \"path\": \"/install.sh\",\n \"contents\": contents,\n \"mode\": \"755\"\n })"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate uninstall. sh and adds it to included files", "response": "def _add_uninstall(self, context):\n \"\"\"\n generates uninstall.sh and adds it to included files\n \"\"\"\n contents = self._render_template('uninstall.sh', context)\n self.config.setdefault('files', []) # file list might be empty\n # add uninstall.sh to list of included files\n self._add_unique_file({\n \"path\": \"/uninstall.sh\",\n \"contents\": contents,\n \"mode\": \"755\"\n })"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating tc_script. sh and adds it to included files", "response": "def _add_tc_script(self):\n \"\"\"\n generates tc_script.sh and adds it to included files\n \"\"\"\n # fill context\n context = dict(tc_options=self.config.get('tc_options', []))\n # import pdb; pdb.set_trace()\n contents = self._render_template('tc_script.sh', context)\n self.config.setdefault('files', []) # file list might be empty\n # add tc_script.sh to list of included files\n self._add_unique_file({\n \"path\": \"/tc_script.sh\",\n \"contents\": contents,\n \"mode\": \"755\"\n })"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _generate_contents(self, tar):\n uci = self.render(files=False)\n # create a list with all the packages (and remove empty entries)\n packages = re.split('package ', uci)\n if '' in packages:\n packages.remove('')\n # create a file for each configuration package used\n for package in packages:\n lines = package.split('\\n')\n package_name = lines[0]\n text_contents = '\\n'.join(lines[2:])\n text_contents = 'package {0}\\n\\n{1}'.format(package_name, text_contents)\n self._add_file(tar=tar,\n name='uci/{0}.conf'.format(package_name),\n contents=text_contents)\n # prepare template context for install and uninstall scripts\n template_context = self._get_install_context()\n # add install.sh to included files\n self._add_install(template_context)\n # add uninstall.sh to included files\n self._add_uninstall(template_context)\n # add vpn up and down scripts\n self._add_openvpn_scripts()\n # add tc_script\n self._add_tc_script()", "response": "Generates the contents of the configuration files for the current user."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef render(self):\n # get jinja2 template\n template_name = '{0}.jinja2'.format(self.get_name())\n template = self.template_env.get_template(template_name)\n # render template and cleanup\n context = getattr(self.backend, 'intermediate_data', {})\n output = template.render(data=context)\n return self.cleanup(output)", "response": "Renders the configuration by using the jinja2 templating engine"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __intermediate_addresses(self, interface):\n address_list = self.get_copy(interface, 'addresses')\n # do not ignore interfaces if they do not contain any address\n if not address_list:\n return [{'proto': 'none'}]\n result = []\n static = {}\n dhcp = []\n for address in address_list:\n family = address.get('family')\n # dhcp\n if address['proto'] == 'dhcp':\n address['proto'] = 'dhcp' if family == 'ipv4' else 'dhcpv6'\n dhcp.append(self.__intermediate_address(address))\n continue\n if 'gateway' in address:\n uci_key = 'gateway' if family == 'ipv4' else 'ip6gw'\n interface[uci_key] = address['gateway']\n # static\n address_key = 'ipaddr' if family == 'ipv4' else 'ip6addr'\n static.setdefault(address_key, [])\n static[address_key].append('{address}/{mask}'.format(**address))\n static.update(self.__intermediate_address(address))\n if static:\n # do not use CIDR notation when using a single ipv4\n # see https://github.com/openwisp/netjsonconfig/issues/54\n if len(static.get('ipaddr', [])) == 1:\n network = ip_interface(six.text_type(static['ipaddr'][0]))\n static['ipaddr'] = str(network.ip)\n static['netmask'] = str(network.netmask)\n # do not use lists when using a single ipv6 address\n # (avoids to change output of existing configuration)\n if len(static.get('ip6addr', [])) == 1:\n static['ip6addr'] = static['ip6addr'][0]\n result.append(static)\n if dhcp:\n result += dhcp\n return result", "response": "converts NetJSON address to intermediate data structure"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef __intermediate_interface(self, interface, uci_name):\n interface.update({\n '.type': 'interface',\n '.name': uci_name,\n 'ifname': interface.pop('name')\n })\n if 'network' in interface:\n del interface['network']\n if 'mac' in interface:\n # mac address of wireless interface must\n # be set in /etc/config/wireless, therfore\n # we can skip this in /etc/config/network\n if interface.get('type') != 'wireless':\n interface['macaddr'] = interface['mac']\n del interface['mac']\n if 'autostart' in interface:\n interface['auto'] = interface['autostart']\n del interface['autostart']\n if 'disabled' in interface:\n interface['enabled'] = not interface['disabled']\n del interface['disabled']\n if 'wireless' in interface:\n del interface['wireless']\n if 'addresses' in interface:\n del interface['addresses']\n return interface", "response": "converts NetJSON interface to intermediate data structure"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove the intermediate keys from the NetJSON address", "response": "def __intermediate_address(self, address):\n \"\"\"\n deletes NetJSON address keys\n \"\"\"\n for key in self._address_keys:\n if key in address:\n del address[key]\n return address"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts NetJSON bridge to intermediate data structure", "response": "def __intermediate_bridge(self, interface, i):\n \"\"\"\n converts NetJSON bridge to\n UCI intermediate data structure\n \"\"\"\n # ensure type \"bridge\" is only given to one logical interface\n if interface['type'] == 'bridge' and i < 2:\n bridge_members = ' '.join(interface.pop('bridge_members'))\n # put bridge members in ifname attribute\n if bridge_members:\n interface['ifname'] = bridge_members\n # if no members, this is an empty bridge\n else:\n interface['bridge_empty'] = True\n del interface['ifname']\n # bridge has already been defined\n # but we need to add more references to it\n elif interface['type'] == 'bridge' and i >= 2:\n # openwrt adds \"br-\" prefix to bridge interfaces\n # we need to take this into account when referring\n # to these physical names\n if 'br-' not in interface['ifname']:\n interface['ifname'] = 'br-{ifname}'.format(**interface)\n # do not repeat bridge attributes (they have already been processed)\n for attr in ['type', 'bridge_members', 'stp', 'gateway']:\n if attr in interface:\n del interface[attr]\n elif interface['type'] != 'bridge':\n del interface['type']\n return interface"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndetermine the intermediate protocol for the UCI interface.", "response": "def __intermediate_proto(self, interface, address):\n \"\"\"\n determines UCI interface \"proto\" option\n \"\"\"\n # proto defaults to static\n address_proto = address.pop('proto', 'static')\n if 'proto' not in interface:\n return address_proto\n else:\n # allow override on interface level\n return interface.pop('proto')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the intermediate DNS servers for the given address.", "response": "def __intermediate_dns_servers(self, uci, address):\n \"\"\"\n determines UCI interface \"dns\" option\n \"\"\"\n # allow override\n if 'dns' in uci:\n return uci['dns']\n # ignore if using DHCP or if \"proto\" is none\n if address['proto'] in ['dhcp', 'dhcpv6', 'none']:\n return None\n dns = self.netjson.get('dns_servers', None)\n if dns:\n return ' '.join(dns)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __intermediate_dns_search(self, uci, address):\n # allow override\n if 'dns_search' in uci:\n return uci['dns_search']\n # ignore if \"proto\" is none\n if address['proto'] == 'none':\n return None\n dns_search = self.netjson.get('dns_search', None)\n if dns_search:\n return ' '.join(dns_search)", "response": "Returns the intermediate DNS search option for the given address."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of violated schema fragments and related error messages", "response": "def _list_errors(e):\n \"\"\"\n Returns a list of violated schema fragments and related error messages\n :param e: ``jsonschema.exceptions.ValidationError`` instance\n \"\"\"\n error_list = []\n for value, error in zip(e.validator_value, e.context):\n error_list.append((value, error.message))\n if error.context:\n error_list += _list_errors(error)\n return error_list"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndetermines the intermediate hwmode of the current antenna.", "response": "def __intermediate_hwmode(self, radio):\n \"\"\"\n possible return values are: 11a, 11b, 11g\n \"\"\"\n protocol = radio['protocol']\n if protocol in ['802.11a', '802.11b', '802.11g']:\n # return 11a, 11b or 11g\n return protocol[4:]\n # determine hwmode depending on channel used\n if radio['channel'] is 0:\n # when using automatic channel selection, we need an\n # additional parameter to determine the frequency band\n return radio.get('hwmode')\n elif radio['channel'] <= 13:\n return '11g'\n else:\n return '11a'"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nintermediate htmode for mac80211 driver", "response": "def __intermediate_htmode(self, radio):\n \"\"\"\n only for mac80211 driver\n \"\"\"\n protocol = radio.pop('protocol')\n channel_width = radio.pop('channel_width')\n # allow overriding htmode\n if 'htmode' in radio:\n return radio['htmode']\n if protocol == '802.11n':\n return 'HT{0}'.format(channel_width)\n elif protocol == '802.11ac':\n return 'VHT{0}'.format(channel_width)\n # disables n\n return 'NONE'"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning NetJSON protocol for the given radio attribute.", "response": "def __netjson_protocol(self, radio):\n \"\"\"\n determines NetJSON protocol radio attribute\n \"\"\"\n htmode = radio.get('htmode')\n hwmode = radio.get('hwmode', None)\n if htmode.startswith('HT'):\n return '802.11n'\n elif htmode.startswith('VHT'):\n return '802.11ac'\n return '802.{0}'.format(hwmode)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __netjson_channel_width(self, radio):\n htmode = radio.pop('htmode')\n if htmode == 'NONE':\n return 20\n channel_width = htmode.replace('VHT', '').replace('HT', '')\n # we need to override htmode\n if '+' in channel_width or '-' in channel_width:\n radio['htmode'] = htmode\n channel_width = channel_width[0:-1]\n return int(channel_width)", "response": "Determine NetJSON channel width from radio attribute."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nremove empty lines from the output.", "response": "def cleanup(self, output):\n \"\"\"\n Generates consistent OpenWRT/LEDE UCI output\n \"\"\"\n # correct indentation\n output = output.replace(' ', '')\\\n .replace('\\noption', '\\n\\toption')\\\n .replace('\\nlist', '\\n\\tlist')\n # convert True to 1 and False to 0\n output = output.replace('True', '1')\\\n .replace('False', '0')\n # max 2 consecutive \\n delimiters\n output = output.replace('\\n\\n\\n', '\\n\\n')\n # if output is present\n # ensure it always ends with 1 new line\n if output.endswith('\\n\\n'):\n return output[0:-1]\n return output"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef auto_client(cls, host, server, ca_path=None, ca_contents=None,\n cert_path=None, cert_contents=None, key_path=None,\n key_contents=None):\n \"\"\"\n Returns a configuration dictionary representing an OpenVPN client configuration\n that is compatible with the passed server configuration.\n\n :param host: remote VPN server\n :param server: dictionary representing a single OpenVPN server configuration\n :param ca_path: optional string representing path to CA, will consequently add\n a file in the resulting configuration dictionary\n :param ca_contents: optional string representing contents of CA file\n :param cert_path: optional string representing path to certificate, will consequently add\n a file in the resulting configuration dictionary\n :param cert_contents: optional string representing contents of cert file\n :param key_path: optional string representing path to key, will consequently add\n a file in the resulting configuration dictionary\n :param key_contents: optional string representing contents of key file\n :returns: dictionary representing a single OpenVPN client configuration\n \"\"\"\n # client defaults\n client = {\n \"mode\": \"p2p\",\n \"nobind\": True,\n \"resolv_retry\": \"infinite\",\n \"tls_client\": True\n }\n # remote\n port = server.get('port') or 1195\n client['remote'] = [{'host': host, 'port': port}]\n # proto\n if server.get('proto') == 'tcp-server':\n client['proto'] = 'tcp-client'\n else:\n client['proto'] = 'udp'\n # determine if pull must be True\n if 'server' in server or 'server_bridge' in server:\n client['pull'] = True\n # tls_client\n if 'tls_server' not in server or not server['tls_server']:\n client['tls_client'] = False\n # ns_cert_type\n ns_cert_type = {None: '',\n '': '',\n 'client': 'server'}\n client['ns_cert_type'] = ns_cert_type[server.get('ns_cert_type')]\n # remote_cert_tls\n remote_cert_tls = {None: '',\n '': '',\n 'client': 'server'}\n client['remote_cert_tls'] = remote_cert_tls[server.get('remote_cert_tls')]\n copy_keys = ['name', 'dev_type', 'dev', 'comp_lzo', 'auth',\n 'cipher', 'ca', 'cert', 'key', 'pkcs12', 'mtu_disc', 'mtu_test',\n 'fragment', 'mssfix', 'keepalive', 'persist_tun', 'mute',\n 'persist_key', 'script_security', 'user', 'group', 'log',\n 'mute_replay_warnings', 'secret', 'reneg_sec', 'tls_timeout',\n 'tls_cipher', 'float', 'fast_io', 'verb']\n for key in copy_keys:\n if key in server:\n client[key] = server[key]\n files = cls._auto_client_files(client, ca_path, ca_contents,\n cert_path, cert_contents,\n key_path, key_contents)\n return {\n 'openvpn': [client],\n 'files': files\n }", "response": "Creates a dictionary representing an OpenVPN client configuration that is compatible with the passed server configuration."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _auto_client_files(cls, client, ca_path=None, ca_contents=None, cert_path=None,\n cert_contents=None, key_path=None, key_contents=None):\n \"\"\"\n returns a list of NetJSON extra files for automatically generated clients\n produces side effects in ``client`` dictionary\n \"\"\"\n files = []\n if ca_path and ca_contents:\n client['ca'] = ca_path\n files.append(dict(path=ca_path,\n contents=ca_contents,\n mode=DEFAULT_FILE_MODE))\n if cert_path and cert_contents:\n client['cert'] = cert_path\n files.append(dict(path=cert_path,\n contents=cert_contents,\n mode=DEFAULT_FILE_MODE))\n if key_path and key_contents:\n client['key'] = key_path\n files.append(dict(path=key_path,\n contents=key_contents,\n mode=DEFAULT_FILE_MODE,))\n return files", "response": "Returns a list of NetJSON extra files for automatically generated clients"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_install_requires():\n requirements = []\n for line in open('requirements.txt').readlines():\n # skip to next iteration if comment or empty line\n if line.startswith('#') or line == '' or line.startswith('http') or line.startswith('git'):\n continue\n # add line to requirements\n requirements.append(line.replace('\\n', ''))\n # add py2-ipaddress if python2\n if sys.version_info.major < 3:\n requirements.append('py2-ipaddress')\n return requirements", "response": "parse requirements. txt ignore links exclude comments"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef events(self, **kwargs):\n return self.__api.events(query=EqualsOperator(\"report\", self.hash_),\n **kwargs)", "response": "Get all events for this report."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef facts(self, **kwargs):\n return self.__api.facts(query=EqualsOperator(\"certname\", self.name),\n **kwargs)", "response": "Get all facts of this node."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fact(self, name):\n facts = self.facts(name=name)\n return next(fact for fact in facts)", "response": "Get a single fact from this node."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets all resources of this node or all resources of the specified type and title.", "response": "def resources(self, type_=None, title=None, **kwargs):\n \"\"\"Get all resources of this node or all resources of the specified\n type. Additional arguments may also be specified that will be passed\n to the query function.\n \"\"\"\n if type_ is None:\n resources = self.__api.resources(\n query=EqualsOperator(\"certname\", self.name),\n **kwargs)\n elif type_ is not None and title is None:\n resources = self.__api.resources(\n type_=type_,\n query=EqualsOperator(\"certname\", self.name),\n **kwargs)\n else:\n resources = self.__api.resources(\n type_=type_,\n title=title,\n query=EqualsOperator(\"certname\", self.name),\n **kwargs)\n return resources"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef resource(self, type_, title, **kwargs):\n resources = self.__api.resources(\n type_=type_,\n title=title,\n query=EqualsOperator(\"certname\", self.name),\n **kwargs)\n return next(resource for resource in resources)", "response": "Get a resource matching the supplied type and title."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef reports(self, **kwargs):\n return self.__api.reports(\n query=EqualsOperator(\"certname\", self.name),\n **kwargs)", "response": "Get all reports for this node."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _url(self, endpoint, path=None):\n\n log.debug('_url called with endpoint: {0} and path: {1}'.format(\n endpoint, path))\n\n try:\n endpoint = ENDPOINTS[endpoint]\n except KeyError:\n # If we reach this we're trying to query an endpoint that doesn't\n # exist. This shouldn't happen unless someone made a booboo.\n raise APIError\n\n url = '{base_url}/{endpoint}'.format(\n base_url=self.base_url,\n endpoint=endpoint,\n )\n\n if path is not None:\n url = '{0}/{1}'.format(url, quote(path))\n\n return url", "response": "Returns the complete URL for the given endpoint and path."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef nodes(self, unreported=2, with_status=False, **kwargs):\n nodes = self._query('nodes', **kwargs)\n now = datetime.datetime.utcnow()\n # If we happen to only get one node back it\n # won't be inside a list so iterating over it\n # goes boom. Therefor we wrap a list around it.\n if type(nodes) == dict:\n nodes = [nodes, ]\n\n if with_status:\n latest_events = self.event_counts(\n query=EqualsOperator(\"latest_report?\", True),\n summarize_by='certname'\n )\n\n for node in nodes:\n node['status_report'] = None\n node['events'] = None\n\n if with_status:\n status = [s for s in latest_events\n if s['subject']['title'] == node['certname']]\n\n try:\n node['status_report'] = node['latest_report_status']\n\n if status:\n node['events'] = status[0]\n except KeyError:\n if status:\n node['events'] = status = status[0]\n if status['successes'] > 0:\n node['status_report'] = 'changed'\n if status['noops'] > 0:\n node['status_report'] = 'noop'\n if status['failures'] > 0:\n node['status_report'] = 'failed'\n else:\n node['status_report'] = 'unchanged'\n\n # node report age\n if node['report_timestamp'] is not None:\n try:\n last_report = json_to_datetime(\n node['report_timestamp'])\n last_report = last_report.replace(tzinfo=None)\n unreported_border = now - timedelta(hours=unreported)\n if last_report < unreported_border:\n delta = (now - last_report)\n node['unreported'] = True\n node['unreported_time'] = '{0}d {1}h {2}m'.format(\n delta.days,\n int(delta.seconds / 3600),\n int((delta.seconds % 3600) / 60)\n )\n except AttributeError:\n node['unreported'] = True\n\n if not node['report_timestamp']:\n node['unreported'] = True\n\n yield Node(self,\n name=node['certname'],\n deactivated=node['deactivated'],\n expired=node['expired'],\n report_timestamp=node['report_timestamp'],\n catalog_timestamp=node['catalog_timestamp'],\n facts_timestamp=node['facts_timestamp'],\n status_report=node['status_report'],\n noop=node.get('latest_report_noop'),\n noop_pending=node.get('latest_report_noop_pending'),\n events=node['events'],\n unreported=node.get('unreported'),\n unreported_time=node.get('unreported_time'),\n report_environment=node['report_environment'],\n catalog_environment=node['catalog_environment'],\n facts_environment=node['facts_environment'],\n latest_report_hash=node.get('latest_report_hash'),\n cached_catalog_status=node.get('cached_catalog_status')\n )", "response": "Query for nodes by either name or query."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets a single node from PuppetDB.", "response": "def node(self, name):\n \"\"\"Gets a single node from PuppetDB.\n\n :param name: The name of the node search.\n :type name: :obj:`string`\n\n :return: An instance of Node\n :rtype: :class:`pypuppetdb.types.Node`\n \"\"\"\n nodes = self.nodes(path=name)\n return next(node for node in nodes)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting the known catalog edges formed between two resources.", "response": "def edges(self, **kwargs):\n \"\"\"Get the known catalog edges, formed between two resources.\n\n :param \\*\\*kwargs: The rest of the keyword arguments are passed\n to the _query function.\n\n :returns: A generating yielding Edges.\n :rtype: :class:`pypuppetdb.types.Edge`\n \"\"\"\n edges = self._query('edges', **kwargs)\n\n for edge in edges:\n identifier_source = edge['source_type'] + \\\n '[' + edge['source_title'] + ']'\n identifier_target = edge['target_type'] + \\\n '[' + edge['target_title'] + ']'\n yield Edge(source=self.resources[identifier_source],\n target=self.resources[identifier_target],\n relationship=edge['relationship'],\n node=edge['certname'])"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nquerying for facts limited by either name value and or query.", "response": "def facts(self, name=None, value=None, **kwargs):\n \"\"\"Query for facts limited by either name, value and/or query.\n\n :param name: (Optional) Only return facts that match this name.\n :type name: :obj:`string`\n :param value: (Optional) Only return facts of `name` that\\\n match this value. Use of this parameter requires the `name`\\\n parameter be set.\n :type value: :obj:`string`\n :param \\*\\*kwargs: The rest of the keyword arguments are passed\n to the _query function\n\n :returns: A generator yielding Facts.\n :rtype: :class:`pypuppetdb.types.Fact`\n \"\"\"\n if name is not None and value is not None:\n path = '{0}/{1}'.format(name, value)\n elif name is not None and value is None:\n path = name\n else:\n path = None\n\n facts = self._query('facts', path=path, **kwargs)\n for fact in facts:\n yield Fact(\n node=fact['certname'],\n name=fact['name'],\n value=fact['value'],\n environment=fact['environment']\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nquery for resources limited by either type or title or query.", "response": "def resources(self, type_=None, title=None, **kwargs):\n \"\"\"Query for resources limited by either type and/or title or query.\n This will yield a Resources object for every returned resource.\n\n :param type_: (Optional) The resource type. This can be any resource\n type referenced in\\\n 'https://docs.puppetlabs.com/references/latest/type.html'\n :type type_: :obj:`string`\n :param title: (Optional) The name of the resource as declared as the\n 'namevar' in the Puppet Manifests. This parameter requires the\\\n `type_` parameter be set.\n :type title: :obj:`string`\n :param \\*\\*kwargs: The rest of the keyword arguments are passed\n to the _query function\n\n :returns: A generator yielding Resources\n :rtype: :class:`pypuppetdb.types.Resource`\n \"\"\"\n path = None\n\n if type_ is not None:\n type_ = self._normalize_resource_type(type_)\n\n if title is not None:\n path = '{0}/{1}'.format(type_, title)\n elif title is None:\n path = type_\n\n resources = self._query('resources', path=path, **kwargs)\n for resource in resources:\n yield Resource(\n node=resource['certname'],\n name=resource['title'],\n type_=resource['type'],\n tags=resource['tags'],\n exported=resource['exported'],\n sourcefile=resource['file'],\n sourceline=resource['line'],\n parameters=resource['parameters'],\n environment=resource['environment'],\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef catalog(self, node):\n catalogs = self.catalogs(path=node)\n return next(x for x in catalogs)", "response": "Get the available catalog for a given node."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the catalog information from the infrastructure based on path and or query results.", "response": "def catalogs(self, **kwargs):\n \"\"\"Get the catalog information from the infrastructure based on path\n and/or query results. It is strongly recommended to include query\n and/or paging parameters for this endpoint to prevent large result\n sets or PuppetDB performance bottlenecks.\n\n :param \\*\\*kwargs: The rest of the keyword arguments are passed\n to the _query function.\n\n :returns: A generator yielding Catalogs\n :rtype: :class:`pypuppetdb.types.Catalog`\n \"\"\"\n catalogs = self._query('catalogs', **kwargs)\n\n if type(catalogs) == dict:\n catalogs = [catalogs, ]\n\n for catalog in catalogs:\n yield Catalog(node=catalog['certname'],\n edges=catalog['edges']['data'],\n resources=catalog['resources']['data'],\n version=catalog['version'],\n transaction_uuid=catalog['transaction_uuid'],\n environment=catalog['environment'],\n code_id=catalog.get('code_id'),\n catalog_uuid=catalog.get('catalog_uuid'))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef aggregate_event_counts(self, summarize_by, query=None,\n count_by=None, count_filter=None):\n \"\"\"Get event counts from puppetdb aggregated into a single map.\n\n :param summarize_by: (Required) The object type to be counted on.\n Valid values are 'containing_class', 'resource'\n and 'certname' or any comma-separated value\n thereof.\n :type summarize_by: :obj:`string`\n :param query: (Optional) The PuppetDB query to filter the results.\n This query is passed to the `events` endpoint.\n :type query: :obj:`string`\n :param count_by: (Optional) The object type that is counted when\n building the counts of 'successes', 'failures',\n 'noops' and 'skips'. Support values are 'certname'\n and 'resource' (default)\n :type count_by: :obj:`string`\n :param count_filter: (Optional) A JSON query that is applied to the\n event-counts output but before the results are\n aggregated. Supported operators are `=`, `>`,\n `<`, `>=`, and `<=`. Supported fields are\n `failures`, `successes`, `noops`, and `skips`.\n :type count_filter: :obj:`string`\n\n :returns: A dictionary of name/value results.\n :rtype: :obj:`dict`\n \"\"\"\n return self._query('aggregate-event-counts',\n query=query, summarize_by=summarize_by,\n count_by=count_by, count_filter=count_filter)", "response": "Get the event counts from puppetdb aggregated into a single map."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef reports(self, **kwargs):\n reports = self._query('reports', **kwargs)\n for report in reports:\n yield Report(\n api=self,\n node=report['certname'],\n hash_=report['hash'],\n start=report['start_time'],\n end=report['end_time'],\n received=report['receive_time'],\n version=report['configuration_version'],\n format_=report['report_format'],\n agent_version=report['puppet_version'],\n transaction=report['transaction_uuid'],\n environment=report['environment'],\n status=report['status'],\n noop=report.get('noop'),\n noop_pending=report.get('noop_pending'),\n metrics=report['metrics']['data'],\n logs=report['logs']['data'],\n code_id=report.get('code_id'),\n catalog_uuid=report.get('catalog_uuid'),\n cached_catalog_status=report.get('cached_catalog_status')\n )", "response": "Get reports for our infrastructure."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting Node and Fact information with an alternative query syntax for structured facts and fact - contents and fact - sets endpoints for many fact - related queries.", "response": "def inventory(self, **kwargs):\n \"\"\"Get Node and Fact information with an alternative query syntax\n for structured facts instead of using the facts, fact-contents and\n factsets endpoints for many fact-related queries.\n\n :param \\*\\*kwargs: The rest of the keyword arguments are passed\n to the _query function.\n\n :returns: A generator yielding Inventory\n :rtype: :class:`pypuppetdb.types.Inventory`\n \"\"\"\n inventory = self._query('inventory', **kwargs)\n for inv in inventory:\n yield Inventory(\n node=inv['certname'],\n time=inv['timestamp'],\n environment=inv['environment'],\n facts=inv['facts'],\n trusted=inv['trusted']\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomparing two objects x and y and returns an integer according to the outcome.", "response": "def versioncmp(v1, v2):\n \"\"\"Compares two objects, x and y, and returns an integer according to the\n outcome. The return value is negative if x < y, zero if x == y and\n positive if x > y.\n\n :param v1: The first object to compare.\n :param v2: The second object to compare.\n\n :returns: -1, 0 or 1.\n :rtype: :obj:`int`\n \"\"\"\n def normalize(v):\n \"\"\"Removes leading zeroes from right of a decimal point from v and\n returns an array of values separated by '.'\n\n :param v: The data to normalize.\n\n :returns: An list representation separated by '.' with all leading\n zeroes stripped.\n :rtype: :obj:`list`\n \"\"\"\n return [int(x) for x in re.sub(r'(\\.0+)*$', '', v).split(\".\")]\n try:\n return cmp(normalize(v1), normalize(v2))\n except NameError:\n return (normalize(v1) > normalize(v2)) - (\n normalize(v1) < normalize(v2))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef connect(host='localhost', port=8080, ssl_verify=False, ssl_key=None,\n ssl_cert=None, timeout=10, protocol=None, url_path='/',\n username=None, password=None, token=None):\n \"\"\"Connect with PuppetDB. This will return an object allowing you\n to query the API through its methods.\n\n :param host: (Default: 'localhost;) Hostname or IP of PuppetDB.\n :type host: :obj:`string`\n\n :param port: (Default: '8080') Port on which to talk to PuppetDB.\n :type port: :obj:`int`\n\n :param ssl_verify: (optional) Verify PuppetDB server certificate.\n :type ssl_verify: :obj:`bool` or :obj:`string` True, False or filesystem \\\n path to CA certificate.\n\n :param ssl_key: (optional) Path to our client secret key.\n :type ssl_key: :obj:`None` or :obj:`string` representing a filesystem\\\n path.\n\n :param ssl_cert: (optional) Path to our client certificate.\n :type ssl_cert: :obj:`None` or :obj:`string` representing a filesystem\\\n path.\n\n :param timeout: (Default: 10) Number of seconds to wait for a response.\n :type timeout: :obj:`int`\n\n :param protocol: (optional) Explicitly specify the protocol to be used\n (especially handy when using HTTPS with ssl_verify=False and\n without certs)\n :type protocol: :obj:`None` or :obj:`string`\n\n :param url_path: (Default: '/') The URL path where PuppetDB is served\n :type url_path: :obj:`None` or :obj:`string`\n\n :param username: (optional) The username to use for HTTP basic\n authentication\n :type username: :obj:`None` or :obj:`string`\n\n :param password: (optional) The password to use for HTTP basic\n authentication\n :type password: :obj:`None` or :obj:`string`\n\n :param token: (optional) The x-auth token to use for X-Authentication\n :type token: :obj:`None` or :obj:`string`\n \"\"\"\n return BaseAPI(host=host, port=port,\n timeout=timeout, ssl_verify=ssl_verify, ssl_key=ssl_key,\n ssl_cert=ssl_cert, protocol=protocol, url_path=url_path,\n username=username, password=password, token=token)", "response": "Connect to the PuppetDB server."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef collection_callback(result=None):\n print(\"Header: {0} | Index: {1} | State: {2} | Status: {3}\".format(\n result.headerIndex,\n result.index,\n opendnp3.CommandPointStateToString(result.state),\n opendnp3.CommandStatusToString(result.status)\n ))", "response": "Callback function for collection command."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef command_callback(result=None):\n print(\"Received command result with summary: {}\".format(opendnp3.TaskCompletionToString(result.summary)))\n result.ForeachItem(collection_callback)", "response": "Callback function for each command result."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef main():\n # app = MyMaster()\n app = MyMaster(log_handler=MyLogger(),\n listener=AppChannelListener(),\n soe_handler=SOEHandler(),\n master_application=MasterApplication())\n _log.debug('Initialization complete. In command loop.')\n # Ad-hoc tests can be performed at this point. See master_cmd.py for examples.\n app.shutdown()\n _log.debug('Exiting.')\n exit()", "response": "The main function of the master."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef send_direct_operate_command(self, command, index, callback=asiodnp3.PrintingCommandCallback.Get(),\n config=opendnp3.TaskConfig().Default()):\n \"\"\"\n Direct operate a single command\n\n :param command: command to operate\n :param index: index of the command\n :param callback: callback that will be invoked upon completion or failure\n :param config: optional configuration that controls normal callbacks and allows the user to be specified for SA\n \"\"\"\n self.master.DirectOperate(command, index, callback, config)", "response": "Send a direct operating command to the master"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef send_direct_operate_command_set(self, command_set, callback=asiodnp3.PrintingCommandCallback.Get(),\n config=opendnp3.TaskConfig().Default()):\n \"\"\"\n Direct operate a set of commands\n\n :param command_set: set of command headers\n :param callback: callback that will be invoked upon completion or failure\n :param config: optional configuration that controls normal callbacks and allows the user to be specified for SA\n \"\"\"\n self.master.DirectOperate(command_set, callback, config)", "response": "Direct operates a set of commands"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef send_select_and_operate_command(self, command, index, callback=asiodnp3.PrintingCommandCallback.Get(),\n config=opendnp3.TaskConfig().Default()):\n \"\"\"\n Select and operate a single command\n\n :param command: command to operate\n :param index: index of the command\n :param callback: callback that will be invoked upon completion or failure\n :param config: optional configuration that controls normal callbacks and allows the user to be specified for SA\n \"\"\"\n self.master.SelectAndOperate(command, index, callback, config)", "response": "Send select and operate command to the master"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send_select_and_operate_command_set(self, command_set, callback=asiodnp3.PrintingCommandCallback.Get(),\n config=opendnp3.TaskConfig().Default()):\n \"\"\"\n Select and operate a set of commands\n\n :param command_set: set of command headers\n :param callback: callback that will be invoked upon completion or failure\n :param config: optional configuration that controls normal callbacks and allows the user to be specified for SA\n \"\"\"\n self.master.SelectAndOperate(command_set, callback, config)", "response": "Send select and operate command set to the master"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef Process(self, info, values):\n visitor_class_types = {\n opendnp3.ICollectionIndexedBinary: VisitorIndexedBinary,\n opendnp3.ICollectionIndexedDoubleBitBinary: VisitorIndexedDoubleBitBinary,\n opendnp3.ICollectionIndexedCounter: VisitorIndexedCounter,\n opendnp3.ICollectionIndexedFrozenCounter: VisitorIndexedFrozenCounter,\n opendnp3.ICollectionIndexedAnalog: VisitorIndexedAnalog,\n opendnp3.ICollectionIndexedBinaryOutputStatus: VisitorIndexedBinaryOutputStatus,\n opendnp3.ICollectionIndexedAnalogOutputStatus: VisitorIndexedAnalogOutputStatus,\n opendnp3.ICollectionIndexedTimeAndInterval: VisitorIndexedTimeAndInterval\n }\n visitor_class = visitor_class_types[type(values)]\n visitor = visitor_class()\n values.Foreach(visitor)\n for index, value in visitor.index_and_value:\n log_string = 'SOEHandler.Process {0}\\theaderIndex={1}\\tdata_type={2}\\tindex={3}\\tvalue={4}'\n _log.debug(log_string.format(info.gv, info.headerIndex, type(values).__name__, index, value))", "response": "Process measurement data.\n\n :param info: HeaderInfo\n :param values: A collection of values received from the Outstation (various data types are possible)."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef main():\n app = OutstationApplication()\n _log.debug('Initialization complete. In command loop.')\n # Ad-hoc tests can be inserted here if desired. See outstation_cmd.py for examples.\n app.shutdown()\n _log.debug('Exiting.')\n exit()", "response": "The main function of the main function."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nset up the OpenDNP3 configuration.", "response": "def configure_stack():\n \"\"\"Set up the OpenDNP3 configuration.\"\"\"\n stack_config = asiodnp3.OutstationStackConfig(opendnp3.DatabaseSizes.AllTypes(10))\n stack_config.outstation.eventBufferConfig = opendnp3.EventBufferConfig().AllTypes(10)\n stack_config.outstation.params.allowUnsolicited = True\n stack_config.link.LocalAddr = 10\n stack_config.link.RemoteAddr = 1\n stack_config.link.KeepAliveTimeout = openpal.TimeDuration().Max()\n return stack_config"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef configure_database(db_config):\n db_config.analog[1].clazz = opendnp3.PointClass.Class2\n db_config.analog[1].svariation = opendnp3.StaticAnalogVariation.Group30Var1\n db_config.analog[1].evariation = opendnp3.EventAnalogVariation.Group32Var7\n db_config.analog[2].clazz = opendnp3.PointClass.Class2\n db_config.analog[2].svariation = opendnp3.StaticAnalogVariation.Group30Var1\n db_config.analog[2].evariation = opendnp3.EventAnalogVariation.Group32Var7\n db_config.binary[1].clazz = opendnp3.PointClass.Class2\n db_config.binary[1].svariation = opendnp3.StaticBinaryVariation.Group1Var2\n db_config.binary[1].evariation = opendnp3.EventBinaryVariation.Group2Var2\n db_config.binary[2].clazz = opendnp3.PointClass.Class2\n db_config.binary[2].svariation = opendnp3.StaticBinaryVariation.Group1Var2\n db_config.binary[2].evariation = opendnp3.EventBinaryVariation.Group2Var2", "response": "Configure the database of the input points."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef GetApplicationIIN(self):\n application_iin = opendnp3.ApplicationIIN()\n application_iin.configCorrupt = False\n application_iin.deviceTrouble = False\n application_iin.localControl = False\n application_iin.needTime = False\n # Just for testing purposes, convert it to an IINField and display the contents of the two bytes.\n iin_field = application_iin.ToIIN()\n _log.debug('OutstationApplication.GetApplicationIIN: IINField LSB={}, MSB={}'.format(iin_field.LSB,\n iin_field.MSB))\n return application_iin", "response": "Return the application - controlled IIN field."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprocesses a received PointValue.", "response": "def process_point_value(cls, command_type, command, index, op_type):\n \"\"\"\n A PointValue was received from the Master. Process its payload.\n\n :param command_type: (string) Either 'Select' or 'Operate'.\n :param command: A ControlRelayOutputBlock or else a wrapped data value (AnalogOutputInt16, etc.).\n :param index: (integer) DNP3 index of the payload's data definition.\n :param op_type: An OperateType, or None if command_type == 'Select'.\n \"\"\"\n _log.debug('Processing received point value for index {}: {}'.format(index, command))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef apply_update(self, value, index):\n _log.debug('Recording {} measurement, index={}, value={}'.format(type(value).__name__, index, value.value))\n builder = asiodnp3.UpdateBuilder()\n builder.Update(value, index)\n update = builder.Build()\n OutstationApplication.get_outstation().Apply(update)", "response": "Record an opendnp3 data value in the opendnp3 database."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nhandles a Select command.", "response": "def Select(self, command, index):\n \"\"\"\n The Master sent a Select command to the Outstation. Handle it.\n\n :param command: ControlRelayOutputBlock,\n AnalogOutputInt16, AnalogOutputInt32, AnalogOutputFloat32, or AnalogOutputDouble64.\n :param index: int\n :return: CommandStatus\n \"\"\"\n OutstationApplication.process_point_value('Select', command, index, None)\n return opendnp3.CommandStatus.SUCCESS"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nhandle an Operate command.", "response": "def Operate(self, command, index, op_type):\n \"\"\"\n The Master sent an Operate command to the Outstation. Handle it.\n\n :param command: ControlRelayOutputBlock,\n AnalogOutputInt16, AnalogOutputInt32, AnalogOutputFloat32, or AnalogOutputDouble64.\n :param index: int\n :param op_type: OperateType\n :return: CommandStatus\n \"\"\"\n OutstationApplication.process_point_value('Operate', command, index, op_type)\n return opendnp3.CommandStatus.SUCCESS"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsend the Master an AnalogInput value. Command syntax is a index value", "response": "def do_a(self, line):\n \"\"\"Send the Master an AnalogInput (group 32) value. Command syntax is: a index value\"\"\"\n index, value_string = self.index_and_value_from_line(line)\n if index and value_string:\n try:\n self.application.apply_update(opendnp3.Analog(float(value_string)), index)\n except ValueError:\n print('Please enter a floating-point value as the second argument.')"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsends the Master a BinaryInput ( group 2 ) value. Command syntax is true or false", "response": "def do_b(self, line):\n \"\"\"Send the Master a BinaryInput (group 2) value. Command syntax is: 'b index true' or 'b index false'\"\"\"\n index, value_string = self.index_and_value_from_line(line)\n if index and value_string:\n if value_string.lower() == 'true' or value_string.lower() == 'false':\n self.application.apply_update(opendnp3.Binary(value_string == 'true'), index)\n else:\n print('Please enter true or false as the second argument.')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef do_b0(self, line):\n self.application.apply_update(opendnp3.Binary(False), index=6)", "response": "Send the Master a BinaryInput value of False at index 6."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending the Master a Counter value. Command syntax is c index value", "response": "def do_c(self, line):\n \"\"\"Send the Master a Counter (group 22) value. Command syntax is: c index value\"\"\"\n index, value_string = self.index_and_value_from_line(line)\n if index and value_string:\n try:\n self.application.apply_update(opendnp3.Counter(int(value_string)), index)\n except ValueError:\n print('Please enter an integer value as the second argument.')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsending the Master a DoubleBitBinaryInput ( group 4 ) value of DETERMINED_ON. Command syntax is d index", "response": "def do_d(self, line):\n \"\"\"Send the Master a DoubleBitBinaryInput (group 4) value of DETERMINED_ON. Command syntax is: d index\"\"\"\n index = self.index_from_line(line)\n if index:\n self.application.apply_update(opendnp3.DoubleBitBinary(opendnp3.DoubleBit.DETERMINED_ON), index)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef do_menu(self, line):\n print('\\ta\\t\\tAnalog measurement.\\tEnter index and value as arguments.')\n print('\\ta2\\t\\tAnalog 2 for MMDC.Vol (index 4).')\n print('\\tb\\t\\tBinary measurement.\\tEnter index and value as arguments.')\n print('\\tb0\\t\\tBinary False for MMDC1.Amp.range (index 6).')\n print('\\tc\\t\\tCounter measurement.\\tEnter index and value as arguments.')\n print('\\td\\t\\tDoubleBit DETERMINED_ON.\\tEnter index as an argument.')\n print('\\thelp\\t\\tDisplay command-line help.')\n print('\\tmenu\\t\\tDisplay this menu.')\n print('\\tquit')", "response": "Display a menu of command - line options."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses an index and value from command line args and return them.", "response": "def index_and_value_from_line(line):\n \"\"\"Parse an index (integer) and value (string) from command line args and return them.\"\"\"\n try:\n index = int(line.split(' ')[0])\n except (ValueError, IndexError):\n print('Please enter an integer index as the first argument.')\n index = None\n try:\n value_string = line.split(' ')[1]\n except (ValueError, IndexError):\n print('Please enter a second argument.')\n value_string = None\n return index, value_string"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse an index from command line args and return it.", "response": "def index_from_line(line):\n \"\"\"Parse an index (integer) from command line args and return it.\"\"\"\n try:\n index = int(line.split(' ')[0])\n except (ValueError, IndexError):\n print('Please enter an integer index as the first argument.')\n index = None\n return index"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef do_menu(self, line):\n print('\\tchan_log_all\\tSet the channel log level to ALL_COMMS.')\n print('\\tchan_log_normal\\tSet the channel log level to NORMAL.')\n print('\\tdisable_unsol\\tPerform the function DISABLE_UNSOLICITED.')\n print('\\thelp\\t\\tDisplay command-line help.')\n print('\\tmast_log_all\\tSet the master log level to ALL_COMMS.')\n print('\\tmast_log_normal\\tSet the master log level to NORMAL.')\n print('\\tmenu\\t\\tDisplay this menu.')\n print('\\to1\\t\\tSend a DirectOperate LATCH_ON command.')\n print('\\to2\\t\\tSend a DirectOperate analog value.')\n print('\\to3\\t\\tSend a DirectOperate CommandSet.')\n print('\\tquit')\n print('\\trestart\\t\\tRequest an outstation cold restart.')\n print('\\ts1\\t\\tSend a SelectAndOperate LATCH_ON command.')\n print('\\ts2\\t\\tSend a SelectAndOperate CommandSet.')\n print('\\tscan_all\\tRead data from the outstation (ScanAllObjects).')\n print('\\tscan_fast\\tDemand immediate execution of the fast (every 1 mins) Class 1 scan.')\n print('\\tscan_range\\tPerform an ad-hoc scan (ScanRange) of GroupVariation 1.2, range 0..3.')\n print('\\tscan_slow\\tDemand immediate execution of the slow (every 30 mins) All-Classes scan.')\n print('\\twrite_time\\tWrite a TimeAndInterval to the outstation.')", "response": "Display a menu of command - line options."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nsetting the channel log level to ALL_COMMS. Command syntax is chan_log_all", "response": "def do_chan_log_all(self, line):\n \"\"\"Set the channel log level to ALL_COMMS. Command syntax is: chan_log_all\"\"\"\n self.application.channel.SetLogFilters(openpal.LogFilters(opendnp3.levels.ALL_COMMS))\n print('Channel log filtering level is now: {0}'.format(opendnp3.levels.ALL_COMMS))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef do_chan_log_normal(self, line):\n self.application.channel.SetLogFilters(openpal.LogFilters(opendnp3.levels.NORMAL))\n print('Channel log filtering level is now: {0}'.format(opendnp3.levels.NORMAL))", "response": "Set the channel log level to NORMAL. Command syntax is chan_log_normal"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef do_disable_unsol(self, line):\n headers = [opendnp3.Header().AllObjects(60, 2),\n opendnp3.Header().AllObjects(60, 3),\n opendnp3.Header().AllObjects(60, 4)]\n self.application.master.PerformFunction(\"disable unsolicited\",\n opendnp3.FunctionCode.DISABLE_UNSOLICITED,\n headers,\n opendnp3.TaskConfig().Default())", "response": "Perform the function DISABLE_UNSOLICITED. Command syntax is : disable unsolicited"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the master log level to ALL_COMMS. Command syntax is mast_log_all", "response": "def do_mast_log_all(self, line):\n \"\"\"Set the master log level to ALL_COMMS. Command syntax is: mast_log_all\"\"\"\n self.application.master.SetLogFilters(openpal.LogFilters(opendnp3.levels.ALL_COMMS))\n _log.debug('Master log filtering level is now: {0}'.format(opendnp3.levels.ALL_COMMS))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the master log level to NORMAL. Command syntax is mast_log_normal", "response": "def do_mast_log_normal(self, line):\n \"\"\"Set the master log level to NORMAL. Command syntax is: mast_log_normal\"\"\"\n self.application.master.SetLogFilters(openpal.LogFilters(opendnp3.levels.NORMAL))\n _log.debug('Master log filtering level is now: {0}'.format(opendnp3.levels.NORMAL))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef do_o1(self, line):\n self.application.send_direct_operate_command(opendnp3.ControlRelayOutputBlock(opendnp3.ControlCode.LATCH_ON),\n 5,\n command_callback)", "response": "Send a DirectOperate Binary Output command to the Outstation. Command syntax is o1"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef do_o3(self, line):\n self.application.send_direct_operate_command_set(opendnp3.CommandSet(\n [\n opendnp3.WithIndex(opendnp3.ControlRelayOutputBlock(opendnp3.ControlCode.LATCH_ON), 0),\n opendnp3.WithIndex(opendnp3.ControlRelayOutputBlock(opendnp3.ControlCode.LATCH_OFF), 1)\n ]),\n command_callback\n )", "response": "Send a DirectOperate BinaryOutput command set to the Outstation."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef do_restart(self, line):\n self.application.master.Restart(opendnp3.RestartType.COLD, restart_callback)", "response": "Request that the Outstation perform a cold restart."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef do_s1(self, line):\n self.application.send_select_and_operate_command(opendnp3.ControlRelayOutputBlock(opendnp3.ControlCode.LATCH_ON),\n 8,\n command_callback)", "response": "Send a SelectAndOperate BinaryOutput command to the Outstation. Command syntax is s1"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsend a SelectAndOperate BinaryOutput command set to the Outstation.", "response": "def do_s2(self, line):\n \"\"\"Send a SelectAndOperate BinaryOutput (group 12) CommandSet to the Outstation. Command syntax is: s2\"\"\"\n self.application.send_select_and_operate_command_set(opendnp3.CommandSet(\n [\n opendnp3.WithIndex(opendnp3.ControlRelayOutputBlock(opendnp3.ControlCode.LATCH_ON), 0)\n ]),\n command_callback\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncall ScanAllObjects. Command syntax is: scan_all", "response": "def do_scan_all(self, line):\n \"\"\"Call ScanAllObjects. Command syntax is: scan_all\"\"\"\n self.application.master.ScanAllObjects(opendnp3.GroupVariationID(2, 1), opendnp3.TaskConfig().Default())"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef do_scan_range(self, line):\n self.application.master.ScanRange(opendnp3.GroupVariationID(1, 2), 0, 3, opendnp3.TaskConfig().Default())", "response": "Do an ad - hoc scan of a range of points."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef do_write_time(self, line):\n millis_since_epoch = int((datetime.now() - datetime.utcfromtimestamp(0)).total_seconds() * 1000.0)\n self.application.master.Write(opendnp3.TimeAndInterval(opendnp3.DNPTime(millis_since_epoch),\n 100,\n opendnp3.IntervalUnits.Seconds),\n 0, # index\n opendnp3.TaskConfig().Default())", "response": "Write a TimeAndInterval to the Outstation. Command syntax is write_time"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwraps function for Bloomberg connection Args: func: function to wrap", "response": "def with_bloomberg(func):\n \"\"\"\n Wrapper function for Bloomberg connection\n\n Args:\n func: function to wrap\n \"\"\"\n @wraps(func)\n def wrapper(*args, **kwargs):\n\n scope = utils.func_scope(func=func)\n param = inspect.signature(func).parameters\n port = kwargs.pop('port', _PORT_)\n timeout = kwargs.pop('timeout', _TIMEOUT_)\n restart = kwargs.pop('restart', False)\n all_kw = {\n k: args[n] if n < len(args) else v.default\n for n, (k, v) in enumerate(param.items()) if k != 'kwargs'\n }\n all_kw.update(kwargs)\n log_level = kwargs.get('log', logs.LOG_LEVEL)\n\n for to_list in ['tickers', 'flds']:\n conv = all_kw.get(to_list, None)\n if hasattr(conv, 'tolist'):\n all_kw[to_list] = getattr(conv, 'tolist')()\n if isinstance(conv, str):\n all_kw[to_list] = [conv]\n\n cached_data = []\n if scope in ['xbbg.blp.bdp', 'xbbg.blp.bds']:\n to_qry = cached.bdp_bds_cache(func=func.__name__, **all_kw)\n cached_data += to_qry.cached_data\n\n if not (to_qry.tickers and to_qry.flds):\n if not cached_data: return pd.DataFrame()\n res = pd.concat(cached_data, sort=False).reset_index(drop=True)\n if not all_kw.get('raw', False):\n res = assist.format_output(\n data=res, source=func.__name__,\n col_maps=all_kw.get('col_maps', dict())\n )\n return res\n\n all_kw['tickers'] = to_qry.tickers\n all_kw['flds'] = to_qry.flds\n\n if scope in ['xbbg.blp.bdib']:\n data_file = storage.hist_file(\n ticker=all_kw['ticker'], dt=all_kw['dt'], typ=all_kw['typ'],\n )\n if files.exists(data_file):\n logger = logs.get_logger(func, level=log_level)\n if all_kw.get('batch', False): return\n logger.debug(f'reading from {data_file} ...')\n return assist.format_intraday(data=pd.read_parquet(data_file), **all_kw)\n\n _, new = create_connection(port=port, timeout=timeout, restart=restart)\n res = func(**{\n k: v for k, v in all_kw.items() if k not in ['raw', 'col_maps']\n })\n if new: delete_connection()\n\n if scope.startswith('xbbg.blp.') and isinstance(res, list):\n final = cached_data + res\n if not final: return pd.DataFrame()\n res = pd.DataFrame(pd.concat(final, sort=False))\n\n if (scope in ['xbbg.blp.bdp', 'xbbg.blp.bds']) \\\n and (not all_kw.get('raw', False)):\n res = assist.format_output(\n data=res.reset_index(drop=True), source=func.__name__,\n col_maps=all_kw.get('col_maps', dict()),\n )\n\n return res\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_connection(port=_PORT_, timeout=_TIMEOUT_, restart=False):\n if _CON_SYM_ in globals():\n if not isinstance(globals()[_CON_SYM_], pdblp.BCon):\n del globals()[_CON_SYM_]\n\n if (_CON_SYM_ in globals()) and (not restart):\n con = globals()[_CON_SYM_]\n if getattr(con, '_session').start(): con.start()\n return con, False\n\n else:\n con = pdblp.BCon(port=port, timeout=timeout)\n globals()[_CON_SYM_] = con\n con.start()\n return con, True", "response": "Create a new Bloomberg connection"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef delete_connection():\n if _CON_SYM_ in globals():\n con = globals().pop(_CON_SYM_)\n if not getattr(con, '_session').start(): con.stop()", "response": "Stop and destroy Bloomberg connection"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef bdp_bds_cache(func, tickers, flds, **kwargs) -> ToQuery:\n cache_data = []\n log_level = kwargs.get('log', logs.LOG_LEVEL)\n logger = logs.get_logger(bdp_bds_cache, level=log_level)\n kwargs['has_date'] = kwargs.pop('has_date', func == 'bds')\n kwargs['cache'] = kwargs.get('cache', True)\n\n tickers = utils.flatten(tickers)\n flds = utils.flatten(flds)\n loaded = pd.DataFrame(data=0, index=tickers, columns=flds)\n\n for ticker, fld in product(tickers, flds):\n data_file = storage.ref_file(\n ticker=ticker, fld=fld, ext='pkl', **{\n k: v for k, v in kwargs.items() if k not in EXC_COLS\n }\n )\n if not files.exists(data_file): continue\n logger.debug(f'reading from {data_file} ...')\n cache_data.append(pd.read_pickle(data_file))\n loaded.loc[ticker, fld] = 1\n\n to_qry = loaded.where(loaded == 0)\\\n .dropna(how='all', axis=1).dropna(how='all', axis=0)\n\n return ToQuery(\n tickers=to_qry.index.tolist(), flds=to_qry.columns.tolist(),\n cached_data=cache_data\n )", "response": "Find cached BDP or BDS queries\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing version of the package", "response": "def parse_version(package):\n \"\"\"\n Parse versions\n \"\"\"\n init_file = f'{PACKAGE_ROOT}/{package}/__init__.py'\n with open(init_file, 'r', encoding='utf-8') as f:\n for line in f.readlines():\n if '__version__' in line:\n return line.split('=')[1].strip()[1:-1]\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse markdown as long description", "response": "def parse_markdown():\n \"\"\"\n Parse markdown as description\n \"\"\"\n readme_file = f'{PACKAGE_ROOT}/README.md'\n if path.exists(readme_file):\n with open(readme_file, 'r', encoding='utf-8') as f:\n long_description = f.read()\n return long_description"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the description in the README file.", "response": "def parse_description(markdown=True):\n \"\"\"\n Parse the description in the README file\n \"\"\"\n if markdown: return parse_markdown()\n\n try:\n from pypandoc import convert\n\n readme_file = f'{PACKAGE_ROOT}/docs/index.rst'\n if not path.exists(readme_file):\n raise ImportError\n return convert(readme_file, 'rst')\n\n except ImportError:\n return parse_markdown()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of tuples containing the base N terms of the base N terms of the base N terms.", "response": "def proc_ovrds(**kwargs):\n \"\"\"\n Bloomberg overrides\n\n Args:\n **kwargs: overrides\n\n Returns:\n list of tuples\n\n Examples:\n >>> proc_ovrds(DVD_Start_Dt='20180101')\n [('DVD_Start_Dt', '20180101')]\n >>> proc_ovrds(DVD_Start_Dt='20180101', cache=True, has_date=True)\n [('DVD_Start_Dt', '20180101')]\n \"\"\"\n return [\n (k, v) for k, v in kwargs.items()\n if k not in list(ELEM_KEYS.keys()) + list(ELEM_KEYS.values()) + PRSV_COLS\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef proc_elms(**kwargs) -> list:\n return [\n (ELEM_KEYS.get(k, k), ELEM_VALS.get(ELEM_KEYS.get(k, k), dict()).get(v, v))\n for k, v in kwargs.items()\n if (k in list(ELEM_KEYS.keys()) + list(ELEM_KEYS.values()))\n and (k not in PRSV_COLS)\n ]", "response": "Return a list of tuples containing all the elements in the current language."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef format_earning(data: pd.DataFrame, header: pd.DataFrame) -> pd.DataFrame:\n if data.dropna(subset=['value']).empty: return pd.DataFrame()\n\n res = pd.concat([\n grp.loc[:, ['value']].set_index(header.value)\n for _, grp in data.groupby(data.position)\n ], axis=1)\n res.index.name = None\n res.columns = res.iloc[0]\n res = res.iloc[1:].transpose().reset_index().apply(\n pd.to_numeric, downcast='float', errors='ignore'\n )\n res.rename(\n columns=lambda vv: '_'.join(vv.lower().split()).replace('fy_', 'fy'),\n inplace=True,\n )\n\n years = res.columns[res.columns.str.startswith('fy')]\n lvl_1 = res.level == 1\n for yr in years:\n res.loc[:, yr] = res.loc[:, yr].round(1)\n pct = f'{yr}_pct'\n res.loc[:, pct] = 0.\n res.loc[lvl_1, pct] = res.loc[lvl_1, pct].astype(float).round(1)\n res.loc[lvl_1, pct] = res.loc[lvl_1, yr] / res.loc[lvl_1, yr].sum() * 100\n sub_pct = []\n for _, snap in res[::-1].iterrows():\n if snap.level > 2: continue\n if snap.level == 1:\n if len(sub_pct) == 0: continue\n sub = pd.concat(sub_pct, axis=1).transpose()\n res.loc[sub.index, pct] = \\\n res.loc[sub.index, yr] / res.loc[sub.index, yr].sum() * 100\n sub_pct = []\n if snap.level == 2: sub_pct.append(snap)\n\n res.set_index('segment_name', inplace=True)\n res.index.name = None\n return res", "response": "Standardized earning outputs and add percentage by each block"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef format_output(data: pd.DataFrame, source, col_maps=None) -> pd.DataFrame:\n if data.empty: return pd.DataFrame()\n if source == 'bdp': req_cols = ['ticker', 'field', 'value']\n else: req_cols = ['ticker', 'field', 'name', 'value', 'position']\n if any(col not in data for col in req_cols): return pd.DataFrame()\n if data.dropna(subset=['value']).empty: return pd.DataFrame()\n\n if source == 'bdp':\n res = pd.DataFrame(pd.concat([\n pd.Series({**{'ticker': t}, **grp.set_index('field').value.to_dict()})\n for t, grp in data.groupby('ticker')\n ], axis=1, sort=False)).transpose().set_index('ticker')\n else:\n res = pd.DataFrame(pd.concat([\n grp.loc[:, ['name', 'value']].set_index('name')\n .transpose().reset_index(drop=True).assign(ticker=t)\n for (t, _), grp in data.groupby(['ticker', 'position'])\n ], sort=False)).reset_index(drop=True).set_index('ticker')\n res.columns.name = None\n\n if col_maps is None: col_maps = dict()\n return res.rename(\n columns=lambda vv: col_maps.get(\n vv, vv.lower().replace(' ', '_').replace('-', '_')\n )\n ).apply(pd.to_numeric, errors='ignore', downcast='float')", "response": "Format pdblp outputs to column - based results\n "} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef info_qry(tickers, flds) -> str:\n full_list = '\\n'.join([f'tickers: {tickers[:8]}'] + [\n f' {tickers[n:(n + 8)]}' for n in range(8, len(tickers), 8)\n ])\n return f'{full_list}\\nfields: {flds}'", "response": "Returns a string with the log info for given tickers and fields."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef bdp(tickers, flds, **kwargs):\n logger = logs.get_logger(bdp, level=kwargs.pop('log', logs.LOG_LEVEL))\n con, _ = create_connection()\n ovrds = assist.proc_ovrds(**kwargs)\n\n logger.info(\n f'loading reference data from Bloomberg:\\n'\n f'{assist.info_qry(tickers=tickers, flds=flds)}'\n )\n data = con.ref(tickers=tickers, flds=flds, ovrds=ovrds)\n if not kwargs.get('cache', False): return [data]\n\n qry_data = []\n for r, snap in data.iterrows():\n subset = [r]\n data_file = storage.ref_file(\n ticker=snap.ticker, fld=snap.field, ext='pkl', **kwargs\n )\n if data_file:\n if not files.exists(data_file): qry_data.append(data.iloc[subset])\n files.create_folder(data_file, is_file=True)\n data.iloc[subset].to_pickle(data_file)\n\n return qry_data", "response": "Bloomberg reference data generator."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef bds(tickers, flds, **kwargs):\n logger = logs.get_logger(bds, level=kwargs.pop('log', logs.LOG_LEVEL))\n con, _ = create_connection()\n ovrds = assist.proc_ovrds(**kwargs)\n\n logger.info(\n f'loading block data from Bloomberg:\\n'\n f'{assist.info_qry(tickers=tickers, flds=flds)}'\n )\n data = con.bulkref(tickers=tickers, flds=flds, ovrds=ovrds)\n if not kwargs.get('cache', False): return [data]\n\n qry_data = []\n for (ticker, fld), grp in data.groupby(['ticker', 'field']):\n data_file = storage.ref_file(\n ticker=ticker, fld=fld, ext='pkl',\n has_date=kwargs.get('has_date', True), **kwargs\n )\n if data_file:\n if not files.exists(data_file): qry_data.append(grp)\n files.create_folder(data_file, is_file=True)\n grp.reset_index(drop=True).to_pickle(data_file)\n\n return qry_data", "response": "Bloomberg data set for a given set of tickers and fields."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a DataFrame containing the earning exposures for a given set of names.", "response": "def earning(\n ticker, by='Geo', typ='Revenue', ccy=None, level=None, **kwargs\n) -> pd.DataFrame:\n \"\"\"\n Earning exposures by Geo or Products\n\n Args:\n ticker: ticker name\n by: [G(eo), P(roduct)]\n typ: type of earning, start with `PG_` in Bloomberg FLDS - default `Revenue`\n ccy: currency of earnings\n level: hierarchy level of earnings\n\n Returns:\n pd.DataFrame\n\n Examples:\n >>> data = earning('AMD US Equity', Eqy_Fund_Year=2017, Number_Of_Periods=1)\n >>> data.round(2)\n level fy2017 fy2017_pct\n Asia-Pacific 1.0 3540.0 66.43\n \u00a0\u00a0\u00a0China 2.0 1747.0 49.35\n \u00a0\u00a0\u00a0Japan 2.0 1242.0 35.08\n \u00a0\u00a0\u00a0Singapore 2.0 551.0 15.56\n United States 1.0 1364.0 25.60\n Europe 1.0 263.0 4.94\n Other Countries 1.0 162.0 3.04\n \"\"\"\n ovrd = 'G' if by[0].upper() == 'G' else 'P'\n new_kw = dict(raw=True, Product_Geo_Override=ovrd)\n header = bds(tickers=ticker, flds='PG_Bulk_Header', **new_kw, **kwargs)\n if ccy: kwargs['Eqy_Fund_Crncy'] = ccy\n if level: kwargs['PG_Hierarchy_Level'] = level\n data = bds(tickers=ticker, flds=f'PG_{typ}', **new_kw, **kwargs)\n return assist.format_earning(data=data, header=header)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dividend(\n tickers, typ='all', start_date=None, end_date=None, **kwargs\n) -> pd.DataFrame:\n \"\"\"\n Bloomberg dividend / split history\n\n Args:\n tickers: list of tickers\n typ: dividend adjustment type\n `all`: `DVD_Hist_All`\n `dvd`: `DVD_Hist`\n `split`: `Eqy_DVD_Hist_Splits`\n `gross`: `Eqy_DVD_Hist_Gross`\n `adjust`: `Eqy_DVD_Adjust_Fact`\n `adj_fund`: `Eqy_DVD_Adj_Fund`\n `with_amt`: `DVD_Hist_All_with_Amt_Status`\n `dvd_amt`: `DVD_Hist_with_Amt_Status`\n `gross_amt`: `DVD_Hist_Gross_with_Amt_Stat`\n `projected`: `BDVD_Pr_Ex_Dts_DVD_Amts_w_Ann`\n start_date: start date\n end_date: end date\n **kwargs: overrides\n\n Returns:\n pd.DataFrame\n\n Examples:\n >>> res = dividend(\n ... tickers=['C US Equity', 'NVDA US Equity', 'MS US Equity'],\n ... start_date='2018-01-01', end_date='2018-05-01'\n ... )\n >>> res.index.name = None\n >>> res.loc[:, ['ex_date', 'rec_date', 'dvd_amt']].round(2)\n ex_date rec_date dvd_amt\n C US Equity 2018-02-02 2018-02-05 0.32\n MS US Equity 2018-04-27 2018-04-30 0.25\n MS US Equity 2018-01-30 2018-01-31 0.25\n NVDA US Equity 2018-02-22 2018-02-23 0.15\n \"\"\"\n if isinstance(tickers, str): tickers = [tickers]\n tickers = [t for t in tickers if ('Equity' in t) and ('=' not in t)]\n\n fld = {\n 'all': 'DVD_Hist_All', 'dvd': 'DVD_Hist',\n 'split': 'Eqy_DVD_Hist_Splits', 'gross': 'Eqy_DVD_Hist_Gross',\n 'adjust': 'Eqy_DVD_Adjust_Fact', 'adj_fund': 'Eqy_DVD_Adj_Fund',\n 'with_amt': 'DVD_Hist_All_with_Amt_Status',\n 'dvd_amt': 'DVD_Hist_with_Amt_Status',\n 'gross_amt': 'DVD_Hist_Gross_with_Amt_Stat',\n 'projected': 'BDVD_Pr_Ex_Dts_DVD_Amts_w_Ann',\n }.get(typ, typ)\n\n if (fld == 'Eqy_DVD_Adjust_Fact') and ('Corporate_Actions_Filter' not in kwargs):\n kwargs['Corporate_Actions_Filter'] = 'NORMAL_CASH|ABNORMAL_CASH|CAPITAL_CHANGE'\n\n if fld in [\n 'DVD_Hist_All', 'DVD_Hist', 'Eqy_DVD_Hist_Gross',\n 'DVD_Hist_All_with_Amt_Status', 'DVD_Hist_with_Amt_Status',\n ]:\n if start_date: kwargs['DVD_Start_Dt'] = utils.fmt_dt(start_date, fmt='%Y%m%d')\n if end_date: kwargs['DVD_End_Dt'] = utils.fmt_dt(end_date, fmt='%Y%m%d')\n\n kwargs['col_maps'] = {\n 'Declared Date': 'dec_date', 'Ex-Date': 'ex_date',\n 'Record Date': 'rec_date', 'Payable Date': 'pay_date',\n 'Dividend Amount': 'dvd_amt', 'Dividend Frequency': 'dvd_freq',\n 'Dividend Type': 'dvd_type', 'Amount Status': 'amt_status',\n 'Adjustment Date': 'adj_date', 'Adjustment Factor': 'adj_factor',\n 'Adjustment Factor Operator Type': 'adj_op',\n 'Adjustment Factor Flag': 'adj_flag',\n 'Amount Per Share': 'amt_ps', 'Projected/Confirmed': 'category',\n }\n\n return bds(tickers=tickers, flds=fld, raw=False, **kwargs)", "response": "A base implementation of the dividend function."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef active_futures(ticker: str, dt) -> str:\n t_info = ticker.split()\n prefix, asset = ' '.join(t_info[:-1]), t_info[-1]\n info = const.market_info(f'{prefix[:-1]}1 {asset}')\n\n f1, f2 = f'{prefix[:-1]}1 {asset}', f'{prefix[:-1]}2 {asset}'\n fut_2 = fut_ticker(gen_ticker=f2, dt=dt, freq=info['freq'])\n fut_1 = fut_ticker(gen_ticker=f1, dt=dt, freq=info['freq'])\n\n fut_tk = bdp(tickers=[fut_1, fut_2], flds='Last_Tradeable_Dt', cache=True)\n\n if pd.Timestamp(dt).month < pd.Timestamp(fut_tk.last_tradeable_dt[0]).month: return fut_1\n\n d1 = bdib(ticker=f1, dt=dt)\n d2 = bdib(ticker=f2, dt=dt)\n\n return fut_1 if d1[f1].volume.sum() > d2[f2].volume.sum() else fut_2", "response": "Returns the name of the active futures contract for the given ticker."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets proper ticker from generic ticker", "response": "def fut_ticker(gen_ticker: str, dt, freq: str, log=logs.LOG_LEVEL) -> str:\n \"\"\"\n Get proper ticker from generic ticker\n\n Args:\n gen_ticker: generic ticker\n dt: date\n freq: futures contract frequency\n log: level of logs\n\n Returns:\n str: exact futures ticker\n \"\"\"\n logger = logs.get_logger(fut_ticker, level=log)\n dt = pd.Timestamp(dt)\n t_info = gen_ticker.split()\n\n asset = t_info[-1]\n if asset in ['Index', 'Curncy', 'Comdty']:\n ticker = ' '.join(t_info[:-1])\n prefix, idx, postfix = ticker[:-1], int(ticker[-1]) - 1, asset\n\n elif asset == 'Equity':\n ticker = t_info[0]\n prefix, idx, postfix = ticker[:-1], int(ticker[-1]) - 1, ' '.join(t_info[1:])\n\n else:\n logger.error(f'unkonwn asset type for ticker: {gen_ticker}')\n return ''\n\n month_ext = 4 if asset == 'Comdty' else 2\n months = pd.date_range(start=dt, periods=max(idx + month_ext, 3), freq=freq)\n logger.debug(f'pulling expiry dates for months: {months}')\n\n def to_fut(month):\n return prefix + const.Futures[month.strftime('%b')] + \\\n month.strftime('%y')[-1] + ' ' + postfix\n\n fut = [to_fut(m) for m in months]\n logger.debug(f'trying futures: {fut}')\n # noinspection PyBroadException\n try:\n fut_matu = bdp(tickers=fut, flds='last_tradeable_dt', cache=True)\n except Exception as e1:\n logger.error(f'error downloading futures contracts (1st trial) {e1}:\\n{fut}')\n # noinspection PyBroadException\n try:\n fut = fut[:-1]\n logger.debug(f'trying futures (2nd trial): {fut}')\n fut_matu = bdp(tickers=fut, flds='last_tradeable_dt', cache=True)\n except Exception as e2:\n logger.error(f'error downloading futures contracts (2nd trial) {e2}:\\n{fut}')\n return ''\n\n sub_fut = fut_matu[pd.DatetimeIndex(fut_matu.last_tradeable_dt) > dt]\n logger.debug(f'futures full chain:\\n{fut_matu.to_string()}')\n logger.debug(f'getting index {idx} from:\\n{sub_fut.to_string()}')\n return sub_fut.index.values[idx]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks exchange hours vs local hours", "response": "def check_hours(tickers, tz_exch, tz_loc=DEFAULT_TZ) -> pd.DataFrame:\n \"\"\"\n Check exchange hours vs local hours\n\n Args:\n tickers: list of tickers\n tz_exch: exchange timezone\n tz_loc: local timezone\n\n Returns:\n Local and exchange hours\n \"\"\"\n cols = ['Trading_Day_Start_Time_EOD', 'Trading_Day_End_Time_EOD']\n con, _ = create_connection()\n hours = con.ref(tickers=tickers, flds=cols)\n cur_dt = pd.Timestamp('today').strftime('%Y-%m-%d ')\n hours.loc[:, 'local'] = hours.value.astype(str).str[:-3]\n hours.loc[:, 'exch'] = pd.DatetimeIndex(\n cur_dt + hours.value.astype(str)\n ).tz_localize(tz_loc).tz_convert(tz_exch).strftime('%H:%M')\n\n hours = pd.concat([\n hours.set_index(['ticker', 'field']).exch.unstack().loc[:, cols],\n hours.set_index(['ticker', 'field']).local.unstack().loc[:, cols],\n ], axis=1)\n hours.columns = ['Exch_Start', 'Exch_End', 'Local_Start', 'Local_End']\n\n return hours"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the full path to the Bloomberg historical data file for the given ticker and date.", "response": "def hist_file(ticker: str, dt, typ='TRADE') -> str:\n \"\"\"\n Data file location for Bloomberg historical data\n\n Args:\n ticker: ticker name\n dt: date\n typ: [TRADE, BID, ASK, BID_BEST, ASK_BEST, BEST_BID, BEST_ASK]\n\n Returns:\n file location\n\n Examples:\n >>> os.environ['BBG_ROOT'] = ''\n >>> hist_file(ticker='ES1 Index', dt='2018-08-01') == ''\n True\n >>> os.environ['BBG_ROOT'] = '/data/bbg'\n >>> hist_file(ticker='ES1 Index', dt='2018-08-01')\n '/data/bbg/Index/ES1 Index/TRADE/2018-08-01.parq'\n \"\"\"\n data_path = os.environ.get(assist.BBG_ROOT, '').replace('\\\\', '/')\n if not data_path: return ''\n asset = ticker.split()[-1]\n proper_ticker = ticker.replace('/', '_')\n cur_dt = pd.Timestamp(dt).strftime('%Y-%m-%d')\n return f'{data_path}/{asset}/{proper_ticker}/{typ}/{cur_dt}.parq'"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfunction to reference the data file for a given field in the current Bloomberg dataset", "response": "def ref_file(\n ticker: str, fld: str, has_date=False, cache=False, ext='parq', **kwargs\n) -> str:\n \"\"\"\n Data file location for Bloomberg reference data\n\n Args:\n ticker: ticker name\n fld: field\n has_date: whether add current date to data file\n cache: if has_date is True, whether to load file from latest cached\n ext: file extension\n **kwargs: other overrides passed to ref function\n\n Returns:\n file location\n\n Examples:\n >>> import shutil\n >>>\n >>> os.environ['BBG_ROOT'] = ''\n >>> ref_file('BLT LN Equity', fld='Crncy') == ''\n True\n >>> os.environ['BBG_ROOT'] = '/data/bbg'\n >>> ref_file('BLT LN Equity', fld='Crncy', cache=True)\n '/data/bbg/Equity/BLT LN Equity/Crncy/ovrd=None.parq'\n >>> ref_file('BLT LN Equity', fld='Crncy')\n ''\n >>> cur_dt = utils.cur_time(tz=utils.DEFAULT_TZ)\n >>> ref_file(\n ... 'BLT LN Equity', fld='DVD_Hist_All', has_date=True, cache=True,\n ... ).replace(cur_dt, '[cur_date]')\n '/data/bbg/Equity/BLT LN Equity/DVD_Hist_All/asof=[cur_date], ovrd=None.parq'\n >>> ref_file(\n ... 'BLT LN Equity', fld='DVD_Hist_All', has_date=True,\n ... cache=True, DVD_Start_Dt='20180101',\n ... ).replace(cur_dt, '[cur_date]')[:-5]\n '/data/bbg/Equity/BLT LN Equity/DVD_Hist_All/asof=[cur_date], DVD_Start_Dt=20180101'\n >>> sample = 'asof=2018-11-02, DVD_Start_Dt=20180101, DVD_End_Dt=20180501.pkl'\n >>> root_path = 'xbbg/tests/data'\n >>> sub_path = f'{root_path}/Equity/AAPL US Equity/DVD_Hist_All'\n >>> os.environ['BBG_ROOT'] = root_path\n >>> for tmp_file in files.all_files(sub_path): os.remove(tmp_file)\n >>> files.create_folder(sub_path)\n >>> sample in shutil.copy(f'{root_path}/{sample}', sub_path)\n True\n >>> new_file = ref_file(\n ... 'AAPL US Equity', 'DVD_Hist_All', DVD_Start_Dt='20180101',\n ... has_date=True, cache=True, ext='pkl'\n ... )\n >>> new_file.split('/')[-1] == f'asof={cur_dt}, DVD_Start_Dt=20180101.pkl'\n True\n >>> old_file = 'asof=2018-11-02, DVD_Start_Dt=20180101, DVD_End_Dt=20180501.pkl'\n >>> old_full = '/'.join(new_file.split('/')[:-1] + [old_file])\n >>> updated_file = old_full.replace('2018-11-02', cur_dt)\n >>> updated_file in shutil.copy(old_full, updated_file)\n True\n >>> exist_file = ref_file(\n ... 'AAPL US Equity', 'DVD_Hist_All', DVD_Start_Dt='20180101',\n ... has_date=True, cache=True, ext='pkl'\n ... )\n >>> exist_file == updated_file\n False\n >>> exist_file = ref_file(\n ... 'AAPL US Equity', 'DVD_Hist_All', DVD_Start_Dt='20180101',\n ... DVD_End_Dt='20180501', has_date=True, cache=True, ext='pkl'\n ... )\n >>> exist_file == updated_file\n True\n \"\"\"\n data_path = os.environ.get(assist.BBG_ROOT, '').replace('\\\\', '/')\n if (not data_path) or (not cache): return ''\n\n proper_ticker = ticker.replace('/', '_')\n cache_days = kwargs.pop('cache_days', 10)\n root = f'{data_path}/{ticker.split()[-1]}/{proper_ticker}/{fld}'\n\n if len(kwargs) > 0: info = utils.to_str(kwargs)[1:-1].replace('|', '_')\n else: info = 'ovrd=None'\n\n # Check date info\n if has_date:\n cur_dt = utils.cur_time()\n missing = f'{root}/asof={cur_dt}, {info}.{ext}'\n to_find = re.compile(rf'{root}/asof=(.*), {info}\\.pkl')\n cur_files = list(filter(to_find.match, sorted(\n files.all_files(path_name=root, keyword=info, ext=ext)\n )))\n if len(cur_files) > 0:\n upd_dt = to_find.match(cur_files[-1]).group(1)\n diff = pd.Timestamp('today') - pd.Timestamp(upd_dt)\n if diff >= pd.Timedelta(days=cache_days): return missing\n return sorted(cur_files)[-1]\n else: return missing\n\n else: return f'{root}/{info}.{ext}'"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef save_intraday(data: pd.DataFrame, ticker: str, dt, typ='TRADE'):\n cur_dt = pd.Timestamp(dt).strftime('%Y-%m-%d')\n logger = logs.get_logger(save_intraday, level='debug')\n info = f'{ticker} / {cur_dt} / {typ}'\n data_file = hist_file(ticker=ticker, dt=dt, typ=typ)\n if not data_file: return\n\n if data.empty:\n logger.warning(f'data is empty for {info} ...')\n return\n\n exch = const.exch_info(ticker=ticker)\n if exch.empty: return\n\n end_time = pd.Timestamp(\n const.market_timing(ticker=ticker, dt=dt, timing='FINISHED')\n ).tz_localize(exch.tz)\n now = pd.Timestamp('now', tz=exch.tz) - pd.Timedelta('1H')\n\n if end_time > now:\n logger.debug(f'skip saving cause market close ({end_time}) < now - 1H ({now}) ...')\n return\n\n logger.info(f'saving data to {data_file} ...')\n files.create_folder(data_file, is_file=True)\n data.to_parquet(data_file)", "response": "Check whether data is done for the day and save it to file"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef exch_info(ticker: str) -> pd.Series:\n logger = logs.get_logger(exch_info, level='debug')\n if ' ' not in ticker.strip():\n ticker = f'XYZ {ticker.strip()} Equity'\n info = param.load_info(cat='exch').get(\n market_info(ticker=ticker).get('exch', ''), dict()\n )\n if ('allday' in info) and ('day' not in info):\n info['day'] = info['allday']\n\n if any(req not in info for req in ['tz', 'allday', 'day']):\n logger.error(f'required exchange info cannot be found in {ticker} ...')\n return pd.Series()\n\n for ss in ValidSessions:\n if ss not in info: continue\n info[ss] = [param.to_hour(num=s) for s in info[ss]]\n\n return pd.Series(info)", "response": "Get exchange info for given ticker"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef market_info(ticker: str) -> dict:\n t_info = ticker.split()\n assets = param.load_info('assets')\n\n # ========================== #\n # Equity #\n # ========================== #\n\n if (t_info[-1] == 'Equity') and ('=' not in t_info[0]):\n exch = t_info[-2]\n for info in assets.get('Equity', [dict()]):\n if 'exch_codes' not in info: continue\n if exch in info['exch_codes']: return info\n return dict()\n\n # ============================ #\n # Currency #\n # ============================ #\n\n if t_info[-1] == 'Curncy':\n for info in assets.get('Curncy', [dict()]):\n if 'tickers' not in info: continue\n if (t_info[0].split('+')[0] in info['tickers']) or \\\n (t_info[0][-1].isdigit() and (t_info[0][:-1] in info['tickers'])):\n return info\n return dict()\n\n if t_info[-1] == 'Comdty':\n for info in assets.get('Comdty', [dict()]):\n if 'tickers' not in info: continue\n if t_info[0][:-1] in info['tickers']: return info\n return dict()\n\n # =================================== #\n # Index / Futures #\n # =================================== #\n\n if (t_info[-1] == 'Index') or (\n (t_info[-1] == 'Equity') and ('=' in t_info[0])\n ):\n if t_info[-1] == 'Equity':\n tck = t_info[0].split('=')[0]\n else:\n tck = ' '.join(t_info[:-1])\n for info in assets.get('Index', [dict()]):\n if 'tickers' not in info: continue\n if (tck[:2] == 'UX') and ('UX' in info['tickers']): return info\n if tck in info['tickers']:\n if t_info[-1] == 'Equity': return info\n if not info.get('is_fut', False): return info\n if tck[:-1].rstrip() in info['tickers']:\n if info.get('is_fut', False): return info\n return dict()\n\n if t_info[-1] == 'Corp':\n for info in assets.get('Corp', [dict()]):\n if 'ticker' not in info: continue\n\n return dict()", "response": "Get info for given market\nONE"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ccy_pair(local, base='USD') -> CurrencyPair:\n ccy_param = param.load_info(cat='ccy')\n if f'{local}{base}' in ccy_param:\n info = ccy_param[f'{local}{base}']\n\n elif f'{base}{local}' in ccy_param:\n info = ccy_param[f'{base}{local}']\n info['factor'] = 1. / info.get('factor', 1.)\n info['power'] = -info.get('power', 1)\n\n elif base.lower() == local.lower():\n info = dict(ticker='')\n info['factor'] = 1.\n if base[-1].lower() == base[-1]:\n info['factor'] /= 100.\n if local[-1].lower() == local[-1]:\n info['factor'] *= 100.\n\n else:\n logger = logs.get_logger(ccy_pair)\n logger.error(f'incorrect currency - local {local} / base {base}')\n return CurrencyPair(ticker='', factor=1., power=1)\n\n if 'factor' not in info: info['factor'] = 1.\n if 'power' not in info: info['power'] = 1\n return CurrencyPair(**info)", "response": "Returns a new pair of CurrencyPair objects for the given local and base."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef market_timing(ticker, dt, timing='EOD', tz='local') -> str:\n logger = logs.get_logger(market_timing)\n exch = pd.Series(exch_info(ticker=ticker))\n if any(req not in exch.index for req in ['tz', 'allday', 'day']):\n logger.error(f'required exchange info cannot be found in {ticker} ...')\n return ''\n\n mkt_time = {\n 'BOD': exch.day[0], 'FINISHED': exch.allday[-1]\n }.get(timing, exch.day[-1])\n\n cur_dt = pd.Timestamp(str(dt)).strftime('%Y-%m-%d')\n if tz == 'local':\n return f'{cur_dt} {mkt_time}'\n\n return timezone.tz_convert(f'{cur_dt} {mkt_time}', to_tz=tz, from_tz=exch.tz)", "response": "Market close time for ticker\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nflattening any array of items to list", "response": "def flatten(iterable, maps=None, unique=False) -> list:\n \"\"\"\n Flatten any array of items to list\n\n Args:\n iterable: any array or value\n maps: map items to values\n unique: drop duplicates\n\n Returns:\n list: flattened list\n\n References:\n https://stackoverflow.com/a/40857703/1332656\n\n Examples:\n >>> flatten('abc')\n ['abc']\n >>> flatten(1)\n [1]\n >>> flatten(1.)\n [1.0]\n >>> flatten(['ab', 'cd', ['xy', 'zz']])\n ['ab', 'cd', 'xy', 'zz']\n >>> flatten(['ab', ['xy', 'zz']], maps={'xy': '0x'})\n ['ab', '0x', 'zz']\n \"\"\"\n if iterable is None: return []\n if maps is None: maps = dict()\n\n if isinstance(iterable, (str, int, float)):\n return [maps.get(iterable, iterable)]\n\n else:\n x = [maps.get(item, item) for item in _to_gen_(iterable)]\n return list(set(x)) if unique else x"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _to_gen_(iterable):\n from collections import Iterable\n\n for elm in iterable:\n if isinstance(elm, Iterable) and not isinstance(elm, (str, bytes)):\n yield from flatten(elm)\n else: yield elm", "response": "Recursively iterate lists and tuples of items and yield them"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef cur_time(typ='date', tz=DEFAULT_TZ) -> (datetime.date, str):\n dt = pd.Timestamp('now', tz=tz)\n\n if typ == 'date': return dt.strftime('%Y-%m-%d')\n if typ == 'time': return dt.strftime('%Y-%m-%d %H:%M:%S')\n if typ == 'time_path': return dt.strftime('%Y-%m-%d/%H-%M-%S')\n if typ == 'raw': return dt\n\n return dt.date()", "response": "Current time is a datetime. date object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert dict to string", "response": "def to_str(\n data: dict, fmt='{key}={value}', sep=', ', public_only=True\n) -> str:\n \"\"\"\n Convert dict to string\n\n Args:\n data: dict\n fmt: how key and value being represented\n sep: how pairs of key and value are seperated\n public_only: if display public members only\n\n Returns:\n str: string representation of dict\n\n Examples:\n >>> test_dict = dict(b=1, a=0, c=2, _d=3)\n >>> to_str(test_dict)\n '{b=1, a=0, c=2}'\n >>> to_str(test_dict, sep='|')\n '{b=1|a=0|c=2}'\n >>> to_str(test_dict, public_only=False)\n '{b=1, a=0, c=2, _d=3}'\n \"\"\"\n if public_only: keys = list(filter(lambda vv: vv[0] != '_', data.keys()))\n else: keys = list(data.keys())\n return '{' + sep.join([\n to_str(data=v, fmt=fmt, sep=sep)\n if isinstance(v, dict) else fstr(fmt=fmt, key=k, value=v)\n for k, v in data.items() if k in keys\n ]) + '}'"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_module(full_path):\n from importlib import util\n\n file_name = full_path.replace('\\\\', '/').split('/')[-1]\n if file_name[-3:] != '.py':\n raise ImportError(f'not a python file: {file_name}')\n module_name = file_name[:-3]\n\n spec = util.spec_from_file_location(name=module_name, location=full_path)\n module = util.module_from_spec(spec=spec)\n spec.loader.exec_module(module=module)\n\n return module", "response": "Load module from full path"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads assets infomation from file", "response": "def _load_yaml_(file_name):\n \"\"\"\n Load assets infomation from file\n\n Args:\n file_name: file name\n\n Returns:\n dict\n \"\"\"\n if not os.path.exists(file_name): return dict()\n\n with open(file_name, 'r', encoding='utf-8') as fp:\n return YAML().load(stream=fp)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting YAML input to hours", "response": "def to_hour(num) -> str:\n \"\"\"\n Convert YAML input to hours\n\n Args:\n num: number in YMAL file, e.g., 900, 1700, etc.\n\n Returns:\n str\n\n Examples:\n >>> to_hour(900)\n '09:00'\n >>> to_hour(1700)\n '17:00'\n \"\"\"\n to_str = str(int(num))\n return pd.Timestamp(f'{to_str[:-2]}:{to_str[-2:]}').strftime('%H:%M')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef abspath(cur_file, parent=0) -> str:\n file_path = os.path.abspath(cur_file).replace('\\\\', '/')\n if os.path.isdir(file_path) and parent == 0: return file_path\n adj = 1 - os.path.isdir(file_path)\n return '/'.join(file_path.split('/')[:-(parent + adj)])", "response": "Absolute path of a file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_folder(path_name: str, is_file=False):\n path_sep = path_name.replace('\\\\', '/').split('/')\n for i in range(1, len(path_sep) + (0 if is_file else 1)):\n cur_path = '/'.join(path_sep[:i])\n if not os.path.exists(cur_path): os.mkdir(cur_path)", "response": "Create folder if not exists"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef all_files(\n path_name, keyword='', ext='', full_path=True,\n has_date=False, date_fmt=DATE_FMT\n) -> list:\n \"\"\"\n Search all files with criteria\n Returned list will be sorted by last modified\n\n Args:\n path_name: full path name\n keyword: keyword to search\n ext: file extensions, split by ','\n full_path: whether return full path (default True)\n has_date: whether has date in file name (default False)\n date_fmt: date format to check for has_date parameter\n\n Returns:\n list: all file names with criteria fulfilled\n \"\"\"\n if not os.path.exists(path=path_name): return []\n path_name = path_name.replace('\\\\', '/')\n\n if keyword or ext:\n keyword = f'*{keyword}*' if keyword else '*'\n if not ext: ext = '*'\n files = sort_by_modified([\n f.replace('\\\\', '/') for f in glob.iglob(f'{path_name}/{keyword}.{ext}')\n if os.path.isfile(f) and (f.replace('\\\\', '/').split('/')[-1][0] != '~')\n ])\n\n else:\n files = sort_by_modified([\n f'{path_name}/{f}' for f in os.listdir(path=path_name)\n if os.path.isfile(f'{path_name}/{f}') and (f[0] != '~')\n ])\n\n if has_date:\n files = filter_by_dates(files, date_fmt=date_fmt)\n\n return files if full_path else [f.split('/')[-1] for f in files]", "response": "Search all files with criteria fulfilled by path_name keyword and ext."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsearching all folders with criteria returns list of all folders", "response": "def all_folders(\n path_name, keyword='', has_date=False, date_fmt=DATE_FMT\n) -> list:\n \"\"\"\n Search all folders with criteria\n Returned list will be sorted by last modified\n\n Args:\n path_name: full path name\n keyword: keyword to search\n has_date: whether has date in file name (default False)\n date_fmt: date format to check for has_date parameter\n\n Returns:\n list: all folder names fulfilled criteria\n \"\"\"\n if not os.path.exists(path=path_name): return []\n path_name = path_name.replace('\\\\', '/')\n\n if keyword:\n folders = sort_by_modified([\n f.replace('\\\\', '/') for f in glob.iglob(f'{path_name}/*{keyword}*')\n if os.path.isdir(f) and (f.replace('\\\\', '/').split('/')[-1][0] != '~')\n ])\n\n else:\n folders = sort_by_modified([\n f'{path_name}/{f}' for f in os.listdir(path=path_name)\n if os.path.isdir(f'{path_name}/{f}') and (f[0] != '~')\n ])\n\n if has_date:\n folders = filter_by_dates(folders, date_fmt=date_fmt)\n\n return folders"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsorts files or folders by modified time", "response": "def sort_by_modified(files_or_folders: list) -> list:\n \"\"\"\n Sort files or folders by modified time\n\n Args:\n files_or_folders: list of files or folders\n\n Returns:\n list\n \"\"\"\n return sorted(files_or_folders, key=os.path.getmtime, reverse=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef filter_by_dates(files_or_folders: list, date_fmt=DATE_FMT) -> list:\n r = re.compile(f'.*{date_fmt}.*')\n return list(filter(\n lambda vv: r.match(vv.replace('\\\\', '/').split('/')[-1]) is not None,\n files_or_folders,\n ))", "response": "Filter files or folders by date format"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the file modified time in python", "response": "def file_modified_time(file_name) -> pd.Timestamp:\n \"\"\"\n File modified time in python\n\n Args:\n file_name: file name\n\n Returns:\n pd.Timestamp\n \"\"\"\n return pd.to_datetime(time.ctime(os.path.getmtime(filename=file_name)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_interval(ticker, session) -> Session:\n if '_' not in session:\n session = f'{session}_normal_0_0'\n interval = Intervals(ticker=ticker)\n ss_info = session.split('_')\n return getattr(interval, f'market_{ss_info.pop(1)}')(*ss_info)", "response": "Get interval from defined ticker and session"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef shift_time(start_time, mins) -> str:\n s_time = pd.Timestamp(start_time)\n e_time = s_time + np.sign(mins) * pd.Timedelta(f'00:{abs(mins)}:00')\n return e_time.strftime('%H:%M')", "response": "Shifts the start time by mins"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef market_open(self, session, mins) -> Session:\n if session not in self.exch: return SessNA\n start_time = self.exch[session][0]\n return Session(start_time, shift_time(start_time, int(mins)))", "response": "Returns a new session for the given session and mins."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a new session for the given session and mins", "response": "def market_close(self, session, mins) -> Session:\n \"\"\"\n Time intervals for market close\n\n Args:\n session: [allday, day, am, pm, night]\n mins: mintues before close\n\n Returns:\n Session of start_time and end_time\n \"\"\"\n if session not in self.exch: return SessNA\n end_time = self.exch[session][-1]\n return Session(shift_time(end_time, -int(mins) + 1), end_time)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a session with the same start time and end time as the given session.", "response": "def market_normal(self, session, after_open, before_close) -> Session:\n \"\"\"\n Time intervals between market\n\n Args:\n session: [allday, day, am, pm, night]\n after_open: mins after open\n before_close: mins before close\n\n Returns:\n Session of start_time and end_time\n \"\"\"\n logger = logs.get_logger(self.market_normal)\n\n if session not in self.exch: return SessNA\n ss = self.exch[session]\n\n s_time = shift_time(ss[0], int(after_open) + 1)\n e_time = shift_time(ss[-1], -int(before_close))\n\n request_cross = pd.Timestamp(s_time) >= pd.Timestamp(e_time)\n session_cross = pd.Timestamp(ss[0]) >= pd.Timestamp(ss[1])\n if request_cross and (not session_cross):\n logger.warning(f'end time {e_time} is earlier than {s_time} ...')\n return SessNA\n\n return Session(s_time, e_time)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a session with the exact start time and end time.", "response": "def market_exact(self, session, start_time: str, end_time: str) -> Session:\n \"\"\"\n Explicitly specify start time and end time\n\n Args:\n session: predefined session\n start_time: start time in terms of HHMM string\n end_time: end time in terms of HHMM string\n\n Returns:\n Session of start_time and end_time\n \"\"\"\n if session not in self.exch: return SessNA\n ss = self.exch[session]\n\n same_day = ss[0] < ss[-1]\n\n if not start_time: s_time = ss[0]\n else:\n s_time = param.to_hour(start_time)\n if same_day: s_time = max(s_time, ss[0])\n\n if not end_time: e_time = ss[-1]\n else:\n e_time = param.to_hour(end_time)\n if same_day: e_time = min(e_time, ss[-1])\n\n if same_day and (s_time > e_time): return SessNA\n return Session(start_time=s_time, end_time=e_time)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_tz(tz) -> str:\n from xbbg.const import exch_info\n\n if tz is None: return DEFAULT_TZ\n\n to_tz = tz\n if isinstance(tz, str):\n if hasattr(TimeZone, tz):\n to_tz = getattr(TimeZone, tz)\n else:\n exch = exch_info(ticker=tz)\n if 'tz' in exch.index:\n to_tz = exch.tz\n\n return to_tz", "response": "Convert tz from ticker or timezone shorthands to Python timzone"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a date time to a tz - aware version of the date", "response": "def tz_convert(dt, to_tz, from_tz=None) -> str:\n \"\"\"\n Convert to tz\n\n Args:\n dt: date time\n to_tz: to tz\n from_tz: from tz - will be ignored if tz from dt is given\n\n Returns:\n str: date & time\n\n Examples:\n >>> dt_1 = pd.Timestamp('2018-09-10 16:00', tz='Asia/Hong_Kong')\n >>> tz_convert(dt_1, to_tz='NY')\n '2018-09-10 04:00:00-04:00'\n >>> dt_2 = pd.Timestamp('2018-01-10 16:00')\n >>> tz_convert(dt_2, to_tz='HK', from_tz='NY')\n '2018-01-11 05:00:00+08:00'\n >>> dt_3 = '2018-09-10 15:00'\n >>> tz_convert(dt_3, to_tz='NY', from_tz='JP')\n '2018-09-10 02:00:00-04:00'\n \"\"\"\n logger = logs.get_logger(tz_convert, level='info')\n f_tz, t_tz = get_tz(from_tz), get_tz(to_tz)\n\n from_dt = pd.Timestamp(str(dt), tz=f_tz)\n logger.debug(f'converting {str(from_dt)} from {f_tz} to {t_tz} ...')\n return str(pd.Timestamp(str(from_dt), tz=t_tz))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef missing_info(**kwargs) -> str:\n func = kwargs.pop('func', 'unknown')\n if 'ticker' in kwargs: kwargs['ticker'] = kwargs['ticker'].replace('/', '_')\n info = utils.to_str(kwargs, fmt='{value}', sep='/')[1:-1]\n return f'{func}/{info}'", "response": "Returns a full infomation for missing query"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef current_missing(**kwargs) -> int:\n data_path = os.environ.get(BBG_ROOT, '').replace('\\\\', '/')\n if not data_path: return 0\n return len(files.all_files(f'{data_path}/Logs/{missing_info(**kwargs)}'))", "response": "Check number of trials for missing values"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update_missing(**kwargs):\n data_path = os.environ.get(BBG_ROOT, '').replace('\\\\', '/')\n if not data_path: return\n if len(kwargs) == 0: return\n\n log_path = f'{data_path}/Logs/{missing_info(**kwargs)}'\n\n cnt = len(files.all_files(log_path)) + 1\n files.create_folder(log_path)\n open(f'{log_path}/{cnt}.log', 'a').close()", "response": "Update the number of trials for missing values\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef public(function):\n orig_func = function\n while isinstance(orig_func, partial):\n orig_func = orig_func.func\n set_view_func_public(orig_func)\n\n return function", "response": "Decorator for public views that require authentication"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef custom_req(session, request):\n # flush event queue in case previous call errored out\n while(session.tryNextEvent()):\n pass\n\n print(\"Sending Request:\\n %s\" % request)\n session.sendRequest(request)\n messages = []\n # Process received events\n while(True):\n # We provide timeout to give the chance for Ctrl+C handling:\n ev = session.nextEvent(500)\n for msg in ev:\n print(\"Message Received:\\n %s\" % msg)\n messages.append(msg)\n if ev.eventType() == blpapi.Event.RESPONSE:\n # Response completely received, so we could exit\n break\n return messages", "response": "Utility for sending a predefined request and printing response as well\n as storing all messages in a list useful for testing"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef to_dict_list(mystr):\n res = _parse(mystr)\n dicts = []\n for res_dict in res:\n dicts.append(res_dict.asDict())\n return dicts", "response": "Translate a string representation of a Bloomberg Open API Request or Response to a list of dictionaries."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef bopen(**kwargs):\n con = BCon(**kwargs)\n con.start()\n try:\n yield con\n finally:\n con.stop()", "response": "A context manager that opens and manage a Bloomberg API session and returns a generator that yields the session object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nstarting the session and initialize services.", "response": "def start(self):\n \"\"\"\n Start connection and initialize session services\n \"\"\"\n\n # flush event queue in defensive way\n logger = _get_logger(self.debug)\n started = self._session.start()\n if started:\n ev = self._session.nextEvent()\n ev_name = _EVENT_DICT[ev.eventType()]\n logger.info('Event Type: {!r}'.format(ev_name))\n for msg in ev:\n logger.info('Message Received:\\n{}'.format(msg))\n if ev.eventType() != blpapi.Event.SESSION_STATUS:\n raise RuntimeError('Expected a \"SESSION_STATUS\" event but '\n 'received a {!r}'.format(ev_name))\n ev = self._session.nextEvent()\n ev_name = _EVENT_DICT[ev.eventType()]\n logger.info('Event Type: {!r}'.format(ev_name))\n for msg in ev:\n logger.info('Message Received:\\n{}'.format(msg))\n if ev.eventType() != blpapi.Event.SESSION_STATUS:\n raise RuntimeError('Expected a \"SESSION_STATUS\" event but '\n 'received a {!r}'.format(ev_name))\n else:\n ev = self._session.nextEvent(self.timeout)\n if ev.eventType() == blpapi.Event.SESSION_STATUS:\n for msg in ev:\n logger.warning('Message Received:\\n{}'.format(msg))\n raise ConnectionError('Could not start blpapi.Session')\n self._init_services()\n return self"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a pandas DataFrame with columns as and indexed by date.", "response": "def bdh(self, tickers, flds, start_date, end_date, elms=None,\n ovrds=None, longdata=False):\n \"\"\"\n Get tickers and fields, return pandas DataFrame with columns as\n MultiIndex with levels \"ticker\" and \"field\" and indexed by \"date\".\n If long data is requested return DataFrame with columns\n [\"date\", \"ticker\", \"field\", \"value\"].\n\n Parameters\n ----------\n tickers: {list, string}\n String or list of strings corresponding to tickers\n flds: {list, string}\n String or list of strings corresponding to FLDS\n start_date: string\n String in format YYYYmmdd\n end_date: string\n String in format YYYYmmdd\n elms: list of tuples\n List of tuples where each tuple corresponds to the other elements\n to be set, e.g. [(\"periodicityAdjustment\", \"ACTUAL\")].\n Refer to the HistoricalDataRequest section in the\n 'Services & schemas reference guide' for more info on these values\n ovrds: list of tuples\n List of tuples where each tuple corresponds to the override\n field and value\n longdata: boolean\n Whether data should be returned in long data format or pivoted\n \"\"\"\n ovrds = [] if not ovrds else ovrds\n elms = [] if not elms else elms\n\n elms = list(elms)\n\n data = self._bdh_list(tickers, flds, start_date, end_date,\n elms, ovrds)\n\n df = pd.DataFrame(data, columns=['date', 'ticker', 'field', 'value'])\n df.loc[:, 'date'] = pd.to_datetime(df.loc[:, 'date'])\n if not longdata:\n cols = ['ticker', 'field']\n df = df.set_index(['date'] + cols).unstack(cols)\n df.columns = df.columns.droplevel(0)\n\n return df"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmake a reference request to get tickers and fields return long pandas DataFrame with columns [ ticker field value", "response": "def ref(self, tickers, flds, ovrds=None):\n \"\"\"\n Make a reference data request, get tickers and fields, return long\n pandas DataFrame with columns [ticker, field, value]\n\n Parameters\n ----------\n tickers: {list, string}\n String or list of strings corresponding to tickers\n flds: {list, string}\n String or list of strings corresponding to FLDS\n ovrds: list of tuples\n List of tuples where each tuple corresponds to the override\n field and value\n\n Example\n -------\n >>> import pdblp\n >>> con = pdblp.BCon()\n >>> con.start()\n >>> con.ref(\"CL1 Comdty\", [\"FUT_GEN_MONTH\"])\n\n Notes\n -----\n This returns reference data which has singleton values. In raw format\n the messages passed back contain data of the form\n\n fieldData = {\n FUT_GEN_MONTH = \"FGHJKMNQUVXZ\"\n }\n \"\"\"\n ovrds = [] if not ovrds else ovrds\n\n logger = _get_logger(self.debug)\n if type(tickers) is not list:\n tickers = [tickers]\n if type(flds) is not list:\n flds = [flds]\n request = self._create_req('ReferenceDataRequest', tickers, flds,\n ovrds, [])\n logger.info('Sending Request:\\n{}'.format(request))\n self._session.sendRequest(request, identity=self._identity)\n data = self._parse_ref(flds)\n data = pd.DataFrame(data)\n data.columns = ['ticker', 'field', 'value']\n return data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef bulkref_hist(self, tickers, flds, dates, ovrds=None,\n date_field='REFERENCE_DATE'):\n \"\"\"\n Make iterative calls to bulkref() and create a long DataFrame with\n columns [date, ticker, field, name, value, position] where each date\n corresponds to overriding a historical data override field.\n\n Parameters\n ----------\n tickers: {list, string}\n String or list of strings corresponding to tickers\n flds: {list, string}\n String or list of strings corresponding to FLDS\n dates: list\n list of date strings in the format YYYYmmdd\n ovrds: list of tuples\n List of tuples where each tuple corresponds to the override\n field and value. This should not include the date_field which will\n be iteratively overridden\n date_field: str\n Field to iteratively override for requesting historical data,\n e.g. REFERENCE_DATE, CURVE_DATE, etc.\n\n Example\n -------\n >>> import pdblp\n >>> con = pdblp.BCon()\n >>> con.start()\n >>> dates = [\"20160625\", \"20160626\"]\n >>> con.bulkref_hist(\"BVIS0587 Index\", \"CURVE_TENOR_RATES\", dates,\n ... date_field=\"CURVE_DATE\")\n\n \"\"\"\n ovrds = [] if not ovrds else ovrds\n\n if type(tickers) is not list:\n tickers = [tickers]\n if type(flds) is not list:\n flds = [flds]\n self._send_hist(tickers, flds, dates, date_field, ovrds)\n data = self._parse_bulkref(flds, keep_corrId=True,\n sent_events=len(dates))\n data = pd.DataFrame(data)\n data.columns = ['ticker', 'field', 'name', 'value', 'position', 'date']\n data = data.sort_values(by=['date', 'position']).reset_index(drop=True)\n data = data.loc[:, ['date', 'ticker', 'field', 'name',\n 'value', 'position']]\n return data", "response": "Bulk - reference the specified set of items into a long DataFrame."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef bdib(self, ticker, start_datetime, end_datetime, event_type, interval,\n elms=None):\n \"\"\"\n Get Open, High, Low, Close, Volume, and numEvents for a ticker.\n Return pandas DataFrame\n\n Parameters\n ----------\n ticker: string\n String corresponding to ticker\n start_datetime: string\n UTC datetime in format YYYY-mm-ddTHH:MM:SS\n end_datetime: string\n UTC datetime in format YYYY-mm-ddTHH:MM:SS\n event_type: string {TRADE, BID, ASK, BID_BEST, ASK_BEST, BEST_BID,\n BEST_ASK}\n Requested data event type\n interval: int {1... 1440}\n Length of time bars\n elms: list of tuples\n List of tuples where each tuple corresponds to the other elements\n to be set. Refer to the IntradayBarRequest section in the\n 'Services & schemas reference guide' for more info on these values\n \"\"\"\n elms = [] if not elms else elms\n\n # flush event queue in case previous call errored out\n logger = _get_logger(self.debug)\n while(self._session.tryNextEvent()):\n pass\n\n # Create and fill the request for the historical data\n request = self.refDataService.createRequest('IntradayBarRequest')\n request.set('security', ticker)\n request.set('eventType', event_type)\n request.set('interval', interval) # bar interval in minutes\n request.set('startDateTime', start_datetime)\n request.set('endDateTime', end_datetime)\n for name, val in elms:\n request.set(name, val)\n\n logger.info('Sending Request:\\n{}'.format(request))\n # Send the request\n self._session.sendRequest(request, identity=self._identity)\n # Process received events\n data = []\n flds = ['open', 'high', 'low', 'close', 'volume', 'numEvents']\n for msg in self._receive_events():\n d = msg['element']['IntradayBarResponse']\n for bar in d['barData']['barTickData']:\n data.append(bar['barTickData'])\n data = pd.DataFrame(data).set_index('time').sort_index().loc[:, flds]\n return data", "response": "Get Open High Low Close Volume and numEvents for a ticker."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef bsrch(self, domain):\n logger = _get_logger(self.debug)\n request = self.exrService.createRequest('ExcelGetGridRequest')\n request.set('Domain', domain)\n logger.info('Sending Request:\\n{}'.format(request))\n self._session.sendRequest(request, identity=self._identity)\n data = []\n for msg in self._receive_events(to_dict=False):\n for v in msg.getElement(\"DataRecords\").values():\n for f in v.getElement(\"DataFields\").values():\n data.append(f.getElementAsString(\"StringValue\"))\n return pd.DataFrame(data)", "response": "This function uses the Bloomberg API to retrieve the BSRCH data for a specific domain."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nassemble one instruction from its textual representation.", "response": "def assemble_one(asmcode, pc=0, fork=DEFAULT_FORK):\n \"\"\" Assemble one EVM instruction from its textual representation.\n\n :param asmcode: assembly code for one instruction\n :type asmcode: str\n :param pc: program counter of the instruction(optional)\n :type pc: int\n :param fork: fork name (optional)\n :type fork: str\n :return: An Instruction object\n :rtype: Instruction\n\n Example use::\n\n >>> print assemble_one('LT')\n\n\n \"\"\"\n try:\n instruction_table = instruction_tables[fork]\n asmcode = asmcode.strip().split(' ')\n instr = instruction_table[asmcode[0].upper()]\n if pc:\n instr.pc = pc\n if instr.operand_size > 0:\n assert len(asmcode) == 2\n instr.operand = int(asmcode[1], 0)\n return instr\n except:\n raise AssembleError(\"Something wrong at pc %d\" % pc)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nassembling a sequence of textual representation of EVM instructions.", "response": "def assemble_all(asmcode, pc=0, fork=DEFAULT_FORK):\n \"\"\" Assemble a sequence of textual representation of EVM instructions\n\n :param asmcode: assembly code for any number of instructions\n :type asmcode: str\n :param pc: program counter of the first instruction(optional)\n :type pc: int\n :param fork: fork name (optional)\n :type fork: str\n :return: An generator of Instruction objects\n :rtype: generator[Instructions]\n\n Example use::\n\n >>> assemble_one('''PUSH1 0x60\\n \\\n PUSH1 0x40\\n \\\n MSTORE\\n \\\n PUSH1 0x2\\n \\\n PUSH2 0x108\\n \\\n PUSH1 0x0\\n \\\n POP\\n \\\n SSTORE\\n \\\n PUSH1 0x40\\n \\\n MLOAD\\n \\\n ''')\n\n \"\"\"\n asmcode = asmcode.split('\\n')\n asmcode = iter(asmcode)\n for line in asmcode:\n if not line.strip():\n continue\n instr = assemble_one(line, pc=pc, fork=fork)\n yield instr\n pc += instr.size"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndisassemble a single instruction from a bytecode stream.", "response": "def disassemble_one(bytecode, pc=0, fork=DEFAULT_FORK):\n \"\"\" Disassemble a single instruction from a bytecode\n\n :param bytecode: the bytecode stream\n :type bytecode: str | bytes | bytearray | iterator\n :param pc: program counter of the instruction(optional)\n :type pc: int\n :param fork: fork name (optional)\n :type fork: str\n :return: an Instruction object\n :rtype: Instruction\n\n Example use::\n\n >>> print disassemble_one('\\x60\\x10')\n\n \"\"\"\n instruction_table = instruction_tables[fork]\n if isinstance(bytecode, bytes):\n bytecode = bytearray(bytecode)\n if isinstance(bytecode, str):\n bytecode = bytearray(bytecode.encode('latin-1'))\n\n bytecode = iter(bytecode)\n try:\n opcode = next(bytecode)\n except StopIteration:\n return\n\n assert isinstance(opcode, int)\n\n instruction = copy.copy(instruction_table.get(opcode, None))\n if instruction is None:\n instruction = Instruction(opcode, 'INVALID', 0, 0, 0, 0, 'Unspecified invalid instruction.')\n instruction.pc = pc\n\n try:\n if instruction.has_operand:\n instruction.parse_operand(bytecode)\n except ParseError:\n instruction = None\n finally:\n return instruction"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef disassemble_all(bytecode, pc=0, fork=DEFAULT_FORK):\n if isinstance(bytecode, bytes):\n bytecode = bytearray(bytecode)\n if isinstance(bytecode, str):\n bytecode = bytearray(bytecode.encode('latin-1'))\n\n bytecode = iter(bytecode)\n while True:\n instr = disassemble_one(bytecode, pc=pc, fork=fork)\n if not instr:\n return\n pc += instr.size\n yield instr", "response": "Disassemble all instructions in bytecode into a list of Instruction objects."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndisassemble an EVM into a text representation of the base EVM.", "response": "def disassemble(bytecode, pc=0, fork=DEFAULT_FORK):\n \"\"\" Disassemble an EVM bytecode\n\n :param bytecode: binary representation of an evm bytecode\n :type bytecode: str | bytes | bytearray\n :param pc: program counter of the first instruction(optional)\n :type pc: int\n :param fork: fork name (optional)\n :type fork: str\n :return: the text representation of the assembler code\n\n Example use::\n\n >>> disassemble(\"\\x60\\x60\\x60\\x40\\x52\\x60\\x02\\x61\\x01\\x00\")\n ...\n PUSH1 0x60\n BLOCKHASH\n MSTORE\n PUSH1 0x2\n PUSH2 0x100\n\n \"\"\"\n return '\\n'.join(map(str, disassemble_all(bytecode, pc=pc, fork=fork)))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef assemble(asmcode, pc=0, fork=DEFAULT_FORK):\n return b''.join(x.bytes for x in assemble_all(asmcode, pc=pc, fork=fork))", "response": "Assemble an EVM program containing a sequence of bytes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef disassemble_hex(bytecode, pc=0, fork=DEFAULT_FORK):\n if bytecode.startswith('0x'):\n bytecode = bytecode[2:]\n bytecode = unhexlify(bytecode)\n return disassemble(bytecode, pc=pc, fork=fork)", "response": "Disassemble an EVM bytecode into a text representation of the current language."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef assemble_hex(asmcode, pc=0, fork=DEFAULT_FORK):\n if isinstance(asmcode, list):\n return '0x' + hexlify(b''.join([x.bytes for x in asmcode])).decode('ascii')\n return '0x' + hexlify(assemble(asmcode, pc=pc, fork=fork)).decode('ascii')", "response": "Assemble an EVM program containing a sequence of instruction and return the hex representation of the bytecode."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a block number to a fork name.", "response": "def block_to_fork(block_number):\n \"\"\" Convert block number to fork name.\n\n :param block_number: block number\n :type block_number: int\n :return: fork name\n :rtype: str\n\n Example use::\n\n >>> block_to_fork(0)\n ...\n \"frontier\"\n >>> block_to_fork(4370000)\n ...\n \"byzantium\"\n >>> block_to_fork(4370001)\n ...\n \"byzantium\"\n \"\"\"\n forks_by_block = {\n 0: \"frontier\",\n 1150000: \"homestead\",\n # 1920000 Dao \n 2463000: \"tangerine_whistle\",\n 2675000: \"spurious_dragon\",\n 4370000: \"byzantium\",\n #7280000: \"constantinople\", # Same Block as petersburg, commented to avoid conflicts\n 7280000: \"petersburg\",\n 9999999: \"serenity\" # to be replaced after Serenity launch\n }\n fork_names = list(forks_by_block.values())\n fork_blocks = list(forks_by_block.keys())\n return fork_names[bisect(fork_blocks, block_number) - 1]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_operand(self, buf):\n buf = iter(buf)\n try:\n operand = 0\n for _ in range(self.operand_size):\n operand <<= 8\n operand |= next(buf)\n self._operand = operand\n except StopIteration:\n raise ParseError(\"Not enough data for decoding\")", "response": "Parses an operand from buf\n "} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nencode the instruction into a byte string.", "response": "def bytes(self):\n \"\"\" Encoded instruction \"\"\"\n b = [bytes([self._opcode])]\n for offset in reversed(range(self.operand_size)):\n b.append(bytes([(self.operand >> offset * 8) & 0xff]))\n return b''.join(b)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreverse the key value in dictionary file", "response": "def reverse(rev_inputs=REV_INPUTS):\n \"\"\"\n reverse the key, value in dictionary file\n :param rev_inputs: the files to be reversed\n :return: None\n \"\"\"\n dirname = os.path.dirname(__file__)\n\n for in_file in rev_inputs:\n reversed_dict = {}\n input_file = in_file + '.txt'\n output_file = in_file + 'Rev.txt'\n input_file = os.path.join(dirname, DICT_DIRECTORY, input_file)\n output_file = os.path.join(dirname, DICT_DIRECTORY, output_file)\n with open(input_file, 'r', encoding='utf-8') as f:\n for line in f:\n line = line.strip()\n split = line.split('\\t')\n if len(split) < 2:\n continue\n term1 = split[0]\n term2 = split[1]\n\n for char in term2.split(' '):\n if char in reversed_dict:\n reversed_dict[char].append(term1)\n else:\n reversed_dict[char] = [term1]\n with open(output_file, 'w', encoding='utf-8') as f:\n for key in reversed_dict:\n line = key + '\\t' + ' '.join(reversed_dict[key]) + '\\n'\n f.write(line)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmerging the phrase files into one file", "response": "def merge(mer_inputs=MER_INPUTS, mer_output=MER_OUTPUT):\n \"\"\"\n merge the phrase files into one file\n :param mer_inputs: the phrase files\n :param mer_output: the output file\n :return: None\n \"\"\"\n dirname = os.path.dirname(__file__)\n output_file = os.path.join(dirname, DICT_DIRECTORY, mer_output)\n lines = []\n for in_file in MER_INPUTS:\n input_file = os.path.join(dirname, DICT_DIRECTORY, in_file)\n with open(input_file, encoding='utf-8') as f:\n for line in f:\n lines.append(line)\n\n with open(output_file, 'w', encoding='utf-8') as f:\n for line in lines:\n f.write(line)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadjust the delay of a resource slot based on the response status.", "response": "def _adjust_delay(self, slot, response):\n \"\"\"Define delay adjustment policy\"\"\"\n if response.status in self.retry_http_codes:\n new_delay = max(slot.delay, 1) * 4\n new_delay = max(new_delay, self.mindelay)\n new_delay = min(new_delay, self.maxdelay)\n slot.delay = new_delay\n self.stats.inc_value('delay_count')\n elif response.status == 200:\n new_delay = max(slot.delay / 2, self.mindelay)\n if new_delay < 0.01:\n new_delay = 0\n slot.delay = new_delay"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef memberness(context):\n '''The likelihood that the context is a \"member\".'''\n if context:\n texts = context.xpath('.//*[local-name()=\"explicitMember\"]/text()').extract()\n text = str(texts).lower()\n\n if len(texts) > 1:\n return 2\n elif 'country' in text:\n return 2\n elif 'member' not in text:\n return 0\n elif 'successor' in text:\n # 'SuccessorMember' is a rare case that shouldn't be treated as member\n return 1\n elif 'parent' in text:\n return 2\n return 3", "response": "The likelihood that the context is a member."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse 10 - Q or 10 - K XML report.", "response": "def parse_10qk(self, response):\n '''Parse 10-Q or 10-K XML report.'''\n loader = ReportItemLoader(response=response)\n item = loader.load_item()\n\n if 'doc_type' in item:\n doc_type = item['doc_type']\n if doc_type in ('10-Q', '10-K'):\n return item\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef camelcase(string):\n\n string = re.sub(r\"^[\\-_\\.]\", '', str(string))\n if not string:\n return string\n return lowercase(string[0]) + re.sub(r\"[\\-_\\.\\s]([a-z])\", lambda matched: uppercase(matched.group(1)), string[1:])", "response": "Convert string into camel case."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef capitalcase(string):\n\n string = str(string)\n if not string:\n return string\n return uppercase(string[0]) + string[1:]", "response": "Convert string into capital case."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert string into path case. Join punctuation with slash.", "response": "def pathcase(string):\n \"\"\"Convert string into path case.\n Join punctuation with slash.\n\n Args:\n string: String to convert.\n\n Returns:\n string: Path cased string.\n\n \"\"\"\n string = snakecase(string)\n if not string:\n return string\n return re.sub(r\"_\", \"/\", string)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef backslashcase(string):\n str1 = re.sub(r\"_\", r\"\\\\\", snakecase(string))\n\n return str1", "response": "Convert string into spinal case. Join punctuation with backslash."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sentencecase(string):\n joiner = ' '\n string = re.sub(r\"[\\-_\\.\\s]\", joiner, str(string))\n if not string:\n return string\n return capitalcase(trimcase(\n re.sub(r\"[A-Z]\", lambda matched: joiner +\n lowercase(matched.group(0)), string)\n ))", "response": "Convert string into sentence case."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting string into snake case.", "response": "def snakecase(string):\n \"\"\"Convert string into snake case.\n Join punctuation with underscore\n\n Args:\n string: String to convert.\n\n Returns:\n string: Snake cased string.\n\n \"\"\"\n\n string = re.sub(r\"[\\-\\.\\s]\", '_', str(string))\n if not string:\n return string\n return lowercase(string[0]) + re.sub(r\"[A-Z]\", lambda matched: '_' + lowercase(matched.group(0)), string[1:])"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _check_input(self, input):\n if isinstance(input, str):\n return 'st'\n elif isinstance(input, list):\n if all(isinstance(item, str) for item in input):\n return 'gst'\n\n raise ValueError(\"String argument should be of type String or\"\n \" a list of strings\")", "response": "Checks the validity of the input."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding the Suffix Tree on the given input.", "response": "def build(self, x):\n \"\"\"Builds the Suffix tree on the given input.\n If the input is of type List of Strings:\n Generalized Suffix Tree is built.\n\n :param x: String or List of Strings\n \"\"\"\n type = self._check_input(x)\n\n if type == 'st':\n x += next(self._terminalSymbolsGenerator())\n self._build(x)\n if type == 'gst':\n self._build_generalized(x)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding a Suffix Tree using McCreight O ( n ) algorithm.", "response": "def _build_McCreight(self, x):\n \"\"\"Builds a Suffix tree using McCreight O(n) algorithm.\n\n Algorithm based on:\n McCreight, Edward M. \"A space-economical suffix tree construction algorithm.\" - ACM, 1976.\n Implementation based on:\n UH CS - 58093 String Processing Algorithms Lecture Notes\n \"\"\"\n u = self.root\n d = 0\n for i in range(len(x)):\n while u.depth == d and u._has_transition(x[d+i]):\n u = u._get_transition_link(x[d+i])\n d = d + 1\n while d < u.depth and x[u.idx + d] == x[i + d]:\n d = d + 1\n if d < u.depth:\n u = self._create_node(x, u, d)\n self._create_leaf(x, i, u, d)\n if not u._get_suffix_link():\n self._compute_slink(x, u)\n u = u._get_suffix_link()\n d = d - 1\n if d < 0:\n d = 0"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _build_generalized(self, xs):\n terminal_gen = self._terminalSymbolsGenerator()\n\n _xs = ''.join([x + next(terminal_gen) for x in xs])\n self.word = _xs\n self._generalized_word_starts(xs)\n self._build(_xs)\n self.root._traverse(self._label_generalized)", "response": "Builds a Generalized Suffix Tree from the array of strings provided."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _label_generalized(self, node):\n if node.is_leaf():\n x = {self._get_word_start_index(node.idx)}\n else:\n x = {n for ns in node.transition_links for n in ns[0].generalized_idxs}\n node.generalized_idxs = x", "response": "Helper method that labels the nodes of GST with indexes of strings\n found in their descendants."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_word_start_index(self, idx):\n i = 0\n for _idx in self.word_starts[1:]:\n if idx < _idx:\n return i\n else:\n i+=1\n return i", "response": "Helper method that returns the index of the string based on node s\n starting index"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the Largest Common Substring of Strings provided in stringIdxs.", "response": "def lcs(self, stringIdxs=-1):\n \"\"\"Returns the Largest Common Substring of Strings provided in stringIdxs.\n If stringIdxs is not provided, the LCS of all strings is returned.\n\n ::param stringIdxs: Optional: List of indexes of strings.\n \"\"\"\n if stringIdxs == -1 or not isinstance(stringIdxs, list):\n stringIdxs = set(range(len(self.word_starts)))\n else:\n stringIdxs = set(stringIdxs)\n\n deepestNode = self._find_lcs(self.root, stringIdxs)\n start = deepestNode.idx\n end = deepestNode.idx + deepestNode.depth\n return self.word[start:end]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _find_lcs(self, node, stringIdxs):\n nodes = [self._find_lcs(n, stringIdxs)\n for (n,_) in node.transition_links\n if n.generalized_idxs.issuperset(stringIdxs)]\n\n if nodes == []:\n return node\n\n deepestNode = max(nodes, key=lambda n: n.depth)\n return deepestNode", "response": "Helper method that finds LCS by traversing the labeled GSD."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _generalized_word_starts(self, xs):\n self.word_starts = []\n i = 0\n for n in range(len(xs)):\n self.word_starts.append(i)\n i += len(xs[n]) + 1", "response": "Helper method returns the starting indexes of strings in GST"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns starting position of the substring y in the string used for building the Suffix tree.", "response": "def find(self, y):\n \"\"\"Returns starting position of the substring y in the string used for\n building the Suffix tree.\n\n :param y: String\n :return: Index of the starting position of string y in the string used for building the Suffix tree\n -1 if y is not a substring.\n \"\"\"\n node = self.root\n while True:\n edge = self._edgeLabel(node, node.parent)\n if edge.startswith(y):\n return node.idx\n \n i = 0\n while(i < len(edge) and edge[i] == y[0]):\n y = y[1:]\n i += 1\n \n if i != 0:\n if i == len(edge) and y != '':\n pass\n else:\n return -1\n \n node = node._get_transition_link(y[0])\n if not node:\n return -1"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _terminalSymbolsGenerator(self):\n py2 = sys.version[0] < '3'\n UPPAs = list(list(range(0xE000,0xF8FF+1)) + list(range(0xF0000,0xFFFFD+1)) + list(range(0x100000, 0x10FFFD+1)))\n for i in UPPAs:\n if py2:\n yield(unichr(i))\n else:\n yield(chr(i))\n raise ValueError(\"To many input strings.\")", "response": "Generator of unique terminal symbols used for building the Generalized Suffix Tree."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing the distance between two sets of points x and y.", "response": "def _dist(self, x, y, A):\n \"(x - y)^T A (x - y)\"\n return scipy.spatial.distance.mahalanobis(x, y, A) ** 2"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef query(self, i, j):\n \"Query the oracle to find out whether i and j should be must-linked\"\n if self.queries_cnt < self.max_queries_cnt:\n self.queries_cnt += 1\n return self.labels[i] == self.labels[j]\n else:\n raise MaximumQueriesExceeded", "response": "Query the oracle to find out whether i and j should be must - linked"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef preprocess_constraints(ml, cl, n):\n \"Create a graph of constraints for both must- and cannot-links\"\n\n # Represent the graphs using adjacency-lists\n ml_graph, cl_graph = {}, {}\n for i in range(n):\n ml_graph[i] = set()\n cl_graph[i] = set()\n\n def add_both(d, i, j):\n d[i].add(j)\n d[j].add(i)\n\n for (i, j) in ml:\n ml_graph[i].add(j)\n ml_graph[j].add(i)\n\n for (i, j) in cl:\n cl_graph[i].add(j)\n cl_graph[j].add(i)\n\n def dfs(i, graph, visited, component):\n visited[i] = True\n for j in graph[i]:\n if not visited[j]:\n dfs(j, graph, visited, component)\n component.append(i)\n\n # Run DFS from each node to get all the graph's components\n # and add an edge for each pair of nodes in the component (create a complete graph)\n # See http://www.techiedelight.com/transitive-closure-graph/ for more details\n visited = [False] * n\n neighborhoods = []\n for i in range(n):\n if not visited[i] and ml_graph[i]:\n component = []\n dfs(i, ml_graph, visited, component)\n for x1 in component:\n for x2 in component:\n if x1 != x2:\n ml_graph[x1].add(x2)\n neighborhoods.append(component)\n\n for (i, j) in cl:\n for x in ml_graph[i]:\n add_both(cl_graph, x, j)\n\n for y in ml_graph[j]:\n add_both(cl_graph, i, y)\n\n for x in ml_graph[i]:\n for y in ml_graph[j]:\n add_both(cl_graph, x, y)\n\n for i in ml_graph:\n for j in ml_graph[i]:\n if j != i and j in cl_graph[i]:\n raise InconsistentConstraintsException('Inconsistent constraints between {} and {}'.format(i, j))\n\n return ml_graph, cl_graph, neighborhoods", "response": "Create a graph of constraints for both must - and cannot - links"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ntranslating a regular Scikit - Learn estimator or pipeline to a PMML pipeline.", "response": "def make_pmml_pipeline(obj, active_fields = None, target_fields = None):\n\t\"\"\"Translates a regular Scikit-Learn estimator or pipeline to a PMML pipeline.\n\n\tParameters:\n\t----------\n\tobj: BaseEstimator\n\t\tThe object.\n\n\tactive_fields: list of strings, optional\n\t\tFeature names. If missing, \"x1\", \"x2\", .., \"xn\" are assumed.\n\n\ttarget_fields: list of strings, optional\n\t\tLabel name(s). If missing, \"y\" is assumed.\n\n\t\"\"\"\n\tsteps = _filter_steps(_get_steps(obj))\n\tpipeline = PMMLPipeline(steps)\n\tif active_fields is not None:\n\t\tpipeline.active_fields = numpy.asarray(active_fields)\n\tif target_fields is not None:\n\t\tpipeline.target_fields = numpy.asarray(target_fields)\n\treturn pipeline"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert a fitted Scikit-Learn pipeline to PMML. Parameters: ---------- pipeline: PMMLPipeline The pipeline. pmml: string The path to where the PMML document should be stored. user_classpath: list of strings, optional The paths to JAR files that provide custom Transformer, Selector and/or Estimator converter classes. The JPMML-SkLearn classpath is constructed by appending user JAR files to package JAR files. with_repr: boolean, optional If true, insert the string representation of pipeline into the PMML document. debug: boolean, optional If true, print information about the conversion process. java_encoding: string, optional The character encoding to use for decoding Java output and error byte streams.", "response": "def sklearn2pmml(pipeline, pmml, user_classpath = [], with_repr = False, debug = False, java_encoding = \"UTF-8\"):\n\t\"\"\"Converts a fitted Scikit-Learn pipeline to PMML.\n\n\tParameters:\n\t----------\n\tpipeline: PMMLPipeline\n\t\tThe pipeline.\n\n\tpmml: string\n\t\tThe path to where the PMML document should be stored.\n\n\tuser_classpath: list of strings, optional\n\t\tThe paths to JAR files that provide custom Transformer, Selector and/or Estimator converter classes.\n\t\tThe JPMML-SkLearn classpath is constructed by appending user JAR files to package JAR files.\n\n\twith_repr: boolean, optional\n\t\tIf true, insert the string representation of pipeline into the PMML document.\n\n\tdebug: boolean, optional\n\t\tIf true, print information about the conversion process.\n\n\tjava_encoding: string, optional\n\t\tThe character encoding to use for decoding Java output and error byte streams.\n\n\t\"\"\"\n\tif debug:\n\t\tjava_version = _java_version(java_encoding)\n\t\tif java_version is None:\n\t\t\tjava_version = (\"java\", \"N/A\")\n\t\tprint(\"python: {0}\".format(platform.python_version()))\n\t\tprint(\"sklearn: {0}\".format(sklearn.__version__))\n\t\tprint(\"sklearn.externals.joblib: {0}\".format(joblib.__version__))\n\t\tprint(\"pandas: {0}\".format(pandas.__version__))\n\t\tprint(\"sklearn_pandas: {0}\".format(sklearn_pandas.__version__))\n\t\tprint(\"sklearn2pmml: {0}\".format(__version__))\n\t\tprint(\"{0}: {1}\".format(java_version[0], java_version[1]))\n\tif not isinstance(pipeline, PMMLPipeline):\n\t\traise TypeError(\"The pipeline object is not an instance of \" + PMMLPipeline.__name__ + \". Use the 'sklearn2pmml.make_pmml_pipeline(obj)' utility function to translate a regular Scikit-Learn estimator or pipeline to a PMML pipeline\")\n\testimator = pipeline._final_estimator\n\tcmd = [\"java\", \"-cp\", os.pathsep.join(_classpath(user_classpath)), \"org.jpmml.sklearn.Main\"]\n\tdumps = []\n\ttry:\n\t\tif with_repr:\n\t\t\tpipeline.repr_ = repr(pipeline)\n\t\t# if isinstance(estimator, H2OEstimator):\n\t\tif hasattr(estimator, \"download_mojo\"):\n\t\t\testimator_mojo = estimator.download_mojo()\n\t\t\tdumps.append(estimator_mojo)\n\t\t\testimator._mojo_path = estimator_mojo\n\t\tpipeline_pkl = _dump(pipeline, \"pipeline\")\n\t\tcmd.extend([\"--pkl-pipeline-input\", pipeline_pkl])\n\t\tdumps.append(pipeline_pkl)\n\t\tcmd.extend([\"--pmml-output\", pmml])\n\t\tif debug:\n\t\t\tprint(\"Executing command:\\n{0}\".format(\" \".join(cmd)))\n\t\ttry:\n\t\t\tprocess = Popen(cmd, stdout = PIPE, stderr = PIPE, bufsize = 1)\n\t\texcept OSError:\n\t\t\traise RuntimeError(\"Java is not installed, or the Java executable is not on system path\")\n\t\toutput, error = process.communicate()\n\t\tretcode = process.poll()\n\t\tif debug or retcode:\n\t\t\tif(len(output) > 0):\n\t\t\t\tprint(\"Standard output:\\n{0}\".format(_decode(output, java_encoding)))\n\t\t\telse:\n\t\t\t\tprint(\"Standard output is empty\")\n\t\t\tif(len(error) > 0):\n\t\t\t\tprint(\"Standard error:\\n{0}\".format(_decode(error, java_encoding)))\n\t\t\telse:\n\t\t\t\tprint(\"Standard error is empty\")\n\t\tif retcode:\n\t\t\traise RuntimeError(\"The JPMML-SkLearn conversion application has failed. The Java executable should have printed more information about the failure into its standard output and/or standard error streams\")\n\tfinally:\n\t\tif debug:\n\t\t\tprint(\"Preserved joblib dump file(s): {0}\".format(\" \".join(dumps)))\n\t\telse:\n\t\t\tfor dump in dumps:\n\t\t\t\tos.remove(dump)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef make_tpot_pmml_config(config, user_classpath = []):\n\ttpot_keys = set(config.keys())\n\tclasses = _supported_classes(user_classpath)\n\tpmml_keys = (set(classes)).union(set([_strip_module(class_) for class_ in classes]))\n\treturn { key : config[key] for key in (tpot_keys).intersection(pmml_keys)}", "response": "Translates a regular TPOT configuration to a PMML - compatible TPOT configuration."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconstruct a new instance of the formset with the formset_kwargs set.", "response": "def construct_formset(self):\n \"\"\"\n Returns an instance of the formset\n \"\"\"\n formset_class = self.get_formset()\n if hasattr(self, 'get_extra_form_kwargs'):\n klass = type(self).__name__\n raise DeprecationWarning(\n 'Calling {0}.get_extra_form_kwargs is no longer supported. '\n 'Set `form_kwargs` in {0}.formset_kwargs or override '\n '{0}.get_formset_kwargs() directly.'.format(klass),\n )\n return formset_class(**self.get_formset_kwargs())"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the keyword arguments for instantiating the formset.", "response": "def get_formset_kwargs(self):\n \"\"\"\n Returns the keyword arguments for instantiating the formset.\n \"\"\"\n kwargs = self.formset_kwargs.copy()\n kwargs.update({\n 'initial': self.get_initial(),\n 'prefix': self.get_prefix(),\n })\n\n if self.request.method in ('POST', 'PUT'):\n kwargs.update({\n 'data': self.request.POST.copy(),\n 'files': self.request.FILES,\n })\n return kwargs"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_factory_kwargs(self):\n # Perform deprecation check\n for attr in ['extra', 'max_num', 'can_order', 'can_delete', 'ct_field',\n 'formfield_callback', 'fk_name', 'widgets', 'ct_fk_field']:\n if hasattr(self, attr):\n klass = type(self).__name__\n raise DeprecationWarning(\n 'Setting `{0}.{1}` at the class level is now deprecated. '\n 'Set `{0}.factory_kwargs` instead.'.format(klass, attr)\n )\n\n kwargs = self.factory_kwargs.copy()\n if self.get_formset_class():\n kwargs['formset'] = self.get_formset_class()\n return kwargs", "response": "Returns the keyword arguments for calling the formset factory\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_success_url(self):\n if self.success_url:\n url = self.success_url\n else:\n # Default to returning to the same page\n url = self.request.get_full_path()\n return url", "response": "Returns the URL of the success page."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the keyword arguments for instantiating the formset.", "response": "def get_formset_kwargs(self):\n \"\"\"\n Returns the keyword arguments for instantiating the formset.\n \"\"\"\n kwargs = super(ModelFormSetMixin, self).get_formset_kwargs()\n kwargs['queryset'] = self.get_queryset()\n return kwargs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef formset_valid(self, formset):\n self.object_list = formset.save()\n return super(ModelFormSetMixin, self).formset_valid(formset)", "response": "Save the associated models."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_formset_kwargs(self):\n # Perform deprecation check\n if hasattr(self, 'save_as_new'):\n klass = type(self).__name__\n raise DeprecationWarning(\n 'Setting `{0}.save_as_new` at the class level is now '\n 'deprecated. Set `{0}.formset_kwargs` instead.'.format(klass)\n )\n kwargs = super(BaseInlineFormSetFactory, self).get_formset_kwargs()\n kwargs['instance'] = self.object\n return kwargs", "response": "Returns the keyword arguments for instantiating the formset."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_factory_kwargs(self):\n kwargs = super(BaseInlineFormSetFactory, self).get_factory_kwargs()\n kwargs.setdefault('fields', self.fields)\n kwargs.setdefault('exclude', self.exclude)\n\n if self.get_form_class():\n kwargs['form'] = self.get_form_class()\n return kwargs", "response": "Returns the keyword arguments for calling the formset factory\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nhandling GET requests and instantiates a blank version of the formset.", "response": "def get(self, request, *args, **kwargs):\n \"\"\"\n Handles GET requests and instantiates a blank version of the formset.\n \"\"\"\n formset = self.construct_formset()\n return self.render_to_response(self.get_context_data(formset=formset))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nhandle POST requests, instantiating a formset instance with the passed POST variables and then checked for validity.", "response": "def post(self, request, *args, **kwargs):\n \"\"\"\n Handles POST requests, instantiating a formset instance with the passed\n POST variables and then checked for validity.\n \"\"\"\n formset = self.construct_formset()\n if formset.is_valid():\n return self.formset_valid(formset)\n else:\n return self.formset_invalid(formset)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a new formset instance with the model class as the model attribute as the formset s model.", "response": "def construct_formset(self):\n \"\"\"\n Overrides construct_formset to attach the model class as\n an attribute of the returned formset instance.\n \"\"\"\n formset = super(InlineFormSetFactory, self).construct_formset()\n formset.model = self.inline_model\n return formset"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef forms_valid(self, form, inlines):\n response = self.form_valid(form)\n for formset in inlines:\n formset.save()\n return response", "response": "Check if the form and inlines are valid and save the associated models."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrendering the context data for the given form and formsets.", "response": "def forms_invalid(self, form, inlines):\n \"\"\"\n If the form or formsets are invalid, re-render the context data with the\n data-filled form and formsets and errors.\n \"\"\"\n return self.render_to_response(self.get_context_data(form=form, inlines=inlines))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconstructs the list of inline formsets for this object.", "response": "def construct_inlines(self):\n \"\"\"\n Returns the inline formset instances\n \"\"\"\n inline_formsets = []\n for inline_class in self.get_inlines():\n inline_instance = inline_class(self.model, self.request, self.object, self.kwargs, self)\n inline_formset = inline_instance.construct_formset()\n inline_formsets.append(inline_formset)\n return inline_formsets"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get(self, request, *args, **kwargs):\n form_class = self.get_form_class()\n form = self.get_form(form_class)\n inlines = self.construct_inlines()\n return self.render_to_response(self.get_context_data(form=form, inlines=inlines, **kwargs))", "response": "Handles GET requests and instantiates a blank version of the form and formsets."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef post(self, request, *args, **kwargs):\n form_class = self.get_form_class()\n form = self.get_form(form_class)\n\n if form.is_valid():\n self.object = form.save(commit=False)\n form_validated = True\n else:\n form_validated = False\n\n inlines = self.construct_inlines()\n\n if all_valid(inlines) and form_validated:\n return self.forms_valid(form, inlines)\n return self.forms_invalid(form, inlines)", "response": "Handles POST requests, instantiating a form and formset instances with the passed\n POST variables and then checked for validity."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd each formset to the context under its corresponding entry in inlines_names.", "response": "def get_context_data(self, **kwargs):\n \"\"\"\n If `inlines_names` has been defined, add each formset to the context under\n its corresponding entry in `inlines_names`\n \"\"\"\n context = {}\n inlines_names = self.get_inlines_names()\n\n if inlines_names:\n # We have formset or inlines in context, but never both\n context.update(zip(inlines_names, kwargs.get('inlines', [])))\n if 'formset' in kwargs:\n context[inlines_names[0]] = kwargs['formset']\n context.update(kwargs)\n return super(NamedFormsetsMixin, self).get_context_data(**context)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ntry to convert word to date", "response": "def try_convert_to_date(self, word):\n \"\"\"\n Tries to convert word to date(datetime) using search_date_formats\n Return None if word fits no one format\n \"\"\"\n for frm in self.search_date_formats:\n try:\n return datetime.datetime.strptime(word, frm).date()\n except ValueError:\n pass\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the url - encoded query string for the given field name.", "response": "def get_params_for_field(self, field_name, sort_type=None):\n \"\"\"\n If sort_type is None - inverse current sort for field, if no sorted - use asc\n \"\"\"\n if not sort_type:\n if self.initial_sort == field_name:\n sort_type = 'desc' if self.initial_sort_type == 'asc' else 'asc'\n else:\n sort_type = 'asc'\n self.initial_params[self.sort_param_name] = self.sort_fields[field_name]\n self.initial_params[self.sort_type_param_name] = sort_type\n return '?%s' % self.initial_params.urlencode()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_start_date(self, obj):\n obj_date = getattr(obj, self.get_date_field())\n try:\n obj_date = obj_date.date()\n except AttributeError:\n # It's a date rather than datetime, so we use it as is\n pass\n return obj_date", "response": "Returns the start date for a model instance"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_end_date(self, obj):\n obj_date = getattr(obj, self.get_end_date_field())\n try:\n obj_date = obj_date.date()\n except AttributeError:\n # It's a date rather than datetime, so we use it as is\n pass\n return obj_date", "response": "Returns the end date for a model instance"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_first_of_week(self):\n if self.first_of_week is None:\n raise ImproperlyConfigured(\"%s.first_of_week is required.\" % self.__class__.__name__)\n if self.first_of_week not in range(7):\n raise ImproperlyConfigured(\"%s.first_of_week must be an integer between 0 and 6.\" % self.__class__.__name__)\n return self.first_of_week", "response": "Returns an integer representing the first day of the week in the assessment."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_queryset(self):\n qs = super(BaseCalendarMonthView, self).get_queryset()\n\n year = self.get_year()\n month = self.get_month()\n\n date_field = self.get_date_field()\n end_date_field = self.get_end_date_field()\n\n date = _date_from_string(year, self.get_year_format(),\n month, self.get_month_format())\n\n since = date\n until = self.get_next_month(date)\n\n # Adjust our start and end dates to allow for next and previous\n # month edges\n if since.weekday() != self.get_first_of_week():\n diff = math.fabs(since.weekday() - self.get_first_of_week())\n since = since - datetime.timedelta(days=diff)\n\n if until.weekday() != ((self.get_first_of_week() + 6) % 7):\n diff = math.fabs(((self.get_first_of_week() + 6) % 7) - until.weekday())\n until = until + datetime.timedelta(days=diff)\n\n if end_date_field:\n # 5 possible conditions for showing an event:\n\n # 1) Single day event, starts after 'since'\n # 2) Multi-day event, starts after 'since' and ends before 'until'\n # 3) Starts before 'since' and ends after 'since' and before 'until'\n # 4) Starts after 'since' but before 'until' and ends after 'until'\n # 5) Starts before 'since' and ends after 'until'\n predicate1 = Q(**{\n '%s__gte' % date_field: since,\n end_date_field: None\n })\n predicate2 = Q(**{\n '%s__gte' % date_field: since,\n '%s__lt' % end_date_field: until\n })\n predicate3 = Q(**{\n '%s__lt' % date_field: since,\n '%s__gte' % end_date_field: since,\n '%s__lt' % end_date_field: until\n })\n predicate4 = Q(**{\n '%s__gte' % date_field: since,\n '%s__lt' % date_field: until,\n '%s__gte' % end_date_field: until\n })\n predicate5 = Q(**{\n '%s__lt' % date_field: since,\n '%s__gte' % end_date_field: until\n })\n return qs.filter(predicate1 | predicate2 | predicate3 | predicate4 | predicate5)\n return qs.filter(**{\n '%s__gte' % date_field: since\n })", "response": "Returns a queryset of models for the requested month."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_context_data(self, **kwargs):\n data = super(BaseCalendarMonthView, self).get_context_data(**kwargs)\n\n year = self.get_year()\n month = self.get_month()\n\n date = _date_from_string(year, self.get_year_format(),\n month, self.get_month_format())\n\n cal = Calendar(self.get_first_of_week())\n\n month_calendar = []\n now = datetime.datetime.utcnow()\n\n date_lists = defaultdict(list)\n multidate_objs = []\n\n for obj in data['object_list']:\n obj_date = self.get_start_date(obj)\n end_date_field = self.get_end_date_field()\n\n if end_date_field:\n end_date = self.get_end_date(obj)\n if end_date and end_date != obj_date:\n multidate_objs.append({\n 'obj': obj,\n 'range': [x for x in daterange(obj_date, end_date)]\n })\n continue # We don't put multi-day events in date_lists\n date_lists[obj_date].append(obj)\n\n for week in cal.monthdatescalendar(date.year, date.month):\n week_range = set(daterange(week[0], week[6]))\n week_events = []\n\n for val in multidate_objs:\n intersect_length = len(week_range.intersection(val['range']))\n\n if intersect_length:\n # Event happens during this week\n slot = 1\n width = intersect_length # How many days is the event during this week?\n nowrap_previous = True # Does the event continue from the previous week?\n nowrap_next = True # Does the event continue to the next week?\n\n if val['range'][0] >= week[0]:\n slot = 1 + (val['range'][0] - week[0]).days\n else:\n nowrap_previous = False\n if val['range'][-1] > week[6]:\n nowrap_next = False\n\n week_events.append({\n 'event': val['obj'],\n 'slot': slot,\n 'width': width,\n 'nowrap_previous': nowrap_previous,\n 'nowrap_next': nowrap_next,\n })\n\n week_calendar = {\n 'events': week_events,\n 'date_list': [],\n }\n for day in week:\n week_calendar['date_list'].append({\n 'day': day,\n 'events': date_lists[day],\n 'today': day == now.date(),\n 'is_current_month': day.month == date.month,\n })\n month_calendar.append(week_calendar)\n\n data['calendar'] = month_calendar\n data['weekdays'] = [DAYS[x] for x in cal.iterweekdays()]\n data['month'] = date\n data['next_month'] = self.get_next_month(date)\n data['previous_month'] = self.get_previous_month(date)\n\n return data", "response": "Returns the context data for the calendar."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_version():\n finder = VersionFinder()\n path = os.path.join(PROJECT_ROOT, 'colorful', '__init__.py')\n with codecs.open(path, 'r', encoding='utf-8') as fp:\n file_data = fp.read().encode('utf-8')\n finder.visit(ast.parse(file_data))\n\n return finder.version", "response": "Read version from __init__. py without loading any files"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef with_setup(self, colormode=None, colorpalette=None, extend_colors=False):\n colorful = Colorful(\n colormode=self.colorful.colormode,\n colorpalette=copy.copy(self.colorful.colorpalette)\n )\n\n colorful.setup(\n colormode=colormode, colorpalette=colorpalette, extend_colors=extend_colors\n )\n yield colorful", "response": "Return a new Colorful object with the given color config."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse the given color files.", "response": "def parse_colors(path):\n \"\"\"Parse the given color files.\n\n Supported are:\n * .txt for X11 colors\n * .json for colornames\n \"\"\"\n if path.endswith(\".txt\"):\n return parse_rgb_txt_file(path)\n elif path.endswith(\".json\"):\n return parse_json_color_file(path)\n\n raise TypeError(\"colorful only supports .txt and .json files for colors\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_rgb_txt_file(path):\n #: Holds the generated color dict\n color_dict = {}\n\n with open(path, 'r') as rgb_txt:\n for line in rgb_txt:\n line = line.strip()\n if not line or line.startswith('!'):\n continue # skip comments\n\n parts = line.split()\n color_dict[\" \".join(parts[3:])] = (int(parts[0]), int(parts[1]), int(parts[2]))\n\n return color_dict", "response": "Parse the given rgb. txt file into a Python dict."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_json_color_file(path):\n with open(path, \"r\") as color_file:\n color_list = json.load(color_file)\n\n # transform raw color list into color dict\n color_dict = {c[\"name\"]: c[\"hex\"] for c in color_list}\n return color_dict", "response": "Parse a JSON color file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sanitize_color_palette(colorpalette):\n new_palette = {}\n\n def __make_valid_color_name(name):\n \"\"\"\n Convert the given name into a valid colorname\n \"\"\"\n if len(name) == 1:\n name = name[0]\n return name[:1].lower() + name[1:]\n\n return name[0].lower() + ''.join(word.capitalize() for word in name[1:])\n\n for key, value in colorpalette.items():\n if isinstance(value, str):\n # we assume it's a hex RGB value\n value = utils.hex_to_rgb(value)\n new_palette[__make_valid_color_name(key.split())] = value\n\n return new_palette", "response": "Sanitize the given color palette so it can be safely used by Colorful."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nshows the modifiers and colors of the current language.", "response": "def show():\n \"\"\"\n Show the modifiers and colors\n \"\"\"\n # modifiers\n sys.stdout.write(colorful.bold('bold') + ' ')\n sys.stdout.write(colorful.dimmed('dimmed') + ' ')\n sys.stdout.write(colorful.italic('italic') + ' ')\n sys.stdout.write(colorful.underlined('underlined') + ' ')\n sys.stdout.write(colorful.inversed('inversed') + ' ')\n sys.stdout.write(colorful.concealed('concealed') + ' ')\n sys.stdout.write(colorful.struckthrough('struckthrough') + '\\n')\n\n # foreground colors\n sys.stdout.write(colorful.red('red') + ' ')\n sys.stdout.write(colorful.green('green') + ' ')\n sys.stdout.write(colorful.yellow('yellow') + ' ')\n sys.stdout.write(colorful.blue('blue') + ' ')\n sys.stdout.write(colorful.magenta('magenta') + ' ')\n sys.stdout.write(colorful.cyan('cyan') + ' ')\n sys.stdout.write(colorful.white('white') + '\\n')\n\n # background colors\n sys.stdout.write(colorful.on_red('red') + ' ')\n sys.stdout.write(colorful.on_green('green') + ' ')\n sys.stdout.write(colorful.on_yellow('yellow') + ' ')\n sys.stdout.write(colorful.on_blue('blue') + ' ')\n sys.stdout.write(colorful.on_magenta('magenta') + ' ')\n sys.stdout.write(colorful.on_cyan('cyan') + ' ')\n sys.stdout.write(colorful.on_white('white') + '\\n')"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nshows the modifiers and colors of the current language.", "response": "def show():\n \"\"\"\n Show the modifiers and colors\n \"\"\"\n with colorful.with_style('monokai') as c:\n # modifiers\n sys.stdout.write(c.bold('bold') + ' ')\n sys.stdout.write(c.dimmed('dimmed') + ' ')\n sys.stdout.write(c.italic('italic') + ' ')\n sys.stdout.write(c.underlined('underlined') + ' ')\n sys.stdout.write(c.inversed('inversed') + ' ')\n sys.stdout.write(c.concealed('concealed') + ' ')\n sys.stdout.write(c.struckthrough('struckthrough') + '\\n')\n\n # foreground colors\n sys.stdout.write(c.orange('orange') + ' ')\n sys.stdout.write(c.magenta('magenta') + ' ')\n sys.stdout.write(c.purple('purple') + ' ')\n sys.stdout.write(c.blue('blue') + ' ')\n sys.stdout.write(c.seaGreen('sea green') + ' ')\n sys.stdout.write(c.green('green') + ' ')\n sys.stdout.write(c.yellow('yellow') + '\\n')\n\n # background colors\n sys.stdout.write(c.on_orange('orange') + ' ')\n sys.stdout.write(c.on_magenta('magenta') + ' ')\n sys.stdout.write(c.on_purple('purple') + ' ')\n sys.stdout.write(c.on_blue('blue') + ' ')\n sys.stdout.write(c.on_seaGreen('sea green') + ' ')\n sys.stdout.write(c.gray_on_green('green') + ' ')\n sys.stdout.write(c.gray_on_yellow('yellow') + '\\n')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndetect what color palettes are supported by the current environment.", "response": "def detect_color_support(env): # noqa\n \"\"\"\n Detect what color palettes are supported.\n It'll return a valid color mode to use\n with colorful.\n\n :param dict env: the environment dict like returned by ``os.envion``\n \"\"\"\n if env.get('COLORFUL_DISABLE', '0') == '1':\n return NO_COLORS\n\n if env.get('COLORFUL_FORCE_8_COLORS', '0') == '1':\n return ANSI_8_COLORS\n\n if env.get('COLORFUL_FORCE_16_COLORS', '0') == '1':\n return ANSI_16_COLORS\n\n if env.get('COLORFUL_FORCE_256_COLORS', '0') == '1':\n return ANSI_256_COLORS\n\n if env.get('COLORFUL_FORCE_TRUE_COLORS', '0') == '1':\n return TRUE_COLORS\n\n # if we are not a tty\n if not sys.stdout.isatty():\n return NO_COLORS\n\n colorterm_env = env.get('COLORTERM')\n if colorterm_env:\n if colorterm_env in {'truecolor', '24bit'}:\n return TRUE_COLORS\n\n if colorterm_env in {'8bit'}:\n return ANSI_256_COLORS\n\n termprog_env = env.get('TERM_PROGRAM')\n if termprog_env:\n if termprog_env in {'iTerm.app', 'Hyper'}:\n return TRUE_COLORS\n\n if termprog_env in {'Apple_Terminal'}:\n return ANSI_256_COLORS\n\n term_env = env.get('TERM')\n if term_env:\n if term_env in {'screen-256', 'screen-256color', 'xterm-256', 'xterm-256color'}:\n return ANSI_256_COLORS\n\n if term_env in {'screen', 'xterm', 'vt100', 'color', 'ansi', 'cygwin', 'linux'}:\n return ANSI_16_COLORS\n\n if colorterm_env:\n # if there was no match with $TERM either but we\n # had one with $COLORTERM, we use it!\n return ANSI_16_COLORS\n\n return ANSI_8_COLORS"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert RGB to ANSI 256 color", "response": "def rgb_to_ansi256(r, g, b):\n \"\"\"\n Convert RGB to ANSI 256 color\n \"\"\"\n if r == g and g == b:\n if r < 8:\n return 16\n if r > 248:\n return 231\n\n return round(((r - 8) / 247.0) * 24) + 232\n\n ansi_r = 36 * round(r / 255.0 * 5.0)\n ansi_g = 6 * round(g / 255.0 * 5.0)\n ansi_b = round(b / 255.0 * 5.0)\n ansi = 16 + ansi_r + ansi_g + ansi_b\n return ansi"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconverting RGB to ANSI 16 color", "response": "def rgb_to_ansi16(r, g, b, use_bright=False):\n \"\"\"\n Convert RGB to ANSI 16 color\n \"\"\"\n ansi_b = round(b / 255.0) << 2\n ansi_g = round(g / 255.0) << 1\n ansi_r = round(r / 255.0)\n ansi = (90 if use_bright else 30) + (ansi_b | ansi_g | ansi_r)\n\n return ansi"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts the given hex string to a valid RGB channel triplet.", "response": "def hex_to_rgb(value):\n \"\"\"\n Convert the given hex string to a\n valid RGB channel triplet.\n \"\"\"\n value = value.lstrip('#')\n check_hex(value)\n\n length = len(value)\n step = int(length / 3)\n return tuple(int(value[i:i+step], 16) for i in range(0, length, step))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_hex(value):\n length = len(value)\n if length not in (3, 6):\n raise ValueError('Hex string #{} is too long'.format(value))\n\n regex = r'[0-9a-f]{{{length}}}'.format(length=length)\n if not re.search(regex, value, re.I):\n raise ValueError('Invalid Hex String: #{}'.format(value))", "response": "Check if the given hex value is a valid RGB color\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef show():\n with colorful.with_style('solarized') as c:\n # modifiers\n sys.stdout.write(c.bold('bold') + ' ')\n sys.stdout.write(c.dimmed('dimmed') + ' ')\n sys.stdout.write(c.italic('italic') + ' ')\n sys.stdout.write(c.underlined('underlined') + ' ')\n sys.stdout.write(c.inversed('inversed') + ' ')\n sys.stdout.write(c.concealed('concealed') + ' ')\n sys.stdout.write(c.struckthrough('struckthrough') + '\\n')\n\n # foreground colors\n sys.stdout.write(c.yellow('yellow') + ' ')\n sys.stdout.write(c.red('orange') + ' ')\n sys.stdout.write(c.red('red') + ' ')\n sys.stdout.write(c.magenta('magenta') + ' ')\n sys.stdout.write(c.magenta('violet') + ' ')\n sys.stdout.write(c.blue('blue') + ' ')\n sys.stdout.write(c.cyan('cyan') + ' ')\n sys.stdout.write(c.green('green') + '\\n')\n\n # background colors\n sys.stdout.write(c.on_yellow('yellow') + ' ')\n sys.stdout.write(c.on_red('orange') + ' ')\n sys.stdout.write(c.on_red('red') + ' ')\n sys.stdout.write(c.on_magenta('magenta') + ' ')\n sys.stdout.write(c.on_magenta('violet') + ' ')\n sys.stdout.write(c.on_blue('blue') + ' ')\n sys.stdout.write(c.on_cyan('cyan') + ' ')\n sys.stdout.write(c.on_green('green') + '\\n')", "response": "Show the modifiers and colors of the current language."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef translate_rgb_to_ansi_code(red, green, blue, offset, colormode):\n if colormode == terminal.NO_COLORS: # colors are disabled, thus return empty string\n return '', ''\n\n if colormode == terminal.ANSI_8_COLORS or colormode == terminal.ANSI_16_COLORS:\n color_code = ansi.rgb_to_ansi16(red, green, blue)\n start_code = ansi.ANSI_ESCAPE_CODE.format(\n code=color_code + offset - ansi.FOREGROUND_COLOR_OFFSET)\n end_code = ansi.ANSI_ESCAPE_CODE.format(code=offset + ansi.COLOR_CLOSE_OFFSET)\n return start_code, end_code\n\n if colormode == terminal.ANSI_256_COLORS:\n color_code = ansi.rgb_to_ansi256(red, green, blue)\n start_code = ansi.ANSI_ESCAPE_CODE.format(code='{base};5;{code}'.format(\n base=8 + offset, code=color_code))\n end_code = ansi.ANSI_ESCAPE_CODE.format(code=offset + ansi.COLOR_CLOSE_OFFSET)\n return start_code, end_code\n\n if colormode == terminal.TRUE_COLORS:\n start_code = ansi.ANSI_ESCAPE_CODE.format(code='{base};2;{red};{green};{blue}'.format(\n base=8 + offset, red=red, green=green, blue=blue))\n end_code = ansi.ANSI_ESCAPE_CODE.format(code=offset + ansi.COLOR_CLOSE_OFFSET)\n return start_code, end_code\n\n raise ColorfulError('invalid color mode \"{0}\"'.format(colormode))", "response": "Translate the given RGB color into the appropriate ANSI escape code."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntranslating a given color name to an ANSI escape code.", "response": "def translate_colorname_to_ansi_code(colorname, offset, colormode, colorpalette):\n \"\"\"\n Translate the given color name to a valid\n ANSI escape code.\n\n :parma str colorname: the name of the color to resolve\n :parma str offset: the offset for the color code\n :param int colormode: the color mode to use. See ``translate_rgb_to_ansi_code``\n :parma dict colorpalette: the color palette to use for the color name mapping\n\n :returns str: the color as ANSI escape code\n\n :raises ColorfulError: if the given color name is invalid\n \"\"\"\n try:\n red, green, blue = colorpalette[colorname]\n except KeyError:\n raise ColorfulError('the color \"{0}\" is unknown. Use a color in your color palette (by default: X11 rgb.txt)'.format( # noqa\n colorname))\n else:\n return translate_rgb_to_ansi_code(red, green, blue, offset, colormode)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef resolve_modifier_to_ansi_code(modifiername, colormode):\n if colormode == terminal.NO_COLORS: # return empty string if colors are disabled\n return '', ''\n\n try:\n start_code, end_code = ansi.MODIFIERS[modifiername]\n except KeyError:\n raise ColorfulError('the modifier \"{0}\" is unknown. Use one of: {1}'.format(\n modifiername, ansi.MODIFIERS.keys()))\n else:\n return ansi.ANSI_ESCAPE_CODE.format(\n code=start_code), ansi.ANSI_ESCAPE_CODE.format(\n code=end_code)", "response": "Resolves the given modifier name to a valid ANSI escape code."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef translate_style(style, colormode, colorpalette):\n style_parts = iter(style.split('_'))\n\n ansi_start_sequence = []\n ansi_end_sequence = []\n\n try:\n # consume all modifiers\n part = None\n for mod_part in style_parts:\n part = mod_part\n if part not in ansi.MODIFIERS:\n break # all modifiers have been consumed\n\n mod_start_code, mod_end_code = resolve_modifier_to_ansi_code(part, colormode)\n ansi_start_sequence.append(mod_start_code)\n ansi_end_sequence.append(mod_end_code)\n else: # we've consumed all parts, thus we can exit\n raise StopIteration()\n\n # next part has to be a foreground color or the 'on' keyword\n # which means we have to consume background colors\n if part != 'on':\n ansi_start_code, ansi_end_code = translate_colorname_to_ansi_code(\n part, ansi.FOREGROUND_COLOR_OFFSET, colormode, colorpalette)\n ansi_start_sequence.append(ansi_start_code)\n ansi_end_sequence.append(ansi_end_code)\n # consume the required 'on' keyword after the foreground color\n next(style_parts)\n\n # next part has to be the background color\n part = next(style_parts)\n ansi_start_code, ansi_end_code = translate_colorname_to_ansi_code(\n part, ansi.BACKGROUND_COLOR_OFFSET, colormode, colorpalette)\n ansi_start_sequence.append(ansi_start_code)\n ansi_end_sequence.append(ansi_end_code)\n except StopIteration: # we've consumed all parts of the styling string\n pass\n\n # construct and return ANSI escape code sequence\n return ''.join(ansi_start_sequence), ''.join(ansi_end_sequence)", "response": "Translate the given style to an ANSI escape code."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a string that is a style of the given string according to the given ANSI style string.", "response": "def style_string(string, ansi_style, colormode, nested=False):\n \"\"\"\n Style the given string according to the given\n ANSI style string.\n\n :param str string: the string to style\n :param tuple ansi_style: the styling string returned by ``translate_style``\n :param int colormode: the color mode to use. See ``translate_rgb_to_ansi_code``\n\n :returns: a string containing proper ANSI sequence\n \"\"\"\n ansi_start_code, ansi_end_code = ansi_style\n\n # replace nest placeholders with the current begin style\n if PY2:\n if isinstance(string, str):\n string = string.decode(DEFAULT_ENCODING)\n string = UNICODE(string).replace(ansi.NEST_PLACEHOLDER, ansi_start_code)\n\n return '{start_code}{string}{end_code}{nest_ph}'.format(\n start_code=ansi_start_code,\n string=string,\n end_code=ansi_end_code,\n nest_ph=ansi.NEST_PLACEHOLDER if nested else '')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef colorpalette(self, colorpalette):\n if isinstance(colorpalette, str): # we assume it's a path to a color file\n colorpalette = colors.parse_colors(colorpalette)\n\n self._colorpalette = colors.sanitize_color_palette(colorpalette)", "response": "Set the colorpalette which should be used by the log file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef setup(self, colormode=None, colorpalette=None, extend_colors=False):\n if colormode:\n self.colormode = colormode\n\n if colorpalette:\n if extend_colors:\n self.update_palette(colorpalette)\n else:\n self.colorpalette = colorpalette", "response": "Setup this colorful object by setting a colormode and colorpalette."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nuses a predefined style as color palette", "response": "def use_style(self, style_name):\n \"\"\"\n Use a predefined style as color palette\n\n :param str style_name: the name of the style\n \"\"\"\n try:\n style = getattr(styles, style_name.upper())\n except AttributeError:\n raise ColorfulError('the style \"{0}\" is undefined'.format(\n style_name))\n else:\n self.colorpalette = style"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef format(self, string, *args, **kwargs):\n return string.format(c=self, *args, **kwargs)", "response": "Format the given string with the given args and kwargs."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef print(self, *objects, **options):\n # NOTE: change signature to same as print() built-in function as\n # soon as Python 2.7 is not supported anymore.\n # If causes problems because of the keyword args after *args\n allowed_options = {'sep', 'end', 'file', 'flush'}\n given_options = set(options.keys())\n if not given_options.issubset(allowed_options):\n raise TypeError('Colorful.print() got unexpected keyword arguments: {0}'.format(\n ', '.join(given_options.difference(allowed_options))))\n\n sep = options.get('sep', ' ')\n end = options.get('end', '\\n')\n file = options.get('file', sys.stdout)\n flush = options.get('flush', False)\n\n styled_objects = [self.format(o) for o in objects]\n print(*styled_objects, sep=sep, end=end, file=file)\n\n # NOTE: if Python 2.7 support is dropped we can directly forward the\n # flush keyword argument to the print() function.\n if flush:\n file.flush()", "response": "Print the given objects to the given file stream."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nread attribute from sysfs and return as string", "response": "def readattr(path, name):\n \"\"\"\n Read attribute from sysfs and return as string\n \"\"\"\n try:\n f = open(USB_SYS_PREFIX + path + \"/\" + name)\n return f.readline().rstrip(\"\\n\")\n except IOError:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef find_ports(device):\n bus_id = device.bus\n dev_id = device.address\n for dirent in os.listdir(USB_SYS_PREFIX):\n matches = re.match(USB_PORTS_STR + '$', dirent)\n if matches:\n bus_str = readattr(dirent, 'busnum')\n if bus_str:\n busnum = float(bus_str)\n else:\n busnum = None\n dev_str = readattr(dirent, 'devnum')\n if dev_str:\n devnum = float(dev_str)\n else:\n devnum = None\n if busnum == bus_id and devnum == dev_id:\n return str(matches.groups()[1])", "response": "Find the port chain a device is plugged on."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets device calibration data based on settings in the temper. conf file.", "response": "def set_calibration_data(self, scale=None, offset=None):\n \"\"\"\n Set device calibration data based on settings in /etc/temper.conf.\n \"\"\"\n if scale is not None and offset is not None:\n self._scale = scale\n self._offset = offset\n elif scale is None and offset is None:\n self._scale = 1.0\n self._offset = 0.0\n try:\n f = open('/etc/temper.conf', 'r')\n except IOError:\n f = None\n if f:\n lines = f.read().split('\\n')\n f.close()\n for line in lines:\n matches = re.match(CALIB_LINE_STR, line)\n if matches:\n bus = int(matches.groups()[0])\n ports = matches.groups()[1]\n scale = float(matches.groups()[2])\n offset = float(matches.groups()[3])\n if (str(ports) == str(self._ports)) and (str(bus) == str(self._bus)):\n self._scale = scale\n self._offset = offset\n else:\n raise RuntimeError(\"Must set both scale and offset, or neither\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting data from the USB device.", "response": "def get_data(self, reset_device=False):\n \"\"\"\n Get data from the USB device.\n \"\"\"\n try:\n if reset_device:\n self._device.reset()\n\n # detach kernel driver from both interfaces if attached, so we can set_configuration()\n for interface in [0,1]:\n if self._device.is_kernel_driver_active(interface):\n LOGGER.debug('Detaching kernel driver for interface %d '\n 'of %r on ports %r', interface, self._device, self._ports)\n self._device.detach_kernel_driver(interface)\n\n self._device.set_configuration()\n\n # Prevent kernel message:\n # \"usbfs: process (python) did not claim interface x before use\"\n # This will become unnecessary once pull-request #124 for\n # PyUSB has been accepted and we depend on a fixed release\n # of PyUSB. Until then, and even with the fix applied, it\n # does not hurt to explicitly claim the interface.\n usb.util.claim_interface(self._device, INTERFACE)\n\n # Turns out we don't actually need that ctrl_transfer.\n # Disabling this reduces number of USBErrors from ~7/30 to 0!\n #self._device.ctrl_transfer(bmRequestType=0x21, bRequest=0x09,\n # wValue=0x0201, wIndex=0x00, data_or_wLength='\\x01\\x01',\n # timeout=TIMEOUT)\n\n\n # Magic: Our TEMPerV1.4 likes to be asked twice. When\n # only asked once, it get's stuck on the next access and\n # requires a reset.\n self._control_transfer(COMMANDS['temp'])\n self._interrupt_read()\n\n # Turns out a whole lot of that magic seems unnecessary.\n #self._control_transfer(COMMANDS['ini1'])\n #self._interrupt_read()\n #self._control_transfer(COMMANDS['ini2'])\n #self._interrupt_read()\n #self._interrupt_read()\n\n # Get temperature\n self._control_transfer(COMMANDS['temp'])\n temp_data = self._interrupt_read()\n\n # Get humidity\n if self._device.product == 'TEMPer1F_H1_V1.4':\n humidity_data = temp_data\n else:\n humidity_data = None\n\n # Combine temperature and humidity data\n data = {'temp_data': temp_data, 'humidity_data': humidity_data}\n\n # Be a nice citizen and undo potential interface claiming.\n # Also see: https://github.com/walac/pyusb/blob/master/docs/tutorial.rst#dont-be-selfish\n usb.util.dispose_resources(self._device)\n return data\n except usb.USBError as err:\n if not reset_device:\n LOGGER.warning(\"Encountered %s, resetting %r and trying again.\", err, self._device)\n return self.get_data(True)\n\n # Catch the permissions exception and add our message\n if \"not permitted\" in str(err):\n raise Exception(\n \"Permission problem accessing USB. \"\n \"Maybe I need to run as root?\")\n else:\n LOGGER.error(err)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_temperature(self, format='celsius', sensor=0):\n results = self.get_temperatures(sensors=[sensor,])\n\n if format == 'celsius':\n return results[sensor]['temperature_c']\n elif format == 'fahrenheit':\n return results[sensor]['temperature_f']\n elif format == 'millicelsius':\n return results[sensor]['temperature_mc']\n else:\n raise ValueError(\"Unknown format\")", "response": "Get the temperature in the specified format."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the temperature of a specific sensor.", "response": "def get_temperatures(self, sensors=None):\n \"\"\"\n Get device temperature reading.\n\n Params:\n - sensors: optional list of sensors to get a reading for, examples:\n [0,] - get reading for sensor 0\n [0, 1,] - get reading for sensors 0 and 1\n None - get readings for all sensors\n \"\"\"\n _sensors = sensors\n if _sensors is None:\n _sensors = list(range(0, self._sensor_count))\n\n if not set(_sensors).issubset(list(range(0, self._sensor_count))):\n raise ValueError(\n 'Some or all of the sensors in the list %s are out of range '\n 'given a sensor_count of %d. Valid range: %s' % (\n _sensors,\n self._sensor_count,\n list(range(0, self._sensor_count)),\n )\n )\n\n data = self.get_data()\n data = data['temp_data']\n\n results = {}\n\n # Interpret device response\n for sensor in _sensors:\n offset = self.lookup_offset(sensor)\n celsius = struct.unpack_from('>h', data, offset)[0] / 256.0\n # Apply scaling and offset (if any)\n celsius = celsius * self._scale + self._offset\n results[sensor] = {\n 'ports': self.get_ports(),\n 'bus': self.get_bus(),\n 'sensor': sensor,\n 'temperature_f': celsius * 1.8 + 32.0,\n 'temperature_c': celsius,\n 'temperature_mc': celsius * 1000,\n 'temperature_k': celsius + 273.15,\n }\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the device humidity reading.", "response": "def get_humidity(self, sensors=None):\n \"\"\"\n Get device humidity reading.\n\n Params:\n - sensors: optional list of sensors to get a reading for, examples:\n [0,] - get reading for sensor 0\n [0, 1,] - get reading for sensors 0 and 1\n None - get readings for all sensors\n \"\"\"\n _sensors = sensors\n if _sensors is None:\n _sensors = list(range(0, self._sensor_count))\n\n if not set(_sensors).issubset(list(range(0, self._sensor_count))):\n raise ValueError(\n 'Some or all of the sensors in the list %s are out of range '\n 'given a sensor_count of %d. Valid range: %s' % (\n _sensors,\n self._sensor_count,\n list(range(0, self._sensor_count)),\n )\n )\n\n data = self.get_data()\n data = data['humidity_data']\n\n results = {}\n\n # Interpret device response\n for sensor in _sensors:\n offset = self.lookup_humidity_offset(sensor)\n if offset is None:\n continue\n humidity = (struct.unpack_from('>H', data, offset)[0] * 32) / 1000.0\n results[sensor] = {\n 'ports': self.get_ports(),\n 'bus': self.get_bus(),\n 'sensor': sensor,\n 'humidity_pc': humidity,\n }\n\n return results"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _control_transfer(self, data):\n LOGGER.debug('Ctrl transfer: %r', data)\n self._device.ctrl_transfer(bmRequestType=0x21, bRequest=0x09,\n wValue=0x0200, wIndex=0x01, data_or_wLength=data, timeout=TIMEOUT)", "response": "Send a control transfer request to the device."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _interrupt_read(self):\n data = self._device.read(ENDPOINT, REQ_INT_LEN, timeout=TIMEOUT)\n LOGGER.debug('Read data: %r', data)\n return data", "response": "Read data from device."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef __check_looks_like_uri(self, uri):\n if uri.split('/')[2] == 'raw.githubusercontent.com':\n return True\n elif uri.split('/')[2] == 'github.com':\n if uri.split('/')[5] == 'raw':\n return True\n else:\n raise GithubFileNotFound('URI %s is not a valid link to a raw file in Github' % uri)", "response": "Checks if the URI looks like a RAW uri in Github"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread the file from Github and returns the content", "response": "def read_file_from_uri(self, uri):\n \"\"\"Reads the file from Github\n\n :param uri: URI of the Github raw File\n\n :returns: UTF-8 text with the content\n \"\"\"\n logger.debug(\"Reading %s\" % (uri))\n\n self.__check_looks_like_uri(uri)\n\n try:\n req = urllib.request.Request(uri)\n req.add_header('Authorization', 'token %s' % self.token)\n r = urllib.request.urlopen(req)\n except urllib.error.HTTPError as err:\n if err.code == 404:\n raise GithubFileNotFound('File %s is not available. Check the URL to ensure it really exists' % uri)\n else:\n raise\n\n return r.read().decode(\"utf-8\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef measure_memory(cls, obj, seen=None):\n size = sys.getsizeof(obj)\n if seen is None:\n seen = set()\n obj_id = id(obj)\n if obj_id in seen:\n return 0\n # Important mark as seen *before* entering recursion to gracefully handle\n # self-referential objects\n seen.add(obj_id)\n if isinstance(obj, dict):\n size += sum([cls.measure_memory(v, seen) for v in obj.values()])\n size += sum([cls.measure_memory(k, seen) for k in obj.keys()])\n elif hasattr(obj, '__dict__'):\n size += cls.measure_memory(obj.__dict__, seen)\n elif hasattr(obj, '__iter__') and not isinstance(obj, (str, bytes, bytearray)):\n size += sum([cls.measure_memory(i, seen) for i in obj])\n return size", "response": "Recursively finds the size of objects and returns it."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __feed_arthur(self):\n\n with self.ARTHUR_FEED_LOCK:\n\n # This is a expensive operation so don't do it always\n if (time.time() - self.ARTHUR_LAST_MEMORY_CHECK) > 5 * self.ARTHUR_LAST_MEMORY_CHECK_TIME:\n self.ARTHUR_LAST_MEMORY_CHECK = time.time()\n logger.debug(\"Measuring the memory used by the raw items dict ...\")\n try:\n memory_size = self.measure_memory(self.arthur_items) / (1024 * 1024)\n except RuntimeError as ex:\n # During memory usage measure, other thread could change the dict\n logger.warning(\"Can't get the memory used by the raw items dict: %s\", ex)\n memory_size = self.ARTHUR_LAST_MEMORY_SIZE\n self.ARTHUR_LAST_MEMORY_CHECK_TIME = time.time() - self.ARTHUR_LAST_MEMORY_CHECK\n logger.debug(\"Arthur items memory size: %0.2f MB (%is to check)\",\n memory_size, self.ARTHUR_LAST_MEMORY_CHECK_TIME)\n self.ARTHUR_LAST_MEMORY_SIZE = memory_size\n\n # Don't feed items from redis if the current python dict is\n # larger than ARTHUR_MAX_MEMORY_SIZE\n\n if self.ARTHUR_LAST_MEMORY_SIZE > self.ARTHUR_MAX_MEMORY_SIZE:\n logger.debug(\"Items queue full. Not collecting items from redis queue.\")\n return\n\n logger.info(\"Collecting items from redis queue\")\n\n db_url = self.config.get_conf()['es_collection']['redis_url']\n\n conn = redis.StrictRedis.from_url(db_url)\n logger.debug(\"Redis connection stablished with %s.\", db_url)\n\n # Get and remove queued items in an atomic transaction\n pipe = conn.pipeline()\n # pipe.lrange(Q_STORAGE_ITEMS, 0, -1)\n pipe.lrange(Q_STORAGE_ITEMS, 0, self.ARTHUR_REDIS_ITEMS - 1)\n pipe.ltrim(Q_STORAGE_ITEMS, self.ARTHUR_REDIS_ITEMS, -1)\n items = pipe.execute()[0]\n\n for item in items:\n arthur_item = pickle.loads(item)\n if arthur_item['tag'] not in self.arthur_items:\n self.arthur_items[arthur_item['tag']] = []\n self.arthur_items[arthur_item['tag']].append(arthur_item)\n\n for tag in self.arthur_items:\n if self.arthur_items[tag]:\n logger.debug(\"Arthur items for %s: %i\", tag, len(self.arthur_items[tag]))", "response": "Feed items from arthur redis queue"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfeeds items from arthur redis queue with backend data collected from arthur redis queue", "response": "def __feed_backend_arthur(self, repo):\n \"\"\" Feed Ocean with backend data collected from arthur redis queue\"\"\"\n\n # Always get pending items from arthur for all data sources\n self.__feed_arthur()\n\n tag = self.backend_tag(repo)\n\n logger.debug(\"Arthur items available for %s\", self.arthur_items.keys())\n\n logger.debug(\"Getting arthur items for %s.\", tag)\n\n if tag in self.arthur_items:\n logger.debug(\"Found items for %s.\", tag)\n while self.arthur_items[tag]:\n yield self.arthur_items[tag].pop()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates the JSON for configuring arthur data", "response": "def __create_arthur_json(self, repo, backend_args):\n \"\"\" Create the JSON for configuring arthur to collect data\n\n https://github.com/grimoirelab/arthur#adding-tasks\n Sample for git:\n\n {\n \"tasks\": [\n {\n \"task_id\": \"arthur.git\",\n \"backend\": \"git\",\n \"backend_args\": {\n \"gitpath\": \"/tmp/arthur_git/\",\n \"uri\": \"https://github.com/grimoirelab/arthur.git\"\n },\n \"category\": \"commit\",\n \"archive_args\": {\n \"archive_path\": '/tmp/test_archives',\n \"fetch_from_archive\": false,\n \"archive_after\": None\n },\n \"scheduler_args\": {\n \"delay\": 10\n }\n }\n ]\n }\n \"\"\"\n\n backend_args = self._compose_arthur_params(self.backend_section, repo)\n if self.backend_section == 'git':\n backend_args['gitpath'] = os.path.join(self.REPOSITORY_DIR, repo)\n backend_args['tag'] = self.backend_tag(repo)\n\n ajson = {\"tasks\": [{}]}\n # This is the perceval tag\n ajson[\"tasks\"][0]['task_id'] = self.backend_tag(repo)\n ajson[\"tasks\"][0]['backend'] = self.backend_section.split(\":\")[0]\n ajson[\"tasks\"][0]['backend_args'] = backend_args\n ajson[\"tasks\"][0]['category'] = backend_args['category']\n ajson[\"tasks\"][0]['archive'] = {}\n ajson[\"tasks\"][0]['scheduler'] = {\"delay\": self.ARTHUR_TASK_DELAY}\n # from-date or offset param must be added\n es_col_url = self._get_collection_url()\n es_index = self.conf[self.backend_section]['raw_index']\n # Get the last activity for the data source\n es = ElasticSearch(es_col_url, es_index)\n connector = get_connector_from_name(self.backend_section)\n\n klass = connector[0] # Backend for the connector\n signature = inspect.signature(klass.fetch)\n\n last_activity = None\n filter_ = {\"name\": \"tag\", \"value\": backend_args['tag']}\n if 'from_date' in signature.parameters:\n last_activity = es.get_last_item_field('metadata__updated_on', [filter_])\n if last_activity:\n ajson[\"tasks\"][0]['backend_args']['from_date'] = last_activity.isoformat()\n elif 'offset' in signature.parameters:\n last_activity = es.get_last_item_field('offset', [filter_])\n if last_activity:\n ajson[\"tasks\"][0]['backend_args']['offset'] = last_activity\n\n if last_activity:\n logging.info(\"Getting raw item with arthur since %s\", last_activity)\n\n return(ajson)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the GitHub SHA for a file in the repository", "response": "def sha_github_file(cls, config, repo_file, repository_api, repository_branch):\n \"\"\" Return the GitHub SHA for a file in the repository \"\"\"\n\n repo_file_sha = None\n\n cfg = config.get_conf()\n github_token = cfg['sortinghat']['identities_api_token']\n headers = {\"Authorization\": \"token \" + github_token}\n\n url_dir = repository_api + \"/git/trees/\" + repository_branch\n logger.debug(\"Gettting sha data from tree: %s\", url_dir)\n raw_repo_file_info = requests.get(url_dir, headers=headers)\n raw_repo_file_info.raise_for_status()\n for rfile in raw_repo_file_info.json()['tree']:\n if rfile['path'] == repo_file:\n logger.debug(\"SHA found: %s, \", rfile[\"sha\"])\n repo_file_sha = rfile[\"sha\"]\n break\n\n return repo_file_sha"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the uuid for a profile name", "response": "def __get_uuids_from_profile_name(self, profile_name):\n \"\"\" Get the uuid for a profile name \"\"\"\n uuids = []\n\n with self.db.connect() as session:\n query = session.query(Profile).\\\n filter(Profile.name == profile_name)\n profiles = query.all()\n if profiles:\n for p in profiles:\n uuids.append(p.uuid)\n return uuids"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef micro_mordred(cfg_path, backend_sections, raw, arthur, identities, enrich, panels):\n\n config = Config(cfg_path)\n\n if raw:\n for backend in backend_sections:\n get_raw(config, backend, arthur)\n\n if identities:\n get_identities(config)\n\n if enrich:\n for backend in backend_sections:\n get_enrich(config, backend)\n\n if panels:\n get_panels(config)", "response": "Execute the raw and enrich phases of a given backend section."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nexecuting the raw phase for a given backend section optionally using Arthur", "response": "def get_raw(config, backend_section, arthur):\n \"\"\"Execute the raw phase for a given backend section, optionally using Arthur\n\n :param config: a Mordred config object\n :param backend_section: the backend section where the raw phase is executed\n :param arthur: if true, it enables Arthur to collect the raw data\n \"\"\"\n\n if arthur:\n task = TaskRawDataArthurCollection(config, backend_section=backend_section)\n else:\n task = TaskRawDataCollection(config, backend_section=backend_section)\n\n TaskProjects(config).execute()\n try:\n task.execute()\n logging.info(\"Loading raw data finished!\")\n except Exception as e:\n logging.error(str(e))\n sys.exit(-1)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_identities(config):\n\n TaskProjects(config).execute()\n task = TaskIdentitiesMerge(config)\n task.execute()\n logging.info(\"Merging identities finished!\")", "response": "Execute the merge identities phase\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_enrich(config, backend_section):\n\n TaskProjects(config).execute()\n task = TaskEnrich(config, backend_section=backend_section)\n try:\n task.execute()\n logging.info(\"Loading enriched data finished!\")\n except Exception as e:\n logging.error(str(e))\n sys.exit(-1)", "response": "Execute the enrich phase for a given backend section"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nexecuting the panels phase", "response": "def get_panels(config):\n \"\"\"Execute the panels phase\n\n :param config: a Mordred config object\n \"\"\"\n\n task = TaskPanels(config)\n task.execute()\n\n task = TaskPanelsMenu(config)\n task.execute()\n\n logging.info(\"Panels creation finished!\")"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconfigure logging level output output", "response": "def config_logging(debug):\n \"\"\"Config logging level output output\"\"\"\n\n if debug:\n logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(message)s')\n logging.debug(\"Debug mode activated\")\n else:\n logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_params_parser():\n\n parser = argparse.ArgumentParser(add_help=False)\n\n parser.add_argument('-g', '--debug', dest='debug',\n action='store_true',\n help=argparse.SUPPRESS)\n parser.add_argument(\"--arthur\", action='store_true', dest='arthur',\n help=\"Enable arthur to collect raw data\")\n parser.add_argument(\"--raw\", action='store_true', dest='raw',\n help=\"Activate raw task\")\n parser.add_argument(\"--enrich\", action='store_true', dest='enrich',\n help=\"Activate enrich task\")\n parser.add_argument(\"--identities\", action='store_true', dest='identities',\n help=\"Activate merge identities task\")\n parser.add_argument(\"--panels\", action='store_true', dest='panels',\n help=\"Activate panels task\")\n\n parser.add_argument(\"--cfg\", dest='cfg_path',\n help=\"Configuration file path\")\n parser.add_argument(\"--backends\", dest='backend_sections', default=[],\n nargs='*', help=\"Backend sections to execute\")\n\n if len(sys.argv) == 1:\n parser.print_help()\n sys.exit(1)\n\n return parser", "response": "Parse command line arguments"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_params():\n\n parser = get_params_parser()\n args = parser.parse_args()\n\n if not args.raw and not args.enrich and not args.identities and not args.panels:\n print(\"No tasks enabled\")\n sys.exit(1)\n\n return args", "response": "Get params to execute the micro - mordred"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __kibiter_version(self):\n version = None\n\n es_url = self.conf['es_enrichment']['url']\n config_url = '.kibana/config/_search'\n url = urijoin(es_url, config_url)\n version = None\n try:\n res = self.grimoire_con.get(url)\n res.raise_for_status()\n version = res.json()['hits']['hits'][0]['_id']\n logger.debug(\"Kibiter version: %s\", version)\n except requests.exceptions.HTTPError:\n logger.warning(\"Can not find Kibiter version\")\n\n return version", "response": "Get the kibiter vesion."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nuploads a panel to Elasticsearch if it does not exist yet.", "response": "def create_dashboard(self, panel_file, data_sources=None, strict=True):\n \"\"\"Upload a panel to Elasticsearch if it does not exist yet.\n\n If a list of data sources is specified, upload only those\n elements (visualizations, searches) that match that data source.\n\n :param panel_file: file name of panel (dashobard) to upload\n :param data_sources: list of data sources\n :param strict: only upload a dashboard if it is newer than the one already existing\n \"\"\"\n es_enrich = self.conf['es_enrichment']['url']\n kibana_url = self.conf['panels']['kibiter_url']\n\n mboxes_sources = set(['pipermail', 'hyperkitty', 'groupsio', 'nntp'])\n if data_sources and any(x in data_sources for x in mboxes_sources):\n data_sources = list(data_sources)\n data_sources.append('mbox')\n if data_sources and ('supybot' in data_sources):\n data_sources = list(data_sources)\n data_sources.append('irc')\n if data_sources and 'google_hits' in data_sources:\n data_sources = list(data_sources)\n data_sources.append('googlehits')\n if data_sources and 'stackexchange' in data_sources:\n # stackexchange is called stackoverflow in panels\n data_sources = list(data_sources)\n data_sources.append('stackoverflow')\n if data_sources and 'phabricator' in data_sources:\n data_sources = list(data_sources)\n data_sources.append('maniphest')\n\n try:\n import_dashboard(es_enrich, kibana_url, panel_file, data_sources=data_sources, strict=strict)\n except ValueError:\n logger.error(\"%s does not include release field. Not loading the panel.\", panel_file)\n except RuntimeError:\n logger.error(\"Can not load the panel %s\", panel_file)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef __upload_title(self, kibiter_major):\n\n if kibiter_major == \"6\":\n resource = \".kibana/doc/projectname\"\n data = {\"projectname\": {\"name\": self.project_name}}\n mapping_resource = \".kibana/_mapping/doc\"\n mapping = {\"dynamic\": \"true\"}\n\n url = urijoin(self.conf['es_enrichment']['url'], resource)\n mapping_url = urijoin(self.conf['es_enrichment']['url'],\n mapping_resource)\n\n logger.debug(\"Adding mapping for dashboard title\")\n res = self.grimoire_con.put(mapping_url, data=json.dumps(mapping),\n headers=ES6_HEADER)\n try:\n res.raise_for_status()\n except requests.exceptions.HTTPError:\n logger.error(\"Couldn't create mapping for dashboard title.\")\n logger.error(res.json())\n\n logger.debug(\"Uploading dashboard title\")\n res = self.grimoire_con.post(url, data=json.dumps(data),\n headers=ES6_HEADER)\n try:\n res.raise_for_status()\n except requests.exceptions.HTTPError:\n logger.error(\"Couldn't create dashboard title.\")\n logger.error(res.json())", "response": "Upload to Kibiter the title for the dashboard."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates the dashboard menu for a Kibiter.", "response": "def __create_dashboard_menu(self, dash_menu, kibiter_major):\n \"\"\"Create the menu definition to access the panels in a dashboard.\n\n :param menu: dashboard menu to upload\n :param kibiter_major: major version of kibiter\n \"\"\"\n logger.info(\"Adding dashboard menu\")\n if kibiter_major == \"6\":\n menu_resource = \".kibana/doc/metadashboard\"\n mapping_resource = \".kibana/_mapping/doc\"\n mapping = {\"dynamic\": \"true\"}\n menu = {'metadashboard': dash_menu}\n else:\n menu_resource = \".kibana/metadashboard/main\"\n mapping_resource = \".kibana/_mapping/metadashboard\"\n mapping = {\"dynamic\": \"true\"}\n menu = dash_menu\n menu_url = urijoin(self.conf['es_enrichment']['url'],\n menu_resource)\n\n mapping_url = urijoin(self.conf['es_enrichment']['url'],\n mapping_resource)\n logger.debug(\"Adding mapping for metadashboard\")\n res = self.grimoire_con.put(mapping_url, data=json.dumps(mapping),\n headers=ES6_HEADER)\n try:\n res.raise_for_status()\n except requests.exceptions.HTTPError:\n logger.error(\"Couldn't create mapping for Kibiter menu.\")\n res = self.grimoire_con.post(menu_url, data=json.dumps(menu),\n headers=ES6_HEADER)\n try:\n res.raise_for_status()\n except requests.exceptions.HTTPError:\n logger.error(\"Couldn't create Kibiter menu.\")\n logger.error(res.json())\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nremoving existing dashboard menu for kibiter.", "response": "def __remove_dashboard_menu(self, kibiter_major):\n \"\"\"Remove existing menu for dashboard, if any.\n\n Usually, we remove the menu before creating a new one.\n\n :param kibiter_major: major version of kibiter\n \"\"\"\n logger.info(\"Removing old dashboard menu, if any\")\n if kibiter_major == \"6\":\n metadashboard = \".kibana/doc/metadashboard\"\n else:\n metadashboard = \".kibana/metadashboard/main\"\n menu_url = urijoin(self.conf['es_enrichment']['url'], metadashboard)\n self.grimoire_con.delete(menu_url)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget the menu entries from the panel definition", "response": "def __get_menu_entries(self, kibiter_major):\n \"\"\" Get the menu entries from the panel definition \"\"\"\n menu_entries = []\n for entry in self.panels_menu:\n if entry['source'] not in self.data_sources:\n continue\n parent_menu_item = {\n 'name': entry['name'],\n 'title': entry['name'],\n 'description': \"\",\n 'type': \"menu\",\n 'dashboards': []\n }\n for subentry in entry['menu']:\n try:\n dash_name = get_dashboard_name(subentry['panel'])\n except FileNotFoundError:\n logging.error(\"Can't open dashboard file %s\", subentry['panel'])\n continue\n # The name for the entry is in self.panels_menu\n child_item = {\n \"name\": subentry['name'],\n \"title\": subentry['name'],\n \"description\": \"\",\n \"type\": \"entry\",\n \"panel_id\": dash_name\n }\n parent_menu_item['dashboards'].append(child_item)\n menu_entries.append(parent_menu_item)\n\n return menu_entries"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the dashboard menu for a given kibiter major number", "response": "def __get_dash_menu(self, kibiter_major):\n \"\"\"Order the dashboard menu\"\"\"\n\n # omenu = OrderedDict()\n omenu = []\n # Start with Overview\n omenu.append(self.menu_panels_common['Overview'])\n\n # Now the data _getsources\n ds_menu = self.__get_menu_entries(kibiter_major)\n\n # Remove the kafka and community menus, they will be included at the end\n kafka_menu = None\n community_menu = None\n\n found_kafka = [pos for pos, menu in enumerate(ds_menu) if menu['name'] == KAFKA_NAME]\n if found_kafka:\n kafka_menu = ds_menu.pop(found_kafka[0])\n\n found_community = [pos for pos, menu in enumerate(ds_menu) if menu['name'] == COMMUNITY_NAME]\n if found_community:\n community_menu = ds_menu.pop(found_community[0])\n\n ds_menu.sort(key=operator.itemgetter('name'))\n omenu += ds_menu\n\n # If kafka and community are present add them before the Data Status and About\n if kafka_menu:\n omenu.append(kafka_menu)\n\n if community_menu:\n omenu.append(community_menu)\n\n # At the end Data Status, About\n omenu.append(self.menu_panels_common['Data Status'])\n omenu.append(self.menu_panels_common['About'])\n\n logger.debug(\"Menu for panels: %s\", json.dumps(ds_menu, indent=4))\n return omenu"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef compose_mbox(projects):\n mbox_archives = '/home/bitergia/mboxes'\n\n mailing_lists_projects = [project for project in projects if 'mailing_lists' in projects[project]]\n for mailing_lists in mailing_lists_projects:\n projects[mailing_lists]['mbox'] = []\n for mailing_list in projects[mailing_lists]['mailing_lists']:\n if 'listinfo' in mailing_list:\n name = mailing_list.split('listinfo/')[1]\n elif 'mailing-list' in mailing_list:\n name = mailing_list.split('mailing-list/')[1]\n else:\n name = mailing_list.split('@')[0]\n\n list_new = \"%s %s/%s.mbox/%s.mbox\" % (name, mbox_archives, name, name)\n projects[mailing_lists]['mbox'].append(list_new)\n\n return projects", "response": "Compose projects. json only for mbox but using the mailing_lists lists\n\n change to emfg - dev"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomposing projects. json for gerrit but using the git lists change the git lists to the git root", "response": "def compose_gerrit(projects):\n \"\"\" Compose projects.json for gerrit, but using the git lists\n\n change: 'http://git.eclipse.org/gitroot/xwt/org.eclipse.xwt.git'\n to: 'git.eclipse.org_xwt/org.eclipse.xwt\n\n :param projects: projects.json\n :return: projects.json with gerrit\n \"\"\"\n git_projects = [project for project in projects if 'git' in projects[project]]\n for project in git_projects:\n repos = [repo for repo in projects[project]['git'] if 'gitroot' in repo]\n if len(repos) > 0:\n projects[project]['gerrit'] = []\n for repo in repos:\n gerrit_project = repo.replace(\"http://git.eclipse.org/gitroot/\", \"\")\n gerrit_project = gerrit_project.replace(\".git\", \"\")\n projects[project]['gerrit'].append(\"git.eclipse.org_\" + gerrit_project)\n\n return projects"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompose projects. json for git", "response": "def compose_git(projects, data):\n \"\"\" Compose projects.json for git\n\n We need to replace '/c/' by '/gitroot/' for instance\n\n change: 'http://git.eclipse.org/c/xwt/org.eclipse.xwt.git'\n to: 'http://git.eclipse.org/gitroot/xwt/org.eclipse.xwt.git'\n\n :param projects: projects.json\n :param data: eclipse JSON\n :return: projects.json with git\n \"\"\"\n for p in [project for project in data if len(data[project]['source_repo']) > 0]:\n repos = []\n for url in data[p]['source_repo']:\n if len(url['url'].split()) > 1: # Error at upstream the project 'tools.corrosion'\n repo = url['url'].split()[1].replace('/c/', '/gitroot/')\n else:\n repo = url['url'].replace('/c/', '/gitroot/')\n\n if repo not in repos:\n repos.append(repo)\n\n projects[p]['git'] = repos\n\n return projects"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncomposes projects. json for mailing lists at upstream", "response": "def compose_mailing_lists(projects, data):\n \"\"\" Compose projects.json for mailing lists\n\n At upstream has two different key for mailing list: 'mailings_lists' and 'dev_list'\n The key 'mailing_lists' is an array with mailing lists\n The key 'dev_list' is a dict with only one mailing list\n\n :param projects: projects.json\n :param data: eclipse JSON\n :return: projects.json with mailing_lists\n \"\"\"\n for p in [project for project in data if len(data[project]['mailing_lists']) > 0]:\n if 'mailing_lists' not in projects[p]:\n projects[p]['mailing_lists'] = []\n\n urls = [url['url'].replace('mailto:', '') for url in data[p]['mailing_lists'] if\n url['url'] not in projects[p]['mailing_lists']]\n projects[p]['mailing_lists'] += urls\n\n for p in [project for project in data if len(data[project]['dev_list']) > 0]:\n if 'mailing_lists' not in projects[p]:\n projects[p]['mailing_lists'] = []\n\n mailing_list = data[p]['dev_list']['url'].replace('mailto:', '')\n projects[p]['mailing_lists'].append(mailing_list)\n\n return projects"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef compose_github(projects, data):\n for p in [project for project in data if len(data[project]['github_repos']) > 0]:\n if 'github' not in projects[p]:\n projects[p]['github'] = []\n\n urls = [url['url'] for url in data[p]['github_repos'] if\n url['url'] not in projects[p]['github']]\n projects[p]['github'] += urls\n\n return projects", "response": "Compose projects. json with github"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef compose_bugzilla(projects, data):\n for p in [project for project in data if len(data[project]['bugzilla']) > 0]:\n if 'bugzilla' not in projects[p]:\n projects[p]['bugzilla'] = []\n\n urls = [url['query_url'] for url in data[p]['bugzilla'] if\n url['query_url'] not in projects[p]['bugzilla']]\n projects[p]['bugzilla'] += urls\n\n return projects", "response": "Compose the bugzilla project list for the project."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef compose_title(projects, data):\n for project in data:\n projects[project] = {\n 'meta': {\n 'title': data[project]['title']\n }\n }\n return projects", "response": "Compose the projects. json file only with the projects titles"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef compose_projects_json(projects, data):\n projects = compose_git(projects, data)\n projects = compose_mailing_lists(projects, data)\n projects = compose_bugzilla(projects, data)\n projects = compose_github(projects, data)\n projects = compose_gerrit(projects)\n projects = compose_mbox(projects)\n\n return projects", "response": "Compose projects. json with all data sources\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nexecute autorefresh for studies", "response": "def __autorefresh_studies(self, cfg):\n \"\"\"Execute autorefresh for areas of code study if configured\"\"\"\n\n if 'studies' not in self.conf[self.backend_section] or \\\n 'enrich_areas_of_code:git' not in self.conf[self.backend_section]['studies']:\n logger.debug(\"Not doing autorefresh for studies, Areas of Code study is not active.\")\n return\n\n aoc_index = self.conf['enrich_areas_of_code:git'].get('out_index', GitEnrich.GIT_AOC_ENRICHED)\n\n # if `out_index` exists but has no value, use default\n if not aoc_index:\n aoc_index = GitEnrich.GIT_AOC_ENRICHED\n\n logger.debug(\"Autorefresh for Areas of Code study index: %s\", aoc_index)\n\n es = Elasticsearch([self.conf['es_enrichment']['url']], timeout=100,\n verify_certs=self._get_enrich_backend().elastic.requests.verify)\n\n if not es.indices.exists(index=aoc_index):\n logger.debug(\"Not doing autorefresh, index doesn't exist for Areas of Code study\")\n return\n\n logger.debug(\"Doing autorefresh for Areas of Code study\")\n\n # Create a GitEnrich backend tweaked to work with AOC index\n aoc_backend = GitEnrich(self.db_sh, None, cfg['projects']['projects_file'],\n self.db_user, self.db_password, self.db_host)\n aoc_backend.mapping = None\n aoc_backend.roles = ['author']\n elastic_enrich = get_elastic(self.conf['es_enrichment']['url'],\n aoc_index, clean=False, backend=aoc_backend)\n aoc_backend.set_elastic(elastic_enrich)\n\n self.__autorefresh(aoc_backend, studies=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexecute the studies configured for the current backend", "response": "def __studies(self, retention_time):\n \"\"\" Execute the studies configured for the current backend \"\"\"\n\n cfg = self.config.get_conf()\n if 'studies' not in cfg[self.backend_section] or not \\\n cfg[self.backend_section]['studies']:\n logger.debug('No studies for %s' % self.backend_section)\n return\n\n studies = [study for study in cfg[self.backend_section]['studies'] if study.strip() != \"\"]\n if not studies:\n logger.debug('No studies for %s' % self.backend_section)\n return\n\n logger.debug(\"Executing studies for %s: %s\" % (self.backend_section, studies))\n time.sleep(2) # Wait so enrichment has finished in ES\n enrich_backend = self._get_enrich_backend()\n ocean_backend = self._get_ocean_backend(enrich_backend)\n\n active_studies = []\n all_studies = enrich_backend.studies\n all_studies_names = [study.__name__ for study in enrich_backend.studies]\n\n # Time to check that configured studies are valid\n logger.debug(\"All studies in %s: %s\", self.backend_section, all_studies_names)\n logger.debug(\"Configured studies %s\", studies)\n cfg_studies_types = [study.split(\":\")[0] for study in studies]\n if not set(cfg_studies_types).issubset(set(all_studies_names)):\n logger.error('Wrong studies names for %s: %s', self.backend_section, studies)\n raise RuntimeError('Wrong studies names ', self.backend_section, studies)\n\n for study in enrich_backend.studies:\n if study.__name__ in cfg_studies_types:\n active_studies.append(study)\n\n enrich_backend.studies = active_studies\n print(\"Executing for %s the studies %s\" % (self.backend_section,\n [study for study in studies]))\n\n studies_args = self.__load_studies()\n\n do_studies(ocean_backend, enrich_backend, studies_args, retention_time=retention_time)\n # Return studies to its original value\n enrich_backend.studies = all_studies"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nretaining the identities in SortingHat based on the retention_time value declared in the setup. cfg.", "response": "def retain_identities(self, retention_time):\n \"\"\"Retain the identities in SortingHat based on the `retention_time`\n value declared in the setup.cfg.\n\n :param retention_time: maximum number of minutes wrt the current date to retain the SortingHat data\n \"\"\"\n enrich_es = self.conf['es_enrichment']['url']\n sortinghat_db = self.db\n current_data_source = self.get_backend(self.backend_section)\n active_data_sources = self.config.get_active_data_sources()\n\n if retention_time is None:\n logger.debug(\"[identities retention] Retention policy disabled, no identities will be deleted.\")\n return\n\n if retention_time <= 0:\n logger.debug(\"[identities retention] Retention time must be greater than 0.\")\n return\n\n retain_identities(retention_time, enrich_es, sortinghat_db, current_data_source, active_data_sources)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the list of repositories for a given backend section", "response": "def get_repos_by_backend_section(cls, backend_section, raw=True):\n \"\"\" return list with the repositories for a backend_section \"\"\"\n repos = []\n projects = TaskProjects.get_projects()\n\n for pro in projects:\n if backend_section in projects[pro]:\n # if the projects.json doesn't contain the `unknown` project, add the repos in the bck section\n if cls.GLOBAL_PROJECT not in projects:\n repos += projects[pro][backend_section]\n else:\n # if the projects.json contains the `unknown` project\n # in the case of the collection phase\n if raw:\n # if the current project is not `unknown`\n if pro != cls.GLOBAL_PROJECT:\n # if the bck section is not in the `unknown` project, add the repos in the bck section\n if backend_section not in projects[cls.GLOBAL_PROJECT]:\n repos += projects[pro][backend_section]\n # if the backend section is in the `unknown` project,\n # add the repo in the bck section under `unknown`\n elif backend_section in projects[pro] and backend_section in projects[cls.GLOBAL_PROJECT]:\n repos += projects[cls.GLOBAL_PROJECT][backend_section]\n # if the current project is `unknown`\n else:\n # if the backend section is only in the `unknown` project,\n # add the repo in the bck section under `unknown`\n not_in_unknown = [projects[pro] for pro in projects if pro != cls.GLOBAL_PROJECT][0]\n if backend_section not in not_in_unknown:\n repos += projects[cls.GLOBAL_PROJECT][backend_section]\n # in the case of the enrichment phase\n else:\n # if the current project is not `unknown`\n if pro != cls.GLOBAL_PROJECT:\n # if the bck section is not in the `unknown` project, add the repos in the bck section\n if backend_section not in projects[cls.GLOBAL_PROJECT]:\n repos += projects[pro][backend_section]\n # if the backend section is in the `unknown` project, add the repos in the bck section\n elif backend_section in projects[pro] and backend_section in projects[cls.GLOBAL_PROJECT]:\n repos += projects[pro][backend_section]\n # if the current project is `unknown`\n else:\n # if the backend section is only in the `unknown` project,\n # add the repo in the bck section under `unknown`\n not_in_unknown_prj = [projects[prj] for prj in projects if prj != cls.GLOBAL_PROJECT]\n not_in_unknown_sections = list(set([section for prj in not_in_unknown_prj\n for section in list(prj.keys())]))\n if backend_section not in not_in_unknown_sections:\n repos += projects[pro][backend_section]\n\n logger.debug(\"List of repos for %s: %s (raw=%s)\", backend_section, repos, raw)\n\n # avoid duplicated repos\n repos = list(set(repos))\n\n return repos"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef convert_from_eclipse(self, eclipse_projects):\n\n projects = {}\n\n # We need the global project for downloading the full Bugzilla and Gerrit\n projects['unknown'] = {\n \"gerrit\": [\"git.eclipse.org\"],\n \"bugzilla\": [\"https://bugs.eclipse.org/bugs/\"]\n }\n\n projects = compose_title(projects, eclipse_projects)\n projects = compose_projects_json(projects, eclipse_projects)\n\n return projects", "response": "Convert from eclipse projects format to grimoire projects json format"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef general_params(cls):\n\n params = {}\n\n # GENERAL CONFIG\n params_general = {\n \"general\": {\n \"min_update_delay\": {\n \"optional\": True,\n \"default\": 60,\n \"type\": int,\n \"description\": \"Short delay between tasks (collect, enrich ...)\"\n },\n \"update\": {\n \"optional\": False,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Execute the tasks in loop\"\n },\n \"short_name\": {\n \"optional\": False,\n \"default\": \"Short name\",\n \"type\": str,\n \"description\": \"Short name of the project\"\n },\n \"debug\": {\n \"optional\": False,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Debug mode (logging mainly)\"\n },\n \"logs_dir\": {\n \"optional\": False,\n \"default\": \"logs\",\n \"type\": str,\n \"description\": \"Directory with the logs of sirmordred\"\n },\n \"log_handler\": {\n \"optional\": True,\n \"default\": \"file\",\n \"type\": str,\n \"description\": \"use rotate for rotating the logs automatically\"\n },\n \"log_max_bytes\": {\n \"optional\": True,\n \"default\": 104857600, # 100MB\n \"type\": int,\n \"description\": \"Max number of bytes per log file\"\n },\n \"log_backup_count\": {\n \"optional\": True,\n \"default\": 5,\n \"type\": int,\n \"description\": \"Number of rotate logs files to preserve\"\n },\n \"bulk_size\": {\n \"optional\": True,\n \"default\": 1000,\n \"type\": int,\n \"description\": \"Number of items to write in Elasticsearch using bulk operations\"\n },\n \"scroll_size\": {\n \"optional\": True,\n \"default\": 100,\n \"type\": int,\n \"description\": \"Number of items to read from Elasticsearch when scrolling\"\n },\n \"aliases_file\": {\n \"optional\": True,\n \"default\": ALIASES_JSON,\n \"type\": str,\n \"description\": \"JSON file to define aliases for raw and enriched indexes\"\n },\n \"menu_file\": {\n \"optional\": True,\n \"default\": MENU_YAML,\n \"type\": str,\n \"description\": \"YAML file to define the menus to be shown in Kibiter\"\n },\n \"retention_time\": {\n \"optional\": True,\n \"default\": None,\n \"type\": int,\n \"description\": \"The maximum number of minutes wrt the current date to retain the data\"\n }\n }\n }\n params_projects = {\n \"projects\": {\n \"projects_file\": {\n \"optional\": True,\n \"default\": PROJECTS_JSON,\n \"type\": str,\n \"description\": \"Projects file path with repositories to be collected group by projects\"\n },\n \"projects_url\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"Projects file URL\"\n },\n \"load_eclipse\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Load the projects from Eclipse\"\n }\n }\n }\n\n params_phases = {\n \"phases\": {\n \"collection\": {\n \"optional\": False,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Activate collection of items\"\n },\n \"enrichment\": {\n \"optional\": False,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Activate enrichment of items\"\n },\n \"identities\": {\n \"optional\": False,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Do the identities tasks\"\n },\n \"panels\": {\n \"optional\": False,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Load panels, create alias and other tasks related\"\n },\n \"track_items\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Track specific items from a gerrit repository\"\n },\n \"report\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Generate the PDF report for a project (alpha)\"\n }\n }\n }\n\n general_config_params = [params_general, params_projects, params_phases]\n\n for section_params in general_config_params:\n params.update(section_params)\n\n # Config provided by tasks\n params_collection = {\n \"es_collection\": {\n \"password\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"Password for connection to Elasticsearch\"\n },\n \"user\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"User for connection to Elasticsearch\"\n },\n \"url\": {\n \"optional\": False,\n \"default\": \"http://172.17.0.1:9200\",\n \"type\": str,\n \"description\": \"Elasticsearch URL\"\n },\n \"arthur\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Use arthur for collecting items from perceval\"\n },\n \"arthur_url\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"URL for the arthur service\"\n },\n \"redis_url\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"URL for the redis service\"\n }\n }\n }\n\n params_enrichment = {\n \"es_enrichment\": {\n \"url\": {\n \"optional\": False,\n \"default\": \"http://172.17.0.1:9200\",\n \"type\": str,\n \"description\": \"Elasticsearch URL\"\n },\n \"autorefresh\": {\n \"optional\": True,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Execute the autorefresh of identities\"\n },\n \"autorefresh_interval\": {\n \"optional\": True,\n \"default\": 2,\n \"type\": int,\n \"description\": \"Set time interval (days) for autorefresh identities\"\n },\n \"user\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"User for connection to Elasticsearch\"\n },\n \"password\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"Password for connection to Elasticsearch\"\n }\n }\n }\n\n params_panels = {\n \"panels\": {\n \"strict\": {\n \"optional\": True,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Enable strict panels loading\"\n },\n \"kibiter_time_from\": {\n \"optional\": True,\n \"default\": \"now-90d\",\n \"type\": str,\n \"description\": \"Default time interval for Kibiter\"\n },\n \"kibiter_default_index\": {\n \"optional\": True,\n \"default\": \"git\",\n \"type\": str,\n \"description\": \"Default index pattern for Kibiter\"\n },\n \"kibiter_url\": {\n \"optional\": False,\n \"default\": None,\n \"type\": str,\n \"description\": \"Kibiter URL\"\n },\n \"kibiter_version\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"Kibiter version\"\n },\n \"community\": {\n \"optional\": True,\n \"default\": True,\n \"type\": bool,\n \"description\": \"Enable community structure menu\"\n },\n \"kafka\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Enable kafka menu\"\n },\n \"github-repos\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Enable GitHub repo stats menu\"\n },\n \"gitlab-issues\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Enable GitLab issues menu\"\n },\n \"gitlab-merges\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Enable GitLab merge requests menu\"\n },\n \"mattermost\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Enable Mattermost menu\"\n }\n }\n }\n\n params_report = {\n \"report\": {\n \"start_date\": {\n \"optional\": False,\n \"default\": \"1970-01-01\",\n \"type\": str,\n \"description\": \"Start date for the report\"\n },\n \"end_date\": {\n \"optional\": False,\n \"default\": \"2100-01-01\",\n \"type\": str,\n \"description\": \"End date for the report\"\n },\n \"interval\": {\n \"optional\": False,\n \"default\": \"quarter\",\n \"type\": str,\n \"description\": \"Interval for the report\"\n },\n \"config_file\": {\n \"optional\": False,\n \"default\": \"report.cfg\",\n \"type\": str,\n \"description\": \"Config file for the report\"\n },\n \"data_dir\": {\n \"optional\": False,\n \"default\": \"report_data\",\n \"type\": str,\n \"description\": \"Directory in which to store the report data\"\n },\n \"filters\": {\n \"optional\": True,\n \"default\": [],\n \"type\": list,\n \"description\": \"General filters to be applied to all queries\"\n },\n \"offset\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"Date offset to be applied to start and end\"\n }\n }\n }\n\n params_sortinghat = {\n \"sortinghat\": {\n \"affiliate\": {\n \"optional\": False,\n \"default\": \"True\",\n \"type\": bool,\n \"description\": \"Affiliate identities to organizations\"\n },\n \"unaffiliated_group\": {\n \"optional\": False,\n \"default\": \"Unknown\",\n \"type\": str,\n \"description\": \"Name of the organization for unaffiliated identities\"\n },\n \"matching\": {\n \"optional\": False,\n \"default\": [\"email\"],\n \"type\": list,\n \"description\": \"Algorithm for matching identities in Sortinghat\"\n },\n \"sleep_for\": {\n \"optional\": False,\n \"default\": 3600,\n \"type\": int,\n \"description\": \"Delay between task identities executions\"\n },\n \"database\": {\n \"optional\": False,\n \"default\": \"sortinghat_db\",\n \"type\": str,\n \"description\": \"Name of the Sortinghat database\"\n },\n \"host\": {\n \"optional\": False,\n \"default\": \"mariadb\",\n \"type\": str,\n \"description\": \"Host with the Sortinghat database\"\n },\n \"user\": {\n \"optional\": False,\n \"default\": \"root\",\n \"type\": str,\n \"description\": \"User to access the Sortinghat database\"\n },\n \"password\": {\n \"optional\": False,\n \"default\": \"\",\n \"type\": str,\n \"description\": \"Password to access the Sortinghat database\"\n },\n \"autoprofile\": {\n \"optional\": False,\n \"default\": [\"customer\", \"git\", \"github\"],\n \"type\": list,\n \"description\": \"Order in which to get the identities information for filling the profile\"\n },\n \"load_orgs\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"deprecated\": \"Load organizations in Sortinghat database\",\n \"description\": \"\"\n },\n \"identities_format\": {\n \"optional\": True,\n \"default\": \"sortinghat\",\n \"type\": str,\n \"description\": \"Format of the identities data to be loaded\"\n },\n \"strict_mapping\": {\n \"optional\": True,\n \"default\": True,\n \"type\": bool,\n \"description\": \"rigorous check of values in identities matching \"\n \"(i.e, well formed email addresses)\"\n },\n \"reset_on_load\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Unmerge and remove affiliations for all identities on load\"\n },\n \"orgs_file\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"File path with the organizations to be loaded in Sortinghat\"\n },\n \"identities_file\": {\n \"optional\": True,\n \"default\": [],\n \"type\": list,\n \"description\": \"File path with the identities to be loaded in Sortinghat\"\n },\n \"identities_export_url\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"URL in which to export the identities in Sortinghat\"\n },\n \"identities_api_token\": {\n \"optional\": True,\n \"default\": None,\n \"type\": str,\n \"description\": \"API token for remote operation with GitHub and Gitlab\"\n },\n \"bots_names\": {\n \"optional\": True,\n \"default\": [],\n \"type\": list,\n \"description\": \"Name of the identities to be marked as bots\"\n },\n \"no_bots_names\": {\n \"optional\": True,\n \"default\": [],\n \"type\": list,\n \"description\": \"Name of the identities to be unmarked as bots\"\n },\n \"autogender\": {\n \"optional\": True,\n \"default\": False,\n \"type\": bool,\n \"description\": \"Add gender to the profiles (executes autogender)\"\n }\n }\n }\n\n params_track_items = {\n \"track_items\": {\n \"project\": {\n \"optional\": False,\n \"default\": \"TrackProject\",\n \"type\": str,\n \"description\": \"Gerrit project to track\"\n },\n \"upstream_raw_es_url\": {\n \"optional\": False,\n \"default\": \"\",\n \"type\": str,\n \"description\": \"URL with the file with the gerrit reviews to track\"\n },\n \"raw_index_gerrit\": {\n \"optional\": False,\n \"default\": \"\",\n \"type\": str,\n \"description\": \"Name of the gerrit raw index\"\n },\n \"raw_index_git\": {\n \"optional\": False,\n \"default\": \"\",\n \"type\": str,\n \"description\": \"Name of the git raw index\"\n }\n }\n }\n\n tasks_config_params = [params_collection, params_enrichment, params_panels,\n params_report, params_sortinghat, params_track_items]\n\n for section_params in tasks_config_params:\n params.update(section_params)\n\n return params", "response": "Define all the possible config params for the class"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nchanging a param in the config", "response": "def set_param(self, section, param, value):\n \"\"\" Change a param in the config \"\"\"\n if section not in self.conf or param not in self.conf[section]:\n logger.error('Config section %s and param %s not exists', section, param)\n else:\n self.conf[section][param] = value"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nadd new configuration to self. conf", "response": "def _add_to_conf(self, new_conf):\n \"\"\"Add new configuration to self.conf.\n\n Adds configuration parameters in new_con to self.conf.\n If they already existed in conf, overwrite them.\n\n :param new_conf: new configuration, to add\n \"\"\"\n\n for section in new_conf:\n if section not in self.conf:\n self.conf[section] = new_conf[section]\n else:\n for param in new_conf[section]:\n self.conf[section][param] = new_conf[section][param]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef es_version(self, url):\n\n try:\n res = self.grimoire_con.get(url)\n res.raise_for_status()\n major = res.json()['version']['number'].split(\".\")[0]\n except Exception:\n logger.error(\"Error retrieving Elasticsearch version: \" + url)\n raise\n return major", "response": "Get the major version of Elasticsearch."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nexecutes the tasks_cls in the non - stop state.", "response": "def execute_nonstop_tasks(self, tasks_cls):\n \"\"\"\n Just a wrapper to the execute_batch_tasks method\n \"\"\"\n self.execute_batch_tasks(tasks_cls,\n self.conf['sortinghat']['sleep_for'],\n self.conf['general']['min_update_delay'], False)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef execute_batch_tasks(self, tasks_cls, big_delay=0, small_delay=0, wait_for_threads=True):\n\n def _split_tasks(tasks_cls):\n \"\"\"\n we internally distinguish between tasks executed by backend\n and tasks executed with no specific backend. \"\"\"\n backend_t = []\n global_t = []\n for t in tasks_cls:\n if t.is_backend_task(t):\n backend_t.append(t)\n else:\n global_t.append(t)\n return backend_t, global_t\n\n backend_tasks, global_tasks = _split_tasks(tasks_cls)\n logger.debug('backend_tasks = %s' % (backend_tasks))\n logger.debug('global_tasks = %s' % (global_tasks))\n\n threads = []\n\n # stopper won't be set unless wait_for_threads is True\n stopper = threading.Event()\n\n # launching threads for tasks by backend\n if len(backend_tasks) > 0:\n repos_backend = self._get_repos_by_backend()\n for backend in repos_backend:\n # Start new Threads and add them to the threads list to complete\n t = TasksManager(backend_tasks, backend, stopper, self.config, small_delay)\n threads.append(t)\n t.start()\n\n # launch thread for global tasks\n if len(global_tasks) > 0:\n # FIXME timer is applied to all global_tasks, does it make sense?\n # All tasks are executed in the same thread sequentially\n gt = TasksManager(global_tasks, \"Global tasks\", stopper, self.config, big_delay)\n threads.append(gt)\n gt.start()\n if big_delay > 0:\n when = datetime.now() + timedelta(seconds=big_delay)\n when_str = when.strftime('%a, %d %b %Y %H:%M:%S %Z')\n logger.info(\"%s will be executed on %s\" % (global_tasks, when_str))\n\n if wait_for_threads:\n time.sleep(1) # Give enough time create and run all threads\n stopper.set() # All threads must stop in the next iteration\n\n # Wait for all threads to complete\n for t in threads:\n t.join()\n\n # Checking for exceptions in threads to log them\n self.__check_queue_for_errors()\n\n logger.debug(\"[thread:main] All threads (and their tasks) are finished\")", "response": "Execute a list of tasks in a batch."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef __execute_initial_load(self):\n\n if self.conf['phases']['panels']:\n tasks_cls = [TaskPanels, TaskPanelsMenu]\n self.execute_tasks(tasks_cls)\n if self.conf['phases']['identities']:\n tasks_cls = [TaskInitSortingHat]\n self.execute_tasks(tasks_cls)\n\n logger.info(\"Loading projects\")\n tasks_cls = [TaskProjects]\n self.execute_tasks(tasks_cls)\n logger.info(\"Done\")\n\n return", "response": "Execute tasks that should be done just one time\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrunning the command line and returns a Result object.", "response": "def run(self, halt_on_nonzero=True, quiet=False, q=False, streaming=False):\n \"\"\"\n After building your commands, call `run()` to have your code executed.\n \"\"\"\n commands = str(self)\n if not (quiet or q):\n self._echo.cmd(commands)\n\n env = self._context[0].get('env', {}) if len(self._context) > 0 else os.environ\n executable = self.current_context.get('executable')\n try:\n process = subprocess.Popen(commands,\n bufsize=1,\n shell=True,\n env=env,\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n executable=executable,\n universal_newlines=True)\n result = Result(process, commands, self._context, streaming, halt_on_nonzero=halt_on_nonzero)\n\n except Exception as e:\n result = Result(None, commands, self._context, exception=e)\n result.dump_exception()\n if halt_on_nonzero:\n raise e\n \n finally:\n self.clear()\n \n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_config(self):\n '''\n Validates the provided config to make sure all the required fields are \n there.\n '''\n # first ensure that all the required fields are there\n for key, key_config in self.params_map.items():\n if key_config['required']:\n if key not in self.config:\n raise ValueError(\"Invalid Configuration! Required parameter '%s' was not provided to Sultan.\")\n \n # second ensure that the fields that were pased were actually fields that\n # can be used\n for key in self.config.keys():\n if key not in self.params_map:\n raise ValueError(\"Invalid Configuration! The parameter '%s' provided is not used by Sultan!\" % key)", "response": "Validates the provided config to make sure all the required fields are in the config and that all the required fields are in the config that are in the params_map."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef stdout(self):\n if self._streaming:\n stdout = []\n while not self.__stdout.empty():\n try:\n line = self.__stdout.get_nowait()\n stdout.append(line)\n except:\n pass\n else:\n stdout = self.__stdout\n return stdout", "response": "Converts the stdout string to a list."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stderr(self):\n if self._streaming:\n stderr = []\n while not self.__stderr.empty():\n try:\n line = self.__stderr.get_nowait()\n stderr.append(line)\n except:\n pass\n else:\n stderr = self.__stderr\n return stderr", "response": "Converts stderr string to a list."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nprinting the stdout to console - if there is any stdout otherwise does nothing.", "response": "def print_stdout(self, always_print=False):\n \"\"\"\n Prints the stdout to console - if there is any stdout, otherwise does nothing.\n :param always_print: print the stdout, even if there is nothing in the buffer (default: false)\n \"\"\"\n if self.__stdout or always_print:\n self.__echo.info(\"--{ STDOUT }---\" + \"-\" * 100)\n self.__format_lines_info(self.stdout)\n self.__echo.info(\"---------------\" + \"-\" * 100)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nprints the stderr to console - if there is any stdout otherwise does nothing.", "response": "def print_stderr(self, always_print=False):\n \"\"\"\n Prints the stderr to console - if there is any stdout, otherwise does nothing.\n :param always_print: print the stderr, even if there is nothing in the buffer (default: false)\n \"\"\"\n if self.__stderr or always_print:\n self.__echo.critical(\"--{ STDERR }---\" + \"-\" * 100)\n self.__format_lines_error(self.stderr)\n self.__echo.critical(\"---------------\" + \"-\" * 100)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nprinting the traceback to console.", "response": "def print_traceback(self, always_print=False):\n \"\"\"\n Prints the traceback to console - if there is any traceback, otherwise does nothing.\n :param always_print: print the traceback, even if there is nothing in the buffer (default: false)\n \"\"\"\n if self._exception or always_print:\n self.__echo.critical(\"--{ TRACEBACK }\" + \"-\" * 100)\n self.__format_lines_error(self.traceback)\n self.__echo.critical(\"---------------\" + \"-\" * 100)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef format(self, record):\n if isinstance(self.fmt, dict):\n self._fmt = self.fmt[record.levelname]\n if sys.version_info > (3, 2):\n # Update self._style because we've changed self._fmt\n # (code based on stdlib's logging.Formatter.__init__())\n if self.style not in logging._STYLES:\n raise ValueError('Style must be one of: %s' % ','.join(\n list(logging._STYLES.keys())))\n self._style = logging._STYLES[self.style][0](self._fmt)\n\n if sys.version_info > (2, 7):\n message = super(LevelFormatter, self).format(record)\n else:\n message = ColoredFormatter.format(self, record)\n\n return message", "response": "Customize the message format based on the log level."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef replace_print(fileobj=sys.stderr):\n printer = _Printer(fileobj)\n\n previous_stdout = sys.stdout\n sys.stdout = printer\n try:\n yield printer\n finally:\n sys.stdout = previous_stdout", "response": "A context manager that replaces sys. stdout with a new printer."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncompacting a list of integers into a comma - separated string of intervals.", "response": "def compact_interval_string(value_list):\n \"\"\"Compact a list of integers into a comma-separated string of intervals.\n\n Args:\n value_list: A list of sortable integers such as a list of numbers\n\n Returns:\n A compact string representation, such as \"1-5,8,12-15\"\n \"\"\"\n\n if not value_list:\n return ''\n\n value_list.sort()\n\n # Start by simply building up a list of separate contiguous intervals\n interval_list = []\n curr = []\n for val in value_list:\n if curr and (val > curr[-1] + 1):\n interval_list.append((curr[0], curr[-1]))\n curr = [val]\n else:\n curr.append(val)\n\n if curr:\n interval_list.append((curr[0], curr[-1]))\n\n # For each interval collapse it down to \"first, last\" or just \"first\" if\n # if first == last.\n return ','.join([\n '{}-{}'.format(pair[0], pair[1]) if pair[0] != pair[1] else str(pair[0])\n for pair in interval_list\n ])"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets a storage client using the provided credentials.", "response": "def _get_storage_service(credentials):\n \"\"\"Get a storage client using the provided credentials or defaults.\"\"\"\n if credentials is None:\n credentials = oauth2client.client.GoogleCredentials.get_application_default(\n )\n return discovery.build('storage', 'v1', credentials=credentials)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn True if we should retry False otherwise.", "response": "def _retry_storage_check(exception):\n \"\"\"Return True if we should retry, False otherwise.\"\"\"\n now = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')\n print_error(\n '%s: Exception %s: %s' % (now, type(exception).__name__, str(exception)))\n return isinstance(exception, oauth2client.client.AccessTokenRefreshError)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload a file from GCS.", "response": "def _load_file_from_gcs(gcs_file_path, credentials=None):\n \"\"\"Load context from a text file in gcs.\n\n Args:\n gcs_file_path: The target file path; should have the 'gs://' prefix.\n credentials: Optional credential to be used to load the file from gcs.\n\n Returns:\n The content of the text file as a string.\n \"\"\"\n gcs_service = _get_storage_service(credentials)\n\n bucket_name, object_name = gcs_file_path[len('gs://'):].split('/', 1)\n request = gcs_service.objects().get_media(\n bucket=bucket_name, object=object_name)\n\n file_handle = io.BytesIO()\n downloader = MediaIoBaseDownload(file_handle, request, chunksize=1024 * 1024)\n done = False\n while not done:\n _, done = _downloader_next_chunk(downloader)\n filevalue = file_handle.getvalue()\n if not isinstance(filevalue, six.string_types):\n filevalue = filevalue.decode()\n return six.StringIO(filevalue)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load_file(file_path, credentials=None):\n if file_path.startswith('gs://'):\n return _load_file_from_gcs(file_path, credentials)\n else:\n return open(file_path, 'r')", "response": "Loads a file from either local or gcs.\n ."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks whether the file exists in GCS.", "response": "def _file_exists_in_gcs(gcs_file_path, credentials=None):\n \"\"\"Check whether the file exists, in GCS.\n\n Args:\n gcs_file_path: The target file path; should have the 'gs://' prefix.\n credentials: Optional credential to be used to load the file from gcs.\n\n Returns:\n True if the file's there.\n \"\"\"\n gcs_service = _get_storage_service(credentials)\n\n bucket_name, object_name = gcs_file_path[len('gs://'):].split('/', 1)\n request = gcs_service.objects().get(\n bucket=bucket_name, object=object_name, projection='noAcl')\n try:\n request.execute()\n return True\n except errors.HttpError:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef file_exists(file_path, credentials=None):\n if file_path.startswith('gs://'):\n return _file_exists_in_gcs(file_path, credentials)\n else:\n return os.path.isfile(file_path)", "response": "Checks whether the file exists on local disk or GCS."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking whether there is a GCS object whose name starts with the prefix.", "response": "def _prefix_exists_in_gcs(gcs_prefix, credentials=None):\n \"\"\"Check whether there is a GCS object whose name starts with the prefix.\n\n Since GCS doesn't actually have folders, this is how we check instead.\n\n Args:\n gcs_prefix: The path; should start with 'gs://'.\n credentials: Optional credential to be used to load the file from gcs.\n\n Returns:\n True if the prefix matches at least one object in GCS.\n\n Raises:\n errors.HttpError: if it can't talk to the server\n \"\"\"\n gcs_service = _get_storage_service(credentials)\n\n bucket_name, prefix = gcs_prefix[len('gs://'):].split('/', 1)\n # documentation in\n # https://cloud.google.com/storage/docs/json_api/v1/objects/list\n request = gcs_service.objects().list(\n bucket=bucket_name, prefix=prefix, maxResults=1)\n response = request.execute()\n return response.get('items', None)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef simple_pattern_exists_in_gcs(file_pattern, credentials=None):\n if '*' not in file_pattern:\n return _file_exists_in_gcs(file_pattern, credentials)\n if not file_pattern.startswith('gs://'):\n raise ValueError('file name must start with gs://')\n gcs_service = _get_storage_service(credentials)\n bucket_name, prefix = file_pattern[len('gs://'):].split('/', 1)\n if '*' in bucket_name:\n raise ValueError('Wildcards may not appear in the bucket name')\n # There is a '*' in prefix because we checked there's one in file_pattern\n # and there isn't one in bucket_name. Hence it must be in prefix.\n assert '*' in prefix\n prefix_no_wildcard = prefix[:prefix.index('*')]\n request = gcs_service.objects().list(\n bucket=bucket_name, prefix=prefix_no_wildcard)\n response = request.execute()\n if 'items' not in response:\n return False\n items_list = [i['name'] for i in response['items']]\n return any(fnmatch.fnmatch(i, prefix) for i in items_list)", "response": "Returns True iff an object exists matching the input GCS pattern."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef outputs_are_present(outputs):\n # outputs are OutputFileParam (see param_util.py)\n\n # If outputs contain a pattern, then there is no way for `dsub` to verify\n # that *all* output is present. The best that `dsub` can do is to verify\n # that *some* output was created for each such parameter.\n for o in outputs:\n if not o.value:\n continue\n if o.recursive:\n if not folder_exists(o.value):\n return False\n else:\n if not simple_pattern_exists_in_gcs(o.value):\n return False\n return True", "response": "True if each output contains at least one file or no output specified."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a dict object representing a pipeline input argument.", "response": "def _build_pipeline_input_file_param(cls, var_name, docker_path):\n \"\"\"Return a dict object representing a pipeline input argument.\"\"\"\n\n # If the filename contains a wildcard, then the target Docker path must\n # be a directory in order to ensure consistency whether the source pattern\n # contains 1 or multiple files.\n #\n # In that case, we set the docker_path to explicitly have a trailing slash\n # (for the Pipelines API \"gsutil cp\" handling, and then override the\n # associated var_name environment variable in the generated Docker command.\n\n path, filename = os.path.split(docker_path)\n if '*' in filename:\n return cls._build_pipeline_file_param(var_name, path + '/')\n else:\n return cls._build_pipeline_file_param(var_name, docker_path)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _build_pipeline_docker_command(cls, script_name, inputs, outputs, envs):\n\n # We upload the user script as an environment argument\n # and write it to SCRIPT_DIR (preserving its local file name).\n #\n # The docker_command:\n # * writes the script body to a file\n # * installs gcloud if there are recursive copies to do\n # * sets environment variables for inputs with wildcards\n # * sets environment variables for recursive input directories\n # * recursively copies input directories\n # * creates output directories\n # * sets environment variables for recursive output directories\n # * sets the DATA_ROOT environment variable to /mnt/data\n # * sets the working directory to ${DATA_ROOT}\n # * executes the user script\n # * recursively copies output directories\n recursive_input_dirs = [\n var for var in inputs if var.recursive and var.value\n ]\n recursive_output_dirs = [\n var for var in outputs if var.recursive and var.value\n ]\n\n install_cloud_sdk = ''\n if recursive_input_dirs or recursive_output_dirs:\n install_cloud_sdk = INSTALL_CLOUD_SDK\n\n export_input_dirs = ''\n copy_input_dirs = ''\n if recursive_input_dirs:\n export_input_dirs = providers_util.build_recursive_localize_env(\n providers_util.DATA_MOUNT_POINT, inputs)\n copy_input_dirs = providers_util.build_recursive_localize_command(\n providers_util.DATA_MOUNT_POINT, inputs, job_model.P_GCS)\n\n export_output_dirs = ''\n copy_output_dirs = ''\n if recursive_output_dirs:\n export_output_dirs = providers_util.build_recursive_gcs_delocalize_env(\n providers_util.DATA_MOUNT_POINT, outputs)\n copy_output_dirs = providers_util.build_recursive_delocalize_command(\n providers_util.DATA_MOUNT_POINT, outputs, job_model.P_GCS)\n\n docker_paths = [\n var.docker_path if var.recursive else os.path.dirname(var.docker_path)\n for var in outputs\n if var.value\n ]\n\n mkdirs = '\\n'.join([\n 'mkdir -p {0}/{1}'.format(providers_util.DATA_MOUNT_POINT, path)\n for path in docker_paths\n ])\n\n inputs_with_wildcards = [\n var for var in inputs if not var.recursive and var.docker_path and\n '*' in os.path.basename(var.docker_path)\n ]\n export_inputs_with_wildcards = '\\n'.join([\n 'export {0}=\"{1}/{2}\"'.format(var.name, providers_util.DATA_MOUNT_POINT,\n var.docker_path)\n for var in inputs_with_wildcards\n ])\n\n export_empty_envs = '\\n'.join([\n 'export {0}=\"\"'.format(var.name)\n for var in envs | inputs | outputs\n if not var.value\n ])\n\n return DOCKER_COMMAND.format(\n mk_runtime_dirs=MK_RUNTIME_DIRS_COMMAND,\n script_path='%s/%s' % (providers_util.SCRIPT_DIR, script_name),\n install_cloud_sdk=install_cloud_sdk,\n export_inputs_with_wildcards=export_inputs_with_wildcards,\n export_input_dirs=export_input_dirs,\n copy_input_dirs=copy_input_dirs,\n mk_output_dirs=mkdirs,\n export_output_dirs=export_output_dirs,\n export_empty_envs=export_empty_envs,\n tmpdir=providers_util.TMP_DIR,\n working_dir=providers_util.WORKING_DIR,\n copy_output_dirs=copy_output_dirs)", "response": "Builds the full pipeline docker command."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef build_pipeline(cls, project, zones, min_cores, min_ram, disk_size,\n boot_disk_size, preemptible, accelerator_type,\n accelerator_count, image, script_name, envs, inputs,\n outputs, pipeline_name):\n \"\"\"Builds a pipeline configuration for execution.\n\n Args:\n project: string name of project.\n zones: list of zone names for jobs to be run at.\n min_cores: int number of CPU cores required per job.\n min_ram: int GB of RAM required per job.\n disk_size: int GB of disk to attach under /mnt/data.\n boot_disk_size: int GB of disk for boot.\n preemptible: use a preemptible VM for the job\n accelerator_type: string GCE defined accelerator type.\n accelerator_count: int number of accelerators of the specified type to\n attach.\n image: string Docker image name in which to run.\n script_name: file name of the script to run.\n envs: list of EnvParam objects specifying environment variables to set\n within each job.\n inputs: list of FileParam objects specifying input variables to set\n within each job.\n outputs: list of FileParam objects specifying output variables to set\n within each job.\n pipeline_name: string name of pipeline.\n\n Returns:\n A nested dictionary with one entry under the key ephemeralPipeline\n containing the pipeline configuration.\n \"\"\"\n if min_cores is None:\n min_cores = job_model.DEFAULT_MIN_CORES\n if min_ram is None:\n min_ram = job_model.DEFAULT_MIN_RAM\n if disk_size is None:\n disk_size = job_model.DEFAULT_DISK_SIZE\n if boot_disk_size is None:\n boot_disk_size = job_model.DEFAULT_BOOT_DISK_SIZE\n if preemptible is None:\n preemptible = job_model.DEFAULT_PREEMPTIBLE\n\n # Format the docker command\n docker_command = cls._build_pipeline_docker_command(script_name, inputs,\n outputs, envs)\n\n # Pipelines inputParameters can be both simple name/value pairs which get\n # set as environment variables, as well as input file paths which the\n # Pipelines controller will automatically localize to the Pipeline VM.\n\n # In the ephemeralPipeline object, the inputParameters are only defined;\n # the values are passed in the pipelineArgs.\n\n # Pipelines outputParameters are only output file paths, which the\n # Pipelines controller can automatically de-localize after the docker\n # command completes.\n\n # The Pipelines API does not support recursive copy of file parameters,\n # so it is implemented within the dsub-generated pipeline.\n # Any inputs or outputs marked as \"recursive\" are completely omitted here;\n # their environment variables will be set in the docker command, and\n # recursive copy code will be generated there as well.\n\n # The Pipelines API does not accept empty environment variables. Set them to\n # empty in DOCKER_COMMAND instead.\n input_envs = [{\n 'name': SCRIPT_VARNAME\n }] + [{\n 'name': env.name\n } for env in envs if env.value]\n\n input_files = [\n cls._build_pipeline_input_file_param(var.name, var.docker_path)\n for var in inputs\n if not var.recursive and var.value\n ]\n\n # Outputs are an array of file parameters\n output_files = [\n cls._build_pipeline_file_param(var.name, var.docker_path)\n for var in outputs\n if not var.recursive and var.value\n ]\n\n # The ephemeralPipeline provides the template for the pipeline.\n # pyformat: disable\n return {\n 'ephemeralPipeline': {\n 'projectId': project,\n 'name': pipeline_name,\n\n # Define the resources needed for this pipeline.\n 'resources': {\n 'minimumCpuCores': min_cores,\n 'minimumRamGb': min_ram,\n 'bootDiskSizeGb': boot_disk_size,\n 'preemptible': preemptible,\n 'zones': google_base.get_zones(zones),\n 'acceleratorType': accelerator_type,\n 'acceleratorCount': accelerator_count,\n\n # Create a data disk that is attached to the VM and destroyed\n # when the pipeline terminates.\n 'disks': [{\n 'name': 'datadisk',\n 'autoDelete': True,\n 'sizeGb': disk_size,\n 'mountPoint': providers_util.DATA_MOUNT_POINT,\n }],\n },\n\n 'inputParameters': input_envs + input_files,\n 'outputParameters': output_files,\n\n 'docker': {\n 'imageName': image,\n 'cmd': docker_command,\n }\n }\n }", "response": "Builds a pipeline for execution."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef build_pipeline_args(cls, project, script, job_params, task_params,\n reserved_labels, preemptible, logging_uri, scopes,\n keep_alive):\n \"\"\"Builds pipeline args for execution.\n\n Args:\n project: string name of project.\n script: Body of the script to execute.\n job_params: dictionary of values for labels, envs, inputs, and outputs\n for this job.\n task_params: dictionary of values for labels, envs, inputs, and outputs\n for this task.\n reserved_labels: dictionary of reserved labels (e.g. task-id,\n task-attempt)\n preemptible: use a preemptible VM for the job\n logging_uri: path for job logging output.\n scopes: list of scope.\n keep_alive: Seconds to keep VM alive on failure\n\n Returns:\n A nested dictionary with one entry under the key pipelineArgs containing\n the pipeline arguments.\n \"\"\"\n # For the Pipelines API, envs and file inputs are all \"inputs\".\n inputs = {}\n inputs.update({SCRIPT_VARNAME: script})\n inputs.update({\n var.name: var.value\n for var in job_params['envs'] | task_params['envs']\n if var.value\n })\n inputs.update({\n var.name: var.uri\n for var in job_params['inputs'] | task_params['inputs']\n if not var.recursive and var.value\n })\n\n # Remove wildcard references for non-recursive output. When the pipelines\n # controller generates a delocalize call, it must point to a bare directory\n # for patterns. The output param OUTFILE=gs://bucket/path/*.bam should\n # delocalize with a call similar to:\n # gsutil cp /mnt/data/output/gs/bucket/path/*.bam gs://bucket/path/\n outputs = {}\n for var in job_params['outputs'] | task_params['outputs']:\n if var.recursive or not var.value:\n continue\n if '*' in var.uri.basename:\n outputs[var.name] = var.uri.path\n else:\n outputs[var.name] = var.uri\n\n labels = {}\n labels.update({\n label.name: label.value if label.value else ''\n for label in (reserved_labels | job_params['labels']\n | task_params['labels'])\n })\n\n # pyformat: disable\n args = {\n 'pipelineArgs': {\n 'projectId': project,\n 'resources': {\n 'preemptible': preemptible,\n },\n 'inputs': inputs,\n 'outputs': outputs,\n 'labels': labels,\n 'serviceAccount': {\n 'email': 'default',\n 'scopes': scopes,\n },\n # Pass the user-specified GCS destination for pipeline logging.\n 'logging': {\n 'gcsPath': logging_uri\n },\n }\n }\n # pyformat: enable\n\n if keep_alive:\n args['pipelineArgs'][\n 'keep_vm_alive_on_failure_duration'] = '%ss' % keep_alive\n\n return args", "response": "Builds the pipeline arguments for execution."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert the integer UTC time value into a local datetime.", "response": "def _datetime_to_utc_int(date):\n \"\"\"Convert the integer UTC time value into a local datetime.\"\"\"\n if date is None:\n return None\n\n # Convert localized datetime to a UTC integer\n epoch = dsub_util.replace_timezone(datetime.utcfromtimestamp(0), pytz.utc)\n return (date - epoch).total_seconds()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_filter(project,\n status=None,\n user_id=None,\n job_id=None,\n job_name=None,\n labels=None,\n task_id=None,\n task_attempt=None,\n create_time_min=None,\n create_time_max=None):\n \"\"\"Return a filter string for operations.list().\"\"\"\n\n ops_filter = ['projectId = %s' % project]\n if status and status != '*':\n ops_filter.append('status = %s' % status)\n\n if user_id != '*':\n ops_filter.append('labels.user-id = %s' % user_id)\n if job_id != '*':\n ops_filter.append('labels.job-id = %s' % job_id)\n if job_name != '*':\n ops_filter.append('labels.job-name = %s' % job_name)\n if task_id != '*':\n ops_filter.append('labels.task-id = %s' % task_id)\n if task_attempt != '*':\n ops_filter.append('labels.task-attempt = %s' % task_attempt)\n\n # Even though labels are nominally 'arbitrary strings' they are trusted\n # since param_util restricts the character set.\n if labels:\n for l in labels:\n ops_filter.append('labels.%s = %s' % (l.name, l.value))\n\n epoch = dsub_util.replace_timezone(datetime.utcfromtimestamp(0), pytz.utc)\n if create_time_min:\n create_time_min_utc_int = (create_time_min - epoch).total_seconds()\n ops_filter.append('createTime >= %d' % create_time_min_utc_int)\n if create_time_max:\n create_time_max_utc_int = (create_time_max - epoch).total_seconds()\n ops_filter.append('createTime <= %d' % create_time_max_utc_int)\n\n return ' AND '.join(ops_filter)", "response": "Return a filter string for operations. list."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_dsub_operation(cls, op):\n if not cls.is_pipelines_operation(op):\n return False\n\n for name in ['job-id', 'job-name', 'user-id']:\n if not cls.get_operation_label(op, name):\n return False\n\n return True", "response": "Determine if a pipelines operation is a dsub request."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef list(cls, service, ops_filter, page_size=0):\n\n page_token = None\n more_operations = True\n documented_default_page_size = 256\n documented_max_page_size = 2048\n\n if not page_size:\n page_size = documented_default_page_size\n page_size = min(page_size, documented_max_page_size)\n\n while more_operations:\n api = service.operations().list(\n name='operations',\n filter=ops_filter,\n pageToken=page_token,\n pageSize=page_size)\n response = google_base.Api.execute(api)\n\n ops = response.get('operations', [])\n for op in ops:\n if cls.is_dsub_operation(op):\n yield GoogleOperation(op)\n\n page_token = response.get('nextPageToken')\n more_operations = bool(page_token)", "response": "Returns the list of operations for the specified filter."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef prepare_job_metadata(self, script, job_name, user_id, create_time):\n return google_base.prepare_job_metadata(script, job_name, user_id,\n create_time)", "response": "Returns a dictionary of metadata fields for the job."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nbuild the ephemeral Pipeline request for the job.", "response": "def _build_pipeline_request(self, task_view):\n \"\"\"Returns a Pipeline objects for the job.\"\"\"\n job_metadata = task_view.job_metadata\n job_params = task_view.job_params\n job_resources = task_view.job_resources\n task_metadata = task_view.task_descriptors[0].task_metadata\n task_params = task_view.task_descriptors[0].task_params\n task_resources = task_view.task_descriptors[0].task_resources\n\n script = task_view.job_metadata['script']\n\n reserved_labels = google_base.build_pipeline_labels(\n job_metadata, task_metadata, task_id_pattern='task-%d')\n\n # Build the ephemeralPipeline for this job.\n # The ephemeralPipeline definition changes for each job because file\n # parameters localCopy.path changes based on the remote_uri.\n pipeline = _Pipelines.build_pipeline(\n project=self._project,\n zones=job_resources.zones,\n min_cores=job_resources.min_cores,\n min_ram=job_resources.min_ram,\n disk_size=job_resources.disk_size,\n boot_disk_size=job_resources.boot_disk_size,\n preemptible=job_resources.preemptible,\n accelerator_type=job_resources.accelerator_type,\n accelerator_count=job_resources.accelerator_count,\n image=job_resources.image,\n script_name=script.name,\n envs=job_params['envs'] | task_params['envs'],\n inputs=job_params['inputs'] | task_params['inputs'],\n outputs=job_params['outputs'] | task_params['outputs'],\n pipeline_name=job_metadata['pipeline-name'])\n\n # Build the pipelineArgs for this job.\n logging_uri = task_resources.logging_path.uri\n scopes = job_resources.scopes or google_base.DEFAULT_SCOPES\n pipeline.update(\n _Pipelines.build_pipeline_args(self._project, script.value, job_params,\n task_params, reserved_labels,\n job_resources.preemptible, logging_uri,\n scopes, job_resources.keep_alive))\n\n return pipeline"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef lookup_job_tasks(self,\n statuses,\n user_ids=None,\n job_ids=None,\n job_names=None,\n task_ids=None,\n task_attempts=None,\n labels=None,\n create_time_min=None,\n create_time_max=None,\n max_tasks=0,\n page_size=0):\n \"\"\"Yields operations based on the input criteria.\n\n If any of the filters are empty or {'*'}, then no filtering is performed on\n that field. Filtering by both a job id list and job name list is\n unsupported.\n\n Args:\n statuses: {'*'}, or a list of job status strings to return. Valid\n status strings are 'RUNNING', 'SUCCESS', 'FAILURE', or 'CANCELED'.\n user_ids: a list of ids for the user(s) who launched the job.\n job_ids: a list of job ids to return.\n job_names: a list of job names to return.\n task_ids: a list of specific tasks within the specified job(s) to return.\n task_attempts: a list of specific attempts within the specified tasks(s)\n to return.\n labels: a list of LabelParam with user-added labels. All labels must\n match the task being fetched.\n create_time_min: a timezone-aware datetime value for the earliest create\n time of a task, inclusive.\n create_time_max: a timezone-aware datetime value for the most recent\n create time of a task, inclusive.\n max_tasks: the maximum number of job tasks to return or 0 for no limit.\n page_size: the page size to use for each query to the pipelins API.\n\n Raises:\n ValueError: if both a job id list and a job name list are provided\n\n Yeilds:\n Genomics API Operations objects.\n \"\"\"\n\n # Server-side, we can filter on status, job_id, user_id, task_id, but there\n # is no OR filter (only AND), so we can't handle lists server side.\n # Therefore we construct a set of queries for each possible combination of\n # these criteria.\n statuses = statuses if statuses else {'*'}\n user_ids = user_ids if user_ids else {'*'}\n job_ids = job_ids if job_ids else {'*'}\n job_names = job_names if job_names else {'*'}\n task_ids = task_ids if task_ids else {'*'}\n task_attempts = task_attempts if task_attempts else {'*'}\n\n # The task-id label value of \"task-n\" instead of just \"n\" is a hold-over\n # from early label value character restrictions.\n # Accept both forms, \"task-n\" and \"n\", for lookups by task-id.\n task_ids = {'task-{}'.format(t) if t.isdigit() else t for t in task_ids}\n\n if job_ids != {'*'} and job_names != {'*'}:\n raise ValueError(\n 'Filtering by both job IDs and job names is not supported')\n\n # AND filter rule arguments.\n labels = labels if labels else set()\n\n # The results of all these queries need to be sorted by create-time\n # (descending). To accomplish this, each query stream (already sorted by\n # create-time) is added to a SortedGeneratorIterator which is a wrapper\n # around a PriorityQueue of iterators (sorted by each stream's newest task's\n # create-time). A sorted list can then be built by stepping through this\n # iterator and adding tasks until none are left or we hit max_tasks.\n\n now = datetime.now()\n\n def _desc_date_sort_key(t):\n return now - dsub_util.replace_timezone(t.get_field('create-time'), None)\n\n query_queue = sorting_util.SortedGeneratorIterator(key=_desc_date_sort_key)\n for status, job_id, job_name, user_id, task_id, task_attempt in (\n itertools.product(statuses, job_ids, job_names, user_ids, task_ids,\n task_attempts)):\n ops_filter = _Operations.get_filter(\n self._project,\n status=status,\n user_id=user_id,\n job_id=job_id,\n job_name=job_name,\n labels=labels,\n task_id=task_id,\n task_attempt=task_attempt,\n create_time_min=create_time_min,\n create_time_max=create_time_max)\n\n # The pipelines API returns operations sorted by create-time date. We can\n # use this sorting guarantee to merge-sort the streams together and only\n # retrieve more tasks as needed.\n stream = _Operations.list(self._service, ops_filter, page_size=page_size)\n query_queue.add_generator(stream)\n\n tasks_yielded = 0\n for task in query_queue:\n yield task\n tasks_yielded += 1\n if 0 < max_tasks <= tasks_yielded:\n break", "response": "Returns a generator that yields all the tasks that match the input criteria."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_jobs(self,\n user_ids,\n job_ids,\n task_ids,\n labels,\n create_time_min=None,\n create_time_max=None):\n \"\"\"Kills the operations associated with the specified job or job.task.\n\n Args:\n user_ids: List of user ids who \"own\" the job(s) to cancel.\n job_ids: List of job_ids to cancel.\n task_ids: List of task-ids to cancel.\n labels: List of LabelParam, each must match the job(s) to be canceled.\n create_time_min: a timezone-aware datetime value for the earliest create\n time of a task, inclusive.\n create_time_max: a timezone-aware datetime value for the most recent\n create time of a task, inclusive.\n\n Returns:\n A list of tasks canceled and a list of error messages.\n \"\"\"\n # Look up the job(s)\n tasks = list(\n self.lookup_job_tasks(\n {'RUNNING'},\n user_ids=user_ids,\n job_ids=job_ids,\n task_ids=task_ids,\n labels=labels,\n create_time_min=create_time_min,\n create_time_max=create_time_max))\n\n print('Found %d tasks to delete.' % len(tasks))\n\n return google_base.cancel(self._service.new_batch_http_request,\n self._service.operations().cancel, tasks)", "response": "Deletes the specified jobs."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a value from the operation for a specific set of field names.", "response": "def get_field(self, field, default=None):\n \"\"\"Returns a value from the operation for a specific set of field names.\n\n Args:\n field: a dsub-specific job metadata key\n default: default value to return if field does not exist or is empty.\n\n Returns:\n A text string for the field or a list for 'inputs'.\n\n Raises:\n ValueError: if the field label is not supported by the operation\n \"\"\"\n\n metadata = self._op.get('metadata')\n\n value = None\n if field == 'internal-id':\n value = self._op['name']\n elif field == 'job-id':\n value = metadata['labels'].get('job-id')\n elif field == 'job-name':\n value = metadata['labels'].get('job-name')\n elif field == 'task-id':\n value = metadata['labels'].get('task-id')\n elif field == 'task-attempt':\n value = metadata['labels'].get('task-attempt')\n elif field == 'user-id':\n value = metadata['labels'].get('user-id')\n elif field == 'dsub-version':\n value = metadata['labels'].get('dsub-version')\n elif field == 'task-status':\n value = self._operation_status()\n elif field == 'logging':\n value = metadata['request']['pipelineArgs']['logging']['gcsPath']\n elif field == 'envs':\n value = self._get_operation_input_field_values(metadata, False)\n elif field == 'labels':\n # Reserved labels are filtered from dsub task output.\n value = {\n k: v\n for k, v in metadata['labels'].items()\n if k not in job_model.RESERVED_LABELS\n }\n elif field == 'inputs':\n value = self._get_operation_input_field_values(metadata, True)\n elif field == 'outputs':\n value = self._get_operation_output_field_values(metadata)\n elif field == 'mounts':\n value = None\n elif field == 'create-time':\n value = google_base.parse_rfc3339_utc_string(metadata['createTime'])\n elif field == 'start-time':\n # Look through the events list for all \"start\" events (only one expected).\n start_events = [\n e for e in metadata.get('events', []) if e['description'] == 'start'\n ]\n # Get the startTime from the last \"start\" event.\n if start_events:\n value = google_base.parse_rfc3339_utc_string(\n start_events[-1]['startTime'])\n elif field == 'end-time':\n if 'endTime' in metadata:\n value = google_base.parse_rfc3339_utc_string(metadata['endTime'])\n elif field == 'status':\n value = self._operation_status()\n elif field in ['status-message', 'status-detail']:\n status, last_update = self._operation_status_message()\n value = status\n elif field == 'last-update':\n status, last_update = self._operation_status_message()\n value = last_update\n elif field == 'provider':\n return _PROVIDER_NAME\n elif field == 'provider-attributes':\n # Use soft getting of keys to address a race condition and to\n # pull the null values found in jobs that fail prior to scheduling.\n gce_data = metadata.get('runtimeMetadata', {}).get('computeEngine', {})\n if 'machineType' in gce_data:\n machine_type = gce_data.get('machineType').rpartition('/')[2]\n else:\n machine_type = None\n instance_name = gce_data.get('instanceName')\n instance_zone = gce_data.get('zone')\n value = {\n 'machine-type': machine_type,\n 'instance-name': instance_name,\n 'zone': instance_zone,\n }\n elif field == 'events':\n events = metadata.get('events', [])\n value = []\n for event in events:\n event_value = {\n 'name':\n event.get('description', ''),\n 'start-time':\n google_base.parse_rfc3339_utc_string(event['startTime'])\n }\n if 'endTime' in event:\n event_value['end-time'] = google_base.parse_rfc3339_utc_string(\n event['endTime'])\n\n value.append(event_value)\n elif field in [\n 'user-project', 'script-name', 'script', 'input-recursives',\n 'output-recursives'\n ]:\n # Supported in local and google-v2 providers.\n value = None\n\n else:\n raise ValueError('Unsupported field: \"%s\"' % field)\n\n return value if value else default"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the most relevant status string and last updated date string.", "response": "def _operation_status_message(self):\n \"\"\"Returns the most relevant status string and last updated date string.\n\n This string is meant for display only.\n\n Returns:\n A printable status string and date string.\n \"\"\"\n metadata = self._op['metadata']\n if not self._op['done']:\n if 'events' in metadata and metadata['events']:\n # Get the last event\n last_event = metadata['events'][-1]\n\n msg = last_event['description']\n ds = last_event['startTime']\n else:\n msg = 'Pending'\n ds = metadata['createTime']\n else:\n ds = metadata['endTime']\n\n if 'error' in self._op:\n msg = self._op['error']['message']\n else:\n msg = 'Success'\n\n return (msg, google_base.parse_rfc3339_utc_string(ds))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_operation_input_field_values(self, metadata, file_input):\n\n # To determine input parameter type, we iterate through the\n # pipeline inputParameters.\n # The values come from the pipelineArgs inputs.\n input_args = metadata['request']['ephemeralPipeline']['inputParameters']\n vals_dict = metadata['request']['pipelineArgs']['inputs']\n\n # Get the names for files or envs\n names = [\n arg['name'] for arg in input_args if ('localCopy' in arg) == file_input\n ]\n\n # Build the return dict\n return {name: vals_dict[name] for name in names if name in vals_dict}", "response": "Returns a dictionary of envs or file inputs for an operation."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning an error message if the operation failed for any reason.", "response": "def error_message(self):\n \"\"\"Returns an error message if the operation failed for any reason.\n\n Failure as defined here means; ended for any reason other than 'success'.\n This means that a successful cancelation will also create an error message\n here.\n\n Returns:\n string, string will be empty if job did not error.\n \"\"\"\n if 'error' in self._op:\n if 'task-id' in self._op['metadata']['labels']:\n job_id = self._op['metadata']['labels']['task-id']\n else:\n job_id = self._op['metadata']['labels']['job-id']\n return 'Error in job %s - code %s: %s' % (job_id,\n self._op['error']['code'],\n self._op['error']['message'])\n else:\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _format_task_name(job_id, task_id, task_attempt):\n docker_name = '%s.%s' % (job_id, 'task' if task_id is None else task_id)\n\n if task_attempt is not None:\n docker_name += '.' + str(task_attempt)\n\n # Docker container names must match: [a-zA-Z0-9][a-zA-Z0-9_.-]\n # So 1) prefix it with \"dsub-\" and 2) change all invalid characters to \"-\".\n return 'dsub-{}'.format(_convert_suffix_to_docker_chars(docker_name))", "response": "Create a task name from a job - id task - id and task - attempt."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrewrites string so that all characters are valid in a docker name suffix.", "response": "def _convert_suffix_to_docker_chars(suffix):\n \"\"\"Rewrite string so that all characters are valid in a docker name suffix.\"\"\"\n # Docker container names must match: [a-zA-Z0-9][a-zA-Z0-9_.-]\n accepted_characters = string.ascii_letters + string.digits + '_.-'\n\n def label_char_transform(char):\n if char in accepted_characters:\n return char\n return '-'\n\n return ''.join(label_char_transform(c) for c in suffix)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _task_sort_function(task):\n return (task.get_field('create-time'), int(task.get_field('task-id', 0)),\n int(task.get_field('task-attempt', 0)))", "response": "Return a tuple for sorting most recent first."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _datetime_in_range(self, dt, dt_min=None, dt_max=None):\n # The pipelines API stores operation create-time with second granularity.\n # We mimic this behavior in the local provider by truncating to seconds.\n dt = dt.replace(microsecond=0)\n if dt_min:\n dt_min = dt_min.replace(microsecond=0)\n else:\n dt_min = dsub_util.replace_timezone(datetime.datetime.min, pytz.utc)\n if dt_max:\n dt_max = dt_max.replace(microsecond=0)\n else:\n dt_max = dsub_util.replace_timezone(datetime.datetime.max, pytz.utc)\n\n return dt_min <= dt <= dt_max", "response": "Determine if the provided time is within the range inclusive."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a Task object with this task s info.", "response": "def _get_task_from_task_dir(self, job_id, user_id, task_id, task_attempt):\n \"\"\"Return a Task object with this task's info.\"\"\"\n\n # We need to be very careful about how we read and interpret the contents\n # of the task directory. The directory could be changing because a new\n # task is being created. The directory could be changing because a task\n # is ending.\n #\n # If the meta.yaml does not exist, the task does not yet exist.\n # If the meta.yaml exists, it means the task is scheduled. It does not mean\n # it is yet running.\n # If the task.pid file exists, it means that the runner.sh was started.\n\n task_dir = self._task_directory(job_id, task_id, task_attempt)\n\n job_descriptor = self._read_task_metadata(task_dir)\n if not job_descriptor:\n return None\n\n # If we read up an old task, the user-id will not be in the job_descriptor.\n if not job_descriptor.job_metadata.get('user-id'):\n job_descriptor.job_metadata['user-id'] = user_id\n\n # Get the pid of the runner\n pid = -1\n try:\n with open(os.path.join(task_dir, 'task.pid'), 'r') as f:\n pid = int(f.readline().strip())\n except (IOError, OSError):\n pass\n\n # Get the script contents\n script = None\n script_name = job_descriptor.job_metadata.get('script-name')\n if script_name:\n script = self._read_script(task_dir, script_name)\n\n # Read the files written by the runner.sh.\n # For new tasks, these may not have been written yet.\n end_time = self._get_end_time_from_task_dir(task_dir)\n last_update = self._get_last_update_time_from_task_dir(task_dir)\n events = self._get_events_from_task_dir(task_dir)\n status = self._get_status_from_task_dir(task_dir)\n log_detail = self._get_log_detail_from_task_dir(task_dir)\n\n # If the status file is not yet written, then mark the task as pending\n if not status:\n status = 'RUNNING'\n log_detail = ['Pending']\n\n return LocalTask(\n task_status=status,\n events=events,\n log_detail=log_detail,\n job_descriptor=job_descriptor,\n end_time=end_time,\n last_update=last_update,\n pid=pid,\n script=script)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a command to delocalize the logging files.", "response": "def _delocalize_logging_command(self, logging_path, user_project):\n \"\"\"Returns a command to delocalize logs.\n\n Args:\n logging_path: location of log files.\n user_project: name of the project to be billed for the request.\n\n Returns:\n eg. 'gs://bucket/path/myfile' or 'gs://bucket/script-foobar-12'\n \"\"\"\n\n # Get the logging prefix (everything up to \".log\")\n logging_prefix = os.path.splitext(logging_path.uri)[0]\n\n # Set the provider-specific mkdir and file copy commands\n if logging_path.file_provider == job_model.P_LOCAL:\n mkdir_cmd = 'mkdir -p \"%s\"\\n' % os.path.dirname(logging_prefix)\n cp_cmd = 'cp'\n elif logging_path.file_provider == job_model.P_GCS:\n mkdir_cmd = ''\n if user_project:\n cp_cmd = 'gsutil -u {} -mq cp'.format(user_project)\n else:\n cp_cmd = 'gsutil -mq cp'\n else:\n assert False\n\n # Construct the copy command\n copy_logs_cmd = textwrap.dedent(\"\"\"\\\n local cp_cmd=\"{cp_cmd}\"\n local prefix=\"{prefix}\"\n \"\"\").format(\n cp_cmd=cp_cmd, prefix=logging_prefix)\n\n # Build up the command\n body = textwrap.dedent(\"\"\"\\\n {mkdir_cmd}\n {copy_logs_cmd}\n \"\"\").format(\n mkdir_cmd=mkdir_cmd, copy_logs_cmd=copy_logs_cmd)\n\n return body"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _task_directory(self, job_id, task_id, task_attempt):\n dir_name = 'task' if task_id is None else str(task_id)\n if task_attempt:\n dir_name = '%s.%s' % (dir_name, task_attempt)\n return self._provider_root() + '/' + job_id + '/' + dir_name", "response": "The local dir for staging files for that particular task."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _make_environment(self, inputs, outputs, mounts):\n env = {}\n env.update(providers_util.get_file_environment_variables(inputs))\n env.update(providers_util.get_file_environment_variables(outputs))\n env.update(providers_util.get_file_environment_variables(mounts))\n return env", "response": "Return a dictionary of environment variables for the container."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _localize_inputs_recursive_command(self, task_dir, inputs):\n data_dir = os.path.join(task_dir, _DATA_SUBDIR)\n provider_commands = [\n providers_util.build_recursive_localize_command(data_dir, inputs,\n file_provider)\n for file_provider in _SUPPORTED_INPUT_PROVIDERS\n ]\n return '\\n'.join(provider_commands)", "response": "Returns a command that will stage recursive inputs."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a directory or file path to be the target for the given file.", "response": "def _get_input_target_path(self, local_file_path):\n \"\"\"Returns a directory or file path to be the target for \"gsutil cp\".\n\n If the filename contains a wildcard, then the target path must\n be a directory in order to ensure consistency whether the source pattern\n contains one or multiple files.\n\n\n Args:\n local_file_path: A full path terminating in a file or a file wildcard.\n\n Returns:\n The path to use as the \"gsutil cp\" target.\n \"\"\"\n\n path, filename = os.path.split(local_file_path)\n if '*' in filename:\n return path + '/'\n else:\n return local_file_path"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a command that will stage inputs.", "response": "def _localize_inputs_command(self, task_dir, inputs, user_project):\n \"\"\"Returns a command that will stage inputs.\"\"\"\n commands = []\n for i in inputs:\n if i.recursive or not i.value:\n continue\n\n source_file_path = i.uri\n local_file_path = task_dir + '/' + _DATA_SUBDIR + '/' + i.docker_path\n dest_file_path = self._get_input_target_path(local_file_path)\n\n commands.append('mkdir -p \"%s\"' % os.path.dirname(local_file_path))\n\n if i.file_provider in [job_model.P_LOCAL, job_model.P_GCS]:\n # The semantics that we expect here are implemented consistently in\n # \"gsutil cp\", and are a bit different than \"cp\" when it comes to\n # wildcard handling, so use it for both local and GCS:\n #\n # - `cp path/* dest/` will error if \"path\" has subdirectories.\n # - `cp \"path/*\" \"dest/\"` will fail (it expects wildcard expansion\n # to come from shell).\n if user_project:\n command = 'gsutil -u %s -mq cp \"%s\" \"%s\"' % (\n user_project, source_file_path, dest_file_path)\n else:\n command = 'gsutil -mq cp \"%s\" \"%s\"' % (source_file_path,\n dest_file_path)\n commands.append(command)\n\n return '\\n'.join(commands)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate the command line for the delocalization of the outputs.", "response": "def _delocalize_outputs_commands(self, task_dir, outputs, user_project):\n \"\"\"Copy outputs from local disk to GCS.\"\"\"\n commands = []\n for o in outputs:\n if o.recursive or not o.value:\n continue\n\n # The destination path is o.uri.path, which is the target directory\n # (rather than o.uri, which includes the filename or wildcard).\n dest_path = o.uri.path\n local_path = task_dir + '/' + _DATA_SUBDIR + '/' + o.docker_path\n\n if o.file_provider == job_model.P_LOCAL:\n commands.append('mkdir -p \"%s\"' % dest_path)\n\n # Use gsutil even for local files (explained in _localize_inputs_command).\n if o.file_provider in [job_model.P_LOCAL, job_model.P_GCS]:\n if user_project:\n command = 'gsutil -u %s -mq cp \"%s\" \"%s\"' % (user_project, local_path,\n dest_path)\n else:\n command = 'gsutil -mq cp \"%s\" \"%s\"' % (local_path, dest_path)\n commands.append(command)\n\n return '\\n'.join(commands)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the dsub version out of the _dsub_version. py source file.", "response": "def get_dsub_version():\n \"\"\"Get the dsub version out of the _dsub_version.py source file.\n\n Setup.py should not import dsub version from dsub directly since ambiguity in\n import order could lead to an old version of dsub setting the version number.\n Parsing the file directly is simpler than using import tools (whose interface\n varies between python 2.7, 3.4, and 3.5).\n\n Returns:\n string of dsub version.\n\n Raises:\n ValueError: if the version is not found.\n \"\"\"\n filename = os.path.join(os.path.dirname(__file__), 'dsub/_dsub_version.py')\n with open(filename, 'r') as versionfile:\n for line in versionfile:\n if line.startswith('DSUB_VERSION ='):\n # Get the version then strip whitespace and quote characters.\n version = line.partition('=')[2]\n return version.strip().strip('\\'\"')\n raise ValueError('Could not find version.')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_filtered_normalized_events(self):\n # Need the user-image to look for the right \"pulling image\" event\n user_image = google_v2_operations.get_action_image(self._op,\n _ACTION_USER_COMMAND)\n\n # Only create an \"ok\" event for operations with SUCCESS status.\n need_ok = google_v2_operations.is_success(self._op)\n\n # Events are keyed by name for easier deletion.\n events = {}\n\n # Events are assumed to be ordered by timestamp (newest to oldest).\n for event in google_v2_operations.get_events(self._op):\n if self._filter(event):\n continue\n\n mapped, match = self._map(event)\n name = mapped['name']\n\n if name == 'ok':\n # If we want the \"ok\" event, we grab the first (most recent).\n if not need_ok or 'ok' in events:\n continue\n\n if name == 'pulling-image':\n if match.group(1) != user_image:\n continue\n\n events[name] = mapped\n\n return sorted(events.values(), key=operator.itemgetter('start-time'))", "response": "Returns a list of events that are filtered down to events of interest."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _map(self, event):\n description = event.get('description', '')\n start_time = google_base.parse_rfc3339_utc_string(\n event.get('timestamp', ''))\n\n for name, regex in _EVENT_REGEX_MAP.items():\n match = regex.match(description)\n if match:\n return {'name': name, 'start-time': start_time}, match\n\n return {'name': description, 'start-time': start_time}, None", "response": "Extract elements from an operation event and map to a named event."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the environment for actions that copy logging files.", "response": "def _get_logging_env(self, logging_uri, user_project):\n \"\"\"Returns the environment for actions that copy logging files.\"\"\"\n if not logging_uri.endswith('.log'):\n raise ValueError('Logging URI must end in \".log\": {}'.format(logging_uri))\n\n logging_prefix = logging_uri[:-len('.log')]\n return {\n 'LOGGING_PATH': '{}.log'.format(logging_prefix),\n 'STDOUT_PATH': '{}-stdout.log'.format(logging_prefix),\n 'STDERR_PATH': '{}-stderr.log'.format(logging_prefix),\n 'USER_PROJECT': user_project,\n }"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_prepare_env(self, script, job_descriptor, inputs, outputs, mounts):\n\n # Add the _SCRIPT_REPR with the repr(script) contents\n # Add the _META_YAML_REPR with the repr(meta) contents\n\n # Add variables for directories that need to be created, for example:\n # DIR_COUNT: 2\n # DIR_0: /mnt/data/input/gs/bucket/path1/\n # DIR_1: /mnt/data/output/gs/bucket/path2\n\n # List the directories in sorted order so that they are created in that\n # order. This is primarily to ensure that permissions are set as we create\n # each directory.\n # For example:\n # mkdir -m 777 -p /root/first/second\n # mkdir -m 777 -p /root/first\n # *may* not actually set 777 on /root/first\n\n docker_paths = sorted([\n var.docker_path if var.recursive else os.path.dirname(var.docker_path)\n for var in inputs | outputs | mounts\n if var.value\n ])\n\n env = {\n _SCRIPT_VARNAME: repr(script.value),\n _META_YAML_VARNAME: repr(job_descriptor.to_yaml()),\n 'DIR_COUNT': str(len(docker_paths))\n }\n\n for idx, path in enumerate(docker_paths):\n env['DIR_{}'.format(idx)] = os.path.join(providers_util.DATA_MOUNT_POINT,\n path)\n\n return env", "response": "Returns a dict with variables for the prepare action."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a dict with variables for the localization action.", "response": "def _get_localization_env(self, inputs, user_project):\n \"\"\"Return a dict with variables for the 'localization' action.\"\"\"\n\n # Add variables for paths that need to be localized, for example:\n # INPUT_COUNT: 1\n # INPUT_0: MY_INPUT_FILE\n # INPUT_RECURSIVE_0: 0\n # INPUT_SRC_0: gs://mybucket/mypath/myfile\n # INPUT_DST_0: /mnt/data/inputs/mybucket/mypath/myfile\n\n non_empty_inputs = [var for var in inputs if var.value]\n env = {'INPUT_COUNT': str(len(non_empty_inputs))}\n\n for idx, var in enumerate(non_empty_inputs):\n env['INPUT_{}'.format(idx)] = var.name\n env['INPUT_RECURSIVE_{}'.format(idx)] = str(int(var.recursive))\n env['INPUT_SRC_{}'.format(idx)] = var.value\n\n # For wildcard paths, the destination must be a directory\n dst = os.path.join(providers_util.DATA_MOUNT_POINT, var.docker_path)\n path, filename = os.path.split(dst)\n if '*' in filename:\n dst = '{}/'.format(path)\n env['INPUT_DST_{}'.format(idx)] = dst\n\n env['USER_PROJECT'] = user_project\n\n return env"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a dict with variables for the delocalization action.", "response": "def _get_delocalization_env(self, outputs, user_project):\n \"\"\"Return a dict with variables for the 'delocalization' action.\"\"\"\n\n # Add variables for paths that need to be delocalized, for example:\n # OUTPUT_COUNT: 1\n # OUTPUT_0: MY_OUTPUT_FILE\n # OUTPUT_RECURSIVE_0: 0\n # OUTPUT_SRC_0: gs://mybucket/mypath/myfile\n # OUTPUT_DST_0: /mnt/data/outputs/mybucket/mypath/myfile\n\n non_empty_outputs = [var for var in outputs if var.value]\n env = {'OUTPUT_COUNT': str(len(non_empty_outputs))}\n\n for idx, var in enumerate(non_empty_outputs):\n env['OUTPUT_{}'.format(idx)] = var.name\n env['OUTPUT_RECURSIVE_{}'.format(idx)] = str(int(var.recursive))\n env['OUTPUT_SRC_{}'.format(idx)] = os.path.join(\n providers_util.DATA_MOUNT_POINT, var.docker_path)\n\n # For wildcard paths, the destination must be a directory\n if '*' in var.uri.basename:\n dst = var.uri.path\n else:\n dst = var.uri\n env['OUTPUT_DST_{}'.format(idx)] = dst\n\n env['USER_PROJECT'] = user_project\n\n return env"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nbuild a dictionary of for the user container environment.", "response": "def _build_user_environment(self, envs, inputs, outputs, mounts):\n \"\"\"Returns a dictionary of for the user container environment.\"\"\"\n envs = {env.name: env.value for env in envs}\n envs.update(providers_util.get_file_environment_variables(inputs))\n envs.update(providers_util.get_file_environment_variables(outputs))\n envs.update(providers_util.get_file_environment_variables(mounts))\n return envs"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a list of two actions per gcs bucket to mount.", "response": "def _get_mount_actions(self, mounts, mnt_datadisk):\n \"\"\"Returns a list of two actions per gcs bucket to mount.\"\"\"\n actions_to_add = []\n for mount in mounts:\n bucket = mount.value[len('gs://'):]\n mount_path = mount.docker_path\n actions_to_add.extend([\n google_v2_pipelines.build_action(\n name='mount-{}'.format(bucket),\n flags=['ENABLE_FUSE', 'RUN_IN_BACKGROUND'],\n image_uri=_GCSFUSE_IMAGE,\n mounts=[mnt_datadisk],\n commands=[\n '--implicit-dirs', '--foreground', '-o ro', bucket,\n os.path.join(providers_util.DATA_MOUNT_POINT, mount_path)\n ]),\n google_v2_pipelines.build_action(\n name='mount-wait-{}'.format(bucket),\n flags=['ENABLE_FUSE'],\n image_uri=_GCSFUSE_IMAGE,\n mounts=[mnt_datadisk],\n commands=[\n 'wait',\n os.path.join(providers_util.DATA_MOUNT_POINT, mount_path)\n ])\n ])\n return actions_to_add"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _build_pipeline_request(self, task_view):\n job_metadata = task_view.job_metadata\n job_params = task_view.job_params\n job_resources = task_view.job_resources\n task_metadata = task_view.task_descriptors[0].task_metadata\n task_params = task_view.task_descriptors[0].task_params\n task_resources = task_view.task_descriptors[0].task_resources\n\n # Set up VM-specific variables\n mnt_datadisk = google_v2_pipelines.build_mount(\n disk=_DATA_DISK_NAME,\n path=providers_util.DATA_MOUNT_POINT,\n read_only=False)\n scopes = job_resources.scopes or google_base.DEFAULT_SCOPES\n\n # Set up the task labels\n labels = {\n label.name: label.value if label.value else '' for label in\n google_base.build_pipeline_labels(job_metadata, task_metadata)\n | job_params['labels'] | task_params['labels']\n }\n\n # Set local variables for the core pipeline values\n script = task_view.job_metadata['script']\n user_project = task_view.job_metadata['user-project'] or ''\n\n envs = job_params['envs'] | task_params['envs']\n inputs = job_params['inputs'] | task_params['inputs']\n outputs = job_params['outputs'] | task_params['outputs']\n mounts = job_params['mounts']\n gcs_mounts = param_util.get_gcs_mounts(mounts)\n\n persistent_disk_mount_params = param_util.get_persistent_disk_mounts(mounts)\n\n persistent_disks = [\n google_v2_pipelines.build_disk(\n name=disk.name.replace('_', '-'), # Underscores not allowed\n size_gb=disk.disk_size or job_model.DEFAULT_MOUNTED_DISK_SIZE,\n source_image=disk.value,\n disk_type=disk.disk_type or job_model.DEFAULT_DISK_TYPE)\n for disk in persistent_disk_mount_params\n ]\n persistent_disk_mounts = [\n google_v2_pipelines.build_mount(\n disk=persistent_disk.get('name'),\n path=os.path.join(providers_util.DATA_MOUNT_POINT,\n persistent_disk_mount_param.docker_path),\n read_only=True)\n for persistent_disk, persistent_disk_mount_param in zip(\n persistent_disks, persistent_disk_mount_params)\n ]\n\n # The list of \"actions\" (1-based) will be:\n # 1- continuous copy of log files off to Cloud Storage\n # 2- prepare the shared mount point (write the user script)\n # 3- localize objects from Cloud Storage to block storage\n # 4- execute user command\n # 5- delocalize objects from block storage to Cloud Storage\n # 6- final copy of log files off to Cloud Storage\n #\n # If the user has requested an SSH server be started, it will be inserted\n # after logging is started, and all subsequent action numbers above will be\n # incremented by 1.\n # If the user has requested to mount one or more buckets, two actions per\n # bucket will be inserted after the prepare step, and all subsequent action\n # numbers will be incremented by the number of actions added.\n #\n # We need to track the action numbers specifically for the user action and\n # the final logging action.\n optional_actions = 0\n if job_resources.ssh:\n optional_actions += 1\n\n mount_actions = self._get_mount_actions(gcs_mounts, mnt_datadisk)\n optional_actions += len(mount_actions)\n\n user_action = 4 + optional_actions\n final_logging_action = 6 + optional_actions\n\n # Set up the commands and environment for the logging actions\n logging_cmd = _LOGGING_CMD.format(\n log_cp_fn=_GSUTIL_CP_FN,\n log_cp_cmd=_LOG_CP_CMD.format(\n user_action=user_action, logging_action='logging_action'))\n continuous_logging_cmd = _CONTINUOUS_LOGGING_CMD.format(\n log_msg_fn=_LOG_MSG_FN,\n log_cp_fn=_GSUTIL_CP_FN,\n log_cp_cmd=_LOG_CP_CMD.format(\n user_action=user_action,\n logging_action='continuous_logging_action'),\n final_logging_action=final_logging_action,\n log_interval=job_resources.log_interval or '60s')\n logging_env = self._get_logging_env(task_resources.logging_path.uri,\n user_project)\n\n # Set up command and environments for the prepare, localization, user,\n # and de-localization actions\n script_path = os.path.join(providers_util.SCRIPT_DIR, script.name)\n prepare_command = _PREPARE_CMD.format(\n log_msg_fn=_LOG_MSG_FN,\n mk_runtime_dirs=_MK_RUNTIME_DIRS_CMD,\n script_var=_SCRIPT_VARNAME,\n python_decode_script=_PYTHON_DECODE_SCRIPT,\n script_path=script_path,\n mk_io_dirs=_MK_IO_DIRS)\n\n prepare_env = self._get_prepare_env(script, task_view, inputs, outputs,\n mounts)\n localization_env = self._get_localization_env(inputs, user_project)\n user_environment = self._build_user_environment(envs, inputs, outputs,\n mounts)\n delocalization_env = self._get_delocalization_env(outputs, user_project)\n\n # Build the list of actions\n actions = []\n actions.append(\n google_v2_pipelines.build_action(\n name='logging',\n flags='RUN_IN_BACKGROUND',\n image_uri=_CLOUD_SDK_IMAGE,\n environment=logging_env,\n entrypoint='/bin/bash',\n commands=['-c', continuous_logging_cmd]))\n\n if job_resources.ssh:\n actions.append(\n google_v2_pipelines.build_action(\n name='ssh',\n image_uri=_SSH_IMAGE,\n mounts=[mnt_datadisk],\n entrypoint='ssh-server',\n port_mappings={_DEFAULT_SSH_PORT: _DEFAULT_SSH_PORT},\n flags='RUN_IN_BACKGROUND'))\n\n actions.append(\n google_v2_pipelines.build_action(\n name='prepare',\n image_uri=_PYTHON_IMAGE,\n mounts=[mnt_datadisk],\n environment=prepare_env,\n entrypoint='/bin/bash',\n commands=['-c', prepare_command]),)\n\n actions.extend(mount_actions)\n\n actions.extend([\n google_v2_pipelines.build_action(\n name='localization',\n image_uri=_CLOUD_SDK_IMAGE,\n mounts=[mnt_datadisk],\n environment=localization_env,\n entrypoint='/bin/bash',\n commands=[\n '-c',\n _LOCALIZATION_CMD.format(\n log_msg_fn=_LOG_MSG_FN,\n recursive_cp_fn=_GSUTIL_RSYNC_FN,\n cp_fn=_GSUTIL_CP_FN,\n cp_loop=_LOCALIZATION_LOOP)\n ]),\n google_v2_pipelines.build_action(\n name='user-command',\n image_uri=job_resources.image,\n mounts=[mnt_datadisk] + persistent_disk_mounts,\n environment=user_environment,\n entrypoint='/usr/bin/env',\n commands=[\n 'bash', '-c',\n _USER_CMD.format(\n tmp_dir=providers_util.TMP_DIR,\n working_dir=providers_util.WORKING_DIR,\n user_script=script_path)\n ]),\n google_v2_pipelines.build_action(\n name='delocalization',\n image_uri=_CLOUD_SDK_IMAGE,\n mounts=[mnt_datadisk],\n environment=delocalization_env,\n entrypoint='/bin/bash',\n commands=[\n '-c',\n _LOCALIZATION_CMD.format(\n log_msg_fn=_LOG_MSG_FN,\n recursive_cp_fn=_GSUTIL_RSYNC_FN,\n cp_fn=_GSUTIL_CP_FN,\n cp_loop=_DELOCALIZATION_LOOP)\n ]),\n google_v2_pipelines.build_action(\n name='final_logging',\n flags='ALWAYS_RUN',\n image_uri=_CLOUD_SDK_IMAGE,\n environment=logging_env,\n entrypoint='/bin/bash',\n commands=['-c', logging_cmd]),\n ])\n\n assert len(actions) - 2 == user_action\n assert len(actions) == final_logging_action\n\n # Prepare the VM (resources) configuration\n disks = [\n google_v2_pipelines.build_disk(\n _DATA_DISK_NAME,\n job_resources.disk_size,\n source_image=None,\n disk_type=job_resources.disk_type or job_model.DEFAULT_DISK_TYPE)\n ]\n disks.extend(persistent_disks)\n network = google_v2_pipelines.build_network(\n job_resources.network, job_resources.subnetwork,\n job_resources.use_private_address)\n if job_resources.machine_type:\n machine_type = job_resources.machine_type\n elif job_resources.min_cores or job_resources.min_ram:\n machine_type = GoogleV2CustomMachine.build_machine_type(\n job_resources.min_cores, job_resources.min_ram)\n else:\n machine_type = job_model.DEFAULT_MACHINE_TYPE\n accelerators = None\n if job_resources.accelerator_type:\n accelerators = [\n google_v2_pipelines.build_accelerator(job_resources.accelerator_type,\n job_resources.accelerator_count)\n ]\n service_account = google_v2_pipelines.build_service_account(\n job_resources.service_account or 'default', scopes)\n\n resources = google_v2_pipelines.build_resources(\n self._project,\n job_resources.regions,\n google_base.get_zones(job_resources.zones),\n google_v2_pipelines.build_machine(\n network=network,\n machine_type=machine_type,\n preemptible=job_resources.preemptible,\n service_account=service_account,\n boot_disk_size_gb=job_resources.boot_disk_size,\n disks=disks,\n accelerators=accelerators,\n nvidia_driver_version=job_resources.nvidia_driver_version,\n labels=labels,\n cpu_platform=job_resources.cpu_platform),\n )\n\n # Build the pipeline request\n pipeline = google_v2_pipelines.build_pipeline(actions, resources, None,\n job_resources.timeout)\n\n return {'pipeline': pipeline, 'labels': labels}", "response": "Builds a request for the pipeline for the task."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsubmits a job to the pipeline.", "response": "def submit_job(self, job_descriptor, skip_if_output_present):\n \"\"\"Submit the job (or tasks) to be executed.\n\n Args:\n job_descriptor: all parameters needed to launch all job tasks\n skip_if_output_present: (boolean) if true, skip tasks whose output\n is present (see --skip flag for more explanation).\n\n Returns:\n A dictionary containing the 'user-id', 'job-id', and 'task-id' list.\n For jobs that are not task array jobs, the task-id list should be empty.\n\n Raises:\n ValueError: if job resources or task data contain illegal values.\n \"\"\"\n # Validate task data and resources.\n param_util.validate_submit_args_or_fail(\n job_descriptor,\n provider_name=_PROVIDER_NAME,\n input_providers=_SUPPORTED_INPUT_PROVIDERS,\n output_providers=_SUPPORTED_OUTPUT_PROVIDERS,\n logging_providers=_SUPPORTED_LOGGING_PROVIDERS)\n\n # Prepare and submit jobs.\n launched_tasks = []\n requests = []\n for task_view in job_model.task_view_generator(job_descriptor):\n\n job_params = task_view.job_params\n task_params = task_view.task_descriptors[0].task_params\n\n outputs = job_params['outputs'] | task_params['outputs']\n if skip_if_output_present:\n # check whether the output's already there\n if dsub_util.outputs_are_present(outputs):\n print('Skipping task because its outputs are present')\n continue\n\n request = self._build_pipeline_request(task_view)\n\n if self._dry_run:\n requests.append(request)\n else:\n task_id = self._submit_pipeline(request)\n launched_tasks.append(task_id)\n\n # If this is a dry-run, emit all the pipeline request objects\n if self._dry_run:\n print(\n json.dumps(\n requests, indent=2, sort_keys=True, separators=(',', ': ')))\n\n if not requests and not launched_tasks:\n return {'job-id': dsub_util.NO_JOB}\n\n return {\n 'job-id': job_descriptor.job_metadata['job-id'],\n 'user-id': job_descriptor.job_metadata['user-id'],\n 'task-id': [task_id for task_id in launched_tasks if task_id],\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the list of operations for the specified filter.", "response": "def _operations_list(self, ops_filter, max_tasks, page_size, page_token):\n \"\"\"Gets the list of operations for the specified filter.\n\n Args:\n ops_filter: string filter of operations to return\n max_tasks: the maximum number of job tasks to return or 0 for no limit.\n page_size: the number of operations to requested on each list operation to\n the pipelines API (if 0 or None, the API default is used)\n page_token: page token returned by a previous _operations_list call.\n\n Returns:\n Operations matching the filter criteria.\n \"\"\"\n\n # We are not using the documented default page size of 256,\n # nor allowing for the maximum page size of 2048 as larger page sizes\n # currently cause the operations.list() API to return an error:\n # HttpError 429 ... Resource has been exhausted (e.g. check quota).\n max_page_size = 128\n\n # Set the page size to the smallest (non-zero) size we can\n page_size = min(sz for sz in [page_size, max_page_size, max_tasks] if sz)\n\n # Execute operations.list() and return all of the dsub operations\n api = self._service.projects().operations().list(\n name='projects/{}/operations'.format(self._project),\n filter=ops_filter,\n pageToken=page_token,\n pageSize=page_size)\n response = google_base.Api.execute(api)\n\n return [\n GoogleOperation(op)\n for op in response.get('operations', [])\n if google_v2_operations.is_dsub_operation(op)\n ], response.get('nextPageToken')"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a list of Genomics API Operations objects that match the input criteria.", "response": "def lookup_job_tasks(self,\n statuses,\n user_ids=None,\n job_ids=None,\n job_names=None,\n task_ids=None,\n task_attempts=None,\n labels=None,\n create_time_min=None,\n create_time_max=None,\n max_tasks=0,\n page_size=0):\n \"\"\"Yields operations based on the input criteria.\n\n If any of the filters are empty or {'*'}, then no filtering is performed on\n that field. Filtering by both a job id list and job name list is\n unsupported.\n\n Args:\n statuses: {'*'}, or a list of job status strings to return. Valid\n status strings are 'RUNNING', 'SUCCESS', 'FAILURE', or 'CANCELED'.\n user_ids: a list of ids for the user(s) who launched the job.\n job_ids: a list of job ids to return.\n job_names: a list of job names to return.\n task_ids: a list of specific tasks within the specified job(s) to return.\n task_attempts: a list of specific attempts within the specified tasks(s)\n to return.\n labels: a list of LabelParam with user-added labels. All labels must\n match the task being fetched.\n create_time_min: a timezone-aware datetime value for the earliest create\n time of a task, inclusive.\n create_time_max: a timezone-aware datetime value for the most recent\n create time of a task, inclusive.\n max_tasks: the maximum number of job tasks to return or 0 for no limit.\n page_size: the page size to use for each query to the pipelins API.\n\n Raises:\n ValueError: if both a job id list and a job name list are provided\n\n Yeilds:\n Genomics API Operations objects.\n \"\"\"\n\n # Build a filter for operations to return\n ops_filter = self._build_query_filter(\n statuses, user_ids, job_ids, job_names, task_ids, task_attempts, labels,\n create_time_min, create_time_max)\n\n # Execute the operations.list() API to get batches of operations to yield\n page_token = None\n tasks_yielded = 0\n while True:\n # If max_tasks is set, let operations.list() know not to send more than\n # we need.\n max_to_fetch = None\n if max_tasks:\n max_to_fetch = max_tasks - tasks_yielded\n ops, page_token = self._operations_list(ops_filter, max_to_fetch,\n page_size, page_token)\n\n for op in ops:\n yield op\n tasks_yielded += 1\n\n assert (max_tasks >= tasks_yielded or not max_tasks)\n if not page_token or 0 < max_tasks <= tasks_yielded:\n break"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _operation_status(self):\n if not google_v2_operations.is_done(self._op):\n return 'RUNNING'\n if google_v2_operations.is_success(self._op):\n return 'SUCCESS'\n if google_v2_operations.is_canceled(self._op):\n return 'CANCELED'\n if google_v2_operations.is_failed(self._op):\n return 'FAILURE'\n\n raise ValueError('Status for operation {} could not be determined'.format(\n self._op['name']))", "response": "Returns the status of this operation."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _operation_status_message(self):\n msg = None\n action = None\n if not google_v2_operations.is_done(self._op):\n last_event = google_v2_operations.get_last_event(self._op)\n if last_event:\n msg = last_event['description']\n action_id = last_event.get('details', {}).get('actionId')\n if action_id:\n action = google_v2_operations.get_action_by_id(self._op, action_id)\n else:\n msg = 'Pending'\n else:\n failed_events = google_v2_operations.get_failed_events(self._op)\n if failed_events:\n failed_event = failed_events[-1]\n msg = failed_event.get('details', {}).get('stderr')\n action_id = failed_event.get('details', {}).get('actionId')\n if action_id:\n action = google_v2_operations.get_action_by_id(self._op, action_id)\n if not msg:\n error = google_v2_operations.get_error(self._op)\n if error:\n msg = error['message']\n else:\n msg = 'Success'\n\n return msg, action", "response": "Returns the most relevant status string and name of failed action."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns an error message if the operation failed for any reason.", "response": "def error_message(self):\n \"\"\"Returns an error message if the operation failed for any reason.\n\n Failure as defined here means ended for any reason other than 'success'.\n This means that a successful cancelation will also return an error message.\n\n Returns:\n string, string will be empty if job did not error.\n \"\"\"\n error = google_v2_operations.get_error(self._op)\n if error:\n job_id = self.get_field('job-id')\n task_id = self.get_field('task-id')\n task_str = job_id if task_id is None else '{} (task: {})'.format(\n job_id, task_id)\n\n return 'Error in {} - code {}: {}'.format(task_str, error['code'],\n error['message'])\n\n return ''"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a value from the operation for a specific field.", "response": "def get_field(self, field, default=None):\n \"\"\"Returns a value from the operation for a specific set of field names.\n\n Args:\n field: a dsub-specific job metadata key\n default: default value to return if field does not exist or is empty.\n\n Returns:\n A text string for the field or a list for 'inputs'.\n\n Raises:\n ValueError: if the field label is not supported by the operation\n \"\"\"\n\n value = None\n if field == 'internal-id':\n value = self._op['name']\n elif field == 'user-project':\n if self._job_descriptor:\n value = self._job_descriptor.job_metadata.get(field)\n elif field in [\n 'job-id', 'job-name', 'task-id', 'task-attempt', 'user-id',\n 'dsub-version'\n ]:\n value = google_v2_operations.get_label(self._op, field)\n elif field == 'task-status':\n value = self._operation_status()\n elif field == 'logging':\n if self._job_descriptor:\n # The job_resources will contain the \"--logging\" value.\n # The task_resources will contain the resolved logging path.\n # Return the resolved logging path.\n task_resources = self._job_descriptor.task_descriptors[0].task_resources\n value = task_resources.logging_path\n\n elif field in ['envs', 'labels']:\n if self._job_descriptor:\n items = providers_util.get_job_and_task_param(\n self._job_descriptor.job_params,\n self._job_descriptor.task_descriptors[0].task_params, field)\n value = {item.name: item.value for item in items}\n elif field in [\n 'inputs', 'outputs', 'input-recursives', 'output-recursives'\n ]:\n if self._job_descriptor:\n value = {}\n items = providers_util.get_job_and_task_param(\n self._job_descriptor.job_params,\n self._job_descriptor.task_descriptors[0].task_params, field)\n value.update({item.name: item.value for item in items})\n elif field == 'mounts':\n if self._job_descriptor:\n items = providers_util.get_job_and_task_param(\n self._job_descriptor.job_params,\n self._job_descriptor.task_descriptors[0].task_params, field)\n value = {item.name: item.value for item in items}\n elif field == 'create-time':\n ds = google_v2_operations.get_create_time(self._op)\n value = google_base.parse_rfc3339_utc_string(ds)\n elif field == 'start-time':\n ds = google_v2_operations.get_start_time(self._op)\n if ds:\n value = google_base.parse_rfc3339_utc_string(ds)\n elif field == 'end-time':\n ds = google_v2_operations.get_end_time(self._op)\n if ds:\n value = google_base.parse_rfc3339_utc_string(ds)\n elif field == 'status':\n value = self._operation_status()\n elif field == 'status-message':\n msg, action = self._operation_status_message()\n if msg.startswith('Execution failed:'):\n # msg may look something like\n # \"Execution failed: action 2: pulling image...\"\n # Emit the actual message (\"pulling image...\")\n msg = msg.split(': ', 2)[-1]\n value = msg\n elif field == 'status-detail':\n msg, action = self._operation_status_message()\n if action:\n value = '{}:\\n{}'.format(action.get('name'), msg)\n else:\n value = msg\n elif field == 'last-update':\n last_update = google_v2_operations.get_last_update(self._op)\n if last_update:\n value = google_base.parse_rfc3339_utc_string(last_update)\n elif field == 'provider':\n return _PROVIDER_NAME\n elif field == 'provider-attributes':\n value = {}\n\n # The VM instance name and zone can be found in the WorkerAssignedEvent.\n # For a given operation, this may have occurred multiple times, so be\n # sure to grab the most recent.\n assigned_events = google_v2_operations.get_worker_assigned_events(\n self._op)\n if assigned_events:\n details = assigned_events[0].get('details', {})\n value['instance-name'] = details.get('instance')\n value['zone'] = details.get('zone')\n\n # The rest of the information comes from the request itself.\n resources = google_v2_operations.get_resources(self._op)\n if 'regions' in resources:\n value['regions'] = resources['regions']\n if 'zones' in resources:\n value['zones'] = resources['zones']\n if 'virtualMachine' in resources:\n vm = resources['virtualMachine']\n value['machine-type'] = vm.get('machineType')\n value['preemptible'] = vm.get('preemptible')\n\n value['boot-disk-size'] = vm.get('bootDiskSizeGb')\n value['network'] = vm.get('network', {}).get('name')\n value['subnetwork'] = vm.get('network', {}).get('subnetwork')\n value['use_private_address'] = vm.get('network',\n {}).get('usePrivateAddress')\n value['cpu_platform'] = vm.get('cpuPlatform')\n value['accelerators'] = vm.get('accelerators')\n value['service-account'] = vm.get('serviceAccount', {}).get('email')\n if 'disks' in vm:\n datadisk = next(\n (d for d in vm['disks'] if d['name'] == _DATA_DISK_NAME))\n if datadisk:\n value['disk-size'] = datadisk.get('sizeGb')\n value['disk-type'] = datadisk.get('type')\n elif field == 'events':\n value = GoogleV2EventMap(self._op).get_filtered_normalized_events()\n elif field == 'script-name':\n if self._job_descriptor:\n value = self._job_descriptor.job_metadata.get(field)\n elif field == 'script':\n value = self._try_op_to_script_body()\n else:\n raise ValueError('Unsupported field: \"%s\"' % field)\n\n return value if value else default"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate the amount of memory required to run the VM.", "response": "def _validate_ram(ram_in_mb):\n \"\"\"Rounds ram up to the nearest multiple of _MEMORY_MULTIPLE.\"\"\"\n return int(GoogleV2CustomMachine._MEMORY_MULTIPLE * math.ceil(\n ram_in_mb / GoogleV2CustomMachine._MEMORY_MULTIPLE))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbuilds a custom machine type string.", "response": "def build_machine_type(cls, min_cores, min_ram):\n \"\"\"Returns a custom machine type string.\"\"\"\n min_cores = min_cores or job_model.DEFAULT_MIN_CORES\n min_ram = min_ram or job_model.DEFAULT_MIN_RAM\n\n # First, min_ram is given in GB. Convert to MB.\n min_ram *= GoogleV2CustomMachine._MB_PER_GB\n\n # Only machine types with 1 vCPU or an even number of vCPUs can be created.\n cores = cls._validate_cores(min_cores)\n # The total memory of the instance must be a multiple of 256 MB.\n ram = cls._validate_ram(min_ram)\n\n # Memory must be between 0.9 GB per vCPU, up to 6.5 GB per vCPU.\n memory_to_cpu_ratio = ram / cores\n\n if memory_to_cpu_ratio < GoogleV2CustomMachine._MIN_MEMORY_PER_CPU:\n # If we're under the ratio, top up the memory.\n adjusted_ram = GoogleV2CustomMachine._MIN_MEMORY_PER_CPU * cores\n ram = cls._validate_ram(adjusted_ram)\n\n elif memory_to_cpu_ratio > GoogleV2CustomMachine._MAX_MEMORY_PER_CPU:\n # If we're over the ratio, top up the CPU.\n adjusted_cores = math.ceil(\n ram / GoogleV2CustomMachine._MAX_MEMORY_PER_CPU)\n cores = cls._validate_cores(adjusted_cores)\n\n else:\n # Ratio is within the restrictions - no adjustments needed.\n pass\n\n return 'custom-{}-{}'.format(int(cores), int(ram))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef build_machine(network=None,\n machine_type=None,\n preemptible=None,\n service_account=None,\n boot_disk_size_gb=None,\n disks=None,\n accelerators=None,\n labels=None,\n cpu_platform=None,\n nvidia_driver_version=None):\n \"\"\"Build a VirtualMachine object for a Pipeline request.\n\n Args:\n network (dict): Network details for the pipeline to run in.\n machine_type (str): GCE Machine Type string for the pipeline.\n preemptible (bool): Use a preemptible VM for the job.\n service_account (dict): Service account configuration for the VM.\n boot_disk_size_gb (int): Boot disk size in GB.\n disks (list[dict]): List of disks to mount.\n accelerators (list[dict]): List of accelerators to attach to the VM.\n labels (dict[string, string]): Labels for the VM.\n cpu_platform (str): The CPU platform to request.\n nvidia_driver_version (str): The NVIDIA driver version to use when attaching\n an NVIDIA GPU accelerator.\n\n Returns:\n An object representing a VirtualMachine.\n \"\"\"\n return {\n 'network': network,\n 'machineType': machine_type,\n 'preemptible': preemptible,\n 'serviceAccount': service_account,\n 'bootDiskSizeGb': boot_disk_size_gb,\n 'disks': disks,\n 'accelerators': accelerators,\n 'labels': labels,\n 'cpuPlatform': cpu_platform,\n 'nvidiaDriverVersion': nvidia_driver_version,\n }", "response": "Builds a GCE VM object for a Pipeline request."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nbuilds an Action object for a Pipeline request.", "response": "def build_action(name=None,\n image_uri=None,\n commands=None,\n entrypoint=None,\n environment=None,\n pid_namespace=None,\n flags=None,\n port_mappings=None,\n mounts=None,\n labels=None):\n \"\"\"Build an Action object for a Pipeline request.\n\n Args:\n name (str): An optional name for the container.\n image_uri (str): The URI to pull the container image from.\n commands (List[str]): commands and arguments to run inside the container.\n entrypoint (str): overrides the ENTRYPOINT specified in the container.\n environment (dict[str,str]): The environment to pass into the container.\n pid_namespace (str): The PID namespace to run the action inside.\n flags (str): Flags that control the execution of this action.\n port_mappings (dict[int, int]): A map of container to host port mappings for\n this container.\n mounts (List): A list of mounts to make available to the action.\n labels (dict[str]): Labels to associate with the action.\n\n Returns:\n An object representing an Action resource.\n \"\"\"\n\n return {\n 'name': name,\n 'imageUri': image_uri,\n 'commands': commands,\n 'entrypoint': entrypoint,\n 'environment': environment,\n 'pidNamespace': pid_namespace,\n 'flags': flags,\n 'portMappings': port_mappings,\n 'mounts': mounts,\n 'labels': labels,\n }"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a list of operations that are part of the job.", "response": "def lookup_job_tasks(self,\n statuses,\n user_ids=None,\n job_ids=None,\n job_names=None,\n task_ids=None,\n task_attempts=None,\n labels=None,\n create_time_min=None,\n create_time_max=None,\n max_tasks=0):\n \"\"\"Return a list of operations. See base.py for additional detail.\"\"\"\n statuses = None if statuses == {'*'} else statuses\n user_ids = None if user_ids == {'*'} else user_ids\n job_ids = None if job_ids == {'*'} else job_ids\n job_names = None if job_names == {'*'} else job_names\n task_ids = None if task_ids == {'*'} else task_ids\n task_attempts = None if task_attempts == {'*'} else task_attempts\n\n if labels or create_time_min or create_time_max:\n raise NotImplementedError(\n 'Lookup by labels and create_time not yet supported by stub.')\n\n operations = [\n x for x in self._operations\n if ((not statuses or x.get_field('status', (None, None))[0] in statuses\n ) and (not user_ids or x.get_field('user', None) in user_ids) and\n (not job_ids or x.get_field('job-id', None) in job_ids) and\n (not job_names or x.get_field('job-name', None) in job_names) and\n (not task_ids or x.get_field('task-id', None) in task_ids) and\n (not task_attempts or\n x.get_field('task-attempt', None) in task_attempts))\n ]\n if max_tasks > 0:\n operations = operations[:max_tasks]\n return operations"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a provider for job submission requests.", "response": "def get_provider(args, resources):\n \"\"\"Returns a provider for job submission requests.\"\"\"\n\n provider = getattr(args, 'provider', 'google')\n\n if provider == 'google':\n return google.GoogleJobProvider(\n getattr(args, 'verbose', False),\n getattr(args, 'dry_run', False), args.project)\n elif provider == 'google-v2':\n return google_v2.GoogleV2JobProvider(\n getattr(args, 'verbose', False), getattr(args, 'dry_run', False),\n args.project)\n elif provider == 'local':\n return local.LocalJobProvider(resources)\n elif provider == 'test-fails':\n return test_fails.FailsJobProvider()\n else:\n raise ValueError('Unknown provider: ' + provider)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates an argument parser with the specified provider.", "response": "def create_parser(prog):\n \"\"\"Create an argument parser, adding in the list of providers.\"\"\"\n parser = argparse.ArgumentParser(prog=prog, formatter_class=DsubHelpFormatter)\n\n parser.add_argument(\n '--provider',\n default='google-v2',\n choices=['local', 'google', 'google-v2', 'test-fails'],\n help=\"\"\"Job service provider. Valid values are \"google-v2\" (Google's\n Pipeline API v2) and \"local\" (local Docker execution). \"test-*\"\n providers are for testing purposes only.\"\"\",\n metavar='PROVIDER')\n\n return parser"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_args(parser, provider_required_args, argv):\n\n # Add the provider required arguments epilog message\n epilog = 'Provider-required arguments:\\n'\n for provider in provider_required_args:\n epilog += ' %s: %s\\n' % (provider, provider_required_args[provider])\n parser.epilog = epilog\n\n # Parse arguments\n args = parser.parse_args(argv)\n\n # For the selected provider, check the required arguments\n for arg in provider_required_args[args.provider]:\n if not args.__getattribute__(arg):\n parser.error('argument --%s is required' % arg)\n\n return args", "response": "Parse the arguments from the command line."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the arguments to point dstat to the same provider + project.", "response": "def get_dstat_provider_args(provider, project):\n \"\"\"A string with the arguments to point dstat to the same provider+project.\"\"\"\n provider_name = get_provider_name(provider)\n\n args = []\n if provider_name == 'google':\n args.append('--project %s' % project)\n elif provider_name == 'google-v2':\n args.append('--project %s' % project)\n elif provider_name == 'local':\n pass\n elif provider_name == 'test-fails':\n pass\n else:\n # New providers should add their dstat required arguments here.\n assert False, 'Provider %s needs get_dstat_provider_args support' % provider\n\n args.insert(0, '--provider %s' % provider_name)\n return ' '.join(args)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _format_task_uri(fmt, job_metadata, task_metadata):\n\n values = {\n 'job-id': None,\n 'task-id': 'task',\n 'job-name': None,\n 'user-id': None,\n 'task-attempt': None\n }\n for key in values:\n values[key] = task_metadata.get(key) or job_metadata.get(key) or values[key]\n\n return fmt.format(**values)", "response": "Returns a URI with placeholders replaced by metadata values."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nformatting the logging URI.", "response": "def format_logging_uri(uri, job_metadata, task_metadata):\n \"\"\"Inserts task metadata into the logging URI.\n\n The core behavior is inspired by the Google Pipelines API:\n (1) If a the uri ends in \".log\", then that is the logging path.\n (2) Otherwise, the uri is treated as \"directory\" for logs and a filename\n needs to be automatically generated.\n\n For (1), if the job is a --tasks job, then the {task-id} is inserted\n before \".log\".\n\n For (2), the file name generated is {job-id}, or for --tasks jobs, it is\n {job-id}.{task-id}.\n\n In both cases .{task-attempt} is inserted before .log for --retries jobs.\n\n In addition, full task metadata substitution is supported. The URI\n may include substitution strings such as\n \"{job-id}\", \"{task-id}\", \"{job-name}\", \"{user-id}\", and \"{task-attempt}\".\n\n Args:\n uri: User-specified logging URI which may contain substitution fields.\n job_metadata: job-global metadata.\n task_metadata: tasks-specific metadata.\n\n Returns:\n The logging_uri formatted as described above.\n \"\"\"\n\n # If the user specifies any formatting (with curly braces), then use that\n # as the format string unchanged.\n fmt = str(uri)\n if '{' not in fmt:\n if uri.endswith('.log'):\n # URI includes a filename. Trim the extension and just use the prefix.\n fmt = os.path.splitext(uri)[0]\n else:\n # URI is a path to a directory. The job-id becomes the filename prefix.\n fmt = os.path.join(uri, '{job-id}')\n\n # If this is a task job, add the task-id.\n if task_metadata.get('task-id') is not None:\n fmt += '.{task-id}'\n\n # If this is a retryable task, add the task-attempt.\n if task_metadata.get('task-attempt') is not None:\n fmt += '.{task-attempt}'\n\n fmt += '.log'\n\n return _format_task_uri(fmt, job_metadata, task_metadata)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _google_v2_parse_arguments(args):\n if (args.zones and args.regions) or (not args.zones and not args.regions):\n raise ValueError('Exactly one of --regions and --zones must be specified')\n\n if args.machine_type and (args.min_cores or args.min_ram):\n raise ValueError(\n '--machine-type not supported together with --min-cores or --min-ram.')", "response": "Validated google - v2 arguments."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses command line arguments and returns a Namespace of parsed arguments.", "response": "def _parse_arguments(prog, argv):\n \"\"\"Parses command line arguments.\n\n Args:\n prog: The path of the program (dsub.py) or an alternate program name to\n display in usage.\n argv: The list of program arguments to parse.\n\n Returns:\n A Namespace of parsed arguments.\n \"\"\"\n # Handle version flag and exit if it was passed.\n param_util.handle_version_flag()\n\n parser = provider_base.create_parser(prog)\n\n # Add dsub core job submission arguments\n parser.add_argument(\n '--version', '-v', default=False, help='Print the dsub version and exit.')\n\n parser.add_argument(\n '--unique-job-id',\n default=False,\n action='store_true',\n help=\"\"\"Experimental: create a unique 32 character UUID for the dsub\n job-id using https://docs.python.org/3/library/uuid.html.\"\"\")\n parser.add_argument(\n '--name',\n help=\"\"\"Name for pipeline. Defaults to the script name or\n first token of the --command if specified.\"\"\")\n parser.add_argument(\n '--tasks',\n nargs='*',\n action=TaskParamAction,\n help=\"\"\"Path to a file of tab separated values (TSV) for task parameters.\n The file may be located in the local filesystem or in a Google Cloud\n Storage bucket.\n\n The first line is a list of column headers specifying an --env,\n --input, --input-recursive, --output or --output-recursive variable,\n and each subsequent line specifies the values for a task.\n\n Optionally specify tasks from the file to submit. Can take the form\n \"m\", \"m-\", or \"m-n\" where m and n are task numbers starting at 1.\"\"\",\n metavar='FILE M-N')\n parser.add_argument(\n '--image',\n default='ubuntu:14.04',\n help=\"\"\"Image name from Docker Hub, Google Container Repository, or other\n Docker image service. The pipeline must have READ access to the\n image.\"\"\")\n parser.add_argument(\n '--dry-run',\n default=False,\n action='store_true',\n help='Print the pipeline(s) that would be run and then exit.')\n parser.add_argument(\n '--command',\n help=\"\"\"Command to run inside the job\\'s Docker container. This argument\n or the --script argument must be provided.\"\"\",\n metavar='COMMAND')\n parser.add_argument(\n '--script',\n help=\"\"\"Path to a script that is located in the local file system or\n inside a Google Cloud Storage bucket. This script will be run inside\n the job\\'s Docker container. This argument or the --command\n argument must be provided.\"\"\",\n metavar='SCRIPT')\n parser.add_argument(\n '--env',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help='Environment variables for the script\\'s execution environment',\n metavar='KEY=VALUE')\n parser.add_argument(\n '--label',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help='Labels to associate to the job.',\n metavar='KEY=VALUE')\n parser.add_argument(\n '--input',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help=\"\"\"Input path arguments to localize into the script's execution\n environment\"\"\",\n metavar='KEY=REMOTE_PATH')\n parser.add_argument(\n '--input-recursive',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help=\"\"\"Input path arguments to localize recursively into the script\\'s\n execution environment\"\"\",\n metavar='KEY=REMOTE_PATH')\n parser.add_argument(\n '--output',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help=\"\"\"Output path arguments to de-localize from the script\\'s execution\n environment\"\"\",\n metavar='KEY=REMOTE_PATH')\n parser.add_argument(\n '--output-recursive',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help=\"\"\"Output path arguments to de-localize recursively from the script's\n execution environment\"\"\",\n metavar='KEY=REMOTE_PATH')\n parser.add_argument(\n '--user',\n '-u',\n help='User submitting the dsub job, defaults to the current OS user.')\n parser.add_argument(\n '--user-project',\n help=\"\"\"Specify a user project to be billed for all requests to Google\n Cloud Storage (logging, localization, delocalization). This flag exists\n to support accessing Requester Pays buckets\"\"\")\n parser.add_argument(\n '--mount',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help=\"\"\"Mount a resource such as a bucket, disk, or directory into your\n Docker container\"\"\",\n metavar='KEY=PATH_SPEC')\n\n # Add dsub job management arguments\n parser.add_argument(\n '--wait',\n action='store_true',\n help='Wait for the job to finish all its tasks.')\n parser.add_argument(\n '--retries',\n default=0,\n type=int,\n help='Number of retries to perform on failed tasks.')\n parser.add_argument(\n '--poll-interval',\n default=10,\n type=int,\n help='Polling interval (in seconds) for checking job status '\n 'when --wait or --after are set.')\n parser.add_argument(\n '--after',\n nargs='+',\n default=[],\n help='Job ID(s) to wait for before starting this job.')\n parser.add_argument(\n '--skip',\n default=False,\n action='store_true',\n help=\"\"\"Do not submit the job if all output specified using the --output\n and --output-recursive parameters already exist. Note that wildcard\n and recursive outputs cannot be strictly verified. See the\n documentation for details.\"\"\")\n\n # Add dsub resource requirement arguments\n parser.add_argument(\n '--min-cores',\n type=int,\n help='Minimum CPU cores for each job')\n parser.add_argument(\n '--min-ram',\n type=float,\n help='Minimum RAM per job in GB')\n parser.add_argument(\n '--disk-size',\n default=job_model.DEFAULT_DISK_SIZE,\n type=int,\n help='Size (in GB) of data disk to attach for each job')\n\n parser.add_argument(\n '--logging',\n help='Cloud Storage path to send logging output'\n ' (either a folder, or file ending in \".log\")')\n\n # Add provider-specific arguments\n\n # Shared arguments between the \"google\" and \"google-v2\" providers\n google_common = parser.add_argument_group(\n title='google-common',\n description='Options common to the \"google\" and \"google-v2\" providers')\n google_common.add_argument(\n '--project', help='Cloud project ID in which to run the pipeline')\n google_common.add_argument(\n '--boot-disk-size',\n default=job_model.DEFAULT_BOOT_DISK_SIZE,\n type=int,\n help='Size (in GB) of the boot disk')\n google_common.add_argument(\n '--preemptible',\n default=False,\n action='store_true',\n help='Use a preemptible VM for the job')\n google_common.add_argument(\n '--zones', nargs='+', help='List of Google Compute Engine zones.')\n google_common.add_argument(\n '--scopes',\n nargs='+',\n help=\"\"\"Space-separated scopes for Google Compute Engine instances.\n If unspecified, provider will use '%s'\"\"\" % ','.join(\n google_base.DEFAULT_SCOPES))\n google_common.add_argument(\n '--accelerator-type',\n help=\"\"\"The Compute Engine accelerator type. By specifying this parameter,\n you will download and install the following third-party software onto\n your job's Compute Engine instances: NVIDIA(R) Tesla(R) drivers and\n NVIDIA(R) CUDA toolkit. Please see\n https://cloud.google.com/compute/docs/gpus/ for supported GPU types\n and\n https://cloud.google.com/genomics/reference/rest/v1alpha2/pipelines#pipelineresources\n for more details.\"\"\")\n google_common.add_argument(\n '--accelerator-count',\n type=int,\n default=0,\n help=\"\"\"The number of accelerators of the specified type to attach.\n By specifying this parameter, you will download and install the\n following third-party software onto your job's Compute Engine\n instances: NVIDIA(R) Tesla(R) drivers and NVIDIA(R) CUDA toolkit.\"\"\")\n\n google = parser.add_argument_group(\n title='\"google\" provider options',\n description='See also the \"google-common\" options listed above')\n google.add_argument(\n '--keep-alive',\n type=int,\n help=\"\"\"Time (in seconds) to keep a tasks's virtual machine (VM) running\n after a localization, docker command, or delocalization failure.\n Allows for connecting to the VM for debugging.\n Default is 0; maximum allowed value is 86400 (1 day).\"\"\")\n\n google_v2 = parser.add_argument_group(\n title='\"google-v2\" provider options',\n description='See also the \"google-common\" options listed above')\n google_v2.add_argument(\n '--regions',\n nargs='+',\n help=\"\"\"List of Google Compute Engine regions.\n Only one of --zones and --regions may be specified.\"\"\")\n google_v2.add_argument(\n '--machine-type', help='Provider-specific machine type')\n google_v2.add_argument(\n '--cpu-platform',\n help=\"\"\"The CPU platform to request. Supported values can be found at\n https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform\"\"\"\n )\n google_v2.add_argument(\n '--network',\n help=\"\"\"The Compute Engine VPC network name to attach the VM's network\n interface to. The value will be prefixed with global/networks/ unless\n it contains a /, in which case it is assumed to be a fully specified\n network resource URL.\"\"\")\n google_v2.add_argument(\n '--subnetwork',\n help=\"\"\"The name of the Compute Engine subnetwork to attach the instance\n to.\"\"\")\n google_v2.add_argument(\n '--use-private-address',\n default=False,\n action='store_true',\n help='If set to true, do not attach a public IP address to the VM.')\n google_v2.add_argument(\n '--timeout',\n help=\"\"\"The maximum amount of time to give the pipeline to complete.\n This includes the time spent waiting for a worker to be allocated.\n Time can be listed using a number followed by a unit. Supported units\n are s (seconds), m (minutes), h (hours), d (days), w (weeks).\n Example: '7d' (7 days).\"\"\")\n google_v2.add_argument(\n '--log-interval',\n help=\"\"\"The amount of time to sleep between copies of log files from\n the pipeline to the logging path.\n Time can be listed using a number followed by a unit. Supported units\n are s (seconds), m (minutes), h (hours).\n Example: '5m' (5 minutes). Default is '1m'.\"\"\")\n google_v2.add_argument(\n '--ssh',\n default=False,\n action='store_true',\n help=\"\"\"If set to true, start an ssh container in the background\n to allow you to log in using SSH and debug in real time.\"\"\")\n google_v2.add_argument(\n '--nvidia-driver-version',\n help=\"\"\"The NVIDIA driver version to use when attaching an NVIDIA GPU\n accelerator. The version specified here must be compatible with the\n GPU libraries contained in the container being executed, and must be\n one of the drivers hosted in the nvidia-drivers-us-public bucket on\n Google Cloud Storage.\"\"\")\n google_v2.add_argument(\n '--service-account',\n type=str,\n help=\"\"\"Email address of the service account to be authorized on the\n Compute Engine VM for each job task. If not specified, the default\n Compute Engine service account for the project will be used.\"\"\")\n google_v2.add_argument(\n '--disk-type',\n help=\"\"\"\n The disk type to use for the data disk. Valid values are pd-standard\n pd-ssd and local-ssd. The default value is pd-standard.\"\"\")\n\n args = provider_base.parse_args(\n parser, {\n 'google': ['project', 'zones', 'logging'],\n 'google-v2': ['project', 'logging'],\n 'test-fails': [],\n 'local': ['logging'],\n }, argv)\n\n if args.provider == 'google':\n _google_parse_arguments(args)\n if args.provider == 'google-v2':\n _google_v2_parse_arguments(args)\n\n return args"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nextracts job - global resources requirements from command - line arguments.", "response": "def _get_job_resources(args):\n \"\"\"Extract job-global resources requirements from input args.\n\n Args:\n args: parsed command-line arguments\n\n Returns:\n Resources object containing the requested resources for the job\n \"\"\"\n logging = param_util.build_logging_param(\n args.logging) if args.logging else None\n timeout = param_util.timeout_in_seconds(args.timeout)\n log_interval = param_util.log_interval_in_seconds(args.log_interval)\n\n return job_model.Resources(\n min_cores=args.min_cores,\n min_ram=args.min_ram,\n machine_type=args.machine_type,\n disk_size=args.disk_size,\n disk_type=args.disk_type,\n boot_disk_size=args.boot_disk_size,\n preemptible=args.preemptible,\n image=args.image,\n regions=args.regions,\n zones=args.zones,\n logging=logging,\n logging_path=None,\n service_account=args.service_account,\n scopes=args.scopes,\n keep_alive=args.keep_alive,\n cpu_platform=args.cpu_platform,\n network=args.network,\n subnetwork=args.subnetwork,\n use_private_address=args.use_private_address,\n accelerator_type=args.accelerator_type,\n accelerator_count=args.accelerator_count,\n nvidia_driver_version=args.nvidia_driver_version,\n timeout=timeout,\n log_interval=log_interval,\n ssh=args.ssh)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nallows provider to extract job - specific metadata from command - line args.", "response": "def _get_job_metadata(provider, user_id, job_name, script, task_ids,\n user_project, unique_job_id):\n \"\"\"Allow provider to extract job-specific metadata from command-line args.\n\n Args:\n provider: job service provider\n user_id: user submitting the job\n job_name: name for the job\n script: the script to run\n task_ids: a set of the task-ids for all tasks in the job\n user_project: name of the project to be billed for the request\n unique_job_id: generate a unique job id\n\n Returns:\n A dictionary of job-specific metadata (such as job id, name, etc.)\n \"\"\"\n create_time = dsub_util.replace_timezone(datetime.datetime.now(), tzlocal())\n user_id = user_id or dsub_util.get_os_user()\n job_metadata = provider.prepare_job_metadata(script.name, job_name, user_id,\n create_time)\n if unique_job_id:\n job_metadata['job-id'] = uuid.uuid4().hex\n\n job_metadata['create-time'] = create_time\n job_metadata['script'] = script\n job_metadata['user-project'] = user_project\n if task_ids:\n job_metadata['task-ids'] = dsub_util.compact_interval_string(list(task_ids))\n\n return job_metadata"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nresolve the logging path from job and task properties.", "response": "def _resolve_task_logging(job_metadata, job_resources, task_descriptors):\n \"\"\"Resolve the logging path from job and task properties.\n\n Args:\n job_metadata: Job metadata, such as job-id, job-name, and user-id.\n job_resources: Resources specified such as ram, cpu, and logging path.\n task_descriptors: Task metadata, parameters, and resources.\n\n Resolve the logging path, which may have substitution parameters such as\n job-id, task-id, user-id, and job-name.\n \"\"\"\n if not job_resources.logging:\n return\n\n for task_descriptor in task_descriptors:\n logging_uri = provider_base.format_logging_uri(\n job_resources.logging.uri, job_metadata, task_descriptor.task_metadata)\n logging_path = job_model.LoggingParam(logging_uri,\n job_resources.logging.file_provider)\n\n if task_descriptor.task_resources:\n task_descriptor.task_resources = task_descriptor.task_resources._replace(\n logging_path=logging_path)\n else:\n task_descriptor.task_resources = job_model.Resources(\n logging_path=logging_path)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nblocking until all of the jobs in the job_ids have completed.", "response": "def _wait_after(provider, job_ids, poll_interval, stop_on_failure):\n \"\"\"Print status info as we wait for those jobs.\n\n Blocks until either all of the listed jobs succeed,\n or one of them fails.\n\n Args:\n provider: job service provider\n job_ids: a set of job IDs (string) to wait for\n poll_interval: integer seconds to wait between iterations\n stop_on_failure: whether to stop waiting if one of the tasks fails.\n\n Returns:\n Empty list if there was no error,\n a list of error messages from the failed tasks otherwise.\n \"\"\"\n\n # Each time through the loop, the job_set is re-set to the jobs remaining to\n # check. Jobs are removed from the list when they complete.\n #\n # We exit the loop when:\n # * No jobs remain are running, OR\n # * stop_on_failure is TRUE AND at least one job returned an error\n\n # remove NO_JOB\n job_ids_to_check = {j for j in job_ids if j != dsub_util.NO_JOB}\n error_messages = []\n while job_ids_to_check and (not error_messages or not stop_on_failure):\n print('Waiting for: %s.' % (', '.join(job_ids_to_check)))\n\n # Poll until any remaining jobs have completed\n jobs_left = _wait_for_any_job(provider, job_ids_to_check, poll_interval)\n\n # Calculate which jobs just completed\n jobs_completed = job_ids_to_check.difference(jobs_left)\n\n # Get all tasks for the newly completed jobs\n tasks_completed = provider.lookup_job_tasks({'*'}, job_ids=jobs_completed)\n\n # We don't want to overwhelm the user with output when there are many\n # tasks per job. So we get a single \"dominant\" task for each of the\n # completed jobs (one that is representative of the job's fate).\n dominant_job_tasks = _dominant_task_for_jobs(tasks_completed)\n if len(dominant_job_tasks) != len(jobs_completed):\n # print info about the jobs we couldn't find\n # (should only occur for \"--after\" where the job ID is a typo).\n jobs_found = dsub_util.tasks_to_job_ids(dominant_job_tasks)\n jobs_not_found = jobs_completed.difference(jobs_found)\n for j in jobs_not_found:\n error = '%s: not found' % j\n print_error(' %s' % error)\n error_messages += [error]\n\n # Print the dominant task for the completed jobs\n for t in dominant_job_tasks:\n job_id = t.get_field('job-id')\n status = t.get_field('task-status')\n print(' %s: %s' % (str(job_id), str(status)))\n if status in ['FAILURE', 'CANCELED']:\n error_messages += [provider.get_tasks_completion_messages([t])]\n\n job_ids_to_check = jobs_left\n\n return error_messages"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _wait_and_retry(provider, job_id, poll_interval, retries, job_descriptor):\n\n while True:\n tasks = provider.lookup_job_tasks({'*'}, job_ids=[job_id])\n\n running_tasks = set()\n completed_tasks = set()\n canceled_tasks = set()\n fully_failed_tasks = set()\n task_fail_count = dict()\n\n # This is an arbitrary task that is either fully failed or canceled (with\n # preference for the former).\n message_task = None\n\n task_dict = dict()\n for t in tasks:\n task_id = job_model.numeric_task_id(t.get_field('task-id'))\n task_dict[task_id] = t\n\n status = t.get_field('task-status')\n if status == 'FAILURE':\n # Could compute this from task-attempt as well.\n task_fail_count[task_id] = task_fail_count.get(task_id, 0) + 1\n if task_fail_count[task_id] > retries:\n fully_failed_tasks.add(task_id)\n message_task = t\n elif status == 'CANCELED':\n canceled_tasks.add(task_id)\n if not message_task:\n message_task = t\n elif status == 'SUCCESS':\n completed_tasks.add(task_id)\n elif status == 'RUNNING':\n running_tasks.add(task_id)\n\n retry_tasks = (\n set(task_fail_count).difference(fully_failed_tasks)\n .difference(running_tasks).difference(completed_tasks)\n .difference(canceled_tasks))\n\n # job completed.\n if not retry_tasks and not running_tasks:\n # If there are any fully failed tasks, return the completion message of an\n # arbitrary one.\n # If not, but there are canceled tasks, return the completion message of\n # an arbitrary one.\n if message_task:\n return [provider.get_tasks_completion_messages([message_task])]\n\n # Otherwise successful completion.\n return []\n\n for task_id in retry_tasks:\n identifier = '{}.{}'.format(job_id, task_id) if task_id else job_id\n print(' {} (attempt {}) failed. Retrying.'.format(\n identifier, task_fail_count[task_id]))\n msg = task_dict[task_id].get_field('status-message')\n print(' Failure message: {}'.format(msg))\n\n _retry_task(provider, job_descriptor, task_id,\n task_fail_count[task_id] + 1)\n\n SLEEP_FUNCTION(poll_interval)", "response": "Wait for a job and retry any tasks that fail."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _retry_task(provider, job_descriptor, task_id, task_attempt):\n td_orig = job_descriptor.find_task_descriptor(task_id)\n\n new_task_descriptors = [\n job_model.TaskDescriptor({\n 'task-id': task_id,\n 'task-attempt': task_attempt\n }, td_orig.task_params, td_orig.task_resources)\n ]\n\n # Update the logging path.\n _resolve_task_resources(job_descriptor.job_metadata,\n job_descriptor.job_resources, new_task_descriptors)\n\n provider.submit_job(\n job_model.JobDescriptor(\n job_descriptor.job_metadata, job_descriptor.job_params,\n job_descriptor.job_resources, new_task_descriptors), False)", "response": "Retry task_id (numeric id) assigning it task_attempt."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _dominant_task_for_jobs(tasks):\n\n per_job = _group_tasks_by_jobid(tasks)\n\n ret = []\n for job_id in per_job.keys():\n tasks_in_salience_order = sorted(per_job[job_id], key=_importance_of_task)\n ret.append(tasks_in_salience_order[0])\n return ret", "response": "Returns a list of dominant tasks for the given tasks."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn importance of a task. Smaller values are more important.", "response": "def _importance_of_task(task):\n \"\"\"Tuple (importance, end-time). Smaller values are more important.\"\"\"\n # The status of a job is going to be determined by the roll-up of its tasks.\n # A FAILURE or CANCELED task means the job has FAILED.\n # If none, then any RUNNING task, the job is still RUNNING.\n # If none, then the job status is SUCCESS.\n #\n # Thus the dominant task for each job is one that exemplifies its\n # status:\n #\n # 1- The first (FAILURE or CANCELED) task, or if none\n # 2- The first RUNNING task, or if none\n # 3- The first SUCCESS task.\n importance = {'FAILURE': 0, 'CANCELED': 0, 'RUNNING': 1, 'SUCCESS': 2}\n return (importance[task.get_field('task-status')], task.get_field(\n 'end-time', datetime.datetime.max))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwaiting until any of the specified jobs are running.", "response": "def _wait_for_any_job(provider, job_ids, poll_interval):\n \"\"\"Waits until any of the listed jobs is not running.\n\n In particular, if any of the jobs sees one of its tasks fail,\n we count the whole job as failing (but do not terminate the remaining\n tasks ourselves).\n\n Args:\n provider: job service provider\n job_ids: a list of job IDs (string) to wait for\n poll_interval: integer seconds to wait between iterations\n\n Returns:\n A set of the jobIDs with still at least one running task.\n \"\"\"\n if not job_ids:\n return\n while True:\n tasks = provider.lookup_job_tasks({'*'}, job_ids=job_ids)\n running_jobs = set()\n failed_jobs = set()\n for t in tasks:\n status = t.get_field('task-status')\n job_id = t.get_field('job-id')\n if status in ['FAILURE', 'CANCELED']:\n failed_jobs.add(job_id)\n if status == 'RUNNING':\n running_jobs.add(job_id)\n remaining_jobs = running_jobs.difference(failed_jobs)\n if failed_jobs or len(remaining_jobs) != len(job_ids):\n return remaining_jobs\n SLEEP_FUNCTION(poll_interval)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nvalidates that the job and task argument names do not overlap.", "response": "def _validate_job_and_task_arguments(job_params, task_descriptors):\n \"\"\"Validates that job and task argument names do not overlap.\"\"\"\n\n if not task_descriptors:\n return\n\n task_params = task_descriptors[0].task_params\n\n # The use case for specifying a label or env/input/output parameter on\n # the command-line and also including it in the --tasks file is not obvious.\n # Should the command-line override the --tasks file? Why?\n # Until this use is articulated, generate an error on overlapping names.\n\n # Check labels\n from_jobs = {label.name for label in job_params['labels']}\n from_tasks = {label.name for label in task_params['labels']}\n\n intersect = from_jobs & from_tasks\n if intersect:\n raise ValueError(\n 'Names for labels on the command-line and in the --tasks file must not '\n 'be repeated: {}'.format(','.join(intersect)))\n\n # Check envs, inputs, and outputs, all of which must not overlap each other\n from_jobs = {\n item.name\n for item in job_params['envs'] | job_params['inputs']\n | job_params['outputs']\n }\n from_tasks = {\n item.name\n for item in task_params['envs'] | task_params['inputs']\n | task_params['outputs']\n }\n\n intersect = from_jobs & from_tasks\n if intersect:\n raise ValueError(\n 'Names for envs, inputs, and outputs on the command-line and in the '\n '--tasks file must not be repeated: {}'.format(','.join(intersect)))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef run_main(args):\n\n if args.command and args.script:\n raise ValueError('Cannot supply both a --command and --script flag')\n\n provider_base.check_for_unsupported_flag(args)\n\n # Set up job parameters and job data from a tasks file or flags.\n input_file_param_util = param_util.InputFileParamUtil(\n DEFAULT_INPUT_LOCAL_PATH)\n output_file_param_util = param_util.OutputFileParamUtil(\n DEFAULT_OUTPUT_LOCAL_PATH)\n mount_param_util = param_util.MountParamUtil(DEFAULT_MOUNT_LOCAL_PATH)\n\n # Get job arguments from the command line\n job_params = param_util.args_to_job_params(\n args.env, args.label, args.input, args.input_recursive, args.output,\n args.output_recursive, args.mount, input_file_param_util,\n output_file_param_util, mount_param_util)\n # If --tasks is on the command-line, then get task-specific data\n if args.tasks:\n task_descriptors = param_util.tasks_file_to_task_descriptors(\n args.tasks, args.retries, input_file_param_util, output_file_param_util)\n\n # Validate job data + task data\n _validate_job_and_task_arguments(job_params, task_descriptors)\n else:\n # Create the implicit task\n task_metadata = {'task-id': None}\n if args.retries:\n task_metadata['task-attempt'] = 1\n task_descriptors = [\n job_model.TaskDescriptor(task_metadata, {\n 'labels': set(),\n 'envs': set(),\n 'inputs': set(),\n 'outputs': set()\n }, job_model.Resources())\n ]\n\n return run(\n provider_base.get_provider(args, resources),\n _get_job_resources(args),\n job_params,\n task_descriptors,\n name=args.name,\n dry_run=args.dry_run,\n command=args.command,\n script=args.script,\n user=args.user,\n user_project=args.user_project,\n wait=args.wait,\n retries=args.retries,\n poll_interval=args.poll_interval,\n after=args.after,\n skip=args.skip,\n project=args.project,\n disable_warning=True,\n unique_job_id=args.unique_job_id)", "response": "Execute job submission from command - line arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run(provider,\n job_resources,\n job_params,\n task_descriptors,\n name=None,\n dry_run=False,\n command=None,\n script=None,\n user=None,\n user_project=None,\n wait=False,\n retries=0,\n poll_interval=10,\n after=None,\n skip=False,\n project=None,\n disable_warning=False,\n unique_job_id=False):\n \"\"\"Actual dsub body, post-stdout-redirection.\"\"\"\n if not dry_run:\n provider_base.emit_provider_message(provider)\n\n if not disable_warning:\n raise ValueError('Do not use this unstable API component!')\n\n if command and script:\n raise ValueError('Cannot supply both a command and script value.')\n\n if command:\n if name:\n command_name = name\n else:\n command_name = _name_for_command(command)\n\n # Add the shebang line to ensure the command is treated as Bash\n script = job_model.Script(command_name, '#!/usr/bin/env bash\\n' + command)\n elif script:\n # Read the script file\n script_file = dsub_util.load_file(script)\n script = job_model.Script(os.path.basename(script), script_file.read())\n else:\n raise ValueError('One of --command or a script name must be supplied')\n\n if retries and not wait:\n raise ValueError('Requesting retries requires requesting wait')\n\n # The contract with providers and downstream code is that the job_params\n # and task_params contain 'labels', 'envs', 'inputs', and 'outputs'.\n job_model.ensure_job_params_are_complete(job_params)\n job_model.ensure_task_params_are_complete(task_descriptors)\n\n task_ids = {\n task_descriptor.task_metadata.get('task-id')\n for task_descriptor in task_descriptors\n if task_descriptor.task_metadata.get('task-id') is not None\n }\n\n # Job and task parameters from the user have been validated.\n # We can now compute some job and task properties, including:\n # job_metadata such as the job-id, create-time, user-id, etc.\n # task_resources such as the logging_path (which may include job-id, task-id)\n job_metadata = _get_job_metadata(provider, user, name, script, task_ids,\n user_project, unique_job_id)\n _resolve_task_resources(job_metadata, job_resources, task_descriptors)\n\n # Job and task properties are now all resolved. Begin execution!\n if not dry_run:\n print('Job: %s' % job_metadata['job-id'])\n\n # Wait for predecessor jobs (if any)\n if after:\n if dry_run:\n print('(Pretend) waiting for: %s.' % after)\n else:\n print('Waiting for predecessor jobs to complete...')\n error_messages = _wait_after(provider, after, poll_interval, True)\n if error_messages:\n for msg in error_messages:\n print_error(msg)\n raise dsub_errors.PredecessorJobFailureError(\n 'One or more predecessor jobs completed but did not succeed.',\n error_messages, None)\n\n # Launch all the job tasks!\n job_descriptor = job_model.JobDescriptor(job_metadata, job_params,\n job_resources, task_descriptors)\n launched_job = provider.submit_job(job_descriptor, skip)\n\n if not dry_run:\n if launched_job['job-id'] == dsub_util.NO_JOB:\n print('Job output already present, skipping new job submission.')\n return {'job-id': dsub_util.NO_JOB}\n print('Launched job-id: %s' % launched_job['job-id'])\n if launched_job.get('task-id'):\n print('%s task(s)' % len(launched_job['task-id']))\n print('To check the status, run:')\n print(\" dstat %s --jobs '%s' --users '%s' --status '*'\" %\n (provider_base.get_dstat_provider_args(provider, project),\n launched_job['job-id'], launched_job['user-id']))\n print('To cancel the job, run:')\n print(\" ddel %s --jobs '%s' --users '%s'\" %\n (provider_base.get_ddel_provider_args(provider, project),\n launched_job['job-id'], launched_job['user-id']))\n\n # Poll for job completion\n if wait:\n print('Waiting for job to complete...')\n\n if retries:\n error_messages = _wait_and_retry(provider, job_metadata['job-id'],\n poll_interval, retries, job_descriptor)\n else:\n error_messages = _wait_after(provider, [job_metadata['job-id']],\n poll_interval, False)\n if error_messages:\n for msg in error_messages:\n print_error(msg)\n raise dsub_errors.JobExecutionError(\n 'One or more jobs finished with status FAILURE or CANCELED'\n ' during wait.', error_messages, launched_job)\n\n return launched_job", "response": "This function runs the job."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncraft a simple command name from the command - line.", "response": "def _name_for_command(command):\n r\"\"\"Craft a simple command name from the command.\n\n The best command strings for this are going to be those where a simple\n command was given; we will use the command to derive the name.\n\n We won't always be able to figure something out and the caller should just\n specify a \"--name\" on the command-line.\n\n For example, commands like \"export VAR=val\\necho ${VAR}\", this function would\n return \"export\".\n\n If the command starts space or a comment, then we'll skip to the first code\n we can find.\n\n If we find nothing, just return \"command\".\n\n >>> _name_for_command('samtools index \"${BAM}\"')\n 'samtools'\n >>> _name_for_command('/usr/bin/sort \"${INFILE}\" > \"${OUTFILE}\"')\n 'sort'\n >>> _name_for_command('# This should be ignored')\n 'command'\n >>> _name_for_command('\\\\\\n\\\\\\n# Bad continuations, but ignore.\\necho hello.')\n 'echo'\n\n Arguments:\n command: the user-provided command\n Returns:\n a proposed name for the task.\n \"\"\"\n\n lines = command.splitlines()\n for line in lines:\n line = line.strip()\n if line and not line.startswith('#') and line != '\\\\':\n return os.path.basename(re.split(r'\\s', line)[0])\n\n return 'command'"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrewrite local file URIs as required by the rewrite_uris method.", "response": "def _local_uri_rewriter(raw_uri):\n \"\"\"Rewrite local file URIs as required by the rewrite_uris method.\n\n Local file paths, unlike GCS paths, may have their raw URI simplified by\n os.path.normpath which collapses extraneous indirect characters.\n\n >>> _local_uri_rewriter('/tmp/a_path/../B_PATH/file.txt')\n ('/tmp/B_PATH/file.txt', 'file/tmp/B_PATH/file.txt')\n >>> _local_uri_rewriter('/myhome/./mydir/')\n ('/myhome/mydir/', 'file/myhome/mydir/')\n\n The local path rewriter will also work to preserve relative paths even\n when creating the docker path. This prevents leaking of information on the\n invoker's system to the remote system. Doing this requires a number of path\n substitutions denoted with the __ convention.\n\n >>> _local_uri_rewriter('./../upper_dir/')[1]\n 'file/_dotdot_/upper_dir/'\n >>> _local_uri_rewriter('~/localdata/*.bam')[1]\n 'file/_home_/localdata/*.bam'\n\n Args:\n raw_uri: (str) the raw file or directory path.\n\n Returns:\n normalized: a simplified and/or expanded version of the uri.\n docker_path: the uri rewritten in the format required for mounting inside\n a docker worker.\n\n \"\"\"\n # The path is split into components so that the filename is not rewritten.\n raw_path, filename = os.path.split(raw_uri)\n # Generate the local path that can be resolved by filesystem operations,\n # this removes special shell characters, condenses indirects and replaces\n # any unnecessary prefix.\n prefix_replacements = [('file:///', '/'), ('~/', os.getenv('HOME')), ('./',\n ''),\n ('file:/', '/')]\n normed_path = raw_path\n for prefix, replacement in prefix_replacements:\n if normed_path.startswith(prefix):\n normed_path = os.path.join(replacement, normed_path[len(prefix):])\n # Because abspath strips the trailing '/' from bare directory references\n # other than root, this ensures that all directory references end with '/'.\n normed_uri = directory_fmt(os.path.abspath(normed_path))\n normed_uri = os.path.join(normed_uri, filename)\n\n # Generate the path used inside the docker image;\n # 1) Get rid of extra indirects: /this/./that -> /this/that\n # 2) Rewrite required indirects as synthetic characters.\n # 3) Strip relative or absolute path leading character.\n # 4) Add 'file/' prefix.\n docker_rewrites = [(r'/\\.\\.', '/_dotdot_'), (r'^\\.\\.', '_dotdot_'),\n (r'^~/', '_home_/'), (r'^file:/', '')]\n docker_path = os.path.normpath(raw_path)\n for pattern, replacement in docker_rewrites:\n docker_path = re.sub(pattern, replacement, docker_path)\n docker_path = docker_path.lstrip('./') # Strips any of '.' './' '/'.\n docker_path = directory_fmt('file/' + docker_path) + filename\n return normed_uri, docker_path"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _get_filtered_mounts(mounts, mount_param_type):\n return set([mount for mount in mounts if isinstance(mount, mount_param_type)])", "response": "Helper function to return an appropriate set of mount parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef split_pair(pair_string, separator, nullable_idx=1):\n\n pair = pair_string.split(separator, 1)\n if len(pair) == 1:\n if nullable_idx == 0:\n return [None, pair[0]]\n elif nullable_idx == 1:\n return [pair[0], None]\n else:\n raise IndexError('nullable_idx should be either 0 or 1.')\n else:\n return pair", "response": "Splits a string into a list of pairs."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse the header from the tasks file into env input and output definitions.", "response": "def parse_tasks_file_header(header, input_file_param_util,\n output_file_param_util):\n \"\"\"Parse the header from the tasks file into env, input, output definitions.\n\n Elements are formatted similar to their equivalent command-line arguments,\n but with associated values coming from the data rows.\n\n Environment variables columns are headered as \"--env \"\n Inputs columns are headered as \"--input \" with the name optional.\n Outputs columns are headered as \"--output \" with the name optional.\n\n For historical reasons, bareword column headers (such as \"JOB_ID\") are\n equivalent to \"--env var_name\".\n\n Args:\n header: Array of header fields\n input_file_param_util: Utility for producing InputFileParam objects.\n output_file_param_util: Utility for producing OutputFileParam objects.\n\n Returns:\n job_params: A list of EnvParams and FileParams for the environment\n variables, LabelParams, input file parameters, and output file parameters.\n\n Raises:\n ValueError: If a header contains a \":\" and the prefix is not supported.\n \"\"\"\n job_params = []\n\n for col in header:\n\n # Reserve the \"-\" and \"--\" namespace.\n # If the column has no leading \"-\", treat it as an environment variable\n col_type = '--env'\n col_value = col\n if col.startswith('-'):\n col_type, col_value = split_pair(col, ' ', 1)\n\n if col_type == '--env':\n job_params.append(job_model.EnvParam(col_value))\n\n elif col_type == '--label':\n job_params.append(job_model.LabelParam(col_value))\n\n elif col_type == '--input' or col_type == '--input-recursive':\n name = input_file_param_util.get_variable_name(col_value)\n job_params.append(\n job_model.InputFileParam(\n name, recursive=(col_type.endswith('recursive'))))\n\n elif col_type == '--output' or col_type == '--output-recursive':\n name = output_file_param_util.get_variable_name(col_value)\n job_params.append(\n job_model.OutputFileParam(\n name, recursive=(col_type.endswith('recursive'))))\n\n else:\n raise ValueError('Unrecognized column header: %s' % col)\n\n return job_params"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tasks_file_to_task_descriptors(tasks, retries, input_file_param_util,\n output_file_param_util):\n \"\"\"Parses task parameters from a TSV.\n\n Args:\n tasks: Dict containing the path to a TSV file and task numbers to run\n variables, input, and output parameters as column headings. Subsequent\n lines specify parameter values, one row per job.\n retries: Number of retries allowed.\n input_file_param_util: Utility for producing InputFileParam objects.\n output_file_param_util: Utility for producing OutputFileParam objects.\n\n Returns:\n task_descriptors: an array of records, each containing the task-id,\n task-attempt, 'envs', 'inputs', 'outputs', 'labels' that defines the set of\n parameters for each task of the job.\n\n Raises:\n ValueError: If no job records were provided\n \"\"\"\n task_descriptors = []\n\n path = tasks['path']\n task_min = tasks.get('min')\n task_max = tasks.get('max')\n\n # Load the file and set up a Reader that tokenizes the fields\n param_file = dsub_util.load_file(path)\n reader = csv.reader(param_file, delimiter='\\t')\n\n # Read the first line and extract the parameters\n header = six.advance_iterator(reader)\n job_params = parse_tasks_file_header(header, input_file_param_util,\n output_file_param_util)\n\n # Build a list of records from the parsed input file\n for row in reader:\n # Tasks are numbered starting at 1 and since the first line of the TSV\n # file is a header, the first task appears on line 2.\n task_id = reader.line_num - 1\n if task_min and task_id < task_min:\n continue\n if task_max and task_id > task_max:\n continue\n\n if len(row) != len(job_params):\n dsub_util.print_error('Unexpected number of fields %s vs %s: line %s' %\n (len(row), len(job_params), reader.line_num))\n\n # Each row can contain \"envs\", \"inputs\", \"outputs\"\n envs = set()\n inputs = set()\n outputs = set()\n labels = set()\n\n for i in range(0, len(job_params)):\n param = job_params[i]\n name = param.name\n if isinstance(param, job_model.EnvParam):\n envs.add(job_model.EnvParam(name, row[i]))\n\n elif isinstance(param, job_model.LabelParam):\n labels.add(job_model.LabelParam(name, row[i]))\n\n elif isinstance(param, job_model.InputFileParam):\n inputs.add(\n input_file_param_util.make_param(name, row[i], param.recursive))\n\n elif isinstance(param, job_model.OutputFileParam):\n outputs.add(\n output_file_param_util.make_param(name, row[i], param.recursive))\n\n task_descriptors.append(\n job_model.TaskDescriptor({\n 'task-id': task_id,\n 'task-attempt': 1 if retries else None\n }, {\n 'labels': labels,\n 'envs': envs,\n 'inputs': inputs,\n 'outputs': outputs\n }, job_model.Resources()))\n\n # Ensure that there are jobs to execute (and not just a header)\n if not task_descriptors:\n raise ValueError('No tasks added from %s' % path)\n\n return task_descriptors", "response": "Parses the input and output parameters from a TSV file and returns a list of task descriptors that define the set of job parameters."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_pair_args(labels, argclass):\n label_data = set()\n for arg in labels:\n name, value = split_pair(arg, '=', nullable_idx=1)\n label_data.add(argclass(name, value))\n return label_data", "response": "Parse flags of key = value pairs and return a list of argclass objects."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nparses the command - line arguments into a set of job parameters and data.", "response": "def args_to_job_params(envs, labels, inputs, inputs_recursive, outputs,\n outputs_recursive, mounts, input_file_param_util,\n output_file_param_util, mount_param_util):\n \"\"\"Parse env, input, and output parameters into a job parameters and data.\n\n Passing arguments on the command-line allows for launching a single job.\n The env, input, and output arguments encode both the definition of the\n job as well as the single job's values.\n\n Env arguments are simple name=value pairs.\n Input and output file arguments can contain name=value pairs or just values.\n Either of the following is valid:\n\n uri\n myfile=uri\n\n Args:\n envs: list of environment variable job parameters\n labels: list of labels to attach to the tasks\n inputs: list of file input parameters\n inputs_recursive: list of recursive directory input parameters\n outputs: list of file output parameters\n outputs_recursive: list of recursive directory output parameters\n mounts: list of gcs buckets to mount\n input_file_param_util: Utility for producing InputFileParam objects.\n output_file_param_util: Utility for producing OutputFileParam objects.\n mount_param_util: Utility for producing MountParam objects.\n\n Returns:\n job_params: a dictionary of 'envs', 'inputs', and 'outputs' that defines the\n set of parameters and data for a job.\n \"\"\"\n # Parse environmental variables and labels.\n env_data = parse_pair_args(envs, job_model.EnvParam)\n label_data = parse_pair_args(labels, job_model.LabelParam)\n\n # For input files, we need to:\n # * split the input into name=uri pairs (name optional)\n # * get the environmental variable name, or automatically set if null.\n # * create the input file param\n input_data = set()\n for (recursive, args) in ((False, inputs), (True, inputs_recursive)):\n for arg in args:\n name, value = split_pair(arg, '=', nullable_idx=0)\n name = input_file_param_util.get_variable_name(name)\n input_data.add(input_file_param_util.make_param(name, value, recursive))\n\n # For output files, we need to:\n # * split the input into name=uri pairs (name optional)\n # * get the environmental variable name, or automatically set if null.\n # * create the output file param\n output_data = set()\n for (recursive, args) in ((False, outputs), (True, outputs_recursive)):\n for arg in args:\n name, value = split_pair(arg, '=', 0)\n name = output_file_param_util.get_variable_name(name)\n output_data.add(output_file_param_util.make_param(name, value, recursive))\n\n mount_data = set()\n for arg in mounts:\n # Mounts can look like `--mount VAR=PATH` or `--mount VAR=PATH {num}`,\n # where num is the size of the disk in Gb. We assume a space is the\n # separator between path and disk size.\n if ' ' in arg:\n key_value_pair, disk_size = arg.split(' ')\n name, value = split_pair(key_value_pair, '=', 1)\n mount_data.add(mount_param_util.make_param(name, value, disk_size))\n else:\n name, value = split_pair(arg, '=', 1)\n mount_data.add(mount_param_util.make_param(name, value, disk_size=None))\n return {\n 'envs': env_data,\n 'inputs': input_data,\n 'outputs': output_data,\n 'labels': label_data,\n 'mounts': mount_data,\n }"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nvalidates that all of the arguments passed to submit_job have valid file providers.", "response": "def validate_submit_args_or_fail(job_descriptor, provider_name, input_providers,\n output_providers, logging_providers):\n \"\"\"Validate that arguments passed to submit_job have valid file providers.\n\n This utility function takes resources and task data args from `submit_job`\n in the base provider. This function will fail with a value error if any of the\n parameters are not valid. See the following example;\n\n >>> job_resources = type('', (object,),\n ... {\"logging\": job_model.LoggingParam('gs://logtemp', job_model.P_GCS)})()\n >>> job_params={'inputs': set(), 'outputs': set(), 'mounts': set()}\n >>> task_descriptors = [\n ... job_model.TaskDescriptor(None, {\n ... 'inputs': {\n ... job_model.FileParam('IN', uri='gs://in/*',\n ... file_provider=job_model.P_GCS)},\n ... 'outputs': set()}, None),\n ... job_model.TaskDescriptor(None, {\n ... 'inputs': set(),\n ... 'outputs': {\n ... job_model.FileParam('OUT', uri='gs://out/*',\n ... file_provider=job_model.P_GCS)}}, None)]\n ...\n >>> validate_submit_args_or_fail(job_model.JobDescriptor(None, job_params,\n ... job_resources, task_descriptors),\n ... provider_name='MYPROVIDER',\n ... input_providers=[job_model.P_GCS],\n ... output_providers=[job_model.P_GCS],\n ... logging_providers=[job_model.P_GCS])\n ...\n >>> validate_submit_args_or_fail(job_model.JobDescriptor(None, job_params,\n ... job_resources, task_descriptors),\n ... provider_name='MYPROVIDER',\n ... input_providers=[job_model.P_GCS],\n ... output_providers=[job_model.P_LOCAL],\n ... logging_providers=[job_model.P_GCS])\n Traceback (most recent call last):\n ...\n ValueError: Unsupported output path (gs://out/*) for provider 'MYPROVIDER'.\n\n Args:\n job_descriptor: instance of job_model.JobDescriptor.\n provider_name: (str) the name of the execution provider.\n input_providers: (string collection) whitelist of file providers for input.\n output_providers: (string collection) whitelist of providers for output.\n logging_providers: (string collection) whitelist of providers for logging.\n\n Raises:\n ValueError: if any file providers do not match the whitelists.\n \"\"\"\n job_resources = job_descriptor.job_resources\n job_params = job_descriptor.job_params\n task_descriptors = job_descriptor.task_descriptors\n\n # Validate logging file provider.\n _validate_providers([job_resources.logging], 'logging', logging_providers,\n provider_name)\n\n # Validate job input and output file providers\n _validate_providers(job_params['inputs'], 'input', input_providers,\n provider_name)\n _validate_providers(job_params['outputs'], 'output', output_providers,\n provider_name)\n\n # Validate input and output file providers.\n for task_descriptor in task_descriptors:\n _validate_providers(task_descriptor.task_params['inputs'], 'input',\n input_providers, provider_name)\n _validate_providers(task_descriptor.task_params['outputs'], 'output',\n output_providers, provider_name)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nhandle the version flag.", "response": "def handle_version_flag():\n \"\"\"If the --version flag is passed, print version to stdout and exit.\n\n Within dsub commands, --version should be the highest priority flag.\n This function supplies a repeatable and DRY way of checking for the\n version flag and printing the version. Callers still need to define a version\n flag in the command's flags so that it shows up in help output.\n \"\"\"\n parser = argparse.ArgumentParser(description='Version parser', add_help=False)\n parser.add_argument('--version', '-v', dest='version', action='store_true')\n parser.set_defaults(version=False)\n args, _ = parser.parse_known_args()\n if args.version:\n print('dsub version: %s' % DSUB_VERSION)\n sys.exit()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef age_to_create_time(age, from_time=None):\n\n if not age:\n return None\n\n if not from_time:\n from_time = dsub_util.replace_timezone(datetime.datetime.now(), tzlocal())\n\n try:\n last_char = age[-1]\n\n if last_char == 's':\n return from_time - datetime.timedelta(seconds=int(age[:-1]))\n elif last_char == 'm':\n return from_time - datetime.timedelta(minutes=int(age[:-1]))\n elif last_char == 'h':\n return from_time - datetime.timedelta(hours=int(age[:-1]))\n elif last_char == 'd':\n return from_time - datetime.timedelta(days=int(age[:-1]))\n elif last_char == 'w':\n return from_time - datetime.timedelta(weeks=int(age[:-1]))\n else:\n # If no unit is given treat the age as seconds from epoch, otherwise apply\n # the correct time unit.\n return dsub_util.replace_timezone(\n datetime.datetime.utcfromtimestamp(int(age)), pytz.utc)\n\n except (ValueError, OverflowError) as e:\n raise ValueError('Unable to parse age string %s: %s' % (age, e))", "response": "Compute the create time for the list filter."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _interval_to_seconds(interval, valid_units='smhdw'):\n if not interval:\n return None\n\n try:\n last_char = interval[-1]\n\n if last_char == 's' and 's' in valid_units:\n return str(float(interval[:-1])) + 's'\n elif last_char == 'm' and 'm' in valid_units:\n return str(float(interval[:-1]) * 60) + 's'\n elif last_char == 'h' and 'h' in valid_units:\n return str(float(interval[:-1]) * 60 * 60) + 's'\n elif last_char == 'd' and 'd' in valid_units:\n return str(float(interval[:-1]) * 60 * 60 * 24) + 's'\n elif last_char == 'w' and 'w' in valid_units:\n return str(float(interval[:-1]) * 60 * 60 * 24 * 7) + 's'\n else:\n raise ValueError(\n 'Unsupported units in interval string %s: %s' % (interval, last_char))\n\n except (ValueError, OverflowError) as e:\n raise ValueError('Unable to parse interval string %s: %s' % (interval, e))", "response": "Convert the timeout duration to seconds."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nproducing a default variable name if none is specified.", "response": "def get_variable_name(self, name):\n \"\"\"Produce a default variable name if none is specified.\"\"\"\n if not name:\n name = '%s%s' % (self._auto_prefix, self._auto_index)\n self._auto_index += 1\n return name"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rewrite_uris(self, raw_uri, file_provider):\n if file_provider == job_model.P_GCS:\n normalized, docker_path = _gcs_uri_rewriter(raw_uri)\n elif file_provider == job_model.P_LOCAL:\n normalized, docker_path = _local_uri_rewriter(raw_uri)\n else:\n raise ValueError('File provider not supported: %r' % file_provider)\n return normalized, os.path.join(self._relative_path, docker_path)", "response": "Accept a raw uri and return rewritten versions."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nfinding the file provider for a URI.", "response": "def parse_file_provider(uri):\n \"\"\"Find the file provider for a URI.\"\"\"\n providers = {'gs': job_model.P_GCS, 'file': job_model.P_LOCAL}\n # URI scheme detector uses a range up to 30 since none of the IANA\n # registered schemes are longer than this.\n provider_found = re.match(r'^([A-Za-z][A-Za-z0-9+.-]{0,29})://', uri)\n if provider_found:\n prefix = provider_found.group(1).lower()\n else:\n # If no provider is specified in the URI, assume that the local\n # filesystem is being used. Availability and validity of the local\n # file/directory will be checked later.\n prefix = 'file'\n if prefix in providers:\n return providers[prefix]\n else:\n raise ValueError('File prefix not supported: %s://' % prefix)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _validate_paths_or_fail(uri, recursive):\n path, filename = os.path.split(uri)\n\n # dsub could support character ranges ([0-9]) with some more work, but for\n # now we assume that basic asterisk wildcards are sufficient. Reject any URI\n # that includes square brackets or question marks, since we know that\n # if they actually worked, it would be accidental.\n if '[' in uri or ']' in uri:\n raise ValueError(\n 'Square bracket (character ranges) are not supported: %s' % uri)\n if '?' in uri:\n raise ValueError('Question mark wildcards are not supported: %s' % uri)\n\n # Only support file URIs and *filename* wildcards\n # Wildcards at the directory level or \"**\" syntax would require better\n # support from the Pipelines API *or* doing expansion here and\n # (potentially) producing a series of FileParams, instead of one.\n if '*' in path:\n raise ValueError(\n 'Path wildcard (*) are only supported for files: %s' % uri)\n if '**' in filename:\n raise ValueError('Recursive wildcards (\"**\") not supported: %s' % uri)\n if filename in ('..', '.'):\n raise ValueError('Path characters \"..\" and \".\" not supported '\n 'for file names: %s' % uri)\n\n # Do not allow non-recursive IO to reference directories.\n if not recursive and not filename:\n raise ValueError('Input or output values that are not recursive must '\n 'reference a filename or wildcard: %s' % uri)", "response": "Do basic validation of the uri return the path and filename."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_uri(self, raw_uri, recursive):\n # Assume recursive URIs are directory paths.\n if recursive:\n raw_uri = directory_fmt(raw_uri)\n # Get the file provider, validate the raw URI, and rewrite the path\n # component of the URI for docker and remote.\n file_provider = self.parse_file_provider(raw_uri)\n self._validate_paths_or_fail(raw_uri, recursive)\n uri, docker_uri = self.rewrite_uris(raw_uri, file_provider)\n uri_parts = job_model.UriParts(\n directory_fmt(os.path.dirname(uri)), os.path.basename(uri))\n return docker_uri, uri_parts, file_provider", "response": "Parse a URI and return a valid docker_path uri and file provider from a flag value."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_param(self, name, raw_uri, recursive):\n if not raw_uri:\n return self.param_class(name, None, None, None, recursive, None)\n docker_path, uri_parts, provider = self.parse_uri(raw_uri, recursive)\n return self.param_class(name, raw_uri, docker_path, uri_parts, recursive,\n provider)", "response": "Return a FileParam given an input uri."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a valid docker_path from a Google Persistent Disk url.", "response": "def _parse_image_uri(self, raw_uri):\n \"\"\"Return a valid docker_path from a Google Persistent Disk url.\"\"\"\n # The string replace is so we don't have colons and double slashes in the\n # mount path. The idea is the resulting mount path would look like:\n # /mnt/data/mount/http/www.googleapis.com/compute/v1/projects/...\n docker_uri = os.path.join(self._relative_path,\n raw_uri.replace('https://', 'https/', 1))\n return docker_uri"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a valid local file path and a docker_uri for a local file path.", "response": "def _parse_local_mount_uri(self, raw_uri):\n \"\"\"Return a valid docker_path for a local file path.\"\"\"\n raw_uri = directory_fmt(raw_uri)\n _, docker_path = _local_uri_rewriter(raw_uri)\n local_path = docker_path[len('file'):]\n docker_uri = os.path.join(self._relative_path, docker_path)\n return local_path, docker_uri"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parse_gcs_uri(self, raw_uri):\n # Assume URI is a directory path.\n raw_uri = directory_fmt(raw_uri)\n _, docker_path = _gcs_uri_rewriter(raw_uri)\n docker_uri = os.path.join(self._relative_path, docker_path)\n return docker_uri", "response": "Return a valid docker_path for a GCS bucket."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef make_param(self, name, raw_uri, disk_size):\n if raw_uri.startswith('https://www.googleapis.com/compute'):\n # Full Image URI should look something like:\n # https://www.googleapis.com/compute/v1/projects//global/images/\n # But don't validate further, should the form of a valid image URI\n # change (v1->v2, for example)\n docker_path = self._parse_image_uri(raw_uri)\n return job_model.PersistentDiskMountParam(\n name, raw_uri, docker_path, disk_size, disk_type=None)\n elif raw_uri.startswith('file://'):\n local_path, docker_path = self._parse_local_mount_uri(raw_uri)\n return job_model.LocalMountParam(name, raw_uri, docker_path, local_path)\n elif raw_uri.startswith('gs://'):\n docker_path = self._parse_gcs_uri(raw_uri)\n return job_model.GCSMountParam(name, raw_uri, docker_path)\n else:\n raise ValueError(\n 'Mount parameter {} must begin with valid prefix.'.format(raw_uri))", "response": "Create a MountParam given a GCS bucket disk image or local path."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef validate_param_name(name, param_type):\n # http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_235\n #\n # 3.235 Name\n # In the shell command language, a word consisting solely of underscores,\n # digits, and alphabetics from the portable character set.\n if not re.match(r'^[a-zA-Z_][a-zA-Z0-9_]*$', name):\n raise ValueError('Invalid %s: %s' % (param_type, name))", "response": "Validate that the name follows posix conventions for env variables."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef validate_bucket_name(bucket):\n if not bucket.startswith('gs://'):\n raise ValueError(\n 'Invalid bucket path \"%s\". Must start with \"gs://\".' % bucket)\n bucket_name = bucket[len('gs://'):]\n if not re.search(r'^\\w[\\w_\\.-]{1,61}\\w$', bucket_name):\n raise ValueError('Invalid bucket name: %s' % bucket)", "response": "Validate that the name is a valid GCS bucket."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convert_to_label_chars(s):\n\n # We want the results to be user-friendly, not just functional.\n # So we can't base-64 encode it.\n # * If upper-case: lower-case it\n # * If the char is not a standard letter or digit. make it a dash\n\n # March 2019 note: underscores are now allowed in labels.\n # However, removing the conversion of underscores to dashes here would\n # create inconsistencies between old jobs and new jobs.\n # With existing code, $USER \"jane_doe\" has a user-id label of \"jane-doe\".\n # If we remove the conversion, the user-id label for new jobs is \"jane_doe\".\n # This makes looking up old jobs more complicated.\n\n accepted_characters = string.ascii_lowercase + string.digits + '-'\n\n def label_char_transform(char):\n if char in accepted_characters:\n return char\n if char in string.ascii_uppercase:\n return char.lower()\n return '-'\n\n return ''.join(label_char_transform(c) for c in s)", "response": "Turn the specified name and value into a valid Google label."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _remove_empty_items(d, required):\n\n new_dict = {}\n for k, v in d.items():\n if k in required:\n new_dict[k] = v\n elif isinstance(v, int) or v:\n # \"if v\" would suppress emitting int(0)\n new_dict[k] = v\n\n return new_dict", "response": "Remove any empty items from a dictionary which\n \u2192 TaskDescriptor."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef task_view_generator(job_descriptor):\n for task_descriptor in job_descriptor.task_descriptors:\n jd = JobDescriptor(job_descriptor.job_metadata, job_descriptor.job_params,\n job_descriptor.job_resources, [task_descriptor])\n yield jd", "response": "Generator that yields a task - specific view of the job."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef numeric_task_id(task_id):\n\n # This function exists to support the legacy \"task-id\" format in the \"google\"\n # provider. Google labels originally could not be numeric. When the google\n # provider is completely replaced by the google-v2 provider, this function can\n # go away.\n\n if task_id is not None:\n if task_id.startswith('task-'):\n return int(task_id[len('task-'):])\n else:\n return int(task_id)", "response": "Converts a task - id to the numeric task - id."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nraises ValueError if the label is invalid.", "response": "def _validate_label(cls, name, value):\n \"\"\"Raise ValueError if the label is invalid.\"\"\"\n # Rules for labels are described in:\n # https://cloud.google.com/compute/docs/labeling-resources#restrictions\n\n # * Keys and values cannot be longer than 63 characters each.\n # * Keys and values can only contain lowercase letters, numeric characters,\n # underscores, and dashes.\n # * International characters are allowed.\n # * Label keys must start with a lowercase letter and international\n # characters are allowed.\n # * Label keys cannot be empty.\n cls._check_label_name(name)\n cls._check_label_value(value)\n\n # Ensure that reserved labels are not being used.\n if not cls._allow_reserved_keys and name in RESERVED_LABELS:\n raise ValueError('Label flag (%s=...) must not use reserved keys: %r' %\n (name, list(RESERVED_LABELS)))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef to_serializable(self):\n\n task_metadata = self.task_metadata\n task_params = self.task_params\n task_resources = self.task_resources\n\n # The only required field is the task-id, even if it is None\n task_id = None\n if task_metadata.get('task-id') is not None:\n task_id = str(task_metadata.get('task-id'))\n\n task = {'task-id': task_id}\n task['create-time'] = task_metadata.get('create-time')\n task['task-attempt'] = task_metadata.get('task-attempt')\n\n if task_resources.logging_path:\n task['logging-path'] = str(task_resources.logging_path.uri)\n\n task['labels'] = {var.name: var.value for var in task_params['labels']}\n\n task['envs'] = {var.name: var.value for var in task_params['envs']}\n\n task['inputs'] = {\n var.name: var.value\n for var in task_params['inputs']\n if not var.recursive\n }\n task['input-recursives'] = {\n var.name: var.value\n for var in task_params['inputs']\n if var.recursive\n }\n task['outputs'] = {\n var.name: var.value\n for var in task_params['outputs']\n if not var.recursive\n }\n task['output-recursives'] = {\n var.name: var.value\n for var in task_params['outputs']\n if var.recursive\n }\n\n return _remove_empty_items(task, ['task-id'])", "response": "Return a dict populated for serialization ( as YAML or JSON."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef to_serializable(self):\n\n job_metadata = self.job_metadata\n job_resources = self.job_resources\n job_params = self.job_params\n task_descriptors = self.task_descriptors\n\n job = {\n 'job-id': job_metadata.get('job-id'),\n 'job-name': job_metadata.get('job-name'),\n 'user-id': job_metadata.get('user-id'),\n 'create-time': job_metadata.get('create-time'),\n 'dsub-version': job_metadata.get('dsub-version'),\n 'user-project': job_metadata.get('user-project'),\n 'task-ids': job_metadata.get('task-ids'),\n 'script-name': job_metadata['script'].name,\n }\n\n # logging is specified as a command-line argument and is typically\n # transformed (substituting job-id). The transformed value is saved\n # on a per-task basis as the 'logging-path'.\n if job_resources.logging:\n job['logging'] = str(job_resources.logging.uri)\n\n job['labels'] = {var.name: var.value for var in job_params['labels']}\n\n job['envs'] = {var.name: var.value for var in job_params['envs']}\n\n job['inputs'] = {\n var.name: var.value\n for var in job_params['inputs']\n if not var.recursive\n }\n job['input-recursives'] = {\n var.name: var.value\n for var in job_params['inputs']\n if var.recursive\n }\n\n job['outputs'] = {\n var.name: var.value\n for var in job_params['outputs']\n if not var.recursive\n }\n job['output-recursives'] = {\n var.name: var.value\n for var in job_params['outputs']\n if var.recursive\n }\n job['mounts'] = {var.name: var.value for var in job_params['mounts']}\n\n tasks = []\n for task_descriptor in task_descriptors:\n tasks.append(task_descriptor.to_serializable())\n\n job['tasks'] = tasks\n\n return _remove_empty_items(job, [])", "response": "Return a dict populated for serialization as YAML or JSON."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\npopulates a JobDescriptor from the local provider s original meta. yaml file.", "response": "def _from_yaml_v0(cls, job):\n \"\"\"Populate a JobDescriptor from the local provider's original meta.yaml.\n\n The local job provider had the first incarnation of a YAML file for each\n task. That idea was extended here in the JobDescriptor and the local\n provider adopted the JobDescriptor.to_yaml() call to write its meta.yaml.\n\n The JobDescriptor.from_yaml() detects if it receives a local provider's\n \"v0\" meta.yaml and calls this function.\n\n Args:\n job: an object produced from decoding meta.yaml.\n\n Returns:\n A JobDescriptor populated as best we can from the old meta.yaml.\n \"\"\"\n\n # The v0 meta.yaml only contained:\n # create-time, job-id, job-name, logging, task-id\n # labels, envs, inputs, outputs\n # It did NOT contain user-id.\n # dsub-version might be there as a label.\n\n job_metadata = {}\n for key in ['job-id', 'job-name', 'create-time']:\n job_metadata[key] = job.get(key)\n\n # Make sure that create-time string is turned into a datetime\n job_metadata['create-time'] = dsub_util.replace_timezone(\n datetime.datetime.strptime(job['create-time'], '%Y-%m-%d %H:%M:%S.%f'),\n tzlocal())\n\n # The v0 meta.yaml contained a \"logging\" field which was the task-specific\n # logging path. It did not include the actual \"--logging\" value the user\n # specified.\n job_resources = Resources()\n\n # The v0 meta.yaml represented a single task.\n # It did not distinguish whether params were job params or task params.\n # We will treat them as either all job params or all task params, based on\n # whether the task-id is empty or an integer value.\n #\n # We also cannot distinguish whether inputs/outputs were recursive or not.\n # Just treat them all as non-recursive.\n params = {}\n\n # The dsub-version may be in the meta.yaml as a label. If so remove it\n # and set it as a top-level job metadata value.\n labels = job.get('labels', {})\n if 'dsub-version' in labels:\n job_metadata['dsub-version'] = labels['dsub-version']\n del labels['dsub-version']\n params['labels'] = cls._label_params_from_dict(labels)\n\n params['envs'] = cls._env_params_from_dict(job.get('envs', {}))\n params['inputs'] = cls._input_file_params_from_dict(\n job.get('inputs', {}), False)\n params['outputs'] = cls._output_file_params_from_dict(\n job.get('outputs', {}), False)\n\n if job.get('task-id') is None:\n job_params = params\n task_metadata = {'task-id': None}\n task_params = {}\n else:\n job_params = {}\n task_metadata = {'task-id': str(job.get('task-id'))}\n task_params = params\n\n task_resources = Resources(logging_path=job.get('logging'))\n\n task_descriptors = [\n TaskDescriptor.get_complete_descriptor(task_metadata, task_params,\n task_resources)\n ]\n\n return JobDescriptor.get_complete_descriptor(\n job_metadata, job_params, job_resources, task_descriptors)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\npopulates and return a JobDescriptor from a YAML string.", "response": "def from_yaml(cls, yaml_string):\n \"\"\"Populate and return a JobDescriptor from a YAML string.\"\"\"\n try:\n job = yaml.full_load(yaml_string)\n except AttributeError:\n # For installations that cannot update their PyYAML version\n job = yaml.load(yaml_string)\n\n # If the YAML does not contain a top-level dsub version, then assume that\n # the string is coming from the local provider, reading an old version of\n # its meta.yaml.\n dsub_version = job.get('dsub-version')\n if not dsub_version:\n return cls._from_yaml_v0(job)\n\n job_metadata = {}\n for key in [\n 'job-id', 'job-name', 'task-ids', 'user-id', 'dsub-version',\n 'user-project', 'script-name'\n ]:\n if job.get(key) is not None:\n job_metadata[key] = job.get(key)\n\n # Make sure that create-time string is turned into a datetime\n job_metadata['create-time'] = dsub_util.replace_timezone(\n job.get('create-time'), pytz.utc)\n\n job_resources = Resources(logging=job.get('logging'))\n\n job_params = {}\n job_params['labels'] = cls._label_params_from_dict(job.get('labels', {}))\n job_params['envs'] = cls._env_params_from_dict(job.get('envs', {}))\n job_params['inputs'] = cls._input_file_params_from_dict(\n job.get('inputs', {}), False)\n job_params['input-recursives'] = cls._input_file_params_from_dict(\n job.get('input-recursives', {}), True)\n job_params['outputs'] = cls._output_file_params_from_dict(\n job.get('outputs', {}), False)\n job_params['output-recursives'] = cls._output_file_params_from_dict(\n job.get('output-recursives', {}), True)\n job_params['mounts'] = cls._mount_params_from_dict(job.get('mounts', {}))\n\n task_descriptors = []\n for task in job.get('tasks', []):\n task_metadata = {'task-id': task.get('task-id')}\n\n # Old instances of the meta.yaml do not have a task create time.\n create_time = task.get('create-time')\n if create_time:\n task_metadata['create-time'] = dsub_util.replace_timezone(\n create_time, pytz.utc)\n\n if task.get('task-attempt') is not None:\n task_metadata['task-attempt'] = task.get('task-attempt')\n\n task_params = {}\n task_params['labels'] = cls._label_params_from_dict(\n task.get('labels', {}))\n task_params['envs'] = cls._env_params_from_dict(task.get('envs', {}))\n task_params['inputs'] = cls._input_file_params_from_dict(\n task.get('inputs', {}), False)\n task_params['input-recursives'] = cls._input_file_params_from_dict(\n task.get('input-recursives', {}), True)\n task_params['outputs'] = cls._output_file_params_from_dict(\n task.get('outputs', {}), False)\n task_params['output-recursives'] = cls._output_file_params_from_dict(\n task.get('output-recursives', {}), True)\n\n task_resources = Resources(logging_path=task.get('logging-path'))\n\n task_descriptors.append(\n TaskDescriptor(task_metadata, task_params, task_resources))\n\n return JobDescriptor(job_metadata, job_params, job_resources,\n task_descriptors)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_task_descriptor(self, task_id):\n\n # It is not guaranteed that the index will be task_id - 1 when --tasks is\n # used with a min/max range.\n for task_descriptor in self.task_descriptors:\n if task_descriptor.task_metadata.get('task-id') == task_id:\n return task_descriptor\n return None", "response": "Returns the task_descriptor corresponding to task_id."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_file_environment_variables(file_params):\n env = {}\n for param in file_params:\n # We have no cases where the environment variable provided to user\n # scripts have a trailing slash, so be sure to always strip it.\n # The case that this is specifically handling is --input-recursive and\n # --output-recursive variables, which are directory values.\n env[param.name] = os.path.join(\n DATA_MOUNT_POINT, param.docker_path.rstrip('/')) if param.value else ''\n return env", "response": "Return a dictionary of environment variables for the user container."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef build_recursive_localize_env(destination, inputs):\n export_input_dirs = '\\n'.join([\n 'export {0}={1}/{2}'.format(var.name, destination.rstrip('/'),\n var.docker_path.rstrip('/'))\n for var in inputs\n if var.recursive and var.docker_path\n ])\n return export_input_dirs", "response": "Build a multi - line string with export statements for the variables that are recursively set in the environment variables\n ."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef build_recursive_localize_command(destination, inputs, file_provider):\n command = _LOCALIZE_COMMAND_MAP[file_provider]\n filtered_inputs = [\n var for var in inputs\n if var.recursive and var.file_provider == file_provider\n ]\n\n copy_input_dirs = '\\n'.join([\n textwrap.dedent(\"\"\"\n mkdir -p {data_mount}/{docker_path}\n for ((i = 0; i < 3; i++)); do\n if {command} {source_uri} {data_mount}/{docker_path}; then\n break\n elif ((i == 2)); then\n 2>&1 echo \"Recursive localization failed.\"\n exit 1\n fi\n done\n chmod -R o+r {data_mount}/{docker_path}\n \"\"\").format(\n command=command,\n source_uri=var.uri,\n data_mount=destination.rstrip('/'),\n docker_path=var.docker_path) for var in filtered_inputs\n ])\n return copy_input_dirs", "response": "Builds a multi - line command that copies the inputs from GCS to the destination folder."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nbuilding a multi - line string with export statements for the variables that are recursively set in the environment variables.", "response": "def build_recursive_gcs_delocalize_env(source, outputs):\n \"\"\"Return a multi-line string with export statements for the variables.\n\n Arguments:\n source: Folder with the data.\n For example /mnt/data\n outputs: a list of OutputFileParam\n\n Returns:\n a multi-line string with a shell script that sets environment variables\n corresponding to the outputs.\n \"\"\"\n filtered_outs = [\n var for var in outputs\n if var.recursive and var.file_provider == job_model.P_GCS\n ]\n return '\\n'.join([\n 'export {0}={1}/{2}'.format(var.name,\n source.rstrip('/'),\n var.docker_path.rstrip('/'))\n for var in filtered_outs\n ])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef build_recursive_delocalize_command(source, outputs, file_provider):\n command = _LOCALIZE_COMMAND_MAP[file_provider]\n filtered_outputs = [\n var for var in outputs\n if var.recursive and var.file_provider == file_provider\n ]\n\n return '\\n'.join([\n textwrap.dedent(\"\"\"\n for ((i = 0; i < 3; i++)); do\n if {command} {data_mount}/{docker_path} {destination_uri}; then\n break\n elif ((i == 2)); then\n 2>&1 echo \"Recursive de-localization failed.\"\n exit 1\n fi\n done\n \"\"\").format(\n command=command,\n data_mount=source.rstrip('/'),\n docker_path=var.docker_path,\n destination_uri=var.uri) for var in filtered_outputs\n ])", "response": "Build a multi - line command that copies the inputs of the source folder to GCS."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef build_mount_env(source, mounts):\n return '\\n'.join([\n 'export {0}={1}/{2}'.format(var.name, source.rstrip('/'),\n var.docker_path.rstrip('/')) for var in mounts\n ])", "response": "Build a multi - line string with export statements for the variables that are set in the environment variables that are set in the mount parameters."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a dict combining the field for job and task params.", "response": "def get_job_and_task_param(job_params, task_params, field):\n \"\"\"Returns a dict combining the field for job and task params.\"\"\"\n return job_params.get(field, set()) | task_params.get(field, set())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _parse_arguments():\n # Handle version flag and exit if it was passed.\n param_util.handle_version_flag()\n\n parser = provider_base.create_parser(sys.argv[0])\n\n parser.add_argument(\n '--version', '-v', default=False, help='Print the dsub version and exit.')\n\n parser.add_argument(\n '--jobs',\n '-j',\n required=True,\n nargs='*',\n help='List of job-ids to delete. Use \"*\" to delete all running jobs.')\n parser.add_argument(\n '--tasks',\n '-t',\n nargs='*',\n help='List of tasks in an array job to delete.')\n parser.add_argument(\n '--users',\n '-u',\n nargs='*',\n default=[],\n help=\"\"\"Deletes only those jobs which were submitted by the list of users.\n Use \"*\" to delete jobs of any user.\"\"\")\n parser.add_argument(\n '--age',\n help=\"\"\"Deletes only those jobs newer than the specified age. Ages can be\n listed using a number followed by a unit. Supported units are\n s (seconds), m (minutes), h (hours), d (days), w (weeks).\n For example: '7d' (7 days). Bare numbers are treated as UTC.\"\"\")\n parser.add_argument(\n '--label',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help='User labels to match. Tasks returned must match all labels.',\n metavar='KEY=VALUE')\n\n # Shared arguments between the \"google\" and \"google-v2\" providers\n google_common = parser.add_argument_group(\n title='google-common',\n description='Options common to the \"google\" and \"google-v2\" providers')\n google_common.add_argument(\n '--project',\n help='Cloud project ID in which to find and delete the job(s)')\n\n return provider_base.parse_args(\n parser, {\n 'google': ['project'],\n 'google-v2': ['project'],\n 'test-fails': [],\n 'local': [],\n }, sys.argv[1:])", "response": "Parses command line arguments and returns a Namespace of parsed arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _emit_search_criteria(user_ids, job_ids, task_ids, labels):\n print('Delete running jobs:')\n print(' user:')\n print(' %s\\n' % user_ids)\n print(' job-id:')\n print(' %s\\n' % job_ids)\n if task_ids:\n print(' task-id:')\n print(' %s\\n' % task_ids)\n # Labels are in a LabelParam namedtuple and must be reformated for printing.\n if labels:\n print(' labels:')\n print(' %s\\n' % repr(labels))", "response": "Print the filters used to delete jobs. Use raw flags as arguments."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ddel_tasks(provider,\n user_ids=None,\n job_ids=None,\n task_ids=None,\n labels=None,\n create_time_min=None,\n create_time_max=None):\n \"\"\"Kill jobs or job tasks.\n\n This function separates ddel logic from flag parsing and user output. Users\n of ddel who intend to access the data programmatically should use this.\n\n Args:\n provider: an instantiated dsub provider.\n user_ids: a set of user ids who \"own\" the job(s) to delete.\n job_ids: a set of job ids to delete.\n task_ids: a set of task ids to delete.\n labels: a set of LabelParam, each must match the job(s) to be cancelled.\n create_time_min: a timezone-aware datetime value for the earliest create\n time of a task, inclusive.\n create_time_max: a timezone-aware datetime value for the most recent create\n time of a task, inclusive.\n\n Returns:\n list of job ids which were deleted.\n \"\"\"\n # Delete the requested jobs\n deleted_tasks, error_messages = provider.delete_jobs(\n user_ids, job_ids, task_ids, labels, create_time_min, create_time_max)\n\n # Emit any errors canceling jobs\n for msg in error_messages:\n print(msg)\n\n return deleted_tasks", "response": "Kill jobs or job tasks."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the action with the given id.", "response": "def get_action_by_id(op, action_id):\n \"\"\"Return the operation's array of actions.\"\"\"\n actions = get_actions(op)\n if actions and 1 <= action_id < len(actions):\n return actions[action_id - 1]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _get_action_by_name(op, name):\n actions = get_actions(op)\n for action in actions:\n if action.get('name') == name:\n return action", "response": "Return the value for the specified action."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the environment for the operation.", "response": "def get_action_environment(op, name):\n \"\"\"Return the environment for the operation.\"\"\"\n action = _get_action_by_name(op, name)\n if action:\n return action.get('environment')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_action_image(op, name):\n action = _get_action_by_name(op, name)\n if action:\n return action.get('imageUri')", "response": "Return the image for the operation."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_failed_events(op):\n events = get_events(op)\n if events:\n return [\n e for e in events if int(e.get('details', {}).get('exitStatus', 0)) != 0\n ]\n return None", "response": "Return the events that have a non - zero exitStatus."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_event_of_type(op, event_type):\n events = get_events(op)\n if not events:\n return None\n\n return [e for e in events if e.get('details', {}).get('@type') == event_type]", "response": "Return all events of a particular type."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_last_update(op):\n last_update = get_end_time(op)\n\n if not last_update:\n last_event = get_last_event(op)\n if last_event:\n last_update = last_event['timestamp']\n\n if not last_update:\n last_update = get_create_time(op)\n\n return last_update", "response": "Return the most recent timestamp in the operation."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_dsub_operation(op):\n if not is_pipeline(op):\n return False\n\n for name in ['dsub-version', 'job-id', 'job-name', 'user-id']:\n if not get_label(op, name):\n return False\n\n return True", "response": "Determine if a pipelines operation is a dsub request."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncreate a new row set that is a summary of the input rows.", "response": "def _prepare_summary_table(rows):\n \"\"\"Create a new table that is a summary of the input rows.\n\n All with the same (job-name or job-id, status) go together.\n\n Args:\n rows: the input rows, a list of dictionaries.\n Returns:\n A new row set of summary information.\n \"\"\"\n if not rows:\n return []\n\n # We either group on the job-name (if present) or fall back to the job-id\n key_field = 'job-name'\n if key_field not in rows[0]:\n key_field = 'job-id'\n\n # Group each of the rows based on (job-name or job-id, status)\n grouped = collections.defaultdict(lambda: collections.defaultdict(lambda: []))\n for row in rows:\n grouped[row.get(key_field, '')][row.get('status', '')] += [row]\n\n # Now that we have the rows grouped, create a summary table.\n # Use the original table as the driver in order to preserve the order.\n new_rows = []\n for job_key in sorted(grouped.keys()):\n group = grouped.get(job_key, None)\n canonical_status = ['RUNNING', 'SUCCESS', 'FAILURE', 'CANCEL']\n # Written this way to ensure that if somehow a new status is introduced,\n # it shows up in our output.\n for status in canonical_status + sorted(group.keys()):\n if status not in group:\n continue\n task_count = len(group[status])\n del group[status]\n if task_count:\n summary_row = collections.OrderedDict()\n summary_row[key_field] = job_key\n summary_row['status'] = status\n summary_row['task-count'] = task_count\n new_rows.append(summary_row)\n\n return new_rows"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprepare a row for the task.", "response": "def _prepare_row(task, full, summary):\n \"\"\"return a dict with the task's info (more if \"full\" is set).\"\"\"\n\n # Would like to include the Job ID in the default set of columns, but\n # it is a long value and would leave little room for status and update time.\n\n row_spec = collections.namedtuple('row_spec',\n ['key', 'required', 'default_value'])\n\n # pyformat: disable\n default_columns = [\n row_spec('job-name', True, None),\n row_spec('task-id', False, None),\n row_spec('last-update', True, None),\n row_spec('status-message', True, None)\n ]\n full_columns = default_columns + [\n row_spec('job-id', True, None),\n row_spec('user-id', True, None),\n row_spec('status', True, None),\n row_spec('status-detail', True, None),\n row_spec('task-attempt', False, None),\n row_spec('create-time', True, None),\n row_spec('start-time', True, None),\n row_spec('end-time', True, None),\n row_spec('internal-id', True, None),\n row_spec('logging', True, None),\n row_spec('labels', True, {}),\n row_spec('envs', True, {}),\n row_spec('inputs', True, {}),\n row_spec('input-recursives', False, {}),\n row_spec('outputs', True, {}),\n row_spec('output-recursives', False, {}),\n row_spec('mounts', True, {}),\n row_spec('provider', True, None),\n row_spec('provider-attributes', True, {}),\n row_spec('events', True, []),\n row_spec('user-project', False, None),\n row_spec('dsub-version', False, None),\n row_spec('script-name', False, None),\n row_spec('script', False, None),\n ]\n summary_columns = default_columns + [\n row_spec('job-id', True, None),\n row_spec('user-id', True, None),\n row_spec('status', True, None),\n ]\n # pyformat: enable\n\n assert not (full and summary), 'Full and summary cannot both be enabled'\n\n if full:\n columns = full_columns\n elif summary:\n columns = summary_columns\n else:\n columns = default_columns\n\n row = {}\n for col in columns:\n key, required, default = col\n\n value = task.get_field(key, default)\n if required or value is not None:\n row[key] = value\n\n return row"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _parse_arguments():\n # Handle version flag and exit if it was passed.\n param_util.handle_version_flag()\n\n parser = provider_base.create_parser(sys.argv[0])\n\n parser.add_argument(\n '--version', '-v', default=False, help='Print the dsub version and exit.')\n\n parser.add_argument(\n '--jobs',\n '-j',\n nargs='*',\n help='A list of jobs IDs on which to check status')\n parser.add_argument(\n '--names',\n '-n',\n nargs='*',\n help='A list of job names on which to check status')\n parser.add_argument(\n '--tasks',\n '-t',\n nargs='*',\n help='A list of task IDs on which to check status')\n parser.add_argument(\n '--attempts',\n nargs='*',\n help='A list of task attempts on which to check status')\n parser.add_argument(\n '--users',\n '-u',\n nargs='*',\n default=[],\n help=\"\"\"Lists only those jobs which were submitted by the list of users.\n Use \"*\" to list jobs of any user.\"\"\")\n parser.add_argument(\n '--status',\n '-s',\n nargs='*',\n default=['RUNNING'],\n choices=['RUNNING', 'SUCCESS', 'FAILURE', 'CANCELED', '*'],\n help=\"\"\"Lists only those jobs which match the specified status(es).\n Choose from {'RUNNING', 'SUCCESS', 'FAILURE', 'CANCELED'}.\n Use \"*\" to list jobs of any status.\"\"\",\n metavar='STATUS')\n parser.add_argument(\n '--age',\n help=\"\"\"List only those jobs newer than the specified age. Ages can be\n listed using a number followed by a unit. Supported units are\n s (seconds), m (minutes), h (hours), d (days), w (weeks).\n For example: '7d' (7 days). Bare numbers are treated as UTC.\"\"\")\n parser.add_argument(\n '--label',\n nargs='*',\n action=param_util.ListParamAction,\n default=[],\n help='User labels to match. Tasks returned must match all labels.',\n metavar='KEY=VALUE')\n parser.add_argument(\n '--poll-interval',\n default=10,\n type=int,\n help='Polling interval (in seconds) for checking job status '\n 'when --wait is set.')\n parser.add_argument(\n '--wait', action='store_true', help='Wait until jobs have all completed.')\n parser.add_argument(\n '--limit',\n default=0,\n type=int,\n help='The maximum number of tasks to list. The default is unlimited.')\n parser.add_argument(\n '--format',\n choices=['text', 'json', 'yaml', 'provider-json'],\n help='Set the output format.')\n output_style = parser.add_mutually_exclusive_group()\n output_style.add_argument(\n '--full',\n '-f',\n action='store_true',\n help='Display output with full task information'\n ' and input parameters.')\n output_style.add_argument(\n '--summary',\n action='store_true',\n help='Display a summary of the results, grouped by (job, status).')\n # Shared arguments between the \"google\" and \"google-v2\" providers\n google_common = parser.add_argument_group(\n title='google-common',\n description='Options common to the \"google\" and \"google-v2\" providers')\n google_common.add_argument(\n '--project',\n help='Cloud project ID in which to find and delete the job(s)')\n\n return provider_base.parse_args(\n parser, {\n 'google': ['project'],\n 'google-v2': ['project'],\n 'test-fails': [],\n 'local': [],\n }, sys.argv[1:])", "response": "Parses command line arguments and returns a Namespace of parsed arguments."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating jobs as lists of task dicts ready for formatting/output. Args: provider: an instantiated dsub provider. statuses: a set of status strings that eligible jobs may match. user_ids: a set of user strings that eligible jobs may match. job_ids: a set of job-id strings eligible jobs may match. job_names: a set of job-name strings eligible jobs may match. task_ids: a set of task-id strings eligible tasks may match. task_attempts: a set of task-attempt strings eligible tasks may match. labels: set of LabelParam that all tasks must match. create_time_min: a timezone-aware datetime value for the earliest create time of a task, inclusive. create_time_max: a timezone-aware datetime value for the most recent create time of a task, inclusive. max_tasks: (int) maximum number of tasks to return per dstat job lookup. full_output: (bool) return all dsub fields. summary_output: (bool) return a summary of the job list. poll_interval: (int) wait time between poll events, dstat will poll jobs until all jobs succeed or fail. Set to zero to disable polling and return after the first lookup. raw_format: (bool) set True to prevent dsub from normalizing the task dict, this defaults to False and should only be set True if a provider-specific view of tasks is absolutely required. (NB: provider interfaces change over time, no transition path will be provided for users depending on this flag). Yields: lists of task dictionaries - each list representing a dstat poll event.", "response": "def dstat_job_producer(provider,\n statuses,\n user_ids=None,\n job_ids=None,\n job_names=None,\n task_ids=None,\n task_attempts=None,\n labels=None,\n create_time_min=None,\n create_time_max=None,\n max_tasks=0,\n full_output=False,\n summary_output=False,\n poll_interval=0,\n raw_format=False):\n \"\"\"Generate jobs as lists of task dicts ready for formatting/output.\n\n Args:\n provider: an instantiated dsub provider.\n statuses: a set of status strings that eligible jobs may match.\n user_ids: a set of user strings that eligible jobs may match.\n job_ids: a set of job-id strings eligible jobs may match.\n job_names: a set of job-name strings eligible jobs may match.\n task_ids: a set of task-id strings eligible tasks may match.\n task_attempts: a set of task-attempt strings eligible tasks may match.\n labels: set of LabelParam that all tasks must match.\n create_time_min: a timezone-aware datetime value for the earliest create\n time of a task, inclusive.\n create_time_max: a timezone-aware datetime value for the most recent create\n time of a task, inclusive.\n max_tasks: (int) maximum number of tasks to return per dstat job lookup.\n full_output: (bool) return all dsub fields.\n summary_output: (bool) return a summary of the job list.\n poll_interval: (int) wait time between poll events, dstat will poll jobs\n until all jobs succeed or fail. Set to zero to disable\n polling and return after the first lookup.\n raw_format: (bool) set True to prevent dsub from normalizing the task dict,\n this defaults to False and should only be set True if a\n provider-specific view of tasks is absolutely required.\n (NB: provider interfaces change over time, no transition path\n will be provided for users depending on this flag).\n\n Yields:\n lists of task dictionaries - each list representing a dstat poll event.\n \"\"\"\n some_job_running = True\n while some_job_running:\n # Get a batch of jobs.\n tasks = provider.lookup_job_tasks(\n statuses,\n user_ids=user_ids,\n job_ids=job_ids,\n job_names=job_names,\n task_ids=task_ids,\n task_attempts=task_attempts,\n labels=labels,\n create_time_min=create_time_min,\n create_time_max=create_time_max,\n max_tasks=max_tasks,\n page_size=max_tasks)\n\n some_job_running = False\n\n formatted_tasks = []\n for task in tasks:\n if 0 < max_tasks <= len(formatted_tasks):\n break\n\n # Format tasks as specified.\n if raw_format:\n formatted_tasks.append(task.raw_task_data())\n else:\n formatted_tasks.append(_prepare_row(task, full_output, summary_output))\n\n # Determine if any of the jobs are running.\n if task.get_field('task-status') == 'RUNNING':\n some_job_running = True\n\n # Yield the tasks and determine if the loop should continue.\n yield formatted_tasks\n if poll_interval and some_job_running:\n time.sleep(poll_interval)\n else:\n break"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef lookup_job_tasks(provider,\n statuses,\n user_ids=None,\n job_ids=None,\n job_names=None,\n task_ids=None,\n task_attempts=None,\n labels=None,\n create_time_min=None,\n create_time_max=None,\n max_tasks=0,\n page_size=0,\n summary_output=False):\n \"\"\"Generate formatted jobs individually, in order of create-time.\n\n Args:\n provider: an instantiated dsub provider.\n statuses: a set of status strings that eligible jobs may match.\n user_ids: a set of user strings that eligible jobs may match.\n job_ids: a set of job-id strings eligible jobs may match.\n job_names: a set of job-name strings eligible jobs may match.\n task_ids: a set of task-id strings eligible tasks may match.\n task_attempts: a set of task-attempt strings eligible tasks may match.\n labels: set of LabelParam that all tasks must match.\n create_time_min: a timezone-aware datetime value for the earliest create\n time of a task, inclusive.\n create_time_max: a timezone-aware datetime value for the most recent create\n time of a task, inclusive.\n max_tasks: (int) maximum number of tasks to return per dstat job lookup.\n page_size: the page size to use for each query to the backend. May be\n ignored by some provider implementations.\n summary_output: (bool) summarize the job list.\n\n Yields:\n Individual task dictionaries with associated metadata\n \"\"\"\n tasks_generator = provider.lookup_job_tasks(\n statuses,\n user_ids=user_ids,\n job_ids=job_ids,\n job_names=job_names,\n task_ids=task_ids,\n task_attempts=task_attempts,\n labels=labels,\n create_time_min=create_time_min,\n create_time_max=create_time_max,\n max_tasks=max_tasks,\n page_size=page_size)\n\n # Yield formatted tasks.\n for task in tasks_generator:\n yield _prepare_row(task, True, summary_output)", "response": "Generates formatted jobs individually in order of create - time."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting types of task fields to python types.", "response": "def prepare_output(self, row):\n \"\"\"Convert types of task fields.\"\"\"\n date_fields = ['last-update', 'create-time', 'start-time', 'end-time']\n int_fields = ['task-attempt']\n\n for col in date_fields:\n if col in row:\n row[col] = self.default_format_date(row[col])\n\n for col in int_fields:\n if col in row and row[col] is not None:\n row[col] = int(row[col])\n\n return row"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef trim_display_field(self, value, max_length):\n if not value:\n return ''\n if len(value) > max_length:\n return value[:max_length - 3] + '...'\n return value", "response": "Return a value for display ; if longer than max_length use ellipsis."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef format_pairs(self, values):\n return ', '.join(\n '%s=%s' % (key, value) for key, value in sorted(values.items()))", "response": "Returns a string of comma - delimited key = value pairs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef string_presenter(self, dumper, data):\n if '\\n' in data:\n return dumper.represent_scalar('tag:yaml.org,2002:str', data, style='|')\n else:\n return dumper.represent_scalar('tag:yaml.org,2002:str', data)", "response": "Presenter to force yaml. dump to use multi - line string style."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_zones(input_list):\n if not input_list:\n return []\n\n output_list = []\n\n for zone in input_list:\n if zone.endswith('*'):\n prefix = zone[:-1]\n output_list.extend([z for z in _ZONES if z.startswith(prefix)])\n else:\n output_list.append(zone)\n\n return output_list", "response": "This function returns a list of zones based on any wildcard input."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef build_pipeline_labels(job_metadata, task_metadata, task_id_pattern=None):\n labels = {\n Label(name, job_metadata[name])\n for name in ['job-name', 'job-id', 'user-id', 'dsub-version']\n }\n\n task_id = task_metadata.get('task-id')\n if task_id is not None: # Check for None (as 0 is conceivably valid)\n if task_id_pattern:\n task_id = task_id_pattern % task_id\n labels.add(Label('task-id', str(task_id)))\n\n task_attempt = task_metadata.get('task-attempt')\n if task_attempt is not None:\n labels.add(Label('task-attempt', str(task_attempt)))\n\n return labels", "response": "Builds a set of standard job and task labels."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef prepare_job_metadata(script, job_name, user_id, create_time):\n\n # The name of the pipeline gets set into the ephemeralPipeline.name as-is.\n # The default name of the pipeline is the script name\n # The name of the job is derived from the job_name and gets set as a\n # 'job-name' label (and so the value must be normalized).\n if job_name:\n pipeline_name = job_name\n job_name_value = job_model.convert_to_label_chars(job_name)\n else:\n pipeline_name = os.path.basename(script)\n job_name_value = job_model.convert_to_label_chars(\n pipeline_name.split('.', 1)[0])\n\n # The user-id will get set as a label\n user_id = job_model.convert_to_label_chars(user_id)\n\n # Now build the job-id. We want the job-id to be expressive while also\n # having a low-likelihood of collisions.\n #\n # For expressiveness, we:\n # * use the job name (truncated at 10 characters).\n # * insert the user-id\n # * add a datetime value\n # To have a high likelihood of uniqueness, the datetime value is out to\n # hundredths of a second.\n #\n # The full job-id is:\n # ----\n job_id = '%s--%s--%s' % (job_name_value[:10], user_id,\n create_time.strftime('%y%m%d-%H%M%S-%f')[:16])\n\n # Standard version is MAJOR.MINOR(.PATCH). This will convert the version\n # string to \"vMAJOR-MINOR(-PATCH)\". Example; \"0.1.0\" -> \"v0-1-0\".\n version = job_model.convert_to_label_chars('v%s' % DSUB_VERSION)\n return {\n 'pipeline-name': pipeline_name,\n 'job-name': job_name_value,\n 'job-id': job_id,\n 'user-id': user_id,\n 'dsub-version': version,\n }", "response": "Prepares the metadata fields for the job."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a datestamp from RFC3339 UTC to a datetime.", "response": "def parse_rfc3339_utc_string(rfc3339_utc_string):\n \"\"\"Converts a datestamp from RFC3339 UTC to a datetime.\n\n Args:\n rfc3339_utc_string: a datetime string in RFC3339 UTC \"Zulu\" format\n\n Returns:\n A datetime.\n \"\"\"\n\n # The timestamp from the Google Operations are all in RFC3339 format, but\n # they are sometimes formatted to millisconds, microseconds, sometimes\n # nanoseconds, and sometimes only seconds:\n # * 2016-11-14T23:05:56Z\n # * 2016-11-14T23:05:56.010Z\n # * 2016-11-14T23:05:56.010429Z\n # * 2016-11-14T23:05:56.010429380Z\n m = re.match(r'(\\d{4})-(\\d{2})-(\\d{2})T(\\d{2}):(\\d{2}):(\\d{2}).?(\\d*)Z',\n rfc3339_utc_string)\n\n # It would be unexpected to get a different date format back from Google.\n # If we raise an exception here, we can break people completely.\n # Instead, let's just return None and people can report that some dates\n # are not showing up.\n # We might reconsider this approach in the future; it was originally\n # established when dates were only used for display.\n if not m:\n return None\n\n groups = m.groups()\n if len(groups[6]) not in (0, 3, 6, 9):\n return None\n\n # Create a UTC datestamp from parsed components\n # 1- Turn components 0-5 from strings to integers\n # 2- If the last component does not exist, set it to 0.\n # If it does exist, make sure to interpret it as milliseconds.\n g = [int(val) for val in groups[:6]]\n\n fraction = groups[6]\n if not fraction:\n micros = 0\n elif len(fraction) == 3:\n micros = int(fraction) * 1000\n elif len(fraction) == 6:\n micros = int(fraction)\n elif len(fraction) == 9:\n # When nanoseconds are provided, we round\n micros = int(round(int(fraction) / 1000))\n else:\n assert False, 'Fraction length not 0, 6, or 9: {}'.len(fraction)\n\n try:\n return datetime(g[0], g[1], g[2], g[3], g[4], g[5], micros, tzinfo=pytz.utc)\n except ValueError as e:\n assert False, 'Could not parse RFC3339 datestring: {} exception: {}'.format(\n rfc3339_utc_string, e)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_operation_full_job_id(op):\n job_id = op.get_field('job-id')\n task_id = op.get_field('task-id')\n if task_id:\n return '%s.%s' % (job_id, task_id)\n else:\n return job_id", "response": "Returns the job - id or task - id for the operation."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncancel a batch of operations.", "response": "def _cancel_batch(batch_fn, cancel_fn, ops):\n \"\"\"Cancel a batch of operations.\n\n Args:\n batch_fn: API-specific batch function.\n cancel_fn: API-specific cancel function.\n ops: A list of operations to cancel.\n\n Returns:\n A list of operations canceled and a list of error messages.\n \"\"\"\n\n # We define an inline callback which will populate a list of\n # successfully canceled operations as well as a list of operations\n # which were not successfully canceled.\n\n canceled = []\n failed = []\n\n def handle_cancel_response(request_id, response, exception):\n \"\"\"Callback for the cancel response.\"\"\"\n del response # unused\n\n if exception:\n # We don't generally expect any failures here, except possibly trying\n # to cancel an operation that is already canceled or finished.\n #\n # If the operation is already finished, provide a clearer message than\n # \"error 400: Bad Request\".\n\n msg = 'error %s: %s' % (exception.resp.status, exception.resp.reason)\n if exception.resp.status == FAILED_PRECONDITION_CODE:\n detail = json.loads(exception.content)\n status = detail.get('error', {}).get('status')\n if status == FAILED_PRECONDITION_STATUS:\n msg = 'Not running'\n\n failed.append({'name': request_id, 'msg': msg})\n else:\n canceled.append({'name': request_id})\n\n return\n\n # Set up the batch object\n batch = batch_fn(callback=handle_cancel_response)\n\n # The callback gets a \"request_id\" which is the operation name.\n # Build a dict such that after the callback, we can lookup the operation\n # objects by name\n ops_by_name = {}\n for op in ops:\n op_name = op.get_field('internal-id')\n ops_by_name[op_name] = op\n batch.add(cancel_fn(name=op_name, body={}), request_id=op_name)\n\n # Cancel the operations\n batch.execute()\n\n # Iterate through the canceled and failed lists to build our return lists\n canceled_ops = [ops_by_name[op['name']] for op in canceled]\n error_messages = []\n for fail in failed:\n op = ops_by_name[fail['name']]\n error_messages.append(\"Error canceling '%s': %s\" %\n (get_operation_full_job_id(op), fail['msg']))\n\n return canceled_ops, error_messages"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncancels a list of operations.", "response": "def cancel(batch_fn, cancel_fn, ops):\n \"\"\"Cancel operations.\n\n Args:\n batch_fn: API-specific batch function.\n cancel_fn: API-specific cancel function.\n ops: A list of operations to cancel.\n\n Returns:\n A list of operations canceled and a list of error messages.\n \"\"\"\n\n # Canceling many operations one-by-one can be slow.\n # The Pipelines API doesn't directly support a list of operations to cancel,\n # but the requests can be performed in batch.\n\n canceled_ops = []\n error_messages = []\n\n max_batch = 256\n total_ops = len(ops)\n for first_op in range(0, total_ops, max_batch):\n batch_canceled, batch_messages = _cancel_batch(\n batch_fn, cancel_fn, ops[first_op:first_op + max_batch])\n canceled_ops.extend(batch_canceled)\n error_messages.extend(batch_messages)\n\n return canceled_ops, error_messages"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks if we should retry the API check.", "response": "def retry_api_check(exception):\n \"\"\"Return True if we should retry. False otherwise.\n\n Args:\n exception: An exception to test for transience.\n\n Returns:\n True if we should retry. False otherwise.\n \"\"\"\n if isinstance(exception, apiclient.errors.HttpError):\n if exception.resp.status in TRANSIENT_HTTP_ERROR_CODES:\n _print_error('Retrying...')\n return True\n\n if isinstance(exception, socket.error):\n if exception.errno in TRANSIENT_SOCKET_ERROR_CODES:\n _print_error('Retrying...')\n return True\n\n if isinstance(exception, oauth2client.client.AccessTokenRefreshError):\n _print_error('Retrying...')\n return True\n\n # For a given installation, this could be a permanent error, but has only\n # been observed as transient.\n if isinstance(exception, SSLError):\n _print_error('Retrying...')\n return True\n\n # This has been observed as a transient error:\n # ServerNotFoundError: Unable to find the server at genomics.googleapis.com\n if isinstance(exception, ServerNotFoundError):\n _print_error('Retrying...')\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if we should retry the auth check.", "response": "def retry_auth_check(exception):\n \"\"\"Specific check for auth error codes.\n\n Return True if we should retry.\n\n False otherwise.\n Args:\n exception: An exception to test for transience.\n\n Returns:\n True if we should retry. False otherwise.\n \"\"\"\n if isinstance(exception, apiclient.errors.HttpError):\n if exception.resp.status in HTTP_AUTH_ERROR_CODES:\n _print_error('Retrying...')\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef setup_service(api_name, api_version, credentials=None):\n if not credentials:\n credentials = oauth2client.client.GoogleCredentials.get_application_default(\n )\n return apiclient.discovery.build(\n api_name, api_version, credentials=credentials)", "response": "Configures the Google Genomics API client."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexecute the base API operation.", "response": "def execute(api):\n \"\"\"Executes operation.\n\n Args:\n api: The base API object\n\n Returns:\n A response body object\n \"\"\"\n try:\n return api.execute()\n except Exception as exception:\n now = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')\n _print_error('%s: Exception %s: %s' % (now, type(exception).__name__,\n str(exception)))\n # Re-raise exception to be handled by retry logic\n raise exception"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nevaluates the type of an argument.", "response": "def _eval_arg_type(arg_type, T=Any, arg=None, sig=None):\n \"\"\"Returns a type from a snippit of python source. Should normally be\n something just like 'str' or 'Object'.\n\n arg_type the source to be evaluated\n T the default type\n arg context of where this type was extracted\n sig context from where the arg was extracted\n\n Returns a type or a Type\n \"\"\"\n try:\n T = eval(arg_type)\n except Exception as e:\n raise ValueError('The type of {0} could not be evaluated in {1} for {2}: {3}' \\\n .format(arg_type, arg, sig, text_type(e)))\n else:\n if type(T) not in (type, Type):\n raise TypeError('{0} is not a valid type in {1} for {2}' \\\n .format(repr(T), arg, sig))\n return T"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef jsonify_status_code(status_code, *args, **kw):\n is_batch = kw.pop('is_batch', False)\n if is_batch:\n response = flask_make_response(json.dumps(*args, **kw))\n response.mimetype = 'application/json'\n response.status_code = status_code\n return response\n response = jsonify(*args, **kw)\n response.status_code = status_code\n return response", "response": "Returns a jsonified response with the specified HTTP status code."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncall by Flask. register_blueprint to register a new blueprint on the application.", "response": "def register(self, app, options, first_registration=False):\n \"\"\"Called by :meth:`Flask.register_blueprint` to register a blueprint\n on the application. This can be overridden to customize the register\n behavior. Keyword arguments from\n :func:`~flask.Flask.register_blueprint` are directly forwarded to this\n method in the `options` dictionary.\n \"\"\"\n self.jsonrpc_site = options.get('jsonrpc_site')\n self._got_registered_once = True\n state = self.make_setup_state(app, options, first_registration)\n if self.has_static_folder and \\\n not self.name + '.static' in state.app.view_functions.keys():\n state.add_url_rule(self.static_url_path + '/',\n view_func=self.send_static_file,\n endpoint='static')\n for deferred in self.deferred_functions:\n deferred(state)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform the actual sending action and returns the result", "response": "def send_payload(self, params):\n \"\"\"Performs the actual sending action and returns the result\n \"\"\"\n data = json.dumps({\n 'jsonrpc': self.version,\n 'method': self.service_name,\n 'params': params,\n 'id': text_type(uuid.uuid4())\n })\n data_binary = data.encode('utf-8')\n url_request = Request(self.service_url, data_binary, headers=self.headers)\n return urlopen(url_request).read()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the return value from a view function to a real response object that is an instance of : attr : response_class.", "response": "def make_response(self, rv):\n \"\"\"Converts the return value from a view function to a real\n response object that is an instance of :attr:`response_class`.\n \"\"\"\n status_or_headers = headers = None\n if isinstance(rv, tuple):\n rv, status_or_headers, headers = rv + (None,) * (3 - len(rv))\n\n if rv is None:\n raise ValueError('View function did not return a response')\n\n if isinstance(status_or_headers, (dict, list)):\n headers, status_or_headers = status_or_headers, None\n\n D = json.loads(extract_raw_data_request(request))\n if type(D) is list:\n raise InvalidRequestError('JSON-RPC batch with decorator (make_response) not is supported')\n else:\n response_obj = self.empty_response(version=D['jsonrpc'])\n response_obj['id'] = D['id']\n response_obj['result'] = rv\n response_obj.pop('error', None)\n rv = jsonify(response_obj)\n\n if status_or_headers is not None:\n if isinstance(status_or_headers, string_types):\n rv.status = status_or_headers\n else:\n rv.status_code = status_or_headers\n if headers:\n rv.headers.extend(headers)\n\n return rv"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the exception data in a format for JSON - RPC AttributeNames", "response": "def json_rpc_format(self):\n \"\"\"Return the Exception data in a format for JSON-RPC\n \"\"\"\n\n error = {\n 'name': text_type(self.__class__.__name__),\n 'code': self.code,\n 'message': '{0}'.format(text_type(self.message)),\n 'data': self.data\n }\n\n if current_app.config['DEBUG']:\n import sys, traceback\n error['stack'] = traceback.format_exc()\n error['executable'] = sys.executable\n\n return error"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef from_file(cls, file):\n if not os.path.exists(file):\n raise ValueError(\"Config file not found.\")\n\n try:\n config_parser = configparser.ConfigParser()\n config_parser.read(file)\n\n configuration = cls(file, config_parser)\n if not configuration.check_config_sanity():\n raise ValueError(\"Error in config file.\")\n else:\n return configuration\n except configparser.Error:\n raise ValueError(\"Config file is invalid.\")", "response": "Try loading given config file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef discover(cls):\n file = os.path.join(Config.config_dir, Config.config_name)\n return cls.from_file(file)", "response": "Make a guess about the config file location an try loading it."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a new config file at the default location.", "response": "def create_config(cls, cfgfile, nick, twtfile, twturl, disclose_identity, add_news):\n \"\"\"Create a new config file at the default location.\n\n :param str cfgfile: path to the config file\n :param str nick: nickname to use for own tweets\n :param str twtfile: path to the local twtxt file\n :param str twturl: URL to the remote twtxt file\n :param bool disclose_identity: if true the users id will be disclosed\n :param bool add_news: if true follow twtxt news feed\n \"\"\"\n cfgfile_dir = os.path.dirname(cfgfile)\n if not os.path.exists(cfgfile_dir):\n os.makedirs(cfgfile_dir)\n\n cfg = configparser.ConfigParser()\n\n cfg.add_section(\"twtxt\")\n cfg.set(\"twtxt\", \"nick\", nick)\n cfg.set(\"twtxt\", \"twtfile\", twtfile)\n cfg.set(\"twtxt\", \"twturl\", twturl)\n cfg.set(\"twtxt\", \"disclose_identity\", str(disclose_identity))\n cfg.set(\"twtxt\", \"character_limit\", \"140\")\n cfg.set(\"twtxt\", \"character_warning\", \"140\")\n\n cfg.add_section(\"following\")\n if add_news:\n cfg.set(\"following\", \"twtxt\", \"https://buckket.org/twtxt_news.txt\")\n\n conf = cls(cfgfile, cfg)\n conf.write_config()\n return conf"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef write_config(self):\n with open(self.config_file, \"w\") as config_file:\n self.cfg.write(config_file)", "response": "Writes self. cfg to self. config_file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef following(self):\n following = []\n try:\n for (nick, url) in self.cfg.items(\"following\"):\n source = Source(nick, url)\n following.append(source)\n except configparser.NoSectionError as e:\n logger.debug(e)\n\n return following", "response": "A list of all Source objects."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nadds a new source to the config s following section.", "response": "def add_source(self, source):\n \"\"\"Adds a new :class:`Source` to the config\u2019s following section.\"\"\"\n if not self.cfg.has_section(\"following\"):\n self.cfg.add_section(\"following\")\n\n self.cfg.set(\"following\", source.nick, source.url)\n self.write_config()"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_source_by_nick(self, nick):\n url = self.cfg.get(\"following\", nick, fallback=None)\n return Source(nick, url) if url else None", "response": "Returns the Source object for the given nick."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving a source from the config", "response": "def remove_source_by_nick(self, nick):\n \"\"\"Removes a :class:`Source` form the config\u2019s following section.\n\n :param str nick: nickname for which will be searched in the config\n \"\"\"\n if not self.cfg.has_section(\"following\"):\n return False\n\n ret_val = self.cfg.remove_option(\"following\", nick)\n self.write_config()\n return ret_val"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef build_default_map(self):\n default_map = {\n \"following\": {\n \"check\": self.check_following,\n \"timeout\": self.timeout,\n \"porcelain\": self.porcelain,\n },\n \"tweet\": {\n \"twtfile\": self.twtfile,\n },\n \"timeline\": {\n \"pager\": self.use_pager,\n \"cache\": self.use_cache,\n \"limit\": self.limit_timeline,\n \"timeout\": self.timeout,\n \"sorting\": self.sorting,\n \"porcelain\": self.porcelain,\n \"twtfile\": self.twtfile,\n \"update_interval\": self.timeline_update_interval,\n },\n \"view\": {\n \"pager\": self.use_pager,\n \"cache\": self.use_cache,\n \"limit\": self.limit_timeline,\n \"timeout\": self.timeout,\n \"sorting\": self.sorting,\n \"porcelain\": self.porcelain,\n \"update_interval\": self.timeline_update_interval,\n }\n }\n return default_map", "response": "Maps config options to the default values used by click returns : class : dict"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the given values in the config file are sane.", "response": "def check_config_sanity(self):\n \"\"\"Checks if the given values in the config file are sane.\"\"\"\n is_sane = True\n\n # This extracts some properties which cannot be checked like \"nick\",\n # but it is definitely better than writing the property names as a\n # string literal.\n properties = [property_name for property_name, obj\n in self.__class__.__dict__.items()\n if isinstance(obj, property)]\n\n for property_name in properties:\n try:\n getattr(self, property_name)\n except ValueError as e:\n click.echo(\"\u2717 Config error on {0} - {1}\".format(property_name, e))\n is_sane = False\n\n return is_sane"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nvalidates a configuration key according to section. item.", "response": "def validate_config_key(ctx, param, value):\n \"\"\"Validate a configuration key according to `section.item`.\"\"\"\n if not value:\n return value\n\n try:\n section, item = value.split(\".\", 1)\n except ValueError:\n raise click.BadArgumentUsage(\"Given key does not contain a section name.\")\n else:\n return section, item"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsearches the given text for mentions and expands them.", "response": "def expand_mentions(text, embed_names=True):\n \"\"\"Searches the given text for mentions and expands them.\n\n For example:\n \"@source.nick\" will be expanded to \"@\".\n \"\"\"\n if embed_names:\n mention_format = \"@<{name} {url}>\"\n else:\n mention_format = \"@<{url}>\"\n\n def handle_mention(match):\n source = get_source_by_name(match.group(1))\n if source is None:\n return \"@{0}\".format(match.group(1))\n return mention_format.format(\n name=source.nick,\n url=source.url)\n\n return short_mention_re.sub(handle_mention, text)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsearches the given text for mentions and returns a human - readable form.", "response": "def format_mentions(text, format_callback=format_mention):\n \"\"\"Searches the given text for mentions generated by `expand_mention()` and returns a human-readable form.\n\n For example:\n \"@\" will result in \"@bob\"\n\n If you follow a source: source.nick will be bold\n If you are the mentioned source: source.nick will be bold and coloured\n If nothing from the above is true: nick will be unstyled\n If nothing from the above is true and nick is not given: url will be used\n \"\"\"\n\n def handle_mention(match):\n name, url = match.groups()\n return format_callback(name, url)\n\n return mention_re.sub(handle_mention, text)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef make_aware(dt):\n return dt if dt.tzinfo else dt.replace(tzinfo=timezone.utc)", "response": "Appends tzinfo and assumes UTC if datetime object has no tzinfo already."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_tweets(raw_tweets, source, now=None):\n if now is None:\n now = datetime.now(timezone.utc)\n\n tweets = []\n for line in raw_tweets:\n try:\n tweet = parse_tweet(line, source, now)\n except (ValueError, OverflowError) as e:\n logger.debug(\"{0} - {1}\".format(source.url, e))\n else:\n tweets.append(tweet)\n\n return tweets", "response": "Parses a list of raw tweets from a twtxt file and returns a list of Tweet objects."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse_tweet(raw_tweet, source, now=None):\n if now is None:\n now = datetime.now(timezone.utc)\n\n raw_created_at, text = raw_tweet.split(\"\\t\", 1)\n created_at = parse_iso8601(raw_created_at)\n\n if created_at > now:\n raise ValueError(\"Tweet is from the future\")\n\n return Tweet(click.unstyle(text.strip()), created_at, source)", "response": "Parses a single raw tweet line from a twtxt file and returns a Tweet object."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ntries loading given cache file.", "response": "def from_file(cls, file, *args, **kwargs):\n \"\"\"Try loading given cache file.\"\"\"\n try:\n cache = shelve.open(file)\n return cls(file, cache, *args, **kwargs)\n except OSError as e:\n logger.debug(\"Loading {0} failed\".format(file))\n raise e"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmake a guess about the cache file location an try loading it.", "response": "def discover(cls, *args, **kwargs):\n \"\"\"Make a guess about the cache file location an try loading it.\"\"\"\n file = os.path.join(Cache.cache_dir, Cache.cache_name)\n return cls.from_file(file, *args, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef is_cached(self, url):\n try:\n return True if url in self.cache else False\n except TypeError:\n return False", "response": "Checks if specified URL is cached."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_tweets(self, url, last_modified, tweets):\n try:\n self.cache[url] = {\"last_modified\": last_modified, \"tweets\": tweets}\n self.mark_updated()\n return True\n except TypeError:\n return False", "response": "Adds new tweets to the cache."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_tweets(self, url, limit=None):\n try:\n tweets = self.cache[url][\"tweets\"]\n self.mark_updated()\n return sorted(tweets, reverse=True)[:limit]\n except KeyError:\n return []", "response": "Retrieves tweets from the cache."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef remove_tweets(self, url):\n try:\n del self.cache[url]\n self.mark_updated()\n return True\n except KeyError:\n return False", "response": "Tries to remove cached tweets. Returns True if successful False otherwise."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndecentralise minimalist microblogging service for hackers.", "response": "def cli(ctx, config, verbose):\n \"\"\"Decentralised, minimalist microblogging service for hackers.\"\"\"\n init_logging(debug=verbose)\n\n if ctx.invoked_subcommand == \"quickstart\":\n return # Skip initializing config file\n\n try:\n if config:\n conf = Config.from_file(config)\n else:\n conf = Config.discover()\n except ValueError as e:\n if \"Error in config file.\" in str(e):\n click.echo(\"\u2717 Please correct the errors mentioned above an run twtxt again.\")\n else:\n click.echo(\"\u2717 Config file not found or not readable. You may want to run twtxt quickstart.\")\n sys.exit()\n\n ctx.default_map = conf.build_default_map()\n ctx.obj = {'conf': conf}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tweet(ctx, created_at, twtfile, text):\n text = expand_mentions(text)\n tweet = Tweet(text, created_at) if created_at else Tweet(text)\n\n pre_tweet_hook = ctx.obj[\"conf\"].pre_tweet_hook\n if pre_tweet_hook:\n run_pre_tweet_hook(pre_tweet_hook, ctx.obj[\"conf\"].options)\n\n if not add_local_tweet(tweet, twtfile):\n click.echo(\"\u2717 Couldn\u2019t write to file.\")\n else:\n post_tweet_hook = ctx.obj[\"conf\"].post_tweet_hook\n if post_tweet_hook:\n run_post_tweet_hook(post_tweet_hook, ctx.obj[\"conf\"].options)", "response": "Append a new tweet to your twtxt file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nretrieving your personal timeline.", "response": "def timeline(ctx, pager, limit, twtfile, sorting, timeout, porcelain, source, cache, force_update):\n \"\"\"Retrieve your personal timeline.\"\"\"\n if source:\n source_obj = ctx.obj[\"conf\"].get_source_by_nick(source)\n if not source_obj:\n logger.debug(\"Not following {0}, trying as URL\".format(source))\n source_obj = Source(source, source)\n sources = [source_obj]\n else:\n sources = ctx.obj[\"conf\"].following\n\n tweets = []\n\n if cache:\n try:\n with Cache.discover(update_interval=ctx.obj[\"conf\"].timeline_update_interval) as cache:\n force_update = force_update or not cache.is_valid\n if force_update:\n tweets = get_remote_tweets(sources, limit, timeout, cache)\n else:\n logger.debug(\"Multiple calls to 'timeline' within {0} seconds. Skipping update\".format(\n cache.update_interval))\n # Behold, almighty list comprehensions! (I might have gone overboard here\u2026)\n tweets = list(chain.from_iterable([cache.get_tweets(source.url) for source in sources]))\n except OSError as e:\n logger.debug(e)\n tweets = get_remote_tweets(sources, limit, timeout)\n else:\n tweets = get_remote_tweets(sources, limit, timeout)\n\n if twtfile and not source:\n source = Source(ctx.obj[\"conf\"].nick, ctx.obj[\"conf\"].twturl, file=twtfile)\n tweets.extend(get_local_tweets(source, limit))\n\n if not tweets:\n return\n\n tweets = sort_and_truncate_tweets(tweets, sorting, limit)\n\n if pager:\n click.echo_via_pager(style_timeline(tweets, porcelain))\n else:\n click.echo(style_timeline(tweets, porcelain))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the list of sources you\u2019re following.", "response": "def following(ctx, check, timeout, porcelain):\n \"\"\"Return the list of sources you\u2019re following.\"\"\"\n sources = ctx.obj['conf'].following\n\n if check:\n sources = get_remote_status(sources, timeout)\n for (source, status) in sources:\n click.echo(style_source_with_status(source, status, porcelain))\n else:\n sources = sorted(sources, key=lambda source: source.nick)\n for source in sources:\n click.echo(style_source(source, porcelain))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nadds a new source to your followings.", "response": "def follow(ctx, nick, url, force):\n \"\"\"Add a new source to your followings.\"\"\"\n source = Source(nick, url)\n sources = ctx.obj['conf'].following\n\n if not force:\n if source.nick in (source.nick for source in sources):\n click.confirm(\"\u27a4 You\u2019re already following {0}. Overwrite?\".format(\n click.style(source.nick, bold=True)), default=False, abort=True)\n\n _, status = get_remote_status([source])[0]\n if not status or status.status_code != 200:\n click.confirm(\"\u27a4 The feed of {0} at {1} is not available. Follow anyway?\".format(\n click.style(source.nick, bold=True),\n click.style(source.url, bold=True)), default=False, abort=True)\n\n ctx.obj['conf'].add_source(source)\n click.echo(\"\u2713 You\u2019re now following {0}.\".format(\n click.style(source.nick, bold=True)))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove an existing source from your followings.", "response": "def unfollow(ctx, nick):\n \"\"\"Remove an existing source from your followings.\"\"\"\n source = ctx.obj['conf'].get_source_by_nick(nick)\n\n try:\n with Cache.discover() as cache:\n cache.remove_tweets(source.url)\n except OSError as e:\n logger.debug(e)\n\n ret_val = ctx.obj['conf'].remove_source_by_nick(nick)\n if ret_val:\n click.echo(\"\u2713 You\u2019ve unfollowed {0}.\".format(\n click.style(source.nick, bold=True)))\n else:\n click.echo(\"\u2717 You\u2019re not following {0}.\".format(\n click.style(nick, bold=True)))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef quickstart():\n width = click.get_terminal_size()[0]\n width = width if width <= 79 else 79\n\n click.secho(\"twtxt - quickstart\", fg=\"cyan\")\n click.secho(\"==================\", fg=\"cyan\")\n click.echo()\n\n help_text = \"This wizard will generate a basic configuration file for twtxt with all mandatory options set. \" \\\n \"You can change all of these later with either twtxt itself or by editing the config file manually. \" \\\n \"Have a look at the docs to get information about the other available options and their meaning.\"\n click.echo(textwrap.fill(help_text, width))\n\n click.echo()\n nick = click.prompt(\"\u27a4 Please enter your desired nick\", default=os.environ.get(\"USER\", \"\"))\n\n def overwrite_check(path):\n if os.path.isfile(path):\n click.confirm(\"\u27a4 '{0}' already exists. Overwrite?\".format(path), abort=True)\n\n cfgfile = click.prompt(\"\u27a4 Please enter the desired location for your config file\",\n os.path.join(Config.config_dir, Config.config_name),\n type=click.Path(readable=True, writable=True, file_okay=True))\n cfgfile = os.path.expanduser(cfgfile)\n overwrite_check(cfgfile)\n\n twtfile = click.prompt(\"\u27a4 Please enter the desired location for your twtxt file\",\n os.path.expanduser(\"~/twtxt.txt\"),\n type=click.Path(readable=True, writable=True, file_okay=True))\n twtfile = os.path.expanduser(twtfile)\n overwrite_check(twtfile)\n\n twturl = click.prompt(\"\u27a4 Please enter the URL your twtxt file will be accessible from\",\n default=\"https://example.org/twtxt.txt\")\n\n disclose_identity = click.confirm(\"\u27a4 Do you want to disclose your identity? Your nick and URL will be shared when \"\n \"making HTTP requests\", default=False)\n\n click.echo()\n add_news = click.confirm(\"\u27a4 Do you want to follow the twtxt news feed?\", default=True)\n\n conf = Config.create_config(cfgfile, nick, twtfile, twturl, disclose_identity, add_news)\n\n twtfile_dir = os.path.dirname(twtfile)\n if not os.path.exists(twtfile_dir):\n os.makedirs(twtfile_dir)\n open(twtfile, \"a\").close()\n\n click.echo()\n click.echo(\"\u2713 Created config file at '{0}'.\".format(click.format_filename(conf.config_file)))\n click.echo(\"\u2713 Created twtxt file at '{0}'.\".format(click.format_filename(twtfile)))", "response": "Quickstart wizard for setting up twtxt."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget or set config item.", "response": "def config(ctx, key, value, remove, edit):\n \"\"\"Get or set config item.\"\"\"\n conf = ctx.obj[\"conf\"]\n\n if not edit and not key:\n raise click.BadArgumentUsage(\"You have to specify either a key or use --edit.\")\n\n if edit:\n return click.edit(filename=conf.config_file)\n\n if remove:\n try:\n conf.cfg.remove_option(key[0], key[1])\n except Exception as e:\n logger.debug(e)\n else:\n conf.write_config()\n return\n\n if not value:\n try:\n click.echo(conf.cfg.get(key[0], key[1]))\n except Exception as e:\n logger.debug(e)\n return\n\n if not conf.cfg.has_section(key[0]):\n conf.cfg.add_section(key[0])\n\n conf.cfg.set(key[0], key[1], value)\n conf.write_config()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn human - readable relative time string.", "response": "def relative_datetime(self):\n \"\"\"Return human-readable relative time string.\"\"\"\n now = datetime.now(timezone.utc)\n tense = \"from now\" if self.created_at > now else \"ago\"\n return \"{0} {1}\".format(humanize.naturaldelta(now - self.created_at), tense)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save(url, *args, **kwargs):\n\n device = heimdallDevice(kwargs.get('device', None))\n\n kwargs['width'] = kwargs.get('width', None) or device.width\n kwargs['height'] = kwargs.get('height', None) or device.height\n kwargs['user_agent'] = kwargs.get('user_agent', None) or device.user_agent\n\n screenshot_image = screenshot(url, **kwargs)\n\n if kwargs.get('optimize'):\n image = Image.open(screenshot_image.path)\n image.save(screenshot_image.path, optimize=True)\n\n return screenshot_image", "response": "Save a screenshot of a URL."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalls PhantomJS with the specified arguments and return the path to the screenshot.", "response": "def screenshot(url, *args, **kwargs):\n \"\"\" Call PhantomJS with the specified flags and options. \"\"\"\n\n phantomscript = os.path.join(os.path.dirname(__file__),\n 'take_screenshot.js')\n\n directory = kwargs.get('save_dir', '/tmp')\n image_name = kwargs.get('image_name', None) or _image_name_from_url(url)\n ext = kwargs.get('format', 'png').lower()\n save_path = os.path.join(directory, image_name) + '.' + ext\n crop_to_visible = kwargs.get('crop_to_visible', False)\n\n cmd_args = [\n 'phantomjs',\n '--ssl-protocol=any',\n phantomscript,\n url,\n '--width',\n str(kwargs['width']),\n '--height',\n str(kwargs['height']),\n '--useragent',\n str(kwargs['user_agent']),\n '--dir',\n directory,\n '--ext',\n ext,\n '--name',\n str(image_name),\n ]\n if crop_to_visible:\n cmd_args.append('--croptovisible')\n\n # TODO:\n # - quality\n # - renderafter\n # - maxexecutiontime\n # - resourcetimeout\n\n output = subprocess.Popen(cmd_args,\n stdout=subprocess.PIPE).communicate()[0]\n\n return Screenshot(save_path, directory, image_name + '.' + ext, ext)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _image_name_from_url(url):\n\n find = r'https?://|[^\\w]'\n replace = '_'\n return re.sub(find, replace, url).strip('_')", "response": "Create a nice image name from the url."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns nearest parent permission for path.", "response": "def get_permissions(self, path):\n \"\"\"\n Return nearest parent permission for `path`.\n\n :param path: path which permission you want to know\n :type path: :py:class:`str` or :py:class:`pathlib.PurePosixPath`\n\n :rtype: :py:class:`aioftp.Permission`\n \"\"\"\n path = pathlib.PurePosixPath(path)\n parents = filter(lambda p: p.is_parent(path), self.permissions)\n perm = min(\n parents,\n key=lambda p: len(path.relative_to(p.path).parts),\n default=Permission(),\n )\n return perm"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nincrement the internal counter by one.", "response": "def release(self):\n \"\"\"\n Release, incrementing the internal counter by one.\n \"\"\"\n if self.value is not None:\n self.value += 1\n if self.value > self.maximum_value:\n raise ValueError(\"Too many releases\")"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def start(self, host=None, port=0, **kwargs):\n self._start_server_extra_arguments = kwargs\n self.connections = {}\n self.server_host = host\n self.server_port = port\n self.server = await asyncio.start_server(\n self.dispatcher,\n host,\n port,\n ssl=self.ssl,\n **self._start_server_extra_arguments,\n )\n for sock in self.server.sockets:\n if sock.family in (socket.AF_INET, socket.AF_INET6):\n host, port, *_ = sock.getsockname()\n if not self.server_port:\n self.server_port = port\n if not self.server_host:\n self.server_host = host\n logger.info(\"serving on %s:%s\", host, port)", "response": ":py:func:`asyncio.coroutine`\n\n Start server.\n\n :param host: ip address to bind for listening.\n :type host: :py:class:`str`\n\n :param port: port number to bind for listening.\n :type port: :py:class:`int`\n\n :param kwargs: keyword arguments, they passed to\n :py:func:`asyncio.start_server`"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting a response to the server.", "response": "async def write_response(self, stream, code, lines=\"\", list=False):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n Complex method for sending response.\n\n :param stream: command connection stream\n :type stream: :py:class:`aioftp.StreamIO`\n\n :param code: server response code\n :type code: :py:class:`str`\n\n :param lines: line or lines, which are response information\n :type lines: :py:class:`str` or :py:class:`collections.Iterable`\n\n :param list: if true, then lines will be sended without code prefix.\n This is useful for **LIST** FTP command and some others.\n :type list: :py:class:`bool`\n \"\"\"\n lines = wrap_with_container(lines)\n write = functools.partial(self.write_line, stream)\n if list:\n head, *body, tail = lines\n await write(code + \"-\" + head)\n for line in body:\n await write(\" \" + line)\n await write(code + \" \" + tail)\n else:\n *body, tail = lines\n for line in body:\n await write(code + \"-\" + line)\n await write(code + \" \" + tail)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def parse_command(self, stream):\n line = await stream.readline()\n if not line:\n raise ConnectionResetError\n s = line.decode(encoding=self.encoding).rstrip()\n logger.info(s)\n cmd, _, rest = s.partition(\" \")\n return cmd.lower(), rest", "response": "Parses the command from the connection steramateram command."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns real and virtual paths for user - view and user requests.", "response": "def get_paths(self, connection, path):\n \"\"\"\n Return *real* and *virtual* paths, resolves \"..\" with \"up\" action.\n *Real* path is path for path_io, when *virtual* deals with\n \"user-view\" and user requests\n\n :param connection: internal options for current connected user\n :type connection: :py:class:`dict`\n\n :param path: received path from user\n :type path: :py:class:`str` or :py:class:`pathlib.PurePosixPath`\n\n :return: (real_path, virtual_path)\n :rtype: (:py:class:`pathlib.Path`, :py:class:`pathlib.PurePosixPath`)\n \"\"\"\n virtual_path = pathlib.PurePosixPath(path)\n if not virtual_path.is_absolute():\n virtual_path = connection.current_directory / virtual_path\n resolved_virtual_path = pathlib.PurePosixPath(\"/\")\n for part in virtual_path.parts[1:]:\n if part == \"..\":\n resolved_virtual_path = resolved_virtual_path.parent\n else:\n resolved_virtual_path /= part\n base_path = connection.user.base_path\n real_path = base_path / resolved_virtual_path.relative_to(\"/\")\n return real_path, resolved_virtual_path"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef bytes2human(n, format=\"%(value).1f%(symbol)s\"):\n symbols = ('B', 'K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y')\n prefix = {}\n for i, s in enumerate(symbols[1:]):\n prefix[s] = 1 << (i + 1) * 10\n for symbol in reversed(symbols[1:]):\n if n >= prefix[symbol]:\n value = float(n) / prefix[symbol]\n return format % locals()\n return format % dict(symbol=symbols[0], value=n)", "response": "Convert a number of bytes into a human readable string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef human2bytes(s):\n symbols = ('B', 'K', 'M', 'G', 'T', 'P', 'E', 'Z', 'Y')\n letter = s[-1:].strip().upper()\n num = s[:-1]\n assert num.isdigit() and letter in symbols, s\n num = float(num)\n prefix = {symbols[0]: 1}\n for i, s in enumerate(symbols[1:]):\n prefix[s] = 1 << (i + 1) * 10\n return int(num * prefix[letter])", "response": "Convert a human readable string to a number of bytes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef register_memory():\n # XXX How to get a reliable representation of memory being used is\n # not clear. (rss - shared) seems kind of ok but we might also use\n # the private working set via get_memory_maps().private*.\n def get_mem(proc):\n if os.name == 'posix':\n mem = proc.memory_info_ex()\n counter = mem.rss\n if 'shared' in mem._fields:\n counter -= mem.shared\n return counter\n else:\n # TODO figure out what to do on Windows\n return proc.get_memory_info().rss\n\n if SERVER_PROC is not None:\n mem = get_mem(SERVER_PROC)\n for child in SERVER_PROC.children():\n mem += get_mem(child)\n server_memory.append(bytes2human(mem))", "response": "Register an approximation of memory used by FTP server process\n and all of its children."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconnect to the FTP server login and return an ftplib. FTP instance.", "response": "def connect():\n \"\"\"Connect to FTP server, login and return an ftplib.FTP instance.\"\"\"\n ftp_class = ftplib.FTP if not SSL else ftplib.FTP_TLS\n ftp = ftp_class(timeout=TIMEOUT)\n ftp.connect(HOST, PORT)\n ftp.login(USER, PASSWORD)\n if SSL:\n ftp.prot_p() # secure data connection\n return ftp"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef retr(ftp):\n ftp.voidcmd('TYPE I')\n with contextlib.closing(ftp.transfercmd(\"RETR \" + TESTFN)) as conn:\n recv_bytes = 0\n while True:\n data = conn.recv(BUFFER_LEN)\n if not data:\n break\n recv_bytes += len(data)\n ftp.voidresp()", "response": "Same as ftplib s retrbinary but discard the received data."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef stor(ftp=None):\n if ftp is None:\n ftp = connect()\n quit = True\n else:\n quit = False\n ftp.voidcmd('TYPE I')\n with contextlib.closing(ftp.transfercmd(\"STOR \" + TESTFN)) as conn:\n chunk = b'x' * BUFFER_LEN\n total_sent = 0\n while True:\n sent = conn.send(chunk)\n total_sent += sent\n if total_sent >= FILE_SIZE:\n break\n ftp.voidresp()\n if quit:\n ftp.quit()\n return ftp", "response": "Same as ftplib s storbinary but just sends dummy data\n instead of reading it from a real file."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef bytes_per_second(ftp, retr=True):\n tot_bytes = 0\n if retr:\n def request_file():\n ftp.voidcmd('TYPE I')\n conn = ftp.transfercmd(\"retr \" + TESTFN)\n return conn\n\n with contextlib.closing(request_file()) as conn:\n register_memory()\n stop_at = time.time() + 1.0\n while stop_at > time.time():\n chunk = conn.recv(BUFFER_LEN)\n if not chunk:\n a = time.time()\n ftp.voidresp()\n conn.close()\n conn = request_file()\n stop_at += time.time() - a\n tot_bytes += len(chunk)\n\n try:\n while chunk:\n chunk = conn.recv(BUFFER_LEN)\n ftp.voidresp()\n conn.close()\n except (ftplib.error_temp, ftplib.error_perm):\n pass\n else:\n ftp.voidcmd('TYPE I')\n with contextlib.closing(ftp.transfercmd(\"STOR \" + TESTFN)) as conn:\n register_memory()\n chunk = b'x' * BUFFER_LEN\n stop_at = time.time() + 1\n while stop_at > time.time():\n tot_bytes += conn.send(chunk)\n ftp.voidresp()\n\n return tot_bytes", "response": "Return the number of bytes transmitted in 1 second."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef universal_exception(coro):\n @functools.wraps(coro)\n async def wrapper(*args, **kwargs):\n try:\n return await coro(*args, **kwargs)\n except (asyncio.CancelledError, NotImplementedError,\n StopAsyncIteration):\n raise\n except Exception:\n raise errors.PathIOError(reason=sys.exc_info())\n\n return wrapper", "response": "Decorator. Reraising any exception except CancelledError and anaioftp. PathIOError with universal exception\n ."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwaiting until the record is available.", "response": "async def wait(self):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n Wait until can do IO\n \"\"\"\n if self._limit is not None and self._limit > 0 and \\\n self._start is not None:\n now = _now()\n end = self._start + self._sum / self._limit\n await asyncio.sleep(max(0, end - now))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nappend data to the internal buffer.", "response": "def append(self, data, start):\n \"\"\"\n Count `data` for throttle\n\n :param data: bytes of data for count\n :type data: :py:class:`bytes`\n\n :param start: start of read/write time from\n :py:meth:`asyncio.BaseEventLoop.time`\n :type start: :py:class:`float`\n \"\"\"\n if self._limit is not None and self._limit > 0:\n if self._start is None:\n self._start = start\n if start - self._start > self.reset_rate:\n self._sum -= round((start - self._start) * self._limit)\n self._start = start\n self._sum += len(data)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef limit(self, value):\n self._limit = value\n self._start = None\n self._sum = 0", "response": "Set the throttle limit for this object"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncloning throttles without memory", "response": "def clone(self):\n \"\"\"\n Clone throttles without memory\n \"\"\"\n return StreamThrottle(\n read=self.read.clone(),\n write=self.write.clone()\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_limits(cls, read_speed_limit=None, write_speed_limit=None):\n return cls(read=Throttle(limit=read_speed_limit),\n write=Throttle(limit=write_speed_limit))", "response": "Create a new StreamThrottle object from the given limits."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def wait(self, name):\n waiters = []\n for throttle in self.throttles.values():\n curr_throttle = getattr(throttle, name)\n if curr_throttle.limit:\n waiters.append(curr_throttle.wait())\n if waiters:\n await asyncio.wait(waiters)", "response": "Wait for all throttles with the given name to be acquired."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef append(self, name, data, start):\n for throttle in self.throttles.values():\n getattr(throttle, name).append(data, start)", "response": "Update timeout for all throttles with the given name and data."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nread count bytes from the stream", "response": "async def read(self, count=-1):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n :py:meth:`aioftp.StreamIO.read` proxy\n \"\"\"\n await self.wait(\"read\")\n start = _now()\n data = await super().read(count)\n self.append(\"read\", data, start)\n return data"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\nasync def write(self, data):\n await self.wait(\"write\")\n start = _now()\n await super().write(data)\n self.append(\"write\", data, start)", "response": "write data to the stream"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if the set of keys matches the given mask.", "response": "def matches(self, mask):\n \"\"\"\n :param mask: Template for comparision. If mask symbol is not digit\n then it passes.\n :type mask: :py:class:`str`\n\n ::\n\n >>> Code(\"123\").matches(\"1\")\n True\n >>> Code(\"123\").matches(\"1x3\")\n True\n \"\"\"\n return all(map(lambda m, c: not m.isdigit() or m == c, mask, self))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfinish the connection and wait for expected_codes response from server passing wait_codes.", "response": "async def finish(self, expected_codes=\"2xx\", wait_codes=\"1xx\"):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n Close connection and wait for `expected_codes` response from server\n passing `wait_codes`.\n\n :param expected_codes: tuple of expected codes or expected code\n :type expected_codes: :py:class:`tuple` of :py:class:`str` or\n :py:class:`str`\n\n :param wait_codes: tuple of wait codes or wait code\n :type wait_codes: :py:class:`tuple` of :py:class:`str` or\n :py:class:`str`\n \"\"\"\n self.close()\n await self.client.command(None, expected_codes, wait_codes)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nparses a single line from the server response stream.", "response": "async def parse_line(self):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n Parsing server response line.\n\n :return: (code, line)\n :rtype: (:py:class:`aioftp.Code`, :py:class:`str`)\n\n :raises ConnectionResetError: if received data is empty (this\n means, that connection is closed)\n :raises asyncio.TimeoutError: if there where no data for `timeout`\n period\n \"\"\"\n line = await self.stream.readline()\n if not line:\n self.stream.close()\n raise ConnectionResetError\n s = line.decode(encoding=self.encoding).rstrip()\n logger.info(s)\n return Code(s[:3]), s[3:]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\nasync def parse_response(self):\n code, rest = await self.parse_line()\n info = [rest]\n curr_code = code\n while rest.startswith(\"-\") or not curr_code.isdigit():\n curr_code, rest = await self.parse_line()\n if curr_code.isdigit():\n info.append(rest)\n if curr_code != code:\n raise errors.StatusCodeError(code, curr_code, info)\n else:\n info.append(curr_code + rest)\n return code, info", "response": "Parses the server response and returns the code and list of lines."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_codes(self, expected_codes, received_code, info):\n if not any(map(received_code.matches, expected_codes)):\n raise errors.StatusCodeError(expected_codes, received_code, info)", "response": "Checks if any of expected codes matches received."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def command(self, command=None, expected_codes=(), wait_codes=()):\n expected_codes = wrap_with_container(expected_codes)\n wait_codes = wrap_with_container(wait_codes)\n if command:\n logger.info(command)\n message = command + END_OF_LINE\n await self.stream.write(message.encode(encoding=self.encoding))\n if expected_codes or wait_codes:\n code, info = await self.parse_response()\n while any(map(code.matches, wait_codes)):\n code, info = await self.parse_response()\n if expected_codes:\n self.check_codes(expected_codes, code, info)\n return code, info", "response": "Send command to the server and wait for response."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nparsing the response from the EPSV command line.", "response": "def parse_epsv_response(s):\n \"\"\"\n Parsing `EPSV` (`message (|||port|)`) response.\n\n :param s: response line\n :type s: :py:class:`str`\n\n :return: (ip, port)\n :rtype: (:py:class:`None`, :py:class:`int`)\n \"\"\"\n matches = tuple(re.finditer(r\"\\((.)\\1\\1\\d+\\1\\)\", s))\n s = matches[-1].group()\n port = int(s[4:-2])\n return None, port"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_pasv_response(s):\n sub, *_ = re.findall(r\"[^(]*\\(([^)]*)\", s)\n nums = tuple(map(int, sub.split(\",\")))\n ip = \".\".join(map(str, nums[:4]))\n port = (nums[4] << 8) | nums[5]\n return ip, port", "response": "Parses a PASV server response line into ip and port tuple."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the response from a directory server response.", "response": "def parse_directory_response(s):\n \"\"\"\n Parsing directory server response.\n\n :param s: response line\n :type s: :py:class:`str`\n\n :rtype: :py:class:`pathlib.PurePosixPath`\n \"\"\"\n seq_quotes = 0\n start = False\n directory = \"\"\n for ch in s:\n if not start:\n if ch == \"\\\"\":\n start = True\n else:\n if ch == \"\\\"\":\n seq_quotes += 1\n else:\n if seq_quotes == 1:\n break\n elif seq_quotes == 2:\n seq_quotes = 0\n directory += '\"'\n directory += ch\n return pathlib.PurePosixPath(directory)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef parse_unix_mode(s):\n parse_rw = {\"rw\": 6, \"r-\": 4, \"-w\": 2, \"--\": 0}\n mode = 0\n mode |= parse_rw[s[0:2]] << 6\n mode |= parse_rw[s[3:5]] << 3\n mode |= parse_rw[s[6:8]]\n if s[2] == \"s\":\n mode |= 0o4100\n elif s[2] == \"x\":\n mode |= 0o0100\n elif s[2] != \"-\":\n raise ValueError\n\n if s[5] == \"s\":\n mode |= 0o2010\n elif s[5] == \"x\":\n mode |= 0o0010\n elif s[5] != \"-\":\n raise ValueError\n\n if s[8] == \"t\":\n mode |= 0o1000\n elif s[8] == \"x\":\n mode |= 0o0001\n elif s[8] != \"-\":\n raise ValueError\n\n return mode", "response": "Parses a unix mode string into a mode integer."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_ls_date(self, s, *, now=None):\n with setlocale(\"C\"):\n try:\n if now is None:\n now = datetime.datetime.now()\n d = datetime.datetime.strptime(s, \"%b %d %H:%M\")\n d = d.replace(year=now.year)\n diff = (now - d).total_seconds()\n if diff > HALF_OF_YEAR_IN_SECONDS:\n d = d.replace(year=now.year + 1)\n elif diff < -HALF_OF_YEAR_IN_SECONDS:\n d = d.replace(year=now.year - 1)\n except ValueError:\n d = datetime.datetime.strptime(s, \"%b %d %Y\")\n return self.format_date_time(d)", "response": "Parse a ls date string into a human readable date string."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing a LIST line from the server.", "response": "def parse_list_line_unix(self, b):\n \"\"\"\n Attempt to parse a LIST line (similar to unix ls utility).\n\n :param b: response line\n :type b: :py:class:`bytes` or :py:class:`str`\n\n :return: (path, info)\n :rtype: (:py:class:`pathlib.PurePosixPath`, :py:class:`dict`)\n \"\"\"\n s = b.decode(encoding=self.encoding).rstrip()\n info = {}\n if s[0] == \"-\":\n info[\"type\"] = \"file\"\n elif s[0] == \"d\":\n info[\"type\"] = \"dir\"\n elif s[0] == \"l\":\n info[\"type\"] = \"link\"\n else:\n info[\"type\"] = \"unknown\"\n\n # TODO: handle symlinks(beware the symlink loop)\n info[\"unix.mode\"] = self.parse_unix_mode(s[1:10])\n s = s[10:].lstrip()\n i = s.index(\" \")\n info[\"unix.links\"] = s[:i]\n\n if not info[\"unix.links\"].isdigit():\n raise ValueError\n\n s = s[i:].lstrip()\n i = s.index(\" \")\n info[\"unix.owner\"] = s[:i]\n s = s[i:].lstrip()\n i = s.index(\" \")\n info[\"unix.group\"] = s[:i]\n s = s[i:].lstrip()\n i = s.index(\" \")\n info[\"size\"] = s[:i]\n\n if not info[\"size\"].isdigit():\n raise ValueError\n\n s = s[i:].lstrip()\n info[\"modify\"] = self.parse_ls_date(s[:12])\n s = s[12:].strip()\n if info[\"type\"] == \"link\":\n i = s.rindex(\" -> \")\n link_dst = s[i + 4:]\n link_src = s[:i]\n i = -2 if link_dst[-1] == \"\\'\" or link_dst[-1] == \"\\\"\" else -1\n info[\"type\"] = \"dir\" if link_dst[i] == \"/\" else \"file\"\n s = link_src\n return pathlib.PurePosixPath(s), info"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nparsing Microsoft Windows dir output line and returns a tuple of path info", "response": "def parse_list_line_windows(self, b):\n \"\"\"\n Parsing Microsoft Windows `dir` output\n\n :param b: response line\n :type b: :py:class:`bytes` or :py:class:`str`\n\n :return: (path, info)\n :rtype: (:py:class:`pathlib.PurePosixPath`, :py:class:`dict`)\n \"\"\"\n line = b.decode(encoding=self.encoding).rstrip(\"\\r\\n\")\n date_time_end = line.index(\"M\")\n date_time_str = line[:date_time_end + 1].strip().split(\" \")\n date_time_str = \" \".join([x for x in date_time_str if len(x) > 0])\n line = line[date_time_end + 1:].lstrip()\n with setlocale(\"C\"):\n strptime = datetime.datetime.strptime\n date_time = strptime(date_time_str, \"%m/%d/%Y %I:%M %p\")\n info = {}\n info[\"modify\"] = self.format_date_time(date_time)\n next_space = line.index(\" \")\n if line.startswith(\"\"):\n info[\"type\"] = \"dir\"\n else:\n info[\"type\"] = \"file\"\n info[\"size\"] = line[:next_space].replace(\",\", \"\")\n if not info[\"size\"].isdigit():\n raise ValueError\n # This here could cause a problem if a filename started with\n # whitespace, but if we were to try to detect such a condition\n # we would have to make strong assumptions about the input format\n filename = line[next_space:].lstrip()\n if filename == \".\" or filename == \"..\":\n raise ValueError\n return pathlib.PurePosixPath(filename), info"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_list_line(self, b):\n ex = []\n parsers = (self.parse_list_line_unix, self.parse_list_line_windows)\n for parser in parsers:\n try:\n return parser(b)\n except (ValueError, KeyError, IndexError) as e:\n ex.append(e)\n raise ValueError(\"All parsers failed to parse\", b, ex)", "response": "Parses LIST response with both Microsoft Windows\u00ae and UNIX parsers and returns a tuple of the path and info."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef parse_mlsx_line(self, b):\n if isinstance(b, bytes):\n s = b.decode(encoding=self.encoding)\n else:\n s = b\n line = s.rstrip()\n facts_found, _, name = line.partition(\" \")\n entry = {}\n for fact in facts_found[:-1].split(\";\"):\n key, _, value = fact.partition(\"=\")\n entry[key.lower()] = value\n return pathlib.PurePosixPath(name), entry", "response": "Parses the MLS response line into a path and info dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def login(self, user=DEFAULT_USER, password=DEFAULT_PASSWORD,\n account=DEFAULT_ACCOUNT):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n Server authentication.\n\n :param user: username\n :type user: :py:class:`str`\n\n :param password: password\n :type password: :py:class:`str`\n\n :param account: account (almost always blank)\n :type account: :py:class:`str`\n\n :raises aioftp.StatusCodeError: if unknown code received\n \"\"\"\n code, info = await self.command(\"USER \" + user, (\"230\", \"33x\"))\n while code.matches(\"33x\"):\n if code == \"331\":\n cmd = \"PASS \" + password\n elif code == \"332\":\n cmd = \"ACCT \" + account\n else:\n raise errors.StatusCodeError(\"33x\", code, info)\n code, info = await self.command(cmd, (\"230\", \"33x\"))", "response": "Login to the server."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\nasync def get_current_directory(self):\n code, info = await self.command(\"PWD\", \"257\")\n directory = self.parse_directory_response(info[-1])\n return directory", "response": "Get current working directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def change_directory(self, path=\"..\"):\n path = pathlib.PurePosixPath(path)\n if path == pathlib.PurePosixPath(\"..\"):\n cmd = \"CDUP\"\n else:\n cmd = \"CWD \" + str(path)\n await self.command(cmd, \"2xx\")", "response": ":py:func:`asyncio.coroutine`\n\n Change current directory. Goes \u00abup\u00bb if no parameters passed.\n\n :param path: new directory, goes \u00abup\u00bb if omitted\n :type path: :py:class:`str` or :py:class:`pathlib.PurePosixPath`"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\nasync def make_directory(self, path, *, parents=True):\n path = pathlib.PurePosixPath(path)\n need_create = []\n while path.name and not await self.exists(path):\n need_create.append(path)\n path = path.parent\n if not parents:\n break\n need_create.reverse()\n for path in need_create:\n await self.command(\"MKD \" + str(path), \"257\")", "response": "Create a new directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef list(self, path=\"\", *, recursive=False, raw_command=None):\n class AsyncLister(AsyncListerMixin):\n stream = None\n\n async def _new_stream(cls, local_path):\n cls.path = local_path\n cls.parse_line = self.parse_mlsx_line\n if raw_command not in [None, \"MLSD\", \"LIST\"]:\n raise ValueError(\"raw_command must be one of MLSD or \"\n f\"LIST, but got {raw_command}\")\n if raw_command in [None, \"MLSD\"]:\n try:\n command = (\"MLSD \" + str(cls.path)).strip()\n return await self.get_stream(command, \"1xx\")\n except errors.StatusCodeError as e:\n code = e.received_codes[-1]\n if not code.matches(\"50x\") or raw_command is not None:\n raise\n if raw_command in [None, \"LIST\"]:\n cls.parse_line = self.parse_list_line\n command = (\"LIST \" + str(cls.path)).strip()\n return await self.get_stream(command, \"1xx\")\n\n def __aiter__(cls):\n cls.directories = collections.deque()\n return cls\n\n async def __anext__(cls):\n if cls.stream is None:\n cls.stream = await cls._new_stream(path)\n while True:\n line = await cls.stream.readline()\n while not line:\n await cls.stream.finish()\n if cls.directories:\n current_path, info = cls.directories.popleft()\n cls.stream = await cls._new_stream(current_path)\n line = await cls.stream.readline()\n else:\n raise StopAsyncIteration\n\n try:\n name, info = cls.parse_line(line)\n except Exception:\n continue\n\n stat = cls.path / name, info\n if info[\"type\"] == \"dir\" and recursive:\n cls.directories.append(stat)\n return stat\n\n return AsyncLister()", "response": "List all files and directories in path."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\nasync def exists(self, path):\n try:\n await self.stat(path)\n return True\n except errors.StatusCodeError as e:\n if e.received_codes[-1].matches(\"550\"):\n return False\n raise", "response": "Check if a path exists."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def rename(self, source, destination):\n await self.command(\"RNFR \" + str(source), \"350\")\n await self.command(\"RNTO \" + str(destination), \"2xx\")", "response": "Rename the name of the current key store."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\nasync def remove(self, path):\n if await self.exists(path):\n info = await self.stat(path)\n if info[\"type\"] == \"file\":\n await self.remove_file(path)\n elif info[\"type\"] == \"dir\":\n for name, info in (await self.list(path)):\n if info[\"type\"] in (\"dir\", \"file\"):\n await self.remove(name)\n await self.remove_directory(path)", "response": "High level remove method for removing path recursively."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef upload_stream(self, destination, *, offset=0):\n return self.get_stream(\n \"STOR \" + str(destination),\n \"1xx\",\n offset=offset,\n )", "response": "Create stream for write data to destination file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate stream for append data to destination file.", "response": "def append_stream(self, destination, *, offset=0):\n \"\"\"\n Create stream for append (write) data to `destination` file.\n\n :param destination: destination path of file on server side\n :type destination: :py:class:`str` or :py:class:`pathlib.PurePosixPath`\n\n :param offset: byte offset for stream start position\n :type offset: :py:class:`int`\n\n :rtype: :py:class:`aioftp.DataConnectionThrottleStreamIO`\n \"\"\"\n return self.get_stream(\n \"APPE \" + str(destination),\n \"1xx\",\n offset=offset,\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a stream for reading data from source file.", "response": "def download_stream(self, source, *, offset=0):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n Create stream for read data from `source` file.\n\n :param source: source path of file on server side\n :type source: :py:class:`str` or :py:class:`pathlib.PurePosixPath`\n\n :param offset: byte offset for stream start position\n :type offset: :py:class:`int`\n\n :rtype: :py:class:`aioftp.DataConnectionThrottleStreamIO`\n \"\"\"\n return self.get_stream(\"RETR \" + str(source), \"1xx\", offset=offset)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ndownloads the contents of the source file or directory into the destination file system.", "response": "async def download(self, source, destination=\"\", *, write_into=False,\n block_size=DEFAULT_BLOCK_SIZE):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n High level download method for downloading files and directories\n recursively and save them to the file system.\n\n :param source: source path of file or directory on server side\n :type source: :py:class:`str` or :py:class:`pathlib.PurePosixPath`\n\n :param destination: destination path of file or directory on client\n side\n :type destination: :py:class:`str` or :py:class:`pathlib.Path`\n\n :param write_into: write source into destination (if you want download\n file and change it name, as well with directories)\n :type write_into: :py:class:`bool`\n\n :param block_size: block size for transaction\n :type block_size: :py:class:`int`\n \"\"\"\n source = pathlib.PurePosixPath(source)\n destination = pathlib.Path(destination)\n if not write_into:\n destination = destination / source.name\n if await self.is_file(source):\n await self.path_io.mkdir(destination.parent,\n parents=True, exist_ok=True)\n async with self.path_io.open(destination, mode=\"wb\") as file_out, \\\n self.download_stream(source) as stream:\n async for block in stream.iter_by_block(block_size):\n await file_out.write(block)\n elif await self.is_dir(source):\n await self.path_io.mkdir(destination, parents=True, exist_ok=True)\n for name, info in (await self.list(source)):\n full = destination / name.relative_to(source)\n if info[\"type\"] in (\"file\", \"dir\"):\n await self.download(name, full, write_into=True,\n block_size=block_size)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting a new passive connection with the server.", "response": "async def get_passive_connection(self, conn_type=\"I\",\n commands=(\"epsv\", \"pasv\")):\n \"\"\"\n :py:func:`asyncio.coroutine`\n\n Getting pair of reader, writer for passive connection with server.\n\n :param conn_type: connection type (\"I\", \"A\", \"E\", \"L\")\n :type conn_type: :py:class:`str`\n\n :param commands: sequence of commands to try to initiate passive\n server creation. First success wins. Default is EPSV, then PASV.\n :type commands: :py:class:`list`\n\n :rtype: (:py:class:`asyncio.StreamReader`,\n :py:class:`asyncio.StreamWriter`)\n \"\"\"\n functions = {\n \"epsv\": self._do_epsv,\n \"pasv\": self._do_pasv,\n }\n if not commands:\n raise ValueError(\"No passive commands provided\")\n await self.command(\"TYPE \" + conn_type, \"200\")\n for i, name in enumerate(commands, start=1):\n name = name.lower()\n if name not in functions:\n raise ValueError(f\"{name!r} not in {set(functions)!r}\")\n try:\n ip, port = await functions[name]()\n break\n except errors.StatusCodeError as e:\n is_last = i == len(commands)\n if is_last or not e.received_codes[-1].matches(\"50x\"):\n raise\n if ip in (\"0.0.0.0\", None):\n ip = self.server_host\n reader, writer = await open_connection(\n ip,\n port,\n self.create_connection,\n self.ssl,\n )\n return reader, writer"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\nasync def get_stream(self, *command_args, conn_type=\"I\", offset=0):\n reader, writer = await self.get_passive_connection(conn_type)\n if offset:\n await self.command(\"REST \" + str(offset), \"350\")\n await self.command(*command_args)\n stream = DataConnectionThrottleStreamIO(\n self,\n reader,\n writer,\n throttles={\"_\": self.throttle},\n timeout=self.socket_timeout,\n )\n return stream", "response": "Get a stream for the current state of the current instance of the class."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef jenks_breaks(values, nb_class):\n\n if not isinstance(values, Iterable) or isinstance(values, (str, bytes)):\n raise TypeError(\"A sequence of numbers is expected\")\n if isinstance(nb_class, float) and int(nb_class) == nb_class:\n nb_class = int(nb_class)\n if not isinstance(nb_class, int):\n raise TypeError(\n \"Number of class have to be a positive integer: \"\n \"expected an instance of 'int' but found {}\"\n .format(type(nb_class)))\n\n nb_values = len(values)\n if np and isinstance(values, np.ndarray):\n values = values[np.argwhere(np.isfinite(values)).reshape(-1)]\n else:\n values = [i for i in values if isfinite(i)]\n \n if len(values) != nb_values:\n warnings.warn('Invalid values encountered (NaN or Inf) were ignored')\n nb_values = len(values)\n \n if nb_class >= nb_values or nb_class < 2:\n raise ValueError(\"Number of class have to be an integer \"\n \"greater than 2 and \"\n \"smaller than the number of values to use\")\n\n return jenks._jenks_breaks(values, nb_class)", "response": "Compute jenks natural breaks on a sequence of values given nb_class."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngrab the image of the current window to a file.", "response": "def grab_to_file(self, filename, bbox=None):\n \"\"\"http://www.pygtk.org/docs/pygtk/class-gdkpixbuf.html.\n\n only \"jpeg\" or \"png\"\n \"\"\"\n\n w = self.gtk.gdk.get_default_root_window()\n# Capture the whole screen.\n if bbox is None:\n sz = w.get_size()\n pb = self.gtk.gdk.Pixbuf(\n self.gtk.gdk.COLORSPACE_RGB, False, 8, sz[0], sz[1]) # 24bit RGB\n pb = pb.get_from_drawable(\n w, w.get_colormap(), 0, 0, 0, 0, sz[0], sz[1])\n# Only capture what we need. The smaller the capture, the faster.\n else:\n sz = [bbox[2] - bbox[0], bbox[3] - bbox[1]]\n pb = self.gtk.gdk.Pixbuf(\n self.gtk.gdk.COLORSPACE_RGB, False, 8, sz[0], sz[1])\n pb = pb.get_from_drawable(\n w, w.get_colormap(), bbox[0], bbox[1], 0, 0, sz[0], sz[1])\n assert pb\n ftype = 'png'\n if filename.endswith('.jpeg'):\n ftype = 'jpeg'\n\n pb.save(filename, ftype)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngrabbing an image directly to a buffer.", "response": "def grab(self, bbox=None):\n \"\"\"Grabs an image directly to a buffer.\n\n :param bbox: Optional tuple or list containing (x1, y1, x2, y2) coordinates\n of sub-region to capture.\n :return: PIL RGB image\n :raises: ValueError, if image data does not have 3 channels (RGB), each with 8\n bits.\n :rtype: Image\n \"\"\"\n w = Gdk.get_default_root_window()\n if bbox is not None:\n g = [bbox[0], bbox[1], bbox[2] - bbox[0], bbox[3] - bbox[1]]\n else:\n g = w.get_geometry()\n pb = Gdk.pixbuf_get_from_window(w, *g)\n if pb.get_bits_per_sample() != 8:\n raise ValueError('Expected 8 bits per pixel.')\n elif pb.get_n_channels() != 3:\n raise ValueError('Expected RGB image.')\n\n # Read the entire buffer into a python bytes object.\n # read_pixel_bytes: New in version 2.32.\n pixel_bytes = pb.read_pixel_bytes().get_data() # type: bytes\n width, height = g[2], g[3]\n\n # Probably for SSE alignment reasons, the pixbuf has extra data in each line.\n # The args after \"raw\" help handle this; see\n # http://effbot.org/imagingbook/decoder.htm#the-raw-decoder\n return Image.frombytes(\n 'RGB', (width, height), pixel_bytes, 'raw', 'RGB', pb.get_rowstride(), 1)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngrabbing the contents of the screen to PIL image memory.", "response": "def grab(bbox=None, childprocess=None, backend=None):\n \"\"\"Copy the contents of the screen to PIL image memory.\n\n :param bbox: optional bounding box (x1,y1,x2,y2)\n :param childprocess: pyscreenshot can cause an error,\n if it is used on more different virtual displays\n and back-end is not in different process.\n Some back-ends are always different processes: scrot, imagemagick\n The default is False if the program was started inside IDLE,\n otherwise it is True.\n :param backend: back-end can be forced if set (examples:scrot, wx,..),\n otherwise back-end is automatic\n \"\"\"\n if childprocess is None:\n childprocess = childprocess_default_value()\n return _grab(\n to_file=False, childprocess=childprocess, backend=backend, bbox=bbox)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef grab_to_file(filename, childprocess=None, backend=None):\n if childprocess is None:\n childprocess = childprocess_default_value()\n return _grab(to_file=True, childprocess=childprocess,\n backend=backend, filename=filename)", "response": "Copy the contents of the screen to a file."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nbacking - end version.", "response": "def backend_version(backend, childprocess=None):\n \"\"\"Back-end version.\n\n :param backend: back-end (examples:scrot, wx,..)\n :param childprocess: see :py:func:`grab`\n :return: version as string\n \"\"\"\n if childprocess is None:\n childprocess = childprocess_default_value()\n if not childprocess:\n return _backend_version(backend)\n else:\n return run_in_childprocess(_backend_version, None, backend)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwrites data from process_tile to PNG file.", "response": "def write(self, process_tile, data):\n \"\"\"\n Write data from process tiles into PNG file(s).\n\n Parameters\n ----------\n process_tile : ``BufferedTile``\n must be member of process ``TilePyramid``\n \"\"\"\n data = self._prepare_array(data)\n\n if data.mask.all():\n logger.debug(\"data empty, nothing to write\")\n else:\n # in case of S3 output, create an boto3 resource\n bucket_resource = get_boto3_bucket(self._bucket) if self._bucket else None\n\n # Convert from process_tile to output_tiles and write\n for tile in self.pyramid.intersecting(process_tile):\n out_path = self.get_path(tile)\n self.prepare_path(tile)\n out_tile = BufferedTile(tile, self.pixelbuffer)\n write_raster_window(\n in_tile=process_tile,\n in_data=data,\n out_profile=self.profile(out_tile),\n out_tile=out_tile,\n out_path=out_path,\n bucket_resource=bucket_resource\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef read(self, output_tile, **kwargs):\n try:\n return ma.masked_values(\n read_raster_no_crs(\n self.get_path(output_tile), indexes=(4 if self.old_band_num else 2)\n ),\n 0\n )\n except FileNotFoundError:\n return self.empty(output_tile)", "response": "Read existing process output."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef profile(self, tile=None):\n dst_metadata = dict(self._profile)\n if tile is not None:\n dst_metadata.update(\n width=tile.width,\n height=tile.height,\n affine=tile.affine, driver=\"PNG\",\n crs=tile.crs\n )\n return dst_metadata", "response": "Create a dictionary that contains the dictionary of metadata for rasterio."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nopens a Mapchete process.", "response": "def open(\n config, mode=\"continue\", zoom=None, bounds=None, single_input_file=None,\n with_cache=False, debug=False\n):\n \"\"\"\n Open a Mapchete process.\n\n Parameters\n ----------\n config : MapcheteConfig object, config dict or path to mapchete file\n Mapchete process configuration\n mode : string\n * ``memory``: Generate process output on demand without reading\n pre-existing data or writing new data.\n * ``readonly``: Just read data without processing new data.\n * ``continue``: (default) Don't overwrite existing output.\n * ``overwrite``: Overwrite existing output.\n zoom : list or integer\n process zoom level or a pair of minimum and maximum zoom level\n bounds : tuple\n left, bottom, right, top process boundaries in output pyramid\n single_input_file : string\n single input file if supported by process\n with_cache : bool\n process output data cached in memory\n\n Returns\n -------\n Mapchete\n a Mapchete process object\n \"\"\"\n return Mapchete(\n MapcheteConfig(\n config, mode=mode, zoom=zoom, bounds=bounds,\n single_input_file=single_input_file, debug=debug),\n with_cache=with_cache)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncount the number of tiles intersecting with geometry.", "response": "def count_tiles(geometry, pyramid, minzoom, maxzoom, init_zoom=0):\n \"\"\"\n Count number of tiles intersecting with geometry.\n\n Parameters\n ----------\n geometry : shapely geometry\n pyramid : TilePyramid\n minzoom : int\n maxzoom : int\n init_zoom : int\n\n Returns\n -------\n number of tiles\n \"\"\"\n if not 0 <= init_zoom <= minzoom <= maxzoom:\n raise ValueError(\"invalid zoom levels given\")\n # tile buffers are not being taken into account\n unbuffered_pyramid = TilePyramid(\n pyramid.grid, tile_size=pyramid.tile_size,\n metatiling=pyramid.metatiling\n )\n # make sure no rounding errors occur\n geometry = geometry.buffer(-0.000000001)\n return _count_tiles(\n [\n unbuffered_pyramid.tile(*tile_id)\n for tile_id in product(\n [init_zoom],\n range(pyramid.matrix_height(init_zoom)),\n range(pyramid.matrix_width(init_zoom))\n )\n ], geometry, minzoom, maxzoom\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_process_tiles(self, zoom=None):\n if zoom or zoom == 0:\n for tile in self.config.process_pyramid.tiles_from_geom(\n self.config.area_at_zoom(zoom), zoom\n ):\n yield tile\n else:\n for zoom in reversed(self.config.zoom_levels):\n for tile in self.config.process_pyramid.tiles_from_geom(\n self.config.area_at_zoom(zoom), zoom\n ):\n yield tile", "response": "Yield process tiles that intersect with the input data bounding boxes and are considered process tiles."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nprocess a large batch of tiles.", "response": "def batch_process(\n self, zoom=None, tile=None, multi=cpu_count(), max_chunksize=1\n ):\n \"\"\"\n Process a large batch of tiles.\n\n Parameters\n ----------\n process : MapcheteProcess\n process to be run\n zoom : list or int\n either single zoom level or list of minimum and maximum zoom level;\n None processes all (default: None)\n tile : tuple\n zoom, row and column of tile to be processed (cannot be used with\n zoom)\n multi : int\n number of workers (default: number of CPU cores)\n max_chunksize : int\n maximum number of process tiles to be queued for each worker;\n (default: 1)\n \"\"\"\n list(self.batch_processor(zoom, tile, multi, max_chunksize))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef batch_processor(\n self, zoom=None, tile=None, multi=cpu_count(), max_chunksize=1\n ):\n \"\"\"\n Process a large batch of tiles and yield report messages per tile.\n\n Parameters\n ----------\n zoom : list or int\n either single zoom level or list of minimum and maximum zoom level;\n None processes all (default: None)\n tile : tuple\n zoom, row and column of tile to be processed (cannot be used with\n zoom)\n multi : int\n number of workers (default: number of CPU cores)\n max_chunksize : int\n maximum number of process tiles to be queued for each worker;\n (default: 1)\n \"\"\"\n if zoom and tile:\n raise ValueError(\"use either zoom or tile\")\n\n # run single tile\n if tile:\n yield _run_on_single_tile(self, tile)\n # run concurrently\n elif multi > 1:\n for process_info in _run_with_multiprocessing(\n self, list(_get_zoom_level(zoom, self)), multi, max_chunksize\n ):\n yield process_info\n # run sequentially\n elif multi == 1:\n for process_info in _run_without_multiprocessing(\n self, list(_get_zoom_level(zoom, self))\n ):\n yield process_info", "response": "A large processor that processes a large set of tiles and yields report messages per tile."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef count_tiles(self, minzoom, maxzoom, init_zoom=0):\n if (minzoom, maxzoom) not in self._count_tiles_cache:\n self._count_tiles_cache[(minzoom, maxzoom)] = count_tiles(\n self.config.area_at_zoom(), self.config.process_pyramid,\n minzoom, maxzoom, init_zoom=0\n )\n return self._count_tiles_cache[(minzoom, maxzoom)]", "response": "Count the number of tiles intersecting with a pyramid."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef execute(self, process_tile, raise_nodata=False):\n if self.config.mode not in [\"memory\", \"continue\", \"overwrite\"]:\n raise ValueError(\"process mode must be memory, continue or overwrite\")\n if isinstance(process_tile, tuple):\n process_tile = self.config.process_pyramid.tile(*process_tile)\n elif isinstance(process_tile, BufferedTile):\n pass\n else:\n raise TypeError(\"process_tile must be tuple or BufferedTile\")\n\n if process_tile.zoom not in self.config.zoom_levels:\n return self.config.output.empty(process_tile)\n\n return self._execute(process_tile, raise_nodata=raise_nodata)", "response": "Execute the Mapchete process."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read(self, output_tile):\n if self.config.mode not in [\"readonly\", \"continue\", \"overwrite\"]:\n raise ValueError(\"process mode must be readonly, continue or overwrite\")\n if isinstance(output_tile, tuple):\n output_tile = self.config.output_pyramid.tile(*output_tile)\n elif isinstance(output_tile, BufferedTile):\n pass\n else:\n raise TypeError(\"output_tile must be tuple or BufferedTile\")\n\n return self.config.output.read(output_tile)", "response": "Reads from the output tile pyramid."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwrites data into output format.", "response": "def write(self, process_tile, data):\n \"\"\"\n Write data into output format.\n\n Parameters\n ----------\n process_tile : BufferedTile or tile index tuple\n process tile\n data : NumPy array or features\n data to be written\n \"\"\"\n if isinstance(process_tile, tuple):\n process_tile = self.config.process_pyramid.tile(*process_tile)\n elif not isinstance(process_tile, BufferedTile):\n raise ValueError(\"invalid process_tile type: %s\" % type(process_tile))\n if self.config.mode not in [\"continue\", \"overwrite\"]:\n raise ValueError(\"cannot write output in current process mode\")\n\n if self.config.mode == \"continue\" and (\n self.config.output.tiles_exist(process_tile)\n ):\n message = \"output exists, not overwritten\"\n logger.debug((process_tile.id, message))\n return ProcessInfo(\n tile=process_tile,\n processed=False,\n process_msg=None,\n written=False,\n write_msg=message\n )\n elif data is None:\n message = \"output empty, nothing written\"\n logger.debug((process_tile.id, message))\n return ProcessInfo(\n tile=process_tile,\n processed=False,\n process_msg=None,\n written=False,\n write_msg=message\n )\n else:\n with Timer() as t:\n self.config.output.write(process_tile=process_tile, data=data)\n message = \"output written in %s\" % t\n logger.debug((process_tile.id, message))\n return ProcessInfo(\n tile=process_tile,\n processed=False,\n process_msg=None,\n written=True,\n write_msg=message\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_raw_output(self, tile, _baselevel_readonly=False):\n if not isinstance(tile, (BufferedTile, tuple)):\n raise TypeError(\"'tile' must be a tuple or BufferedTile\")\n if isinstance(tile, tuple):\n tile = self.config.output_pyramid.tile(*tile)\n if _baselevel_readonly:\n tile = self.config.baselevels[\"tile_pyramid\"].tile(*tile.id)\n\n # Return empty data if zoom level is outside of process zoom levels.\n if tile.zoom not in self.config.zoom_levels:\n return self.config.output.empty(tile)\n\n # TODO implement reprojection\n if tile.crs != self.config.process_pyramid.crs:\n raise NotImplementedError(\n \"reprojection between processes not yet implemented\"\n )\n\n if self.config.mode == \"memory\":\n # Determine affected process Tile and check whether it is already\n # cached.\n process_tile = self.config.process_pyramid.intersecting(tile)[0]\n return self._extract(\n in_tile=process_tile,\n in_data=self._execute_using_cache(process_tile),\n out_tile=tile\n )\n\n # TODO: cases where tile intersects with multiple process tiles\n process_tile = self.config.process_pyramid.intersecting(tile)[0]\n\n # get output_tiles that intersect with current tile\n if tile.pixelbuffer > self.config.output.pixelbuffer:\n output_tiles = list(self.config.output_pyramid.tiles_from_bounds(\n tile.bounds, tile.zoom\n ))\n else:\n output_tiles = self.config.output_pyramid.intersecting(tile)\n\n if self.config.mode == \"readonly\" or _baselevel_readonly:\n if self.config.output.tiles_exist(process_tile):\n return self._read_existing_output(tile, output_tiles)\n else:\n return self.config.output.empty(tile)\n elif self.config.mode == \"continue\" and not _baselevel_readonly:\n if self.config.output.tiles_exist(process_tile):\n return self._read_existing_output(tile, output_tiles)\n else:\n return self._process_and_overwrite_output(tile, process_tile)\n elif self.config.mode == \"overwrite\" and not _baselevel_readonly:\n return self._process_and_overwrite_output(tile, process_tile)", "response": "Get raw output data for a given tile."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nextracts data from tile.", "response": "def _extract(self, in_tile=None, in_data=None, out_tile=None):\n \"\"\"Extract data from tile.\"\"\"\n return self.config.output.extract_subset(\n input_data_tiles=[(in_tile, in_data)],\n out_tile=out_tile\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read(self, **kwargs):\n if self.tile.pixelbuffer > self.config.output.pixelbuffer:\n output_tiles = list(self.config.output_pyramid.tiles_from_bounds(\n self.tile.bounds, self.tile.zoom\n ))\n else:\n output_tiles = self.config.output_pyramid.intersecting(self.tile)\n return self.config.output.extract_subset(\n input_data_tiles=[\n (output_tile, self.config.output.read(output_tile))\n for output_tile in output_tiles\n ],\n out_tile=self.tile,\n )", "response": "Read existing output data from a previous run."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef open(self, input_id, **kwargs):\n if not isinstance(input_id, str):\n return input_id.open(self.tile, **kwargs)\n if input_id not in self.params[\"input\"]:\n raise ValueError(\"%s not found in config as input file\" % input_id)\n return self.params[\"input\"][input_id].open(self.tile, **kwargs)", "response": "Open input data.\n\n Parameters\n ----------\n input_id : string\n input identifier from configuration file or file path\n kwargs : driver specific parameters (e.g. resampling)\n\n Returns\n -------\n tiled input data : InputTile\n reprojected input data within tile"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate hillshading from elevation data.", "response": "def hillshade(\n self, elevation, azimuth=315.0, altitude=45.0, z=1.0, scale=1.0\n ):\n \"\"\"\n Calculate hillshading from elevation data.\n\n Parameters\n ----------\n elevation : array\n input elevation data\n azimuth : float\n horizontal angle of light source (315: North-West)\n altitude : float\n vertical angle of light source (90 would result in slope shading)\n z : float\n vertical exaggeration factor\n scale : float\n scale factor of pixel size units versus height units (insert 112000\n when having elevation values in meters in a geodetic projection)\n\n Returns\n -------\n hillshade : array\n \"\"\"\n return commons_hillshade.hillshade(\n elevation, self, azimuth, altitude, z, scale)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a list of contour lines from the elevation data.", "response": "def contours(\n self, elevation, interval=100, field='elev', base=0\n ):\n \"\"\"\n Extract contour lines from elevation data.\n\n Parameters\n ----------\n elevation : array\n input elevation data\n interval : integer\n elevation value interval when drawing contour lines\n field : string\n output field name containing elevation value\n base : integer\n elevation base value the intervals are computed from\n\n Returns\n -------\n contours : iterable\n contours as GeoJSON-like pairs of properties and geometry\n \"\"\"\n return commons_contours.extract_contours(\n elevation, self.tile, interval=interval, field=field, base=base)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nclipping array by geometry.", "response": "def clip(\n self, array, geometries, inverted=False, clip_buffer=0\n ):\n \"\"\"\n Clip array by geometry.\n\n Parameters\n ----------\n array : array\n raster data to be clipped\n geometries : iterable\n geometries used to clip source array\n inverted : bool\n invert clipping (default: False)\n clip_buffer : int\n buffer (in pixels) geometries before applying clip\n\n Returns\n -------\n clipped array : array\n \"\"\"\n return commons_clip.clip_array_with_vector(\n array, self.tile.affine, geometries,\n inverted=inverted, clip_buffer=clip_buffer*self.tile.pixel_x_size)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef clip_array_with_vector(\n array, array_affine, geometries, inverted=False, clip_buffer=0\n):\n \"\"\"\n Clip input array with a vector list.\n\n Parameters\n ----------\n array : array\n input raster data\n array_affine : Affine\n Affine object describing the raster's geolocation\n geometries : iterable\n iterable of dictionaries, where every entry has a 'geometry' and\n 'properties' key.\n inverted : bool\n invert clip (default: False)\n clip_buffer : integer\n buffer (in pixels) geometries before clipping\n\n Returns\n -------\n clipped array : array\n \"\"\"\n # buffer input geometries and clean up\n buffered_geometries = []\n for feature in geometries:\n feature_geom = to_shape(feature[\"geometry\"])\n if feature_geom.is_empty:\n continue\n if feature_geom.geom_type == \"GeometryCollection\":\n # for GeometryCollections apply buffer to every subgeometry\n # and make union\n buffered_geom = unary_union([\n g.buffer(clip_buffer) for g in feature_geom])\n else:\n buffered_geom = feature_geom.buffer(clip_buffer)\n if not buffered_geom.is_empty:\n buffered_geometries.append(buffered_geom)\n\n # mask raster by buffered geometries\n if buffered_geometries:\n if array.ndim == 2:\n return ma.masked_array(\n array, geometry_mask(\n buffered_geometries, array.shape, array_affine,\n invert=inverted))\n elif array.ndim == 3:\n mask = geometry_mask(\n buffered_geometries, (array.shape[1], array.shape[2]),\n array_affine, invert=inverted)\n return ma.masked_array(\n array, mask=np.stack((mask for band in array)))\n\n # if no geometries, return unmasked array\n else:\n fill = False if inverted else True\n return ma.masked_array(\n array, mask=np.full(array.shape, fill, dtype=bool))", "response": "Clip input array with a vector list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate tile pyramid out of input raster.", "response": "def pyramid(\n input_raster,\n output_dir,\n pyramid_type=None,\n output_format=None,\n resampling_method=None,\n scale_method=None,\n zoom=None,\n bounds=None,\n overwrite=False,\n debug=False\n):\n \"\"\"Create tile pyramid out of input raster.\"\"\"\n bounds = bounds if bounds else None\n options = dict(\n pyramid_type=pyramid_type,\n scale_method=scale_method,\n output_format=output_format,\n resampling=resampling_method,\n zoom=zoom,\n bounds=bounds,\n overwrite=overwrite\n )\n raster2pyramid(input_raster, output_dir, options)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a tile pyramid from an input raster dataset.", "response": "def raster2pyramid(input_file, output_dir, options):\n \"\"\"Create a tile pyramid out of an input raster dataset.\"\"\"\n pyramid_type = options[\"pyramid_type\"]\n scale_method = options[\"scale_method\"]\n output_format = options[\"output_format\"]\n resampling = options[\"resampling\"]\n zoom = options[\"zoom\"]\n bounds = options[\"bounds\"]\n mode = \"overwrite\" if options[\"overwrite\"] else \"continue\"\n\n # Prepare process parameters\n minzoom, maxzoom = _get_zoom(zoom, input_file, pyramid_type)\n with rasterio.open(input_file, \"r\") as input_raster:\n output_bands = input_raster.count\n input_dtype = input_raster.dtypes[0]\n output_dtype = input_raster.dtypes[0]\n nodataval = input_raster.nodatavals[0]\n nodataval = nodataval if nodataval else 0\n if output_format == \"PNG\" and output_bands > 3:\n output_bands = 3\n output_dtype = 'uint8'\n scales_minmax = ()\n if scale_method == \"dtype_scale\":\n for index in range(1, output_bands+1):\n scales_minmax += (DTYPE_RANGES[input_dtype], )\n elif scale_method == \"minmax_scale\":\n for index in range(1, output_bands+1):\n band = input_raster.read(index)\n scales_minmax += ((band.min(), band.max()), )\n elif scale_method == \"crop\":\n for index in range(1, output_bands+1):\n scales_minmax += ((0, 255), )\n if input_dtype == \"uint8\":\n scale_method = None\n scales_minmax = ()\n for index in range(1, output_bands+1):\n scales_minmax += ((None, None), )\n\n # Create configuration\n config = dict(\n process=\"mapchete.processes.pyramid.tilify\",\n output={\n \"path\": output_dir,\n \"format\": output_format,\n \"bands\": output_bands,\n \"dtype\": output_dtype\n },\n pyramid=dict(pixelbuffer=5, grid=pyramid_type),\n scale_method=scale_method,\n scales_minmax=scales_minmax,\n input={\"raster\": input_file},\n config_dir=os.getcwd(),\n zoom_levels=dict(min=minzoom, max=maxzoom),\n nodataval=nodataval,\n resampling=resampling,\n bounds=bounds,\n baselevel={\"zoom\": maxzoom, \"resampling\": resampling},\n mode=mode\n )\n\n # create process\n with mapchete.open(config, zoom=zoom, bounds=bounds) as mp:\n # prepare output directory\n if not os.path.exists(output_dir):\n os.makedirs(output_dir)\n # run process\n mp.batch_process(zoom=[minzoom, maxzoom])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndetermining minimum and maximum zoomlevel.", "response": "def _get_zoom(zoom, input_raster, pyramid_type):\n \"\"\"Determine minimum and maximum zoomlevel.\"\"\"\n if not zoom:\n minzoom = 1\n maxzoom = get_best_zoom_level(input_raster, pyramid_type)\n elif len(zoom) == 1:\n minzoom = zoom[0]\n maxzoom = zoom[0]\n elif len(zoom) == 2:\n if zoom[0] < zoom[1]:\n minzoom = zoom[0]\n maxzoom = zoom[1]\n else:\n minzoom = zoom[1]\n maxzoom = zoom[0]\n return minzoom, maxzoom"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef validate_values(config, values):\n if not isinstance(config, dict):\n raise TypeError(\"config must be a dictionary\")\n for value, vtype in values:\n if value not in config:\n raise ValueError(\"%s not given\" % value)\n if not isinstance(config[value], vtype):\n raise TypeError(\"%s must be %s\" % (value, vtype))\n return True", "response": "Validate whether values are found in config and has the right type."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns hash of x.", "response": "def get_hash(x):\n \"\"\"Return hash of x.\"\"\"\n if isinstance(x, str):\n return hash(x)\n elif isinstance(x, dict):\n return hash(yaml.dump(x))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_zoom_levels(process_zoom_levels=None, init_zoom_levels=None):\n process_zoom_levels = _validate_zooms(process_zoom_levels)\n if init_zoom_levels is None:\n return process_zoom_levels\n else:\n init_zoom_levels = _validate_zooms(init_zoom_levels)\n if not set(init_zoom_levels).issubset(set(process_zoom_levels)):\n raise MapcheteConfigError(\n \"init zooms must be a subset of process zoom\")\n return init_zoom_levels", "response": "Validate and return zoom levels."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef snap_bounds(bounds=None, pyramid=None, zoom=None):\n if not isinstance(bounds, (tuple, list)):\n raise TypeError(\"bounds must be either a tuple or a list\")\n if len(bounds) != 4:\n raise ValueError(\"bounds has to have exactly four values\")\n if not isinstance(pyramid, BufferedTilePyramid):\n raise TypeError(\"pyramid has to be a BufferedTilePyramid\")\n\n bounds = Bounds(*bounds)\n lb = pyramid.tile_from_xy(bounds.left, bounds.bottom, zoom, on_edge_use=\"rt\").bounds\n rt = pyramid.tile_from_xy(bounds.right, bounds.top, zoom, on_edge_use=\"lb\").bounds\n return Bounds(lb.left, lb.bottom, rt.right, rt.top)", "response": "Snaps bounds to tiles boundaries of specific zoom level."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nclips bounds by clip.", "response": "def clip_bounds(bounds=None, clip=None):\n \"\"\"\n Clips bounds by clip.\n\n Parameters\n ----------\n bounds : bounds to be clipped\n clip : clip bounds\n\n Returns\n -------\n Bounds(left, bottom, right, top)\n \"\"\"\n bounds = Bounds(*bounds)\n clip = Bounds(*clip)\n return Bounds(\n max(bounds.left, clip.left),\n max(bounds.bottom, clip.bottom),\n min(bounds.right, clip.right),\n min(bounds.top, clip.top)\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload the process pyramid of a raw mapchete configuration as dictionary.", "response": "def raw_conf_process_pyramid(raw_conf):\n \"\"\"\n Loads the process pyramid of a raw configuration.\n\n Parameters\n ----------\n raw_conf : dict\n Raw mapchete configuration as dictionary.\n\n Returns\n -------\n BufferedTilePyramid\n \"\"\"\n return BufferedTilePyramid(\n raw_conf[\"pyramid\"][\"grid\"],\n metatiling=raw_conf[\"pyramid\"].get(\"metatiling\", 1),\n pixelbuffer=raw_conf[\"pyramid\"].get(\"pixelbuffer\", 0)\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads the process pyramid of a raw mapchete configuration.", "response": "def bounds_from_opts(\n wkt_geometry=None, point=None, bounds=None, zoom=None, raw_conf=None\n):\n \"\"\"\n Loads the process pyramid of a raw configuration.\n\n Parameters\n ----------\n raw_conf : dict\n Raw mapchete configuration as dictionary.\n\n Returns\n -------\n BufferedTilePyramid\n \"\"\"\n if wkt_geometry:\n return wkt.loads(wkt_geometry).bounds\n elif point:\n x, y = point\n zoom_levels = get_zoom_levels(\n process_zoom_levels=raw_conf[\"zoom_levels\"],\n init_zoom_levels=zoom\n )\n tp = raw_conf_process_pyramid(raw_conf)\n return tp.tile_from_xy(x, y, max(zoom_levels)).bounds\n else:\n return bounds"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nvalidates the zoom levels.", "response": "def _validate_zooms(zooms):\n \"\"\"\n Return a list of zoom levels.\n\n Following inputs are converted:\n - int --> [int]\n - dict{min, max} --> range(min, max + 1)\n - [int] --> [int]\n - [int, int] --> range(smaller int, bigger int + 1)\n \"\"\"\n if isinstance(zooms, dict):\n if any([a not in zooms for a in [\"min\", \"max\"]]):\n raise MapcheteConfigError(\"min and max zoom required\")\n zmin = _validate_zoom(zooms[\"min\"])\n zmax = _validate_zoom(zooms[\"max\"])\n if zmin > zmax:\n raise MapcheteConfigError(\n \"max zoom must not be smaller than min zoom\")\n return list(range(zmin, zmax + 1))\n elif isinstance(zooms, list):\n if len(zooms) == 1:\n return zooms\n elif len(zooms) == 2:\n zmin, zmax = sorted([_validate_zoom(z) for z in zooms])\n return list(range(zmin, zmax + 1))\n else:\n return zooms\n else:\n return [_validate_zoom(zooms)]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _raw_at_zoom(config, zooms):\n params_per_zoom = {}\n for zoom in zooms:\n params = {}\n for name, element in config.items():\n if name not in _RESERVED_PARAMETERS:\n out_element = _element_at_zoom(name, element, zoom)\n if out_element is not None:\n params[name] = out_element\n params_per_zoom[zoom] = params\n return params_per_zoom", "response": "Return parameter dictionary per zoom level."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the element at a given zoom level.", "response": "def _element_at_zoom(name, element, zoom):\n \"\"\"\n Return the element filtered by zoom level.\n\n - An input integer or float gets returned as is.\n - An input string is checked whether it starts with \"zoom\". Then, the\n provided zoom level gets parsed and compared with the actual zoom\n level. If zoom levels match, the element gets returned.\n TODOs/gotchas:\n - Elements are unordered, which can lead to unexpected results when\n defining the YAML config.\n - Provided zoom levels for one element in config file are not allowed\n to \"overlap\", i.e. there is not yet a decision mechanism implemented\n which handles this case.\n \"\"\"\n # If element is a dictionary, analyze subitems.\n if isinstance(element, dict):\n if \"format\" in element:\n # we have an input or output driver here\n return element\n out_elements = {}\n for sub_name, sub_element in element.items():\n out_element = _element_at_zoom(sub_name, sub_element, zoom)\n if name == \"input\":\n out_elements[sub_name] = out_element\n elif out_element is not None:\n out_elements[sub_name] = out_element\n # If there is only one subelement, collapse unless it is\n # input. In such case, return a dictionary.\n if len(out_elements) == 1 and name != \"input\":\n return next(iter(out_elements.values()))\n # If subelement is empty, return None\n if len(out_elements) == 0:\n return None\n return out_elements\n # If element is a zoom level statement, filter element.\n elif isinstance(name, str):\n if name.startswith(\"zoom\"):\n return _filter_by_zoom(\n conf_string=name.strip(\"zoom\").strip(), zoom=zoom,\n element=element)\n # If element is a string but not a zoom level statement, return\n # element.\n else:\n return element\n # Return all other types as they are.\n else:\n return element"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _filter_by_zoom(element=None, conf_string=None, zoom=None):\n for op_str, op_func in [\n # order of operators is important:\n # prematurely return in cases of \"<=\" or \">=\", otherwise\n # _strip_zoom() cannot parse config strings starting with \"<\"\n # or \">\"\n (\"=\", operator.eq),\n (\"<=\", operator.le),\n (\">=\", operator.ge),\n (\"<\", operator.lt),\n (\">\", operator.gt),\n ]:\n if conf_string.startswith(op_str):\n return element if op_func(zoom, _strip_zoom(conf_string, op_str)) else None", "response": "Return element only if zoom condition matches with config string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning zoom level as integer or throw error.", "response": "def _strip_zoom(input_string, strip_string):\n \"\"\"Return zoom level as integer or throw error.\"\"\"\n try:\n return int(input_string.strip(strip_string))\n except Exception as e:\n raise MapcheteConfigError(\"zoom level could not be determined: %s\" % e)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _flatten_tree(tree, old_path=None):\n flat_tree = []\n for key, value in tree.items():\n new_path = \"/\".join([old_path, key]) if old_path else key\n if isinstance(value, dict) and \"format\" not in value:\n flat_tree.extend(_flatten_tree(value, old_path=new_path))\n else:\n flat_tree.append((new_path, value))\n return flat_tree", "response": "Flatten dict tree into dictionary where keys are paths of old dict."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef bounds(self):\n if self._raw[\"bounds\"] is None:\n return self.process_pyramid.bounds\n else:\n return Bounds(*_validate_bounds(self._raw[\"bounds\"]))", "response": "Process bounds as defined in the configuration."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nprocessing bounds this process is currently initialized with. This gets triggered by using the ``init_bounds`` kwarg. If not set, it will be equal to self.bounds.", "response": "def init_bounds(self):\n \"\"\"\n Process bounds this process is currently initialized with.\n\n This gets triggered by using the ``init_bounds`` kwarg. If not set, it will\n be equal to self.bounds.\n \"\"\"\n if self._raw[\"init_bounds\"] is None:\n return self.bounds\n else:\n return Bounds(*_validate_bounds(self._raw[\"init_bounds\"]))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef effective_bounds(self):\n return snap_bounds(\n bounds=clip_bounds(bounds=self.init_bounds, clip=self.process_pyramid.bounds),\n pyramid=self.process_pyramid,\n zoom=min(\n self.baselevels[\"zooms\"]\n ) if self.baselevels else min(\n self.init_zoom_levels\n )\n )", "response": "Return effective process bounds for process."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn object of driver.", "response": "def output(self):\n \"\"\"Output object of driver.\"\"\"\n output_params = dict(\n self._raw[\"output\"],\n grid=self.output_pyramid.grid,\n pixelbuffer=self.output_pyramid.pixelbuffer,\n metatiling=self.output_pyramid.metatiling\n )\n if \"path\" in output_params:\n output_params.update(\n path=absolute_path(path=output_params[\"path\"], base_dir=self.config_dir)\n )\n\n if \"format\" not in output_params:\n raise MapcheteConfigError(\"output format not specified\")\n\n if output_params[\"format\"] not in available_output_formats():\n raise MapcheteConfigError(\n \"format %s not available in %s\" % (\n output_params[\"format\"], str(available_output_formats())\n )\n )\n writer = load_output_writer(output_params)\n try:\n writer.is_valid_with_config(output_params)\n except Exception as e:\n logger.exception(e)\n raise MapcheteConfigError(\n \"driver %s not compatible with configuration: %s\" % (\n writer.METADATA[\"driver_name\"], e\n )\n )\n return writer"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef input(self):\n # the delimiters are used by some input drivers\n delimiters = dict(\n zoom=self.init_zoom_levels,\n bounds=self.init_bounds,\n process_bounds=self.bounds,\n effective_bounds=self.effective_bounds\n )\n\n # get input items only of initialized zoom levels\n raw_inputs = {\n # convert input definition to hash\n get_hash(v): v\n for zoom in self.init_zoom_levels\n if \"input\" in self._params_at_zoom[zoom]\n # to preserve file groups, \"flatten\" the input tree and use\n # the tree paths as keys\n for key, v in _flatten_tree(self._params_at_zoom[zoom][\"input\"])\n if v is not None\n }\n\n initalized_inputs = {}\n for k, v in raw_inputs.items():\n\n # for files and tile directories\n if isinstance(v, str):\n logger.debug(\"load input reader for simple input %s\", v)\n try:\n reader = load_input_reader(\n dict(\n path=absolute_path(path=v, base_dir=self.config_dir),\n pyramid=self.process_pyramid,\n pixelbuffer=self.process_pyramid.pixelbuffer,\n delimiters=delimiters\n ),\n readonly=self.mode == \"readonly\")\n except Exception as e:\n logger.exception(e)\n raise MapcheteDriverError(\"error when loading input %s: %s\" % (v, e))\n logger.debug(\"input reader for simple input %s is %s\", v, reader)\n\n # for abstract inputs\n elif isinstance(v, dict):\n logger.debug(\"load input reader for abstract input %s\", v)\n try:\n reader = load_input_reader(\n dict(\n abstract=deepcopy(v),\n pyramid=self.process_pyramid,\n pixelbuffer=self.process_pyramid.pixelbuffer,\n delimiters=delimiters,\n conf_dir=self.config_dir\n ),\n readonly=self.mode == \"readonly\")\n except Exception as e:\n logger.exception(e)\n raise MapcheteDriverError(\"error when loading input %s: %s\" % (v, e))\n logger.debug(\"input reader for abstract input %s is %s\", v, reader)\n else:\n raise MapcheteConfigError(\"invalid input type %s\", type(v))\n # trigger bbox creation\n reader.bbox(out_crs=self.process_pyramid.crs)\n initalized_inputs[k] = reader\n\n return initalized_inputs", "response": "Load all input items from the input tree and store them in a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a dictionary of baselevels configuration parameters.", "response": "def baselevels(self):\n \"\"\"\n Optional baselevels configuration.\n\n baselevels:\n min: \n max: \n lower: \n higher: \n \"\"\"\n if \"baselevels\" not in self._raw:\n return {}\n baselevels = self._raw[\"baselevels\"]\n minmax = {k: v for k, v in baselevels.items() if k in [\"min\", \"max\"]}\n\n if not minmax:\n raise MapcheteConfigError(\"no min and max values given for baselevels\")\n for v in minmax.values():\n if not isinstance(v, int) or v < 0:\n raise MapcheteConfigError(\n \"invalid baselevel zoom parameter given: %s\" % minmax.values()\n )\n\n zooms = list(range(\n minmax.get(\"min\", min(self.zoom_levels)),\n minmax.get(\"max\", max(self.zoom_levels)) + 1)\n )\n\n if not set(self.zoom_levels).difference(set(zooms)):\n raise MapcheteConfigError(\"baselevels zooms fully cover process zooms\")\n\n return dict(\n zooms=zooms,\n lower=baselevels.get(\"lower\", \"nearest\"),\n higher=baselevels.get(\"higher\", \"nearest\"),\n tile_pyramid=BufferedTilePyramid(\n self.output_pyramid.grid,\n pixelbuffer=self.output_pyramid.pixelbuffer,\n metatiling=self.process_pyramid.metatiling\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning configuration parameters snapshot for zoom level dependent process configuration.", "response": "def params_at_zoom(self, zoom):\n \"\"\"\n Return configuration parameters snapshot for zoom as dictionary.\n\n Parameters\n ----------\n zoom : int\n zoom level\n\n Returns\n -------\n configuration snapshot : dictionary\n zoom level dependent process configuration\n \"\"\"\n if zoom not in self.init_zoom_levels:\n raise ValueError(\n \"zoom level not available with current configuration\")\n out = dict(self._params_at_zoom[zoom], input={}, output=self.output)\n if \"input\" in self._params_at_zoom[zoom]:\n flat_inputs = {}\n for k, v in _flatten_tree(self._params_at_zoom[zoom][\"input\"]):\n if v is None:\n flat_inputs[k] = None\n else:\n flat_inputs[k] = self.input[get_hash(v)]\n out[\"input\"] = _unflatten_tree(flat_inputs)\n else:\n out[\"input\"] = {}\n return out"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the area of the process at a given zoom level.", "response": "def area_at_zoom(self, zoom=None):\n \"\"\"\n Return process bounding box for zoom level.\n\n Parameters\n ----------\n zoom : int or None\n if None, the union of all zoom level areas is returned\n\n Returns\n -------\n process area : shapely geometry\n \"\"\"\n if zoom is None:\n if not self._cache_full_process_area:\n logger.debug(\"calculate process area ...\")\n self._cache_full_process_area = cascaded_union([\n self._area_at_zoom(z) for z in self.init_zoom_levels]\n ).buffer(0)\n return self._cache_full_process_area\n else:\n if zoom not in self.init_zoom_levels:\n raise ValueError(\n \"zoom level not available with current configuration\")\n return self._area_at_zoom(zoom)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef bounds_at_zoom(self, zoom=None):\n return () if self.area_at_zoom(zoom).is_empty else Bounds(\n *self.area_at_zoom(zoom).bounds)", "response": "Return process bounds at zoom level."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef process_file(self):\n warnings.warn(DeprecationWarning(\"'self.process_file' is deprecated\"))\n return os.path.join(self._raw[\"config_dir\"], self._raw[\"process\"])", "response": "Return the path to the process file."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ngenerate index files for given zoom level.", "response": "def zoom_index_gen(\n mp=None,\n out_dir=None,\n zoom=None,\n geojson=False,\n gpkg=False,\n shapefile=False,\n txt=False,\n vrt=False,\n fieldname=\"location\",\n basepath=None,\n for_gdal=True,\n threading=False,\n):\n \"\"\"\n Generate indexes for given zoom level.\n\n Parameters\n ----------\n mp : Mapchete object\n process output to be indexed\n out_dir : path\n optionally override process output directory\n zoom : int\n zoom level to be processed\n geojson : bool\n generate GeoJSON index (default: False)\n gpkg : bool\n generate GeoPackage index (default: False)\n shapefile : bool\n generate Shapefile index (default: False)\n txt : bool\n generate tile path list textfile (default: False)\n vrt : bool\n GDAL-style VRT file (default: False)\n fieldname : str\n field name which contains paths of tiles (default: \"location\")\n basepath : str\n if set, use custom base path instead of output path\n for_gdal : bool\n use GDAL compatible remote paths, i.e. add \"/vsicurl/\" before path\n (default: True)\n \"\"\"\n for zoom in get_zoom_levels(process_zoom_levels=zoom):\n with ExitStack() as es:\n # get index writers for all enabled formats\n index_writers = []\n if geojson:\n index_writers.append(\n es.enter_context(\n VectorFileWriter(\n driver=\"GeoJSON\",\n out_path=_index_file_path(out_dir, zoom, \"geojson\"),\n crs=mp.config.output_pyramid.crs,\n fieldname=fieldname\n )\n )\n )\n if gpkg:\n index_writers.append(\n es.enter_context(\n VectorFileWriter(\n driver=\"GPKG\",\n out_path=_index_file_path(out_dir, zoom, \"gpkg\"),\n crs=mp.config.output_pyramid.crs,\n fieldname=fieldname\n )\n )\n )\n if shapefile:\n index_writers.append(\n es.enter_context(\n VectorFileWriter(\n driver=\"ESRI Shapefile\",\n out_path=_index_file_path(out_dir, zoom, \"shp\"),\n crs=mp.config.output_pyramid.crs,\n fieldname=fieldname\n )\n )\n )\n if txt:\n index_writers.append(\n es.enter_context(\n TextFileWriter(out_path=_index_file_path(out_dir, zoom, \"txt\"))\n )\n )\n if vrt:\n index_writers.append(\n es.enter_context(\n VRTFileWriter(\n out_path=_index_file_path(out_dir, zoom, \"vrt\"),\n output=mp.config.output,\n out_pyramid=mp.config.output_pyramid\n )\n )\n )\n\n logger.debug(\"use the following index writers: %s\", index_writers)\n\n def _worker(tile):\n # if there are indexes to write to, check if output exists\n tile_path = _tile_path(\n orig_path=mp.config.output.get_path(tile),\n basepath=basepath,\n for_gdal=for_gdal\n )\n indexes = [\n i for i in index_writers\n if not i.entry_exists(tile=tile, path=tile_path)\n ]\n if indexes:\n output_exists = mp.config.output.tiles_exist(output_tile=tile)\n else:\n output_exists = None\n return tile, tile_path, indexes, output_exists\n\n with concurrent.futures.ThreadPoolExecutor() as executor:\n for task in concurrent.futures.as_completed(\n (\n executor.submit(_worker, i)\n for i in mp.config.output_pyramid.tiles_from_geom(\n mp.config.area_at_zoom(zoom), zoom\n )\n )\n ):\n tile, tile_path, indexes, output_exists = task.result()\n # only write entries if there are indexes to write to and output\n # exists\n if indexes and output_exists:\n logger.debug(\"%s exists\", tile_path)\n logger.debug(\"write to %s indexes\" % len(indexes))\n for index in indexes:\n index.write(tile, tile_path)\n # yield tile for progress information\n yield tile"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_segmentize_value(input_file=None, tile_pyramid=None):\n warnings.warn(DeprecationWarning(\"get_segmentize_value() has moved to mapchete.io\"))\n return io.get_segmentize_value(input_file, tile_pyramid)", "response": "Returns the recommended segmentation value in input file units."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the bounding box of the data in the specified CRS.", "response": "def bbox(self, out_crs=None):\n \"\"\"\n Return data bounding box.\n\n Parameters\n ----------\n out_crs : ``rasterio.crs.CRS``\n rasterio CRS object (default: CRS of process pyramid)\n\n Returns\n -------\n bounding box : geometry\n Shapely geometry object\n \"\"\"\n out_crs = self.pyramid.crs if out_crs is None else out_crs\n with rasterio.open(self.path) as inp:\n inp_crs = inp.crs\n out_bbox = bbox = box(*inp.bounds)\n # If soucre and target CRSes differ, segmentize and reproject\n if inp_crs != out_crs:\n # estimate segmentize value (raster pixel size * tile size)\n # and get reprojected bounding box\n return reproject_geometry(\n segmentize_geometry(\n bbox, inp.transform[0] * self.pyramid.tile_size\n ),\n src_crs=inp_crs, dst_crs=out_crs\n )\n else:\n return out_bbox"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef read(self, indexes=None, **kwargs):\n return read_raster_window(\n self.raster_file.path,\n self.tile,\n indexes=self._get_band_indexes(indexes),\n resampling=self.resampling,\n gdal_opts=self.gdal_opts\n )", "response": "Read reprojected & resampled input data."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck if the data within this tile is empty.", "response": "def is_empty(self, indexes=None):\n \"\"\"\n Check if there is data within this tile.\n\n Returns\n -------\n is empty : bool\n \"\"\"\n # empty if tile does not intersect with file bounding box\n return not self.tile.bbox.intersects(\n self.raster_file.bbox(out_crs=self.tile.crs)\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_band_indexes(self, indexes=None):\n if indexes:\n if isinstance(indexes, list):\n return indexes\n else:\n return [indexes]\n else:\n return range(1, self.raster_file.profile[\"count\"] + 1)", "response": "Return valid band indexes."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read(self, output_tile, **kwargs):\n path = self.get_path(output_tile)\n try:\n with fiona.open(path, \"r\") as src:\n return list(src)\n except DriverError as e:\n for i in (\"does not exist in the file system\", \"No such file or directory\"):\n if i in str(e):\n return self.empty(output_tile)\n else:\n raise", "response": "Read existing process output."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites data from process tiles into GeoJSON file.", "response": "def write(self, process_tile, data):\n \"\"\"\n Write data from process tiles into GeoJSON file(s).\n\n Parameters\n ----------\n process_tile : ``BufferedTile``\n must be member of process ``TilePyramid``\n \"\"\"\n if data is None or len(data) == 0:\n return\n if not isinstance(data, (list, types.GeneratorType)):\n raise TypeError(\n \"GeoJSON driver data has to be a list or generator of GeoJSON objects\"\n )\n\n data = list(data)\n\n if not len(data):\n logger.debug(\"no features to write\")\n else:\n # in case of S3 output, create an boto3 resource\n bucket_resource = get_boto3_bucket(self._bucket) if self._bucket else None\n\n # Convert from process_tile to output_tiles\n for tile in self.pyramid.intersecting(process_tile):\n out_path = self.get_path(tile)\n self.prepare_path(tile)\n out_tile = BufferedTile(tile, self.pixelbuffer)\n write_vector_window(\n in_data=data,\n out_schema=self.output_params[\"schema\"],\n out_tile=out_tile,\n out_path=out_path,\n bucket_resource=bucket_resource\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncheck if output format is valid with other process parameters.", "response": "def is_valid_with_config(self, config):\n \"\"\"\n Check if output format is valid with other process parameters.\n\n Parameters\n ----------\n config : dictionary\n output configuration parameters\n\n Returns\n -------\n is_valid : bool\n \"\"\"\n validate_values(config, [(\"schema\", dict), (\"path\", str)])\n validate_values(config[\"schema\"], [(\"properties\", dict), (\"geometry\", str)])\n if config[\"schema\"][\"geometry\"] not in [\n \"Geometry\", \"Point\", \"MultiPoint\", \"Line\", \"MultiLine\",\n \"Polygon\", \"MultiPolygon\"\n ]:\n raise TypeError(\"invalid geometry type\")\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef read(self, validity_check=True, no_neighbors=False, **kwargs):\n if no_neighbors:\n raise NotImplementedError()\n return self._from_cache(validity_check=validity_check)", "response": "Read data from process output."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nexecuting a Mapchete process.", "response": "def execute(\n mapchete_files,\n zoom=None,\n bounds=None,\n point=None,\n wkt_geometry=None,\n tile=None,\n overwrite=False,\n multi=None,\n input_file=None,\n logfile=None,\n verbose=False,\n no_pbar=False,\n debug=False,\n max_chunksize=None,\n vrt=False,\n idx_out_dir=None\n):\n \"\"\"Execute a Mapchete process.\"\"\"\n multi = multi if multi else cpu_count()\n mode = \"overwrite\" if overwrite else \"continue\"\n # send verbose messages to /dev/null if not activated\n if debug or not verbose:\n verbose_dst = open(os.devnull, 'w')\n else:\n verbose_dst = sys.stdout\n\n for mapchete_file in mapchete_files:\n\n tqdm.tqdm.write(\"preparing to process %s\" % mapchete_file, file=verbose_dst)\n\n with click_spinner.spinner(disable=debug) as spinner:\n\n # process single tile\n if tile:\n tile = raw_conf_process_pyramid(raw_conf(mapchete_file)).tile(*tile)\n\n with mapchete.open(\n mapchete_file,\n mode=mode,\n bounds=tile.bounds,\n zoom=tile.zoom,\n single_input_file=input_file\n ) as mp:\n spinner.stop()\n tqdm.tqdm.write(\"processing 1 tile\", file=verbose_dst)\n\n # run process on tile\n for result in mp.batch_processor(tile=tile):\n utils.write_verbose_msg(result, dst=verbose_dst)\n tqdm.tqdm.write(\n \"processing %s finished\" % mapchete_file, file=verbose_dst\n )\n\n # write VRT index\n if vrt:\n tqdm.tqdm.write(\"creating VRT\", file=verbose_dst)\n for tile in tqdm.tqdm(\n zoom_index_gen(\n mp=mp,\n zoom=tile.zoom,\n out_dir=(\n idx_out_dir if idx_out_dir else mp.config.output.path\n ),\n vrt=vrt,\n ),\n total=mp.count_tiles(tile.zoom, tile.zoom),\n unit=\"tile\",\n disable=debug or no_pbar\n ):\n logger.debug(\"%s indexed\", tile)\n tqdm.tqdm.write(\n \"VRT(s) creation for %s finished\" % mapchete_file,\n file=verbose_dst\n )\n\n # process area\n else:\n with mapchete.open(\n mapchete_file,\n mode=mode,\n zoom=zoom,\n bounds=bounds_from_opts(\n wkt_geometry=wkt_geometry,\n point=point,\n bounds=bounds,\n raw_conf=raw_conf(mapchete_file)\n ),\n single_input_file=input_file\n ) as mp:\n spinner.stop()\n tiles_count = mp.count_tiles(\n min(mp.config.init_zoom_levels),\n max(mp.config.init_zoom_levels)\n )\n tqdm.tqdm.write(\n \"processing %s tile(s) on %s worker(s)\" % (tiles_count, multi),\n file=verbose_dst\n )\n\n # run process on tiles\n for process_info in tqdm.tqdm(\n mp.batch_processor(\n multi=multi, zoom=zoom, max_chunksize=max_chunksize\n ),\n total=tiles_count,\n unit=\"tile\",\n disable=debug or no_pbar\n ):\n utils.write_verbose_msg(process_info, dst=verbose_dst)\n tqdm.tqdm.write(\n \"processing %s finished\" % mapchete_file, file=verbose_dst\n )\n\n # write VRT index\n if vrt:\n tqdm.tqdm.write(\"creating VRT(s)\", file=verbose_dst)\n for tile in tqdm.tqdm(\n zoom_index_gen(\n mp=mp,\n zoom=mp.config.init_zoom_levels,\n out_dir=(\n idx_out_dir if idx_out_dir\n else mp.config.output.path\n ),\n vrt=vrt\n ),\n total=mp.count_tiles(\n min(mp.config.init_zoom_levels),\n max(mp.config.init_zoom_levels)\n ),\n unit=\"tile\",\n disable=debug or no_pbar\n ):\n logger.debug(\"%s indexed\", tile)\n tqdm.tqdm.write(\n \"VRT(s) creation for %s finished\" % mapchete_file,\n file=verbose_dst\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef available_output_formats():\n output_formats = []\n for v in pkg_resources.iter_entry_points(DRIVERS_ENTRY_POINT):\n driver_ = v.load()\n if hasattr(driver_, \"METADATA\") and (\n driver_.METADATA[\"mode\"] in [\"w\", \"rw\"]\n ):\n output_formats.append(driver_.METADATA[\"driver_name\"])\n return output_formats", "response": "Return all available output formats."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns all available input formats.", "response": "def available_input_formats():\n \"\"\"\n Return all available input formats.\n\n Returns\n -------\n formats : list\n all available input formats\n \"\"\"\n input_formats = []\n for v in pkg_resources.iter_entry_points(DRIVERS_ENTRY_POINT):\n logger.debug(\"driver found: %s\", v)\n driver_ = v.load()\n if hasattr(driver_, \"METADATA\") and (driver_.METADATA[\"mode\"] in [\"r\", \"rw\"]):\n input_formats.append(driver_.METADATA[\"driver_name\"])\n return input_formats"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load_output_writer(output_params, readonly=False):\n if not isinstance(output_params, dict):\n raise TypeError(\"output_params must be a dictionary\")\n driver_name = output_params[\"format\"]\n for v in pkg_resources.iter_entry_points(DRIVERS_ENTRY_POINT):\n _driver = v.load()\n if all(\n [hasattr(_driver, attr) for attr in [\"OutputData\", \"METADATA\"]]\n ) and (\n _driver.METADATA[\"driver_name\"] == driver_name\n ):\n return _driver.OutputData(output_params, readonly=readonly)\n raise MapcheteDriverError(\"no loader for driver '%s' could be found.\" % driver_name)", "response": "Load the output class of driver."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nload input class of driver.", "response": "def load_input_reader(input_params, readonly=False):\n \"\"\"\n Return input class of driver.\n\n Returns\n -------\n input_params : ``InputData``\n input parameters\n \"\"\"\n logger.debug(\"find input reader with params %s\", input_params)\n if not isinstance(input_params, dict):\n raise TypeError(\"input_params must be a dictionary\")\n if \"abstract\" in input_params:\n driver_name = input_params[\"abstract\"][\"format\"]\n elif \"path\" in input_params:\n if os.path.splitext(input_params[\"path\"])[1]:\n input_file = input_params[\"path\"]\n driver_name = driver_from_file(input_file)\n else:\n logger.debug(\"%s is a directory\", input_params[\"path\"])\n driver_name = \"TileDirectory\"\n else:\n raise MapcheteDriverError(\"invalid input parameters %s\" % input_params)\n for v in pkg_resources.iter_entry_points(DRIVERS_ENTRY_POINT):\n driver_ = v.load()\n if hasattr(driver_, \"METADATA\") and (\n driver_.METADATA[\"driver_name\"] == driver_name\n ):\n return v.load().InputData(input_params, readonly=readonly)\n raise MapcheteDriverError(\"no loader for driver '%s' could be found.\" % driver_name)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef driver_from_file(input_file):\n file_ext = os.path.splitext(input_file)[1].split(\".\")[1]\n if file_ext not in _file_ext_to_driver():\n raise MapcheteDriverError(\n \"no driver could be found for file extension %s\" % file_ext\n )\n driver = _file_ext_to_driver()[file_ext]\n if len(driver) > 1:\n warnings.warn(\n DeprecationWarning(\n \"more than one driver for file found, taking %s\" % driver[0]\n )\n )\n return driver[0]", "response": "Guess driver from file extension."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndump output JSON and verify parameters if output metadata exist.", "response": "def write_output_metadata(output_params):\n \"\"\"Dump output JSON and verify parameters if output metadata exist.\"\"\"\n if \"path\" in output_params:\n metadata_path = os.path.join(output_params[\"path\"], \"metadata.json\")\n logger.debug(\"check for output %s\", metadata_path)\n try:\n existing_params = read_output_metadata(metadata_path)\n logger.debug(\"%s exists\", metadata_path)\n logger.debug(\"existing output parameters: %s\", pformat(existing_params))\n existing_tp = existing_params[\"pyramid\"]\n current_params = params_to_dump(output_params)\n logger.debug(\"current output parameters: %s\", pformat(current_params))\n current_tp = BufferedTilePyramid(**current_params[\"pyramid\"])\n if existing_tp != current_tp:\n raise MapcheteConfigError(\n \"pyramid definitions between existing and new output do not match: \"\n \"%s != %s\" % (existing_tp, current_tp)\n )\n existing_format = existing_params[\"driver\"][\"format\"]\n current_format = current_params[\"driver\"][\"format\"]\n if existing_format != current_format:\n raise MapcheteConfigError(\n \"existing output format does not match new output format: \"\n \"%s != %s\" % (\n (existing_format, current_format)\n )\n )\n except FileNotFoundError:\n logger.debug(\"%s does not exist\", metadata_path)\n dump_params = params_to_dump(output_params)\n # dump output metadata\n write_json(metadata_path, dump_params)\n else:\n logger.debug(\"no path parameter found\")"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlisting input and output formats.", "response": "def formats(input_formats, output_formats, debug=False):\n \"\"\"List input and/or output formats.\"\"\"\n if input_formats == output_formats:\n show_inputs, show_outputs = True, True\n else:\n show_inputs, show_outputs = input_formats, output_formats\n\n if show_inputs:\n click.echo(\"input formats:\")\n for driver in available_input_formats():\n click.echo(\"- %s\" % driver)\n if show_outputs:\n click.echo(\"output formats:\")\n for driver in available_output_formats():\n click.echo(\"- %s\" % driver)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nopen a new input tile and return a new InputTile object.", "response": "def open(\n self,\n tile,\n tile_directory_zoom=None,\n matching_method=\"gdal\",\n matching_max_zoom=None,\n matching_precision=8,\n fallback_to_higher_zoom=False,\n resampling=\"nearest\",\n **kwargs\n ):\n \"\"\"\n Return InputTile object.\n\n Parameters\n ----------\n tile : ``Tile``\n tile_directory_zoom : None\n If set, data will be read from exactly this zoom level\n matching_method : str ('gdal' or 'min') (default: 'gdal')\n gdal: Uses GDAL's standard method. Here, the target resolution is calculated\n by averaging the extent's pixel sizes over both x and y axes. This\n approach returns a zoom level which may not have the best quality but will\n speed up reading significantly.\n min: Returns the zoom level which matches the minimum resolution of the\n extents four corner pixels. This approach returns the zoom level with the\n best possible quality but with low performance. If the tile extent is\n outside of the destination pyramid, a TopologicalError will be raised.\n matching_max_zoom : int (default: None)\n If set, it will prevent reading from zoom levels above the maximum.\n matching_precision : int\n Round resolutions to n digits before comparing.\n fallback_to_higher_zoom : bool (default: False)\n In case no data is found at zoom level, try to read data from higher zoom\n levels. Enabling this setting can lead to many IO requests in areas with no\n data.\n resampling : string\n raster file: one of \"nearest\", \"average\", \"bilinear\" or \"lanczos\"\n\n Returns\n -------\n input tile : ``InputTile``\n tile view of input data\n \"\"\"\n # determine tile bounds in TileDirectory CRS\n td_bounds = reproject_geometry(\n tile.bbox,\n src_crs=tile.tp.crs,\n dst_crs=self.td_pyramid.crs\n ).bounds\n\n # find target zoom level\n if tile_directory_zoom is not None:\n zoom = tile_directory_zoom\n else:\n zoom = tile_to_zoom_level(\n tile, dst_pyramid=self.td_pyramid, matching_method=matching_method,\n precision=matching_precision\n )\n if matching_max_zoom is not None:\n zoom = min([zoom, matching_max_zoom])\n\n if fallback_to_higher_zoom:\n tiles_paths = []\n # check if tiles exist otherwise try higher zoom level\n while len(tiles_paths) == 0 and zoom >= 0:\n tiles_paths = _get_tiles_paths(\n basepath=self.path,\n ext=self._ext,\n pyramid=self.td_pyramid,\n bounds=td_bounds,\n zoom=zoom\n )\n logger.debug(\"%s existing tiles found at zoom %s\", len(tiles_paths), zoom)\n zoom -= 1\n else:\n tiles_paths = _get_tiles_paths(\n basepath=self.path,\n ext=self._ext,\n pyramid=self.td_pyramid,\n bounds=td_bounds,\n zoom=zoom\n )\n logger.debug(\"%s existing tiles found at zoom %s\", len(tiles_paths), zoom)\n return InputTile(\n tile,\n tiles_paths=tiles_paths,\n file_type=self._file_type,\n profile=self._profile,\n td_crs=self.td_pyramid.crs,\n resampling=resampling,\n read_as_tiledir_func=self._read_as_tiledir_func,\n **kwargs\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef bbox(self, out_crs=None):\n return reproject_geometry(\n box(*self._bounds),\n src_crs=self.td_pyramid.crs,\n dst_crs=self.pyramid.crs if out_crs is None else out_crs\n )", "response": "Return data bounding box."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nread the data from the target dataset into a list of tuples.", "response": "def read(\n self,\n validity_check=False,\n indexes=None,\n resampling=None,\n dst_nodata=None,\n gdal_opts=None,\n **kwargs\n ):\n \"\"\"\n Read reprojected & resampled input data.\n\n Parameters\n ----------\n validity_check : bool\n vector file: also run checks if reprojected geometry is valid,\n otherwise throw RuntimeError (default: True)\n\n indexes : list or int\n raster file: a list of band numbers; None will read all.\n dst_nodata : int or float, optional\n raster file: if not set, the nodata value from the source dataset\n will be used\n gdal_opts : dict\n raster file: GDAL options passed on to rasterio.Env()\n\n Returns\n -------\n data : list for vector files or numpy array for raster files\n \"\"\"\n return self._read_as_tiledir(\n data_type=self._file_type,\n out_tile=self.tile,\n td_crs=self._td_crs,\n tiles_paths=self._tiles_paths,\n profile=self._profile,\n validity_check=validity_check,\n indexes=indexes,\n resampling=resampling if resampling else self._resampling,\n dst_nodata=dst_nodata,\n gdal_opts=gdal_opts,\n **{k: v for k, v in kwargs.items() if k != \"data_type\"}\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nextract contour lines from an array.", "response": "def extract_contours(array, tile, interval=100, field='elev', base=0):\n \"\"\"\n Extract contour lines from an array.\n\n Parameters\n ----------\n array : array\n input elevation data\n tile : Tile\n tile covering the array\n interval : integer\n elevation value interval when drawing contour lines\n field : string\n output field name containing elevation value\n base : integer\n elevation base value the intervals are computed from\n\n Returns\n -------\n contours : iterable\n contours as GeoJSON-like pairs of properties and geometry\n \"\"\"\n import matplotlib.pyplot as plt\n levels = _get_contour_values(\n array.min(), array.max(), interval=interval, base=base)\n if not levels:\n return []\n contours = plt.contour(array, levels)\n index = 0\n out_contours = []\n for level in range(len(contours.collections)):\n elevation = levels[index]\n index += 1\n paths = contours.collections[level].get_paths()\n for path in paths:\n out_coords = [\n (\n tile.left + (y * tile.pixel_x_size),\n tile.top - (x * tile.pixel_y_size),\n )\n for x, y in zip(path.vertices[:, 1], path.vertices[:, 0])\n ]\n if len(out_coords) >= 2:\n out_contours.append(\n dict(\n properties={field: elevation},\n geometry=mapping(LineString(out_coords))\n )\n )\n return out_contours"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of values between min and max within an interval.", "response": "def _get_contour_values(min_val, max_val, base=0, interval=100):\n \"\"\"Return a list of values between min and max within an interval.\"\"\"\n i = base\n out = []\n if min_val < base:\n while i >= min_val:\n i -= interval\n while i <= max_val:\n if i >= min_val:\n out.append(i)\n i += interval\n return out"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates an empty Mapchete and process file in a given directory.", "response": "def create(\n mapchete_file,\n process_file,\n out_format,\n out_path=None,\n pyramid_type=None,\n force=False\n):\n \"\"\"Create an empty Mapchete and process file in a given directory.\"\"\"\n if os.path.isfile(process_file) or os.path.isfile(mapchete_file):\n if not force:\n raise IOError(\"file(s) already exists\")\n\n out_path = out_path if out_path else os.path.join(os.getcwd(), \"output\")\n\n # copy file template to target directory\n process_template = pkg_resources.resource_filename(\n \"mapchete.static\", \"process_template.py\"\n )\n process_file = os.path.join(os.getcwd(), process_file)\n copyfile(process_template, process_file)\n\n # modify and copy mapchete file template to target directory\n mapchete_template = pkg_resources.resource_filename(\n \"mapchete.static\", \"mapchete_template.mapchete\"\n )\n\n output_options = dict(\n format=out_format, path=out_path, **FORMAT_MANDATORY[out_format]\n )\n\n pyramid_options = {'grid': pyramid_type}\n\n substitute_elements = {\n 'process_file': process_file,\n 'output': dump({'output': output_options}, default_flow_style=False),\n 'pyramid': dump({'pyramid': pyramid_options}, default_flow_style=False)\n }\n with open(mapchete_template, 'r') as config_template:\n config = Template(config_template.read())\n customized_config = config.substitute(substitute_elements)\n with open(mapchete_file, 'w') as target_config:\n target_config.write(customized_config)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nchecks whether output tiles of a tile exist.", "response": "def tiles_exist(self, process_tile=None, output_tile=None):\n \"\"\"\n Check whether output tiles of a tile (either process or output) exists.\n\n Parameters\n ----------\n process_tile : ``BufferedTile``\n must be member of process ``TilePyramid``\n output_tile : ``BufferedTile``\n must be member of output ``TilePyramid``\n\n Returns\n -------\n exists : bool\n \"\"\"\n if process_tile and output_tile:\n raise ValueError(\"just one of 'process_tile' and 'output_tile' allowed\")\n if process_tile:\n return any(\n path_exists(self.get_path(tile))\n for tile in self.pyramid.intersecting(process_tile)\n )\n if output_tile:\n return path_exists(self.get_path(output_tile))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_path(self, tile):\n return os.path.join(*[\n self.path,\n str(tile.zoom),\n str(tile.row),\n str(tile.col) + self.file_extension\n ])", "response": "Determine target file path."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef prepare_path(self, tile):\n makedirs(os.path.dirname(self.get_path(tile)))", "response": "Create directory and subdirectory if necessary."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef output_is_valid(self, process_data):\n if self.METADATA[\"data_type\"] == \"raster\":\n return (\n is_numpy_or_masked_array(process_data) or\n is_numpy_or_masked_array_with_tags(process_data)\n )\n elif self.METADATA[\"data_type\"] == \"vector\":\n return is_feature_list(process_data)", "response": "Check whether process output is allowed with output driver."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn verified and cleaned output. Parameters ---------- process_data : raw process output Returns ------- NumPy array or list of features.", "response": "def output_cleaned(self, process_data):\n \"\"\"\n Return verified and cleaned output.\n\n Parameters\n ----------\n process_data : raw process output\n\n Returns\n -------\n NumPy array or list of features.\n \"\"\"\n if self.METADATA[\"data_type\"] == \"raster\":\n if is_numpy_or_masked_array(process_data):\n return process_data\n elif is_numpy_or_masked_array_with_tags(process_data):\n data, tags = process_data\n return self.output_cleaned(data), tags\n elif self.METADATA[\"data_type\"] == \"vector\":\n return list(process_data)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef extract_subset(self, input_data_tiles=None, out_tile=None):\n if self.METADATA[\"data_type\"] == \"raster\":\n mosaic = create_mosaic(input_data_tiles)\n return extract_from_array(\n in_raster=prepare_array(\n mosaic.data,\n nodata=self.nodata,\n dtype=self.output_params[\"dtype\"]\n ),\n in_affine=mosaic.affine,\n out_tile=out_tile\n )\n elif self.METADATA[\"data_type\"] == \"vector\":\n return [\n feature for feature in list(\n chain.from_iterable([features for _, features in input_data_tiles])\n )\n if shape(feature[\"geometry\"]).intersects(out_tile.bbox)\n ]", "response": "Extract subset from multiple tiles."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _read_as_tiledir(\n self,\n out_tile=None,\n td_crs=None,\n tiles_paths=None,\n profile=None,\n validity_check=False,\n indexes=None,\n resampling=None,\n dst_nodata=None,\n gdal_opts=None,\n **kwargs\n ):\n \"\"\"\n Read reprojected & resampled input data.\n\n Parameters\n ----------\n validity_check : bool\n vector file: also run checks if reprojected geometry is valid,\n otherwise throw RuntimeError (default: True)\n\n indexes : list or int\n raster file: a list of band numbers; None will read all.\n dst_nodata : int or float, optional\n raster file: if not set, the nodata value from the source dataset\n will be used\n gdal_opts : dict\n raster file: GDAL options passed on to rasterio.Env()\n\n Returns\n -------\n data : list for vector files or numpy array for raster files\n \"\"\"\n return _read_as_tiledir(\n data_type=self.METADATA[\"data_type\"],\n out_tile=out_tile,\n td_crs=td_crs,\n tiles_paths=tiles_paths,\n profile=profile,\n validity_check=validity_check,\n indexes=indexes,\n resampling=resampling,\n dst_nodata=dst_nodata,\n gdal_opts=gdal_opts,\n **{k: v for k, v in kwargs.items() if k != \"data_type\"}\n )", "response": "Read the data from the reprojected and resampled input files into a list of tile directories."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the slope and aspect map of a single resource.", "response": "def calculate_slope_aspect(elevation, xres, yres, z=1.0, scale=1.0):\n \"\"\"\n Calculate slope and aspect map.\n\n Return a pair of arrays 2 pixels smaller than the input elevation array.\n\n Slope is returned in radians, from 0 for sheer face to pi/2 for\n flat ground. Aspect is returned in radians, counterclockwise from -pi\n at north around to pi.\n\n Logic here is borrowed from hillshade.cpp:\n http://www.perrygeo.net/wordpress/?p=7\n\n Parameters\n ----------\n elevation : array\n input elevation data\n xres : float\n column width\n yres : float\n row height\n z : float\n vertical exaggeration factor\n scale : float\n scale factor of pixel size units versus height units (insert 112000\n when having elevation values in meters in a geodetic projection)\n\n Returns\n -------\n slope shade : array\n \"\"\"\n z = float(z)\n scale = float(scale)\n height, width = elevation.shape[0] - 2, elevation.shape[1] - 2\n window = [\n z * elevation[row:(row + height), col:(col + width)]\n for (row, col) in product(range(3), range(3))\n ]\n x = (\n (window[0] + window[3] + window[3] + window[6])\n - (window[2] + window[5] + window[5] + window[8])\n ) / (8.0 * xres * scale)\n y = (\n (window[6] + window[7] + window[7] + window[8])\n - (window[0] + window[1] + window[1] + window[2])\n ) / (8.0 * yres * scale)\n # in radians, from 0 to pi/2\n slope = math.pi/2 - np.arctan(np.sqrt(x*x + y*y))\n # in radians counterclockwise, from -pi at north back to pi\n aspect = np.arctan2(x, y)\n return slope, aspect"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the hillshaded numpy array.", "response": "def hillshade(elevation, tile, azimuth=315.0, altitude=45.0, z=1.0, scale=1.0):\n \"\"\"\n Return hillshaded numpy array.\n\n Parameters\n ----------\n elevation : array\n input elevation data\n tile : Tile\n tile covering the array\n z : float\n vertical exaggeration factor\n scale : float\n scale factor of pixel size units versus height units (insert 112000\n when having elevation values in meters in a geodetic projection)\n \"\"\"\n azimuth = float(azimuth)\n altitude = float(altitude)\n z = float(z)\n scale = float(scale)\n xres = tile.tile.pixel_x_size\n yres = -tile.tile.pixel_y_size\n slope, aspect = calculate_slope_aspect(\n elevation, xres, yres, z=z, scale=scale)\n deg2rad = math.pi / 180.0\n shaded = np.sin(altitude * deg2rad) * np.sin(slope) \\\n + np.cos(altitude * deg2rad) * np.cos(slope) \\\n * np.cos((azimuth - 90.0) * deg2rad - aspect)\n # shaded now has values between -1.0 and +1.0\n # stretch to 0 - 255 and invert\n shaded = (((shaded+1.0)/2)*-255.0).astype(\"uint8\")\n # add one pixel padding using the edge values\n return ma.masked_array(\n data=np.pad(shaded, 1, mode='edge'), mask=elevation.mask\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef tile(self, zoom, row, col):\n tile = self.tile_pyramid.tile(zoom, row, col)\n return BufferedTile(tile, pixelbuffer=self.pixelbuffer)", "response": "Return a buffered tile object of this BufferedTilePyramid."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef tiles_from_bounds(self, bounds, zoom):\n for tile in self.tiles_from_bbox(box(*bounds), zoom):\n yield self.tile(*tile.id)", "response": "Yield all tiles intersecting with bounds."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nyields all metatiles intersecting with given bounding box.", "response": "def tiles_from_bbox(self, geometry, zoom):\n \"\"\"\n All metatiles intersecting with given bounding box.\n\n Parameters\n ----------\n geometry : ``shapely.geometry``\n zoom : integer\n zoom level\n\n Yields\n ------\n intersecting tiles : generator\n generates ``BufferedTiles``\n \"\"\"\n for tile in self.tile_pyramid.tiles_from_bbox(geometry, zoom):\n yield self.tile(*tile.id)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tiles_from_geom(self, geometry, zoom):\n for tile in self.tile_pyramid.tiles_from_geom(geometry, zoom):\n yield self.tile(*tile.id)", "response": "Yield all tiles intersecting with input geometry."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning all BufferedTiles intersecting with tile.", "response": "def intersecting(self, tile):\n \"\"\"\n Return all BufferedTiles intersecting with tile.\n\n Parameters\n ----------\n tile : ``BufferedTile``\n another tile\n \"\"\"\n return [\n self.tile(*intersecting_tile.id)\n for intersecting_tile in self.tile_pyramid.intersecting(tile)\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn dictionary representation of pyramid parameters.", "response": "def to_dict(self):\n \"\"\"\n Return dictionary representation of pyramid parameters.\n \"\"\"\n return dict(\n grid=self.grid.to_dict(),\n metatiling=self.metatiling,\n tile_size=self.tile_size,\n pixelbuffer=self.pixelbuffer\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting the children of this tile.", "response": "def get_children(self):\n \"\"\"\n Get tile children (intersecting tiles in next zoom level).\n\n Returns\n -------\n children : list\n a list of ``BufferedTiles``\n \"\"\"\n return [BufferedTile(t, self.pixelbuffer) for t in self._tile.get_children()]"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns list of BufferedTiles that are connected to the current tile.", "response": "def get_neighbors(self, connectedness=8):\n \"\"\"\n Return tile neighbors.\n\n Tile neighbors are unique, i.e. in some edge cases, where both the left\n and right neighbor wrapped around the antimeridian is the same. Also,\n neighbors ouside the northern and southern TilePyramid boundaries are\n excluded, because they are invalid.\n\n -------------\n | 8 | 1 | 5 |\n -------------\n | 4 | x | 2 |\n -------------\n | 7 | 3 | 6 |\n -------------\n\n Parameters\n ----------\n connectedness : int\n [4 or 8] return four direct neighbors or all eight.\n\n Returns\n -------\n list of BufferedTiles\n \"\"\"\n return [\n BufferedTile(t, self.pixelbuffer)\n for t in self._tile.get_neighbors(connectedness=connectedness)\n ]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_on_edge(self):\n return (\n self.left <= self.tile_pyramid.left or # touches_left\n self.bottom <= self.tile_pyramid.bottom or # touches_bottom\n self.right >= self.tile_pyramid.right or # touches_right\n self.top >= self.tile_pyramid.top # touches_top\n )", "response": "Determine whether tile touches or goes over pyramid edge."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef execute(\n mp,\n resampling=\"nearest\",\n scale_method=None,\n scales_minmax=None\n):\n \"\"\"\n Read, stretch and return raster data.\n\n Inputs:\n -------\n raster\n raster file\n\n Parameters:\n -----------\n resampling : str\n rasterio.Resampling method\n scale_method : str\n - dtype_scale: use dtype minimum and maximum values\n - minmax_scale: use dataset bands minimum and maximum values\n - crop: clip data to output dtype\n scales_minmax : tuple\n tuple of band specific scale values\n\n Output:\n -------\n np.ndarray\n \"\"\"\n with mp.open(\"raster\", resampling=resampling) as raster_file:\n\n # exit if input tile is empty\n if raster_file.is_empty():\n return \"empty\"\n\n # actually read data and iterate through bands\n scaled = ()\n mask = ()\n raster_data = raster_file.read()\n if raster_data.ndim == 2:\n raster_data = ma.expand_dims(raster_data, axis=0)\n if not scale_method:\n scales_minmax = [(i, i) for i in range(len(raster_data))]\n\n for band, (scale_min, scale_max) in zip(raster_data, scales_minmax):\n if scale_method in [\"dtype_scale\", \"minmax_scale\"]:\n scaled += (_stretch_array(band, scale_min, scale_max), )\n elif scale_method == \"crop\":\n scaled += (np.clip(band, scale_min, scale_max), )\n else:\n scaled += (band, )\n mask += (band.mask, )\n\n return ma.masked_array(np.stack(scaled), np.stack(mask))", "response": "Execute the raster file read stretch and return raster data."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read(self, output_tile, **kwargs):\n try:\n return read_raster_no_crs(self.get_path(output_tile))\n except FileNotFoundError:\n return self.empty(output_tile)", "response": "Read existing process output."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrite data from process_tile to GeoTIFF file.", "response": "def write(self, process_tile, data):\n \"\"\"\n Write data from process tiles into GeoTIFF file(s).\n\n Parameters\n ----------\n process_tile : ``BufferedTile``\n must be member of process ``TilePyramid``\n data : ``np.ndarray``\n \"\"\"\n if (\n isinstance(data, tuple) and\n len(data) == 2 and\n isinstance(data[1], dict)\n ):\n data, tags = data\n else:\n tags = {}\n data = prepare_array(\n data,\n masked=True,\n nodata=self.nodata,\n dtype=self.profile(process_tile)[\"dtype\"]\n )\n\n if data.mask.all():\n logger.debug(\"data empty, nothing to write\")\n else:\n # in case of S3 output, create an boto3 resource\n bucket_resource = get_boto3_bucket(self._bucket) if self._bucket else None\n\n # Convert from process_tile to output_tiles and write\n for tile in self.pyramid.intersecting(process_tile):\n out_path = self.get_path(tile)\n self.prepare_path(tile)\n out_tile = BufferedTile(tile, self.pixelbuffer)\n write_raster_window(\n in_tile=process_tile,\n in_data=data,\n out_profile=self.profile(out_tile),\n out_tile=out_tile,\n out_path=out_path,\n tags=tags,\n bucket_resource=bucket_resource\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef profile(self, tile=None):\n dst_metadata = GTIFF_DEFAULT_PROFILE\n dst_metadata.pop(\"transform\", None)\n dst_metadata.update(\n count=self.output_params[\"bands\"],\n dtype=self.output_params[\"dtype\"],\n driver=\"GTiff\"\n )\n if tile is not None:\n dst_metadata.update(\n crs=tile.crs, width=tile.width, height=tile.height,\n affine=tile.affine)\n else:\n for k in [\"crs\", \"width\", \"height\", \"affine\"]:\n dst_metadata.pop(k, None)\n if \"nodata\" in self.output_params:\n dst_metadata.update(nodata=self.output_params[\"nodata\"])\n try:\n if \"compression\" in self.output_params:\n warnings.warn(\n DeprecationWarning(\"use 'compress' instead of 'compression'\")\n )\n dst_metadata.update(compress=self.output_params[\"compression\"])\n else:\n dst_metadata.update(compress=self.output_params[\"compress\"])\n dst_metadata.update(predictor=self.output_params[\"predictor\"])\n except KeyError:\n pass\n return dst_metadata", "response": "Create a dictionary of dictionary of dictionary of dictionary of image data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning an empty array.", "response": "def empty(self, process_tile):\n \"\"\"\n Return empty data.\n\n Parameters\n ----------\n process_tile : ``BufferedTile``\n must be member of process ``TilePyramid``\n\n Returns\n -------\n empty data : array\n empty array with data type provided in output profile\n \"\"\"\n profile = self.profile(process_tile)\n return ma.masked_array(\n data=np.full(\n (profile[\"count\"], ) + process_tile.shape, profile[\"nodata\"],\n dtype=profile[\"dtype\"]),\n mask=True\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef for_web(self, data):\n return memory_file(\n prepare_array(\n data, masked=True, nodata=self.nodata, dtype=self.profile()[\"dtype\"]\n ),\n self.profile()\n ), \"image/tiff\"", "response": "Convert data to web output."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef open(self, tile, process, **kwargs):\n return InputTile(tile, process, kwargs.get(\"resampling\", None))", "response": "Open process output as input for other process."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreading reprojected & resampled input data.", "response": "def read(self, indexes=None, **kwargs):\n \"\"\"\n Read reprojected & resampled input data.\n\n Parameters\n ----------\n indexes : integer or list\n band number or list of band numbers\n\n Returns\n -------\n data : array\n \"\"\"\n band_indexes = self._get_band_indexes(indexes)\n arr = self.process.get_raw_output(self.tile)\n if len(band_indexes) == 1:\n return arr[band_indexes[0] - 1]\n else:\n return ma.concatenate([ma.expand_dims(arr[i - 1], 0) for i in band_indexes])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nchecking if there is data within this tile.", "response": "def is_empty(self, indexes=None):\n \"\"\"\n Check if there is data within this tile.\n\n Returns\n -------\n is empty : bool\n \"\"\"\n # empty if tile does not intersect with file bounding box\n return not self.tile.bbox.intersects(self.process.config.area_at_zoom())"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_band_indexes(self, indexes=None):\n if indexes:\n if isinstance(indexes, list):\n return indexes\n else:\n return [indexes]\n else:\n return range(1, self.process.config.output.profile(self.tile)[\"count\"] + 1)", "response": "Return valid band indexes."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a dictionary of metadata for rasterio.", "response": "def profile(self, tile=None):\n \"\"\"\n Create a metadata dictionary for rasterio.\n\n Parameters\n ----------\n tile : ``BufferedTile``\n\n Returns\n -------\n metadata : dictionary\n output profile dictionary used for rasterio.\n \"\"\"\n dst_metadata = PNG_DEFAULT_PROFILE\n dst_metadata.pop(\"transform\", None)\n if tile is not None:\n dst_metadata.update(\n width=tile.width, height=tile.height, affine=tile.affine,\n crs=tile.crs)\n try:\n dst_metadata.update(count=self.output_params[\"count\"])\n except KeyError:\n pass\n return dst_metadata"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts data to web output.", "response": "def for_web(self, data):\n \"\"\"\n Convert data to web output.\n\n Parameters\n ----------\n data : array\n\n Returns\n -------\n web data : array\n \"\"\"\n rgba = self._prepare_array_for_png(data)\n data = ma.masked_where(rgba == self.nodata, rgba)\n return memory_file(data, self.profile()), 'image/png'"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns empty data. Parameters ---------- process_tile : ``BufferedTile`` must be member of process ``TilePyramid`` Returns ------- empty data : array empty array with data type given in output parameters", "response": "def empty(self, process_tile):\n \"\"\"\n Return empty data.\n\n Parameters\n ----------\n process_tile : ``BufferedTile``\n must be member of process ``TilePyramid``\n\n Returns\n -------\n empty data : array\n empty array with data type given in output parameters\n \"\"\"\n bands = (\n self.output_params[\"bands\"]\n if \"bands\" in self.output_params\n else PNG_DEFAULT_PROFILE[\"count\"]\n )\n return ma.masked_array(\n data=ma.zeros((bands, ) + process_tile.shape),\n mask=ma.zeros((bands, ) + process_tile.shape),\n dtype=PNG_DEFAULT_PROFILE[\"dtype\"]\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nserves a Mapchete process.", "response": "def serve(\n mapchete_file,\n port=None,\n internal_cache=None,\n zoom=None,\n bounds=None,\n overwrite=False,\n readonly=False,\n memory=False,\n input_file=None,\n debug=False,\n logfile=None\n):\n \"\"\"\n Serve a Mapchete process.\n\n Creates the Mapchete host and serves both web page with OpenLayers and the\n WMTS simple REST endpoint.\n \"\"\"\n app = create_app(\n mapchete_files=[mapchete_file], zoom=zoom,\n bounds=bounds, single_input_file=input_file,\n mode=_get_mode(memory, readonly, overwrite), debug=debug\n )\n if os.environ.get(\"MAPCHETE_TEST\") == \"TRUE\":\n logger.debug(\"don't run flask app, MAPCHETE_TEST environment detected\")\n else:\n app.run(\n threaded=True, debug=True, port=port, host='0.0.0.0',\n extra_files=[mapchete_file]\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_app(\n mapchete_files=None, zoom=None, bounds=None, single_input_file=None,\n mode=\"continue\", debug=None\n):\n \"\"\"Configure and create Flask app.\"\"\"\n from flask import Flask, render_template_string\n app = Flask(__name__)\n mapchete_processes = {\n os.path.splitext(os.path.basename(mapchete_file))[0]: mapchete.open(\n mapchete_file, zoom=zoom, bounds=bounds,\n single_input_file=single_input_file, mode=mode, with_cache=True,\n debug=debug)\n for mapchete_file in mapchete_files\n }\n\n mp = next(iter(mapchete_processes.values()))\n pyramid_type = mp.config.process_pyramid.grid\n pyramid_srid = mp.config.process_pyramid.crs.to_epsg()\n process_bounds = \",\".join([str(i) for i in mp.config.bounds_at_zoom()])\n grid = \"g\" if pyramid_srid == 3857 else \"WGS84\"\n web_pyramid = BufferedTilePyramid(pyramid_type)\n\n @app.route('/', methods=['GET'])\n def index():\n \"\"\"Render and hosts the appropriate OpenLayers instance.\"\"\"\n return render_template_string(\n pkgutil.get_data(\n 'mapchete.static', 'index.html').decode(\"utf-8\"),\n srid=pyramid_srid,\n process_bounds=process_bounds,\n is_mercator=(pyramid_srid == 3857),\n process_names=mapchete_processes.keys()\n )\n\n @app.route(\n \"/\".join([\n \"\", \"wmts_simple\", \"1.0.0\", \"\", \"default\",\n grid, \"\", \"\", \".\"]),\n methods=['GET'])\n def get(mp_name, zoom, row, col, file_ext):\n \"\"\"Return processed, empty or error (in pink color) tile.\"\"\"\n logger.debug(\n \"received tile (%s, %s, %s) for process %s\", zoom, row, col,\n mp_name)\n # convert zoom, row, col into tile object using web pyramid\n return _tile_response(\n mapchete_processes[mp_name], web_pyramid.tile(zoom, row, col),\n debug)\n\n return app", "response": "Configure and create Flask app."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads a window of raster files from a single file.", "response": "def read_raster_window(\n input_files,\n tile,\n indexes=None,\n resampling=\"nearest\",\n src_nodata=None,\n dst_nodata=None,\n gdal_opts=None\n):\n \"\"\"\n Return NumPy arrays from an input raster.\n\n NumPy arrays are reprojected and resampled to tile properties from input\n raster. If tile boundaries cross the antimeridian, data on the other side\n of the antimeridian will be read and concatenated to the numpy array\n accordingly.\n\n Parameters\n ----------\n input_files : string or list\n path to a raster file or list of paths to multiple raster files readable by\n rasterio.\n tile : Tile\n a Tile object\n indexes : list or int\n a list of band numbers; None will read all.\n resampling : string\n one of \"nearest\", \"average\", \"bilinear\" or \"lanczos\"\n src_nodata : int or float, optional\n if not set, the nodata value from the source dataset will be used\n dst_nodata : int or float, optional\n if not set, the nodata value from the source dataset will be used\n gdal_opts : dict\n GDAL options passed on to rasterio.Env()\n\n Returns\n -------\n raster : MaskedArray\n \"\"\"\n with rasterio.Env(\n **get_gdal_options(\n gdal_opts,\n is_remote=path_is_remote(\n input_files[0] if isinstance(input_files, list) else input_files, s3=True\n )\n )\n ) as env:\n logger.debug(\"reading %s with GDAL options %s\", input_files, env.options)\n return _read_raster_window(\n input_files,\n tile,\n indexes=indexes,\n resampling=resampling,\n src_nodata=src_nodata,\n dst_nodata=dst_nodata\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nextract a numpy array from a raster file.", "response": "def _get_warped_array(\n input_file=None,\n indexes=None,\n dst_bounds=None,\n dst_shape=None,\n dst_crs=None,\n resampling=None,\n src_nodata=None,\n dst_nodata=None\n):\n \"\"\"Extract a numpy array from a raster file.\"\"\"\n try:\n return _rasterio_read(\n input_file=input_file,\n indexes=indexes,\n dst_bounds=dst_bounds,\n dst_shape=dst_shape,\n dst_crs=dst_crs,\n resampling=resampling,\n src_nodata=src_nodata,\n dst_nodata=dst_nodata\n )\n except Exception as e:\n logger.exception(\"error while reading file %s: %s\", input_file, e)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nwrap function around rasterio. open. read.", "response": "def read_raster_no_crs(input_file, indexes=None, gdal_opts=None):\n \"\"\"\n Wrapper function around rasterio.open().read().\n\n Parameters\n ----------\n input_file : str\n Path to file\n indexes : int or list\n Band index or list of band indexes to be read.\n\n Returns\n -------\n MaskedArray\n\n Raises\n ------\n FileNotFoundError if file cannot be found.\n \"\"\"\n with warnings.catch_warnings():\n warnings.simplefilter(\"ignore\")\n try:\n with rasterio.Env(\n **get_gdal_options(\n gdal_opts, is_remote=path_is_remote(input_file, s3=True)\n )\n ):\n with rasterio.open(input_file, \"r\") as src:\n return src.read(indexes=indexes, masked=True)\n except RasterioIOError as e:\n for i in (\"does not exist in the file system\", \"No such file or directory\"):\n if i in str(e):\n raise FileNotFoundError(\"%s not found\" % input_file)\n else:\n raise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting a window from a numpy array to a GeoTIFF file.", "response": "def write_raster_window(\n in_tile=None, in_data=None, out_profile=None, out_tile=None, out_path=None,\n tags=None, bucket_resource=None\n):\n \"\"\"\n Write a window from a numpy array to an output file.\n\n Parameters\n ----------\n in_tile : ``BufferedTile``\n ``BufferedTile`` with a data attribute holding NumPy data\n in_data : array\n out_profile : dictionary\n metadata dictionary for rasterio\n out_tile : ``Tile``\n provides output boundaries; if None, in_tile is used\n out_path : string\n output path to write to\n tags : optional tags to be added to GeoTIFF file\n bucket_resource : boto3 bucket resource to write to in case of S3 output\n \"\"\"\n if not isinstance(out_path, str):\n raise TypeError(\"out_path must be a string\")\n logger.debug(\"write %s\", out_path)\n if out_path == \"memoryfile\":\n raise DeprecationWarning(\n \"Writing to memoryfile with write_raster_window() is deprecated. \"\n \"Please use RasterWindowMemoryFile.\"\n )\n out_tile = in_tile if out_tile is None else out_tile\n _validate_write_window_params(in_tile, out_tile, in_data, out_profile)\n\n # extract data\n window_data = extract_from_array(\n in_raster=in_data,\n in_affine=in_tile.affine,\n out_tile=out_tile\n ) if in_tile != out_tile else in_data\n\n # use transform instead of affine\n if \"affine\" in out_profile:\n out_profile[\"transform\"] = out_profile.pop(\"affine\")\n\n # write if there is any band with non-masked data\n if window_data.all() is not ma.masked:\n\n try:\n if out_path.startswith(\"s3://\"):\n with RasterWindowMemoryFile(\n in_tile=out_tile,\n in_data=window_data,\n out_profile=out_profile,\n out_tile=out_tile,\n tags=tags\n ) as memfile:\n logger.debug((out_tile.id, \"upload tile\", out_path))\n bucket_resource.put_object(\n Key=\"/\".join(out_path.split(\"/\")[3:]),\n Body=memfile\n )\n else:\n with rasterio.open(out_path, 'w', **out_profile) as dst:\n logger.debug((out_tile.id, \"write tile\", out_path))\n dst.write(window_data.astype(out_profile[\"dtype\"], copy=False))\n _write_tags(dst, tags)\n except Exception as e:\n logger.exception(\"error while writing file %s: %s\", out_path, e)\n raise\n else:\n logger.debug((out_tile.id, \"array window empty\", out_path))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef extract_from_array(in_raster=None, in_affine=None, out_tile=None):\n if isinstance(in_raster, ReferencedRaster):\n in_affine = in_raster.affine\n in_raster = in_raster.data\n\n # get range within array\n minrow, maxrow, mincol, maxcol = bounds_to_ranges(\n out_bounds=out_tile.bounds, in_affine=in_affine, in_shape=in_raster.shape\n )\n # if output window is within input window\n if (\n minrow >= 0 and\n mincol >= 0 and\n maxrow <= in_raster.shape[-2] and\n maxcol <= in_raster.shape[-1]\n ):\n return in_raster[..., minrow:maxrow, mincol:maxcol]\n # raise error if output is not fully within input\n else:\n raise ValueError(\"extraction fails if output shape is not within input\")", "response": "Extract raster data window array."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nextract and resample from array to target tile.", "response": "def resample_from_array(\n in_raster=None,\n in_affine=None,\n out_tile=None,\n in_crs=None,\n resampling=\"nearest\",\n nodataval=0\n):\n \"\"\"\n Extract and resample from array to target tile.\n\n Parameters\n ----------\n in_raster : array\n in_affine : ``Affine``\n out_tile : ``BufferedTile``\n resampling : string\n one of rasterio's resampling methods (default: nearest)\n nodataval : integer or float\n raster nodata value (default: 0)\n\n Returns\n -------\n resampled array : array\n \"\"\"\n # TODO rename function\n if isinstance(in_raster, ma.MaskedArray):\n pass\n if isinstance(in_raster, np.ndarray):\n in_raster = ma.MaskedArray(in_raster, mask=in_raster == nodataval)\n elif isinstance(in_raster, ReferencedRaster):\n in_affine = in_raster.affine\n in_crs = in_raster.crs\n in_raster = in_raster.data\n elif isinstance(in_raster, tuple):\n in_raster = ma.MaskedArray(\n data=np.stack(in_raster),\n mask=np.stack([\n band.mask\n if isinstance(band, ma.masked_array)\n else np.where(band == nodataval, True, False)\n for band in in_raster\n ]),\n fill_value=nodataval\n )\n else:\n raise TypeError(\"wrong input data type: %s\" % type(in_raster))\n if in_raster.ndim == 2:\n in_raster = ma.expand_dims(in_raster, axis=0)\n elif in_raster.ndim == 3:\n pass\n else:\n raise TypeError(\"input array must have 2 or 3 dimensions\")\n if in_raster.fill_value != nodataval:\n ma.set_fill_value(in_raster, nodataval)\n out_shape = (in_raster.shape[0], ) + out_tile.shape\n dst_data = np.empty(out_shape, in_raster.dtype)\n in_raster = ma.masked_array(\n data=in_raster.filled(), mask=in_raster.mask, fill_value=nodataval\n )\n reproject(\n in_raster,\n dst_data,\n src_transform=in_affine,\n src_crs=in_crs if in_crs else out_tile.crs,\n dst_transform=out_tile.affine,\n dst_crs=out_tile.crs,\n resampling=Resampling[resampling]\n )\n return ma.MaskedArray(dst_data, mask=dst_data == nodataval)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_mosaic(tiles, nodata=0):\n if isinstance(tiles, GeneratorType):\n tiles = list(tiles)\n elif not isinstance(tiles, list):\n raise TypeError(\"tiles must be either a list or generator\")\n if not all([isinstance(pair, tuple) for pair in tiles]):\n raise TypeError(\"tiles items must be tuples\")\n if not all([\n all([isinstance(tile, BufferedTile), isinstance(data, np.ndarray)])\n for tile, data in tiles\n ]):\n raise TypeError(\"tuples must be pairs of BufferedTile and array\")\n if len(tiles) == 0:\n raise ValueError(\"tiles list is empty\")\n\n logger.debug(\"create mosaic from %s tile(s)\", len(tiles))\n # quick return if there is just one tile\n if len(tiles) == 1:\n tile, data = tiles[0]\n return ReferencedRaster(\n data=data,\n affine=tile.affine,\n bounds=tile.bounds,\n crs=tile.crs\n )\n\n # assert all tiles have same properties\n pyramid, resolution, dtype = _get_tiles_properties(tiles)\n # just handle antimeridian on global pyramid types\n shift = _shift_required(tiles)\n # determine mosaic shape and reference\n m_left, m_bottom, m_right, m_top = None, None, None, None\n for tile, data in tiles:\n num_bands = data.shape[0] if data.ndim > 2 else 1\n left, bottom, right, top = tile.bounds\n if shift:\n # shift by half of the grid width\n left += pyramid.x_size / 2\n right += pyramid.x_size / 2\n # if tile is now shifted outside pyramid bounds, move within\n if right > pyramid.right:\n right -= pyramid.x_size\n left -= pyramid.x_size\n m_left = min([left, m_left]) if m_left is not None else left\n m_bottom = min([bottom, m_bottom]) if m_bottom is not None else bottom\n m_right = max([right, m_right]) if m_right is not None else right\n m_top = max([top, m_top]) if m_top is not None else top\n height = int(round((m_top - m_bottom) / resolution))\n width = int(round((m_right - m_left) / resolution))\n # initialize empty mosaic\n mosaic = ma.MaskedArray(\n data=np.full((num_bands, height, width), dtype=dtype, fill_value=nodata),\n mask=np.ones((num_bands, height, width))\n )\n # create Affine\n affine = Affine(resolution, 0, m_left, 0, -resolution, m_top)\n # fill mosaic array with tile data\n for tile, data in tiles:\n data = prepare_array(data, nodata=nodata, dtype=dtype)\n t_left, t_bottom, t_right, t_top = tile.bounds\n if shift:\n t_left += pyramid.x_size / 2\n t_right += pyramid.x_size / 2\n # if tile is now shifted outside pyramid bounds, move within\n if t_right > pyramid.right:\n t_right -= pyramid.x_size\n t_left -= pyramid.x_size\n minrow, maxrow, mincol, maxcol = bounds_to_ranges(\n out_bounds=(t_left, t_bottom, t_right, t_top),\n in_affine=affine,\n in_shape=(height, width)\n )\n mosaic[:, minrow:maxrow, mincol:maxcol] = data\n mosaic.mask[:, minrow:maxrow, mincol:maxcol] = data.mask\n if shift:\n # shift back output mosaic\n affine = Affine(resolution, 0, m_left - pyramid.x_size / 2, 0, -resolution, m_top)\n return ReferencedRaster(\n data=mosaic,\n affine=affine,\n bounds=Bounds(m_left, m_bottom, m_right, m_top),\n crs=tile.crs\n )", "response": "Create a mosaic from a list of tiles."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns range values from geolocated input.", "response": "def bounds_to_ranges(out_bounds=None, in_affine=None, in_shape=None):\n \"\"\"\n Return bounds range values from geolocated input.\n\n Parameters\n ----------\n out_bounds : tuple\n left, bottom, right, top\n in_affine : Affine\n input geolocation\n in_shape : tuple\n input shape\n\n Returns\n -------\n minrow, maxrow, mincol, maxcol\n \"\"\"\n return itertools.chain(\n *from_bounds(\n *out_bounds, transform=in_affine, height=in_shape[-2], width=in_shape[-1]\n ).round_lengths(pixel_precision=0).round_offsets(pixel_precision=0).toranges()\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tiles_to_affine_shape(tiles):\n if not tiles:\n raise TypeError(\"no tiles provided\")\n pixel_size = tiles[0].pixel_x_size\n left, bottom, right, top = (\n min([t.left for t in tiles]),\n min([t.bottom for t in tiles]),\n max([t.right for t in tiles]),\n max([t.top for t in tiles]),\n )\n return (\n Affine(pixel_size, 0, left, 0, -pixel_size, top),\n Shape(\n width=int(round((right - left) / pixel_size, 0)),\n height=int(round((top - bottom) / pixel_size, 0)),\n )\n )", "response": "Converts a list of BufferedTiles into a single Affine and Shape object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _shift_required(tiles):\n if tiles[0][0].tile_pyramid.is_global:\n # get set of tile columns\n tile_cols = sorted(list(set([t[0].col for t in tiles])))\n # if tile columns are an unbroken sequence, tiles are connected and are not\n # passing the Antimeridian\n if tile_cols == list(range(min(tile_cols), max(tile_cols) + 1)):\n return False\n else:\n # look at column gaps and try to determine the smallest distance\n def gen_groups(items):\n \"\"\"Groups tile columns by sequence.\"\"\"\n j = items[0]\n group = [j]\n for i in items[1:]:\n # item is next in expected sequence\n if i == j + 1:\n group.append(i)\n # gap occured, so yield existing group and create new one\n else:\n yield group\n group = [i]\n j = i\n yield group\n\n groups = list(gen_groups(tile_cols))\n # in case there is only one group, don't shift\n if len(groups) == 1:\n return False\n # distance between first column of first group and last column of last group\n normal_distance = groups[-1][-1] - groups[0][0]\n # distance between last column of first group and last column of first group\n # but crossing the antimeridian\n antimeridian_distance = (\n groups[0][-1] + tiles[0][0].tile_pyramid.matrix_width(tiles[0][0].zoom)\n ) - groups[-1][0]\n # return whether distance over antimeridian is shorter\n return antimeridian_distance < normal_distance\n else:\n return False", "response": "Determine if the distance over an antimeridian is shorter than normal distance."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef memory_file(data=None, profile=None):\n memfile = MemoryFile()\n profile.update(width=data.shape[-2], height=data.shape[-1])\n with memfile.open(**profile) as dataset:\n dataset.write(data)\n return memfile", "response": "Returns a MemoryFile instance from input."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef prepare_array(data, masked=True, nodata=0, dtype=\"int16\"):\n # input is iterable\n if isinstance(data, (list, tuple)):\n return _prepare_iterable(data, masked, nodata, dtype)\n\n # special case if a 2D single band is provided\n elif isinstance(data, np.ndarray) and data.ndim == 2:\n data = ma.expand_dims(data, axis=0)\n\n # input is a masked array\n if isinstance(data, ma.MaskedArray):\n return _prepare_masked(data, masked, nodata, dtype)\n\n # input is a NumPy array\n elif isinstance(data, np.ndarray):\n if masked:\n return ma.masked_values(data.astype(dtype, copy=False), nodata, copy=False)\n else:\n return data.astype(dtype, copy=False)\n else:\n raise ValueError(\n \"data must be array, masked array or iterable containing arrays.\"\n )", "response": "Turn input data into a proper array for further usage."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning data bounding box of the current object.", "response": "def bbox(self, out_crs=None):\n \"\"\"\n Return data bounding box.\n\n Parameters\n ----------\n out_crs : ``rasterio.crs.CRS``\n rasterio CRS object (default: CRS of process pyramid)\n\n Returns\n -------\n bounding box : geometry\n Shapely geometry object\n \"\"\"\n out_crs = self.pyramid.crs if out_crs is None else out_crs\n with fiona.open(self.path) as inp:\n inp_crs = CRS(inp.crs)\n bbox = box(*inp.bounds)\n # TODO find a way to get a good segmentize value in bbox source CRS\n return reproject_geometry(bbox, src_crs=inp_crs, dst_crs=out_crs)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef read(self, validity_check=True, **kwargs):\n return [] if self.is_empty() else self._read_from_cache(validity_check)", "response": "Read reprojected and resampled input data."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks if the cache is empty", "response": "def is_empty(self):\n \"\"\"\n Check if there is data within this tile.\n\n Returns\n -------\n is empty : bool\n \"\"\"\n if not self.tile.bbox.intersects(self.vector_file.bbox()):\n return True\n return len(self._read_from_cache(True)) == 0"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef reproject_geometry(\n geometry, src_crs=None, dst_crs=None, error_on_clip=False, validity_check=True,\n antimeridian_cutting=False\n):\n \"\"\"\n Reproject a geometry to target CRS.\n\n Also, clips geometry if it lies outside the destination CRS boundary.\n Supported destination CRSes for clipping: 4326 (WGS84), 3857 (Spherical\n Mercator) and 3035 (ETRS89 / ETRS-LAEA).\n\n Parameters\n ----------\n geometry : ``shapely.geometry``\n src_crs : ``rasterio.crs.CRS`` or EPSG code\n CRS of source data\n dst_crs : ``rasterio.crs.CRS`` or EPSG code\n target CRS\n error_on_clip : bool\n raises a ``RuntimeError`` if a geometry is outside of CRS bounds\n (default: False)\n validity_check : bool\n checks if reprojected geometry is valid and throws ``TopologicalError``\n if invalid (default: True)\n antimeridian_cutting : bool\n cut geometry at Antimeridian; can result in a multipart output geometry\n\n Returns\n -------\n geometry : ``shapely.geometry``\n \"\"\"\n src_crs = _validated_crs(src_crs)\n dst_crs = _validated_crs(dst_crs)\n\n def _reproject_geom(geometry, src_crs, dst_crs):\n if geometry.is_empty:\n return geometry\n else:\n out_geom = to_shape(\n transform_geom(\n src_crs.to_dict(),\n dst_crs.to_dict(),\n mapping(geometry),\n antimeridian_cutting=antimeridian_cutting\n )\n )\n return _repair(out_geom) if validity_check else out_geom\n\n # return repaired geometry if no reprojection needed\n if src_crs == dst_crs or geometry.is_empty:\n return _repair(geometry)\n\n # geometry needs to be clipped to its CRS bounds\n elif (\n dst_crs.is_epsg_code and # just in case for an CRS with EPSG code\n dst_crs.get(\"init\") in CRS_BOUNDS and # if CRS has defined bounds\n dst_crs.get(\"init\") != \"epsg:4326\" # and is not WGS84 (does not need clipping)\n ):\n wgs84_crs = CRS().from_epsg(4326)\n # get dst_crs boundaries\n crs_bbox = box(*CRS_BOUNDS[dst_crs.get(\"init\")])\n # reproject geometry to WGS84\n geometry_4326 = _reproject_geom(geometry, src_crs, wgs84_crs)\n # raise error if geometry has to be clipped\n if error_on_clip and not geometry_4326.within(crs_bbox):\n raise RuntimeError(\"geometry outside target CRS bounds\")\n # clip geometry dst_crs boundaries and return\n return _reproject_geom(crs_bbox.intersection(geometry_4326), wgs84_crs, dst_crs)\n\n # return without clipping if destination CRS does not have defined bounds\n else:\n return _reproject_geom(geometry, src_crs, dst_crs)", "response": "Reproject a geometry to target CRS."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef segmentize_geometry(geometry, segmentize_value):\n if geometry.geom_type != \"Polygon\":\n raise TypeError(\"segmentize geometry type must be Polygon\")\n\n return Polygon(\n LinearRing([\n p\n # pick polygon linestrings\n for l in map(\n lambda x: LineString([x[0], x[1]]),\n zip(geometry.exterior.coords[:-1], geometry.exterior.coords[1:])\n )\n # interpolate additional points in between and don't forget end point\n for p in [\n l.interpolate(segmentize_value * i).coords[0]\n for i in range(int(l.length / segmentize_value))\n ] + [l.coords[1]]\n ])\n )", "response": "Segmentize Polygon outer ring by segmentize value."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nread a window of an input vector dataset.", "response": "def read_vector_window(input_files, tile, validity_check=True):\n \"\"\"\n Read a window of an input vector dataset.\n\n Also clips geometry.\n\n Parameters:\n -----------\n input_file : string\n path to vector file\n tile : ``Tile``\n tile extent to read data from\n validity_check : bool\n checks if reprojected geometry is valid and throws ``RuntimeError`` if\n invalid (default: True)\n\n Returns\n -------\n features : list\n a list of reprojected GeoJSON-like features\n \"\"\"\n if not isinstance(input_files, list):\n input_files = [input_files]\n return [\n feature\n for feature in chain.from_iterable([\n _read_vector_window(path, tile, validity_check=validity_check)\n for path in input_files\n ])\n ]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nwrite vector window to GeoJSON file.", "response": "def write_vector_window(\n in_data=None, out_schema=None, out_tile=None, out_path=None, bucket_resource=None\n):\n \"\"\"\n Write features to GeoJSON file.\n\n Parameters\n ----------\n in_data : features\n out_schema : dictionary\n output schema for fiona\n out_tile : ``BufferedTile``\n tile used for output extent\n out_path : string\n output path for GeoJSON file\n \"\"\"\n # Delete existing file.\n try:\n os.remove(out_path)\n except OSError:\n pass\n\n out_features = []\n for feature in in_data:\n try:\n # clip feature geometry to tile bounding box and append for writing\n # if clipped feature still\n for out_geom in multipart_to_singleparts(\n clean_geometry_type(\n to_shape(feature[\"geometry\"]).intersection(out_tile.bbox),\n out_schema[\"geometry\"]\n )\n ):\n out_features.append({\n \"geometry\": mapping(out_geom),\n \"properties\": feature[\"properties\"]\n })\n except Exception as e:\n logger.warning(\"failed to prepare geometry for writing: %s\", e)\n continue\n\n # write if there are output features\n if out_features:\n\n try:\n if out_path.startswith(\"s3://\"):\n # write data to remote file\n with VectorWindowMemoryFile(\n tile=out_tile,\n features=out_features,\n schema=out_schema,\n driver=\"GeoJSON\"\n ) as memfile:\n logger.debug((out_tile.id, \"upload tile\", out_path))\n bucket_resource.put_object(\n Key=\"/\".join(out_path.split(\"/\")[3:]),\n Body=memfile\n )\n else:\n # write data to local file\n with fiona.open(\n out_path, 'w', schema=out_schema, driver=\"GeoJSON\",\n crs=out_tile.crs.to_dict()\n ) as dst:\n logger.debug((out_tile.id, \"write tile\", out_path))\n dst.writerecords(out_features)\n except Exception as e:\n logger.error(\"error while writing file %s: %s\", out_path, e)\n raise\n\n else:\n logger.debug((out_tile.id, \"nothing to write\", out_path))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a new geometry of a specific type if possible.", "response": "def clean_geometry_type(geometry, target_type, allow_multipart=True):\n \"\"\"\n Return geometry of a specific type if possible.\n\n Filters and splits up GeometryCollection into target types. This is\n necessary when after clipping and/or reprojecting the geometry types from\n source geometries change (i.e. a Polygon becomes a LineString or a\n LineString becomes Point) in some edge cases.\n\n Parameters\n ----------\n geometry : ``shapely.geometry``\n target_type : string\n target geometry type\n allow_multipart : bool\n allow multipart geometries (default: True)\n\n Returns\n -------\n cleaned geometry : ``shapely.geometry``\n returns None if input geometry type differs from target type\n\n Raises\n ------\n GeometryTypeError : if geometry type does not match target_type\n \"\"\"\n multipart_geoms = {\n \"Point\": MultiPoint,\n \"LineString\": MultiLineString,\n \"Polygon\": MultiPolygon,\n \"MultiPoint\": MultiPoint,\n \"MultiLineString\": MultiLineString,\n \"MultiPolygon\": MultiPolygon\n }\n\n if target_type not in multipart_geoms.keys():\n raise TypeError(\"target type is not supported: %s\" % target_type)\n\n if geometry.geom_type == target_type:\n return geometry\n\n elif allow_multipart:\n target_multipart_type = multipart_geoms[target_type]\n if geometry.geom_type == \"GeometryCollection\":\n return target_multipart_type([\n clean_geometry_type(g, target_type, allow_multipart)\n for g in geometry])\n elif any([\n isinstance(geometry, target_multipart_type),\n multipart_geoms[geometry.geom_type] == target_multipart_type\n ]):\n return geometry\n\n raise GeometryTypeError(\n \"geometry type does not match: %s, %s\" % (geometry.geom_type, target_type)\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef multipart_to_singleparts(geom):\n if isinstance(geom, base.BaseGeometry):\n if hasattr(geom, \"geoms\"):\n for subgeom in geom:\n yield subgeom\n else:\n yield geom", "response": "Yield shapely single part geometries if geom is multipart otherwise yield geom."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef execute(\n mp,\n td_resampling=\"nearest\",\n td_matching_method=\"gdal\",\n td_matching_max_zoom=None,\n td_matching_precision=8,\n td_fallback_to_higher_zoom=False,\n clip_pixelbuffer=0,\n **kwargs\n):\n \"\"\"\n Convert and optionally clip input raster data.\n\n Inputs:\n -------\n raster\n singleband or multiband data input\n clip (optional)\n vector data used to clip output\n\n Parameters\n ----------\n td_resampling : str (default: 'nearest')\n Resampling used when reading from TileDirectory.\n td_matching_method : str ('gdal' or 'min') (default: 'gdal')\n gdal: Uses GDAL's standard method. Here, the target resolution is\n calculated by averaging the extent's pixel sizes over both x and y\n axes. This approach returns a zoom level which may not have the\n best quality but will speed up reading significantly.\n min: Returns the zoom level which matches the minimum resolution of the\n extents four corner pixels. This approach returns the zoom level\n with the best possible quality but with low performance. If the\n tile extent is outside of the destination pyramid, a\n TopologicalError will be raised.\n td_matching_max_zoom : int (optional, default: None)\n If set, it will prevent reading from zoom levels above the maximum.\n td_matching_precision : int (default: 8)\n Round resolutions to n digits before comparing.\n td_fallback_to_higher_zoom : bool (default: False)\n In case no data is found at zoom level, try to read data from higher\n zoom levels. Enabling this setting can lead to many IO requests in\n areas with no data.\n clip_pixelbuffer : int\n Use pixelbuffer when clipping output by geometry. (default: 0)\n\n Output\n ------\n np.ndarray\n \"\"\"\n # read clip geometry\n if \"clip\" in mp.params[\"input\"]:\n clip_geom = mp.open(\"clip\").read()\n if not clip_geom:\n logger.debug(\"no clip data over tile\")\n return \"empty\"\n else:\n clip_geom = []\n\n with mp.open(\n \"raster\",\n matching_method=td_matching_method,\n matching_max_zoom=td_matching_max_zoom,\n matching_precision=td_matching_precision,\n fallback_to_higher_zoom=td_fallback_to_higher_zoom,\n resampling=td_resampling\n ) as raster:\n raster_data = raster.read()\n if raster.is_empty() or raster_data[0].mask.all():\n logger.debug(\"raster empty\")\n return \"empty\"\n\n if clip_geom:\n # apply original nodata mask and clip\n clipped = mp.clip(\n np.where(raster_data[0].mask, mp.params[\"output\"].nodata, raster_data),\n clip_geom,\n clip_buffer=clip_pixelbuffer,\n inverted=True\n )\n return np.where(clipped.mask, clipped, mp.params[\"output\"].nodata)\n else:\n return np.where(raster_data[0].mask, mp.params[\"output\"].nodata, raster_data)", "response": "Execute the main function of the base class."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_best_zoom_level(input_file, tile_pyramid_type):\n tile_pyramid = BufferedTilePyramid(tile_pyramid_type)\n with rasterio.open(input_file, \"r\") as src:\n xmin, ymin, xmax, ymax = reproject_geometry(\n segmentize_geometry(\n box(\n src.bounds.left, src.bounds.bottom, src.bounds.right,\n src.bounds.top\n ),\n get_segmentize_value(input_file, tile_pyramid)\n ),\n src_crs=src.crs, dst_crs=tile_pyramid.crs\n ).bounds\n x_dif = xmax - xmin\n y_dif = ymax - ymin\n size = float(src.width + src.height)\n avg_resolution = (\n (x_dif / float(src.width)) * (float(src.width) / size) +\n (y_dif / float(src.height)) * (float(src.height) / size)\n )\n\n for zoom in range(0, 40):\n if tile_pyramid.pixel_x_size(zoom) <= avg_resolution:\n return zoom-1", "response": "Determine the best base zoom level for a raster."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_segmentize_value(input_file=None, tile_pyramid=None):\n with rasterio.open(input_file, \"r\") as input_raster:\n pixelsize = input_raster.transform[0]\n return pixelsize * tile_pyramid.tile_size", "response": "Returns the recommended segmentation value in input file units."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef tile_to_zoom_level(tile, dst_pyramid=None, matching_method=\"gdal\", precision=8):\n def width_height(bounds):\n try:\n l, b, r, t = reproject_geometry(\n box(*bounds), src_crs=tile.crs, dst_crs=dst_pyramid.crs\n ).bounds\n except ValueError:\n raise TopologicalError(\"bounds cannot be translated into target CRS\")\n return r - l, t - b\n\n if tile.tp.crs == dst_pyramid.crs:\n return tile.zoom\n else:\n if matching_method == \"gdal\":\n # use rasterio/GDAL method to calculate default warp target properties\n transform, width, height = calculate_default_transform(\n tile.tp.crs,\n dst_pyramid.crs,\n tile.width,\n tile.height,\n *tile.bounds\n )\n # this is the resolution the tile would have in destination TilePyramid CRS\n tile_resolution = round(transform[0], precision)\n elif matching_method == \"min\":\n # calculate the minimum pixel size from the four tile corner pixels\n l, b, r, t = tile.bounds\n x = tile.pixel_x_size\n y = tile.pixel_y_size\n res = []\n for bounds in [\n (l, t - y, l + x, t), # left top\n (l, b, l + x, b + y), # left bottom\n (r - x, b, r, b + y), # right bottom\n (r - x, t - y, r, t) # right top\n ]:\n try:\n w, h = width_height(bounds)\n res.extend([w, h])\n except TopologicalError:\n logger.debug(\"pixel outside of destination pyramid\")\n if res:\n tile_resolution = round(min(res), precision)\n else:\n raise TopologicalError(\"tile outside of destination pyramid\")\n else:\n raise ValueError(\"invalid method given: %s\", matching_method)\n logger.debug(\n \"we are looking for a zoom level interpolating to %s resolution\",\n tile_resolution\n )\n zoom = 0\n while True:\n td_resolution = round(dst_pyramid.pixel_x_size(zoom), precision)\n if td_resolution <= tile_resolution:\n break\n zoom += 1\n logger.debug(\"target zoom for %s: %s (%s)\", tile_resolution, zoom, td_resolution)\n return zoom", "response": "Given a tile and dst_pyramid returns a new tile with the highest zoom level."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef path_is_remote(path, s3=True):\n prefixes = (\"http://\", \"https://\", \"/vsicurl/\")\n if s3:\n prefixes += (\"s3://\", \"/vsis3/\")\n return path.startswith(prefixes)", "response": "Determines whether a path is remote or local."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef path_exists(path):\n if path.startswith((\"http://\", \"https://\")):\n try:\n urlopen(path).info()\n return True\n except HTTPError as e:\n if e.code == 404:\n return False\n else:\n raise\n elif path.startswith(\"s3://\"):\n bucket = get_boto3_bucket(path.split(\"/\")[2])\n key = \"/\".join(path.split(\"/\")[3:])\n for obj in bucket.objects.filter(Prefix=key):\n if obj.key == key:\n return True\n else:\n return False\n else:\n logger.debug(\"%s exists: %s\", path, os.path.exists(path))\n return os.path.exists(path)", "response": "Check if file exists either remote or local."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns absolute path if path is local.", "response": "def absolute_path(path=None, base_dir=None):\n \"\"\"\n Return absolute path if path is local.\n\n Parameters:\n -----------\n path : path to file\n base_dir : base directory used for absolute path\n\n Returns:\n --------\n absolute path\n \"\"\"\n if path_is_remote(path):\n return path\n else:\n if os.path.isabs(path):\n return path\n else:\n if base_dir is None or not os.path.isabs(base_dir):\n raise TypeError(\"base_dir must be an absolute path.\")\n return os.path.abspath(os.path.join(base_dir, path))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn relative path to file or remote.", "response": "def relative_path(path=None, base_dir=None):\n \"\"\"\n Return relative path if path is local.\n\n Parameters:\n -----------\n path : path to file\n base_dir : directory where path sould be relative to\n\n Returns:\n --------\n relative path\n \"\"\"\n if path_is_remote(path) or not os.path.isabs(path):\n return path\n else:\n return os.path.relpath(path, base_dir)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_json(path, params):\n logger.debug(\"write %s to %s\", params, path)\n if path.startswith(\"s3://\"):\n bucket = get_boto3_bucket(path.split(\"/\")[2])\n key = \"/\".join(path.split(\"/\")[3:])\n logger.debug(\"upload %s\", key)\n bucket.put_object(\n Key=key,\n Body=json.dumps(params, sort_keys=True, indent=4)\n )\n else:\n makedirs(os.path.dirname(path))\n with open(path, 'w') as dst:\n json.dump(params, dst, sort_keys=True, indent=4)", "response": "Write json to local or remote."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read_json(path):\n if path.startswith((\"http://\", \"https://\")):\n try:\n return json.loads(urlopen(path).read().decode())\n except HTTPError:\n raise FileNotFoundError(\"%s not found\", path)\n elif path.startswith(\"s3://\"):\n bucket = get_boto3_bucket(path.split(\"/\")[2])\n key = \"/\".join(path.split(\"/\")[3:])\n for obj in bucket.objects.filter(Prefix=key):\n if obj.key == key:\n return json.loads(obj.get()['Body'].read().decode())\n raise FileNotFoundError(\"%s not found\", path)\n else:\n try:\n with open(path, \"r\") as src:\n return json.loads(src.read())\n except:\n raise FileNotFoundError(\"%s not found\", path)", "response": "Read local or remote."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns a merged set of GDAL options.", "response": "def get_gdal_options(opts, is_remote=False):\n \"\"\"\n Return a merged set of custom and default GDAL/rasterio Env options.\n\n If is_remote is set to True, the default GDAL_HTTP_OPTS are appended.\n\n Parameters\n ----------\n opts : dict or None\n Explicit GDAL options.\n is_remote : bool\n Indicate whether Env is for a remote file.\n\n Returns\n -------\n dictionary\n \"\"\"\n user_opts = {} if opts is None else dict(**opts)\n if is_remote:\n return dict(GDAL_HTTP_OPTS, **user_opts)\n else:\n return user_opts"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nopens a new input tile.", "response": "def open(self, tile, **kwargs):\n \"\"\"\n Return InputTile object.\n\n Parameters\n ----------\n tile : ``Tile``\n\n Returns\n -------\n input tile : ``InputTile``\n tile view of input data\n \"\"\"\n return self.process.config.output.open(tile, self.process, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn data bounding box.", "response": "def bbox(self, out_crs=None):\n \"\"\"\n Return data bounding box.\n\n Parameters\n ----------\n out_crs : ``rasterio.crs.CRS``\n rasterio CRS object (default: CRS of process pyramid)\n\n Returns\n -------\n bounding box : geometry\n Shapely geometry object\n \"\"\"\n return reproject_geometry(\n self.process.config.area_at_zoom(),\n src_crs=self.process.config.process_pyramid.crs,\n dst_crs=self.pyramid.crs if out_crs is None else out_crs\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef win_activate(title, **kwargs):\n text = kwargs.get(\"text\", \"\")\n ret = AUTO_IT.AU3_WinActivate(LPCWSTR(title), LPCWSTR(text))\n return ret", "response": "Activates (gives focus to) a window.\n :param title:\n :param text:\n :return:"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef win_exists(title, **kwargs):\n text = kwargs.get(\"text\", \"\")\n ret = AUTO_IT.AU3_WinExists(LPCWSTR(title), LPCWSTR(text))\n return ret", "response": "Checks to see if a specified window exists."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef win_get_caret_pos():\n p = POINT()\n AUTO_IT.AU3_WinGetCaretPos(byref(p))\n return p.x, p.y", "response": "Returns the coordinates of the caret in the foreground window\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef win_get_state(title, **kwargs):\n text = kwargs.get(\"text\", \"\")\n res = AUTO_IT.AU3_WinGetState(LPCWSTR(title), LPCWSTR(text))\n return res", "response": "Retrieves the state of a given window."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef win_menu_select_item(title, *items, **kwargs):\n text = kwargs.get(\"text\", \"\")\n if not (0 < len(items) < 8):\n raise ValueError(\"accepted none item or number of items exceed eight\")\n f_items = [LPCWSTR(item) for item in items]\n for i in xrange(8 - len(f_items)):\n f_items.append(LPCWSTR(\"\"))\n\n ret = AUTO_IT.AU3_WinMenuSelectItem(LPCWSTR(title), LPCWSTR(text),\n *f_items)\n return ret", "response": "This function will select one or more items from the menu."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef win_set_trans(title, trans, **kwargs):\n text = kwargs.get(\"text\", \"\")\n\n ret = AUTO_IT.AU3_WinSetTrans(LPCWSTR(title), LPCWSTR(text), INT(trans))\n return ret", "response": "Sets the transparency of a window."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef auto_it_set_option(option, param):\n pre_value = AUTO_IT.AU3_AutoItSetOption(LPCWSTR(option), INT(param))\n return pre_value", "response": "Changes the operation of various AutoIt functions and parameters\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchanges the priority of a process.", "response": "def process_set_priority(process, priority):\n \"\"\"\n Changes the priority of a process\n :param process: The name or PID of the process to check.\n :param priority:A flag which determines what priority to set\n 0 - Idle/Low\n 1 - Below Normal (Not supported on Windows 95/98/ME)\n 2 - Normal\n 3 - Above Normal (Not supported on Windows 95/98/ME)\n 4 - High\n 5 - Realtime (Use with caution, may make the system unstable)\n :return:\n \"\"\"\n ret = AUTO_IT.AU3_ProcessSetPriority(LPCWSTR(process), INT(priority))\n if ret == 0:\n if error() == 1:\n raise AutoItError(\"set priority failed\")\n elif error() == 2:\n raise AutoItError(\"unsupported priority class be used\")\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef process_wait(process, timeout=0):\n ret = AUTO_IT.AU3_ProcessWait(LPCWSTR(process), INT(timeout))\n return ret", "response": "Waits until a given process exists."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npause script execution until a given process does not exist.", "response": "def process_wait_close(process, timeout=0):\n \"\"\"\n Pauses script execution until a given process does not exist.\n :param process:\n :param timeout:\n :return:\n \"\"\"\n ret = AUTO_IT.AU3_ProcessWaitClose(LPCWSTR(process), INT(timeout))\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef run_as(user, domain, password, filename, logon_flag=1, work_dir=\"\",\n show_flag=Properties.SW_SHOWNORMAL):\n \"\"\"\n Runs an external program.\n :param user: username The user name to use.\n :param domain: The domain name to use.\n :param password: The password to use.\n :param logon_flag: 0 = do not load the user profile, 1 = (default) load\n the user profile, 2 = use for net credentials only\n :param filename: The name of the executable (EXE, BAT, COM, or PIF) to run.\n :param work_dir: The working directory.\n :param show_flag: The \"show\" flag of the executed program:\n SW_HIDE = Hidden window\n SW_MINIMIZE = Minimized window\n SW_MAXIMIZE = Maximized window\n :return:\n \"\"\"\n ret = AUTO_IT.AU3_RunAs(\n LPCWSTR(user), LPCWSTR(domain), LPCWSTR(password), INT(logon_flag),\n LPCWSTR(filename), LPCWSTR(work_dir), INT(show_flag)\n )\n return ret", "response": "Runs an external program."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrunning an external program as a wait.", "response": "def run_as_wait(user, domain, password, filename, logon_flag=1, work_dir=\"\",\n show_flag=Properties.SW_SHOWNORMAL):\n \"\"\"\n Runs an external program.\n :param user: username The user name to use.\n :param domain: The domain name to use.\n :param password: The password to use.\n :param logon_flag: 0 = do not load the user profile, 1 = (default) load\n the user profile, 2 = use for net credentials only\n :param filename: The name of the executable (EXE, BAT, COM, or PIF) to run.\n :param work_dir: The working directory.\n :param show_flag: The \"show\" flag of the executed program:\n SW_HIDE = Hidden window\n SW_MINIMIZE = Minimized window\n SW_MAXIMIZE = Maximized window\n :return:\n \"\"\"\n ret = AUTO_IT.AU3_RunAsWait(\n LPCWSTR(user), LPCWSTR(domain), LPCWSTR(password), INT(logon_flag),\n LPCWSTR(filename), LPCWSTR(work_dir), INT(show_flag)\n )\n return ret"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef hook(self, event_type='push'):\n\n def decorator(func):\n self._hooks[event_type].append(func)\n return func\n\n return decorator", "response": "Decorator for registering a function as a hook."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _get_digest(self):\n\n return hmac.new(\n self._secret, request.data, hashlib.sha1).hexdigest() if self._secret else None", "response": "Return message digest if a secret key was provided"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating rst document for PyPi.", "response": "def long_description():\n \"\"\"Generate .rst document for PyPi.\"\"\"\n import argparse\n parser = argparse.ArgumentParser()\n parser.add_argument('--doc', dest=\"doc\",\n action=\"store_true\", default=False)\n args, sys.argv = parser.parse_known_args(sys.argv)\n if args.doc:\n import doc2md, pypandoc\n md = doc2md.doc2md(doc2md.__doc__, \"doc2md\", toc=False)\n long_description = pypandoc.convert(md, 'rst', format='md')\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving common indentation from string.", "response": "def unindent(lines):\n \"\"\"\n Remove common indentation from string.\n\n Unlike doctrim there is no special treatment of the first line.\n\n \"\"\"\n try:\n # Determine minimum indentation:\n indent = min(len(line) - len(line.lstrip())\n for line in lines if line)\n except ValueError:\n return lines\n else:\n return [line[indent:] for line in lines]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfinds all section names and return a list with their names.", "response": "def find_sections(lines):\n \"\"\"\n Find all section names and return a list with their names.\n \"\"\"\n sections = []\n for line in lines:\n if is_heading(line):\n sections.append(get_heading(line))\n return sections"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating table of contents for array of section names.", "response": "def make_toc(sections, maxdepth=0):\n \"\"\"\n Generate table of contents for array of section names.\n \"\"\"\n if not sections:\n return []\n outer = min(n for n,t in sections)\n refs = []\n for ind,sec in sections:\n if maxdepth and ind-outer+1 > maxdepth:\n continue\n ref = sec.lower()\n ref = ref.replace('`', '')\n ref = ref.replace(' ', '-')\n ref = ref.replace('?', '')\n refs.append(\" \"*(ind-outer) + \"- [%s](#%s)\" % (sec, ref))\n return refs"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef doc2md(docstr, title, min_level=1, more_info=False, toc=True, maxdepth=0):\n text = doctrim(docstr)\n lines = text.split('\\n')\n\n sections = find_sections(lines)\n if sections:\n level = min(n for n,t in sections) - 1\n else:\n level = 1\n\n shiftlevel = 0\n if level < min_level:\n shiftlevel = min_level - level\n level = min_level\n sections = [(lev+shiftlevel, tit) for lev,tit in sections]\n\n head = next((i for i, l in enumerate(lines) if is_heading(l)), 0)\n md = [\n make_heading(level, title),\n \"\",\n ] + lines[:head]\n if toc:\n md += make_toc(sections, maxdepth)\n md += ['']\n md += _doc2md(lines[head:], shiftlevel)\n if more_info:\n return (md, sections)\n else:\n return \"\\n\".join(md)", "response": "Convert a docstring to a markdown text."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef mod2md(module, title, title_api_section, toc=True, maxdepth=0):\n docstr = module.__doc__\n\n text = doctrim(docstr)\n lines = text.split('\\n')\n\n sections = find_sections(lines)\n if sections:\n level = min(n for n,t in sections) - 1\n else:\n level = 1\n\n api_md = []\n api_sec = []\n if title_api_section and module.__all__:\n sections.append((level+1, title_api_section))\n for name in module.__all__:\n api_sec.append((level+2, \"`\" + name + \"`\"))\n api_md += ['', '']\n entry = module.__dict__[name]\n if entry.__doc__:\n md, sec = doc2md(entry.__doc__, \"`\" + name + \"`\",\n min_level=level+2, more_info=True, toc=False)\n api_sec += sec\n api_md += md\n\n sections += api_sec\n\n # headline\n head = next((i for i, l in enumerate(lines) if is_heading(l)), 0)\n md = [\n make_heading(level, title),\n \"\",\n ] + lines[:head]\n\n # main sections\n if toc:\n md += make_toc(sections, maxdepth)\n md += ['']\n md += _doc2md(lines[head:])\n\n # API section\n md += [\n '',\n '',\n make_heading(level+1, title_api_section),\n ]\n if toc:\n md += ['']\n md += make_toc(api_sec, 1)\n md += api_md\n\n return \"\\n\".join(md)", "response": "Generate markdown document from module including API section."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef largest_finite_distance(self):\n block_start_distances = [block.distance_start for block in self._profile_blocks if\n block.distance_start < float('inf')]\n block_end_distances = [block.distance_end for block in self._profile_blocks if\n block.distance_end < float('inf')]\n distances = block_start_distances + block_end_distances\n if len(distances) > 0:\n return max(distances)\n else:\n return None", "response": "Compute the maximum finite distance."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _temporal_distance_cdf(self):\n distance_split_points = set()\n for block in self._profile_blocks:\n if block.distance_start != float('inf'):\n distance_split_points.add(block.distance_end)\n distance_split_points.add(block.distance_start)\n\n distance_split_points_ordered = numpy.array(sorted(list(distance_split_points)))\n temporal_distance_split_widths = distance_split_points_ordered[1:] - distance_split_points_ordered[:-1]\n trip_counts = numpy.zeros(len(temporal_distance_split_widths))\n delta_peaks = defaultdict(lambda: 0)\n\n for block in self._profile_blocks:\n if block.distance_start == block.distance_end:\n delta_peaks[block.distance_end] += block.width()\n else:\n start_index = numpy.searchsorted(distance_split_points_ordered, block.distance_end)\n end_index = numpy.searchsorted(distance_split_points_ordered, block.distance_start)\n trip_counts[start_index:end_index] += 1\n\n unnormalized_cdf = numpy.array([0] + list(numpy.cumsum(temporal_distance_split_widths * trip_counts)))\n if not (numpy.isclose(\n [unnormalized_cdf[-1]],\n [self._end_time - self._start_time - sum(delta_peaks.values())], atol=1E-4\n ).all()):\n print(unnormalized_cdf[-1], self._end_time - self._start_time - sum(delta_peaks.values()))\n raise RuntimeError(\"Something went wrong with cdf computation!\")\n\n if len(delta_peaks) > 0:\n for peak in delta_peaks.keys():\n if peak == float('inf'):\n continue\n index = numpy.nonzero(distance_split_points_ordered == peak)[0][0]\n unnormalized_cdf = numpy.insert(unnormalized_cdf, index, unnormalized_cdf[index])\n distance_split_points_ordered = numpy.insert(distance_split_points_ordered, index,\n distance_split_points_ordered[index])\n # walk_waiting_time_fraction = walk_total_time / (self.end_time_dep - self.start_time_dep)\n unnormalized_cdf[(index + 1):] = unnormalized_cdf[(index + 1):] + delta_peaks[peak]\n\n norm_cdf = unnormalized_cdf / (unnormalized_cdf[-1] + delta_peaks[float('inf')])\n return distance_split_points_ordered, norm_cdf", "response": "Computes the temporal distance cumulative density function for the current time - series of time - series."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _temporal_distance_pdf(self):\n temporal_distance_split_points_ordered, norm_cdf = self._temporal_distance_cdf()\n delta_peak_loc_to_probability_mass = {}\n\n non_delta_peak_split_points = [temporal_distance_split_points_ordered[0]]\n non_delta_peak_densities = []\n for i in range(0, len(temporal_distance_split_points_ordered) - 1):\n left = temporal_distance_split_points_ordered[i]\n right = temporal_distance_split_points_ordered[i + 1]\n width = right - left\n prob_mass = norm_cdf[i + 1] - norm_cdf[i]\n if width == 0.0:\n delta_peak_loc_to_probability_mass[left] = prob_mass\n else:\n non_delta_peak_split_points.append(right)\n non_delta_peak_densities.append(prob_mass / float(width))\n assert (len(non_delta_peak_densities) == len(non_delta_peak_split_points) - 1)\n return numpy.array(non_delta_peak_split_points), \\\n numpy.array(non_delta_peak_densities), delta_peak_loc_to_probability_mass", "response": "Returns the temporal distance probability density function."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves all trips that are fully outside the buffer.", "response": "def remove_all_trips_fully_outside_buffer(db_conn, center_lat, center_lon, buffer_km, update_secondary_data=True):\n \"\"\"\n Not used in the regular filter process for the time being.\n\n Parameters\n ----------\n db_conn: sqlite3.Connection\n connection to the GTFS object\n center_lat: float\n center_lon: float\n buffer_km: float\n \"\"\"\n distance_function_str = add_wgs84_distance_function_to_db(db_conn)\n stops_within_buffer_query_sql = \"SELECT stop_I FROM stops WHERE CAST(\" + distance_function_str + \\\n \"(lat, lon, {lat} , {lon}) AS INT) < {d_m}\"\\\n .format(lat=float(center_lat), lon=float(center_lon), d_m=int(1000*buffer_km))\n select_all_trip_Is_where_stop_I_is_within_buffer_sql = \"SELECT distinct(trip_I) FROM stop_times WHERE stop_I IN (\" + stops_within_buffer_query_sql + \")\"\n trip_Is_to_remove_sql = \"SELECT trip_I FROM trips WHERE trip_I NOT IN ( \" + select_all_trip_Is_where_stop_I_is_within_buffer_sql + \")\"\n trip_Is_to_remove = pandas.read_sql(trip_Is_to_remove_sql, db_conn)[\"trip_I\"].values\n trip_Is_to_remove_string = \",\".join([str(trip_I) for trip_I in trip_Is_to_remove])\n remove_all_trips_fully_outside_buffer_sql = \"DELETE FROM trips WHERE trip_I IN (\" + trip_Is_to_remove_string + \")\"\n remove_all_stop_times_where_trip_I_fully_outside_buffer_sql = \"DELETE FROM stop_times WHERE trip_I IN (\" + trip_Is_to_remove_string + \")\"\n db_conn.execute(remove_all_trips_fully_outside_buffer_sql)\n db_conn.execute(remove_all_stop_times_where_trip_I_fully_outside_buffer_sql)\n delete_stops_not_in_stop_times_and_not_as_parent_stop(db_conn)\n db_conn.execute(DELETE_ROUTES_NOT_PRESENT_IN_TRIPS_SQL)\n db_conn.execute(DELETE_SHAPES_NOT_REFERENCED_IN_TRIPS_SQL)\n db_conn.execute(DELETE_DAYS_ENTRIES_NOT_PRESENT_IN_TRIPS_SQL)\n db_conn.execute(DELETE_DAY_TRIPS2_ENTRIES_NOT_PRESENT_IN_TRIPS_SQL)\n db_conn.execute(DELETE_CALENDAR_ENTRIES_FOR_NON_REFERENCE_SERVICE_IS_SQL)\n db_conn.execute(DELETE_CALENDAR_DATES_ENTRIES_FOR_NON_REFERENCE_SERVICE_IS_SQL)\n db_conn.execute(DELETE_FREQUENCIES_ENTRIES_NOT_PRESENT_IN_TRIPS)\n db_conn.execute(DELETE_AGENCIES_NOT_REFERENCED_IN_ROUTES_SQL)\n if update_secondary_data:\n update_secondary_data_copies(db_conn)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nremoves dangling entries from the shapes directory.", "response": "def remove_dangling_shapes(db_conn):\n \"\"\"\n Remove dangling entries from the shapes directory.\n\n Parameters\n ----------\n db_conn: sqlite3.Connection\n connection to the GTFS object\n \"\"\"\n db_conn.execute(DELETE_SHAPES_NOT_REFERENCED_IN_TRIPS_SQL)\n SELECT_MIN_MAX_SHAPE_BREAKS_BY_TRIP_I_SQL = \\\n \"SELECT trips.trip_I, shape_id, min(shape_break) as min_shape_break, max(shape_break) as max_shape_break FROM trips, stop_times WHERE trips.trip_I=stop_times.trip_I GROUP BY trips.trip_I\"\n trip_min_max_shape_seqs= pandas.read_sql(SELECT_MIN_MAX_SHAPE_BREAKS_BY_TRIP_I_SQL, db_conn)\n\n rows = []\n for row in trip_min_max_shape_seqs.itertuples():\n shape_id, min_shape_break, max_shape_break = row.shape_id, row.min_shape_break, row.max_shape_break\n if min_shape_break is None or max_shape_break is None:\n min_shape_break = float('-inf')\n max_shape_break = float('-inf')\n rows.append( (shape_id, min_shape_break, max_shape_break) )\n DELETE_SQL_BASE = \"DELETE FROM shapes WHERE shape_id=? AND (seq?)\"\n db_conn.executemany(DELETE_SQL_BASE, rows)\n remove_dangling_shapes_references(db_conn)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _delete_rows_by_start_and_end_date(self):\n # filter by start_time_ut and end_date_ut:\n if (self.start_date is not None) and (self.end_date is not None):\n start_date_ut = self.gtfs.get_day_start_ut(self.start_date)\n end_date_ut = self.gtfs.get_day_start_ut(self.end_date)\n if self.copy_db_conn.execute(\"SELECT count(*) FROM day_trips2 WHERE start_time_ut IS null \"\n \"OR end_time_ut IS null\").fetchone() != (0,):\n raise ValueError(\"Missing information in day_trips2 (start_time_ut and/or end_time_ut), \"\n \"check trips.start_time_ds and trips.end_time_ds.\")\n logging.info(\"Filtering based on start_time_ut and end_time_ut\")\n table_to_preserve_map = {\n \"calendar\": \"start_date < date({filter_end_ut}, 'unixepoch', 'localtime') \"\n \"AND \"\n \"end_date >= date({filter_start_ut}, 'unixepoch', 'localtime') \",\n \"calendar_dates\": \"date >= date({filter_start_ut}, 'unixepoch', 'localtime') \"\n \"AND \"\n \"date < date({filter_end_ut}, 'unixepoch', 'localtime') \",\n \"day_trips2\": 'start_time_ut < {filter_end_ut} '\n 'AND '\n 'end_time_ut > {filter_start_ut} ',\n \"days\": \"day_start_ut >= {filter_start_ut} \"\n \"AND \"\n \"day_start_ut < {filter_end_ut} \"\n }\n table_to_remove_map = {key: \"WHERE NOT ( \" + to_preserve + \" );\"\n for key, to_preserve in table_to_preserve_map.items() }\n\n # Ensure that process timezone is correct as we rely on 'localtime' in the SQL statements.\n GTFS(self.copy_db_conn).set_current_process_time_zone()\n # remove the 'source' entries from tables\n for table, query_template in table_to_remove_map.items():\n param_dict = {\"filter_start_ut\": str(start_date_ut),\n \"filter_end_ut\": str(end_date_ut)}\n query = \"DELETE FROM \" + table + \" \" + \\\n query_template.format(**param_dict)\n self.copy_db_conn.execute(query)\n self.copy_db_conn.commit()\n\n return FILTERED\n else:\n return NOT_FILTERED", "response": "This method deletes rows from the sqlite database copy that are out of the time span defined by start_date and end_date."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfiltering the calendar table s services by start and end dates.", "response": "def _filter_by_calendar(self):\n \"\"\"\n update calendar table's services\n :param copy_db_conn:\n :param start_date:\n :param end_date:\n :return:\n \"\"\"\n if (self.start_date is not None) and (self.end_date is not None):\n logging.info(\"Making date extract\")\n\n start_date_query = \"UPDATE calendar \" \\\n \"SET start_date='{start_date}' \" \\\n \"WHERE start_date<'{start_date}' \".format(start_date=self.start_date)\n self.copy_db_conn.execute(start_date_query)\n\n end_date_query = \"UPDATE calendar \" \\\n \"SET end_date='{end_date_to_include}' \" \\\n \"WHERE end_date>'{end_date_to_include}' \" \\\n .format(end_date_to_include=self.end_date_to_include_str)\n self.copy_db_conn.execute(end_date_query)\n\n # then recursively delete further data:\n self.copy_db_conn.execute(DELETE_TRIPS_NOT_IN_DAYS_SQL)\n self.copy_db_conn.execute(DELETE_SHAPES_NOT_REFERENCED_IN_TRIPS_SQL)\n self.copy_db_conn.execute(DELETE_STOP_TIMES_NOT_REFERENCED_IN_TRIPS_SQL)\n delete_stops_not_in_stop_times_and_not_as_parent_stop(self.copy_db_conn)\n self.copy_db_conn.execute(DELETE_STOP_DISTANCE_ENTRIES_WITH_NONEXISTENT_STOPS_SQL)\n self.copy_db_conn.execute(DELETE_ROUTES_NOT_PRESENT_IN_TRIPS_SQL)\n self.copy_db_conn.execute(DELETE_AGENCIES_NOT_REFERENCED_IN_ROUTES_SQL)\n self.copy_db_conn.commit()\n return FILTERED\n else:\n return NOT_FILTERED"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nfilter by agency ids", "response": "def _filter_by_agency(self):\n \"\"\"\n filter by agency ids\n :param copy_db_conn:\n :param agency_ids_to_preserve:\n :return:\n \"\"\"\n if self.agency_ids_to_preserve is not None:\n logging.info(\"Filtering based on agency_ids\")\n agency_ids_to_preserve = list(self.agency_ids_to_preserve)\n agencies = pandas.read_sql(\"SELECT * FROM agencies\", self.copy_db_conn)\n agencies_to_remove = []\n for idx, row in agencies.iterrows():\n if row['agency_id'] not in agency_ids_to_preserve:\n agencies_to_remove.append(row['agency_id'])\n for agency_id in agencies_to_remove:\n self.copy_db_conn.execute('DELETE FROM agencies WHERE agency_id=?', (agency_id,))\n # and remove recursively related to the agencies:\n self.copy_db_conn.execute('DELETE FROM routes WHERE '\n 'agency_I NOT IN (SELECT agency_I FROM agencies)')\n self.copy_db_conn.execute('DELETE FROM trips WHERE '\n 'route_I NOT IN (SELECT route_I FROM routes)')\n self.copy_db_conn.execute('DELETE FROM calendar WHERE '\n 'service_I NOT IN (SELECT service_I FROM trips)')\n self.copy_db_conn.execute('DELETE FROM calendar_dates WHERE '\n 'service_I NOT IN (SELECT service_I FROM trips)')\n self.copy_db_conn.execute('DELETE FROM days WHERE '\n 'trip_I NOT IN (SELECT trip_I FROM trips)')\n self.copy_db_conn.execute('DELETE FROM stop_times WHERE '\n 'trip_I NOT IN (SELECT trip_I FROM trips)')\n self.copy_db_conn.execute('DELETE FROM stop_times WHERE '\n 'trip_I NOT IN (SELECT trip_I FROM trips)')\n self.copy_db_conn.execute('DELETE FROM shapes WHERE '\n 'shape_id NOT IN (SELECT shape_id FROM trips)')\n self.copy_db_conn.execute('DELETE FROM day_trips2 WHERE '\n 'trip_I NOT IN (SELECT trip_I FROM trips)')\n self.copy_db_conn.commit()\n return FILTERED\n else:\n return NOT_FILTERED"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfilter the feed based on the distance of the buffer and the number of intermediate stops that are within the buffer.", "response": "def _filter_spatially(self):\n \"\"\"\n Filter the feed based on self.buffer_distance_km from self.buffer_lon and self.buffer_lat.\n\n 1. First include all stops that are within self.buffer_distance_km from self.buffer_lon and self.buffer_lat.\n 2. Then include all intermediate stops that are between any of the included stop pairs with some PT trip.\n 3. Repeat step 2 until no more stops are to be included.\n\n As a summary this process should get rid of PT network tendrils, but should preserve the PT network intact\n at its core.\n \"\"\"\n if self.buffer_lat is None or self.buffer_lon is None or self.buffer_distance_km is None:\n return NOT_FILTERED\n\n print(\"filtering with lat: \" + str(self.buffer_lat) +\n \" lon: \" + str(self.buffer_lon) +\n \" buffer distance: \" + str(self.buffer_distance_km))\n remove_all_trips_fully_outside_buffer(self.copy_db_conn,\n self.buffer_lat,\n self.buffer_lon,\n self.buffer_distance_km,\n update_secondary_data=False)\n logging.info(\"Making spatial extract\")\n\n find_distance_func_name = add_wgs84_distance_function_to_db(self.copy_db_conn)\n assert find_distance_func_name == \"find_distance\"\n\n # select all stops that are within the buffer and have some stop_times assigned.\n stop_distance_filter_sql_base = (\n \"SELECT DISTINCT stops.stop_I FROM stops, stop_times\" +\n \" WHERE CAST(find_distance(lat, lon, {buffer_lat}, {buffer_lon}) AS INT) < {buffer_distance_meters}\" +\n \" AND stops.stop_I=stop_times.stop_I\"\n )\n stops_within_buffer_sql = stop_distance_filter_sql_base.format(\n buffer_lat=float(self.buffer_lat),\n buffer_lon=float(self.buffer_lon),\n buffer_distance_meters=int(self.buffer_distance_km * 1000)\n )\n stops_within_buffer = set(row[0] for row in self.copy_db_conn.execute(stops_within_buffer_sql))\n\n # For each trip_I, find smallest (min_seq) and largest (max_seq) stop sequence numbers that\n # are within the soft buffer_distance from the buffer_lon and buffer_lat, and add them into the\n # list of stops to preserve.\n # Note that if a trip is OUT-IN-OUT-IN-OUT, this process preserves (at least) the part IN-OUT-IN of the trip.\n # Repeat until no more stops are found.\n\n stops_within_buffer_string = \"(\" +\",\".join(str(stop_I) for stop_I in stops_within_buffer) + \")\"\n trip_min_max_include_seq_sql = (\n 'SELECT trip_I, min(seq) AS min_seq, max(seq) AS max_seq FROM stop_times, stops '\n 'WHERE stop_times.stop_I = stops.stop_I '\n ' AND stops.stop_I IN {stop_I_list}'\n ' GROUP BY trip_I'\n ).format(stop_I_list=stops_within_buffer_string)\n trip_I_min_seq_max_seq_df = pandas.read_sql(trip_min_max_include_seq_sql, self.copy_db_conn)\n\n for trip_I_seq_row in trip_I_min_seq_max_seq_df.itertuples():\n trip_I = trip_I_seq_row.trip_I\n min_seq = trip_I_seq_row.min_seq\n max_seq = trip_I_seq_row.max_seq\n # DELETE FROM STOP_TIMES\n if min_seq == max_seq:\n # Only one entry in stop_times to be left, remove whole trip.\n self.copy_db_conn.execute(\"DELETE FROM stop_times WHERE trip_I={trip_I}\".format(trip_I=trip_I))\n self.copy_db_conn.execute(\"DELETE FROM trips WHERE trip_i={trip_I}\".format(trip_I=trip_I))\n else:\n # DELETE STOP_TIME ENTRIES BEFORE ENTERING AND AFTER DEPARTING THE BUFFER AREA\n DELETE_STOP_TIME_ENTRIES_SQL = \\\n \"DELETE FROM stop_times WHERE trip_I={trip_I} AND (seq<{min_seq} OR seq>{max_seq})\"\\\n .format(trip_I=trip_I, max_seq=max_seq, min_seq=min_seq)\n self.copy_db_conn.execute(DELETE_STOP_TIME_ENTRIES_SQL)\n\n STOPS_NOT_WITHIN_BUFFER__FOR_TRIP_SQL = \\\n \"SELECT seq, stop_I IN {stops_within_hard_buffer} AS within FROM stop_times WHERE trip_I={trip_I} ORDER BY seq\"\\\n .format(stops_within_hard_buffer=stops_within_buffer_string, trip_I=trip_I)\n stop_times_within_buffer_df = pandas.read_sql(STOPS_NOT_WITHIN_BUFFER__FOR_TRIP_SQL, self.copy_db_conn)\n if stop_times_within_buffer_df['within'].all():\n continue\n else:\n _split_trip(self.copy_db_conn, trip_I, stop_times_within_buffer_df)\n\n\n # Delete all shapes that are not fully within the buffer to avoid shapes going outside\n # the buffer area in a some cases.\n # This could probably be done in some more sophisticated way though (per trip)\n SHAPE_IDS_NOT_WITHIN_BUFFER_SQL = \\\n \"SELECT DISTINCT shape_id FROM SHAPES \" \\\n \"WHERE CAST(find_distance(lat, lon, {buffer_lat}, {buffer_lon}) AS INT) > {buffer_distance_meters}\" \\\n .format(buffer_lat=self.buffer_lat,\n buffer_lon=self.buffer_lon,\n buffer_distance_meters=self.buffer_distance_km * 1000)\n DELETE_ALL_SHAPE_IDS_NOT_WITHIN_BUFFER_SQL = \"DELETE FROM shapes WHERE shape_id IN (\" \\\n + SHAPE_IDS_NOT_WITHIN_BUFFER_SQL + \")\"\n self.copy_db_conn.execute(DELETE_ALL_SHAPE_IDS_NOT_WITHIN_BUFFER_SQL)\n SET_SHAPE_ID_TO_NULL_FOR_HARD_BUFFER_FILTERED_SHAPE_IDS = \\\n \"UPDATE trips SET shape_id=NULL WHERE trips.shape_id IN (\" + SHAPE_IDS_NOT_WITHIN_BUFFER_SQL + \")\"\n self.copy_db_conn.execute(SET_SHAPE_ID_TO_NULL_FOR_HARD_BUFFER_FILTERED_SHAPE_IDS)\n\n\n # Delete trips with only one stop\n self.copy_db_conn.execute('DELETE FROM stop_times WHERE '\n 'trip_I IN (SELECT trip_I FROM '\n '(SELECT trip_I, count(*) AS N_stops from stop_times '\n 'GROUP BY trip_I) q1 '\n 'WHERE N_stops = 1)')\n\n # Delete trips with only one stop but several instances in stop_times\n self.copy_db_conn.execute('DELETE FROM stop_times WHERE '\n 'trip_I IN (SELECT q1.trip_I AS trip_I FROM '\n '(SELECT trip_I, stop_I, count(*) AS stops_per_stop FROM stop_times '\n 'GROUP BY trip_I, stop_I) q1, '\n '(SELECT trip_I, count(*) as n_stops FROM stop_times '\n 'GROUP BY trip_I) q2 '\n 'WHERE q1.trip_I = q2.trip_I AND n_stops = stops_per_stop)')\n\n # Delete all stop_times for uncovered stops\n delete_stops_not_in_stop_times_and_not_as_parent_stop(self.copy_db_conn)\n # Consecutively delete all the rest remaining.\n self.copy_db_conn.execute(DELETE_TRIPS_NOT_REFERENCED_IN_STOP_TIMES)\n self.copy_db_conn.execute(DELETE_ROUTES_NOT_PRESENT_IN_TRIPS_SQL)\n self.copy_db_conn.execute(DELETE_AGENCIES_NOT_REFERENCED_IN_ROUTES_SQL)\n self.copy_db_conn.execute(DELETE_SHAPES_NOT_REFERENCED_IN_TRIPS_SQL)\n self.copy_db_conn.execute(DELETE_STOP_DISTANCE_ENTRIES_WITH_NONEXISTENT_STOPS_SQL)\n self.copy_db_conn.execute(DELETE_FREQUENCIES_ENTRIES_NOT_PRESENT_IN_TRIPS)\n remove_dangling_shapes(self.copy_db_conn)\n self.copy_db_conn.commit()\n return FILTERED"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngives a set of transit events and a static walking network compute the pseudo - connections for each event in the transit network.", "response": "def compute_pseudo_connections(transit_connections, start_time_dep,\n end_time_dep, transfer_margin,\n walk_network, walk_speed):\n \"\"\"\n Given a set of transit events and the static walk network,\n \"transform\" the static walking network into a set of \"pseudo-connections\".\n\n As a first approximation, we add pseudo-connections to depart after each arrival of a transit connection\n to it's arrival stop.\n\n Parameters\n ----------\n transit_connections: list[Connection]\n start_time_dep : int\n start time in unixtime seconds\n end_time_dep: int\n end time in unixtime seconds (no new connections will be scanned after this time)\n transfer_margin: int\n required extra margin required for transfers in seconds\n walk_speed: float\n walking speed between stops in meters / second\n walk_network: networkx.Graph\n each edge should have the walking distance as a data attribute (\"d_walk\") expressed in meters\n\n Returns\n -------\n pseudo_connections: set[Connection]\n \"\"\"\n # A pseudo-connection should be created after (each) arrival to a transit_connection's arrival stop.\n pseudo_connection_set = set() # use a set to ignore possible duplicates\n for c in transit_connections:\n if start_time_dep <= c.departure_time <= end_time_dep:\n walk_arr_stop = c.departure_stop\n walk_arr_time = c.departure_time - transfer_margin\n for _, walk_dep_stop, data in walk_network.edges(nbunch=[walk_arr_stop], data=True):\n walk_dep_time = walk_arr_time - data['d_walk'] / float(walk_speed)\n if walk_dep_time > end_time_dep or walk_dep_time < start_time_dep:\n continue\n pseudo_connection = Connection(walk_dep_stop,\n walk_arr_stop,\n walk_dep_time,\n walk_arr_time,\n Connection.WALK_TRIP_ID,\n Connection.WALK_SEQ,\n is_walk=True)\n pseudo_connection_set.add(pseudo_connection)\n return pseudo_connection_set"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the earliest visit time of the stop.", "response": "def get_min_visit_time(self):\n \"\"\"\n Get the earliest visit time of the stop.\n \"\"\"\n if not self.visit_events:\n return float('inf')\n else:\n return min(self.visit_events, key=lambda event: event.arr_time_ut).arr_time_ut"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef visit(self, event):\n to_visit = False\n if event.arr_time_ut <= self.min_transfer_time+self.get_min_visit_time():\n to_visit = True\n else:\n for ve in self.visit_events:\n if (event.trip_I == ve.trip_I) and event.arr_time_ut < ve.arr_time_ut:\n to_visit = True\n\n if to_visit:\n self.visit_events.append(event)\n min_time = self.get_min_visit_time()\n # remove any visits that are 'too old'\n self.visit_events = [v for v in self.visit_events if v.arr_time_ut <= min_time+self.min_transfer_time]\n return to_visit", "response": "Visit the stop if it has not been visited already by an event with arr_time_ut < self. min_transfer_time"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if the event can be infected using this event.", "response": "def can_infect(self, event):\n \"\"\"\n Whether the spreading stop can infect using this event.\n \"\"\"\n if event.from_stop_I != self.stop_I:\n return False\n\n if not self.has_been_visited():\n return False\n else:\n time_sep = event.dep_time_ut-self.get_min_visit_time()\n # if the gap between the earliest visit_time and current time is\n # smaller than the min. transfer time, the stop can pass the spreading\n # forward\n if (time_sep >= self.min_transfer_time) or (event.trip_I == -1 and time_sep >= 0):\n return True\n else:\n for visit in self.visit_events:\n # if no transfer, please hop-on\n if (event.trip_I == visit.trip_I) and (time_sep >= 0):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a list of Connection objects for each trip in a GTFS.", "response": "def get_transit_connections(gtfs, start_time_ut, end_time_ut):\n \"\"\"\n Parameters\n ----------\n gtfs: gtfspy.GTFS\n end_time_ut: int\n start_time_ut: int\n\n Returns\n -------\n list[Connection]\n \"\"\"\n if start_time_ut + 20 * 3600 < end_time_ut:\n warn(\"Note that it is possible that same trip_I's can take place during multiple days, \"\n \"which could (potentially) affect the outcomes of the CSA routing!\")\n assert (isinstance(gtfs, GTFS))\n events_df = temporal_network(gtfs, start_time_ut=start_time_ut, end_time_ut=end_time_ut)\n assert (isinstance(events_df, pandas.DataFrame))\n return list(map(lambda e: Connection(e.from_stop_I, e.to_stop_I, e.dep_time_ut, e.arr_time_ut, e.trip_I, e.seq),\n events_df.itertuples()\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_walk_network(gtfs, max_link_distance_m=1000):\n assert (isinstance(gtfs, GTFS))\n return walk_transfer_stop_to_stop_network(gtfs, max_link_distance=max_link_distance_m)", "response": "Returns a walk network from a GTFS object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the shape points corresponding to each stop in a trip.", "response": "def calculate_trip_shape_breakpoints(conn):\n \"\"\"Pre-compute the shape points corresponding to each trip's stop.\n\n Depends: shapes\"\"\"\n from gtfspy import shapes\n\n cur = conn.cursor()\n breakpoints_cache = {}\n\n # Counters for problems - don't print every problem.\n count_bad_shape_ordering = 0\n count_bad_shape_fit = 0\n count_no_shape_fit = 0\n\n trip_Is = [x[0] for x in\n cur.execute('SELECT DISTINCT trip_I FROM stop_times').fetchall()]\n for trip_I in trip_Is:\n # Get the shape points\n row = cur.execute('''SELECT shape_id\n FROM trips WHERE trip_I=?''', (trip_I,)).fetchone()\n if row is None:\n continue\n shape_id = row[0]\n if shape_id is None or shape_id == '':\n continue\n\n # Get the stop points\n cur.execute('''SELECT seq, lat, lon, stop_id\n FROM stop_times LEFT JOIN stops USING (stop_I)\n WHERE trip_I=?\n ORDER BY seq''',\n (trip_I,))\n #print '%20s, %s'%(run_code, datetime.fromtimestamp(run_sch_starttime))\n stop_points = [dict(seq=row[0],\n lat=row[1],\n lon=row[2],\n stop_I=row[3])\n for row in cur if row[1] and row[2]]\n # Calculate a cache key for this sequence.\n # If both shape_id, and all stop_Is are same, then we can re-use existing breakpoints:\n cache_key = (shape_id, tuple(x['stop_I'] for x in stop_points))\n if cache_key in breakpoints_cache:\n breakpoints = breakpoints_cache[cache_key]\n else:\n # Must re-calculate breakpoints:\n\n shape_points = shapes.get_shape_points(cur, shape_id)\n breakpoints, badness \\\n = shapes.find_segments(stop_points, shape_points)\n if breakpoints != sorted(breakpoints):\n # route_name, route_id, route_I, trip_id, trip_I = \\\n # cur.execute('''SELECT name, route_id, route_I, trip_id, trip_I\n # FROM trips LEFT JOIN routes USING (route_I)\n # WHERE trip_I=? LIMIT 1''', (trip_I,)).fetchone()\n # print \"Ignoring: Route with bad shape ordering:\", route_name, route_id, route_I, trip_id, trip_I\n count_bad_shape_ordering += 1\n # select * from stop_times where trip_I=NNNN order by shape_break;\n breakpoints_cache[cache_key] = None\n continue # Do not set shape_break for this trip.\n # Add it to cache\n breakpoints_cache[cache_key] = breakpoints\n\n if badness > 30 * len(breakpoints):\n #print \"bad shape fit: %s (%s, %s, %s)\" % (badness, trip_I, shape_id, len(breakpoints))\n count_bad_shape_fit += 1\n\n if breakpoints is None:\n continue\n\n if len(breakpoints) == 0:\n # No valid route could be identified.\n #print \"Ignoring: No shape identified for trip_I=%s, shape_id=%s\" % (trip_I, shape_id)\n count_no_shape_fit += 1\n continue\n\n # breakpoints is the corresponding points for each stop\n assert len(breakpoints) == len(stop_points)\n cur.executemany('UPDATE stop_times SET shape_break=? '\n 'WHERE trip_I=? AND seq=? ',\n ((int(bkpt), int(trip_I), int(stpt['seq']))\n for bkpt, stpt in zip(breakpoints, stop_points)))\n if count_bad_shape_fit > 0:\n print(\" Shape trip breakpoints: %s bad fits\" % count_bad_shape_fit)\n if count_bad_shape_ordering > 0:\n print(\" Shape trip breakpoints: %s bad shape orderings\" % count_bad_shape_ordering)\n if count_no_shape_fit > 0:\n print(\" Shape trip breakpoints: %s no shape fits\" % count_no_shape_fit)\n conn.commit()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nimport journey data for a target stop.", "response": "def import_journey_data_for_target_stop(self, target_stop_I, origin_stop_I_to_journey_labels, enforce_synchronous_writes=False):\n \"\"\"\n Parameters\n ----------\n origin_stop_I_to_journey_labels: dict\n key: origin_stop_Is\n value: list of labels\n target_stop_I: int\n \"\"\"\n cur = self.conn.cursor()\n self.conn.isolation_level = 'EXCLUSIVE'\n # if not enforce_synchronous_writes:\n cur.execute('PRAGMA synchronous = 0;')\n\n if self.track_route:\n self._insert_journeys_with_route_into_db(origin_stop_I_to_journey_labels, target_stop=int(target_stop_I))\n else:\n self._insert_journeys_into_db_no_route(origin_stop_I_to_journey_labels, target_stop=int(target_stop_I))\n print(\"Finished import process\")\n self.conn.commit()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _insert_journeys_into_db_no_route(self, stop_profiles, target_stop=None):\n # TODO: Change the insertion so that the check last journey id and insertions are in the same transaction block\n \"\"\"\n con.isolation_level = 'EXCLUSIVE'\n con.execute('BEGIN EXCLUSIVE')\n #exclusive access starts here. Nothing else can r/w the db, do your magic here.\n con.commit()\n \"\"\"\n print(\"Collecting journey data\")\n journey_id = 1\n journey_list = []\n tot = len(stop_profiles)\n for i, (origin_stop, labels) in enumerate(stop_profiles.items(), start=1):\n #print(\"\\r Stop \" + str(i) + \" of \" + str(tot), end='', flush=True)\n for label in labels:\n assert (isinstance(label, LabelTimeWithBoardingsCount))\n if self.multitarget_routing:\n target_stop = None\n else:\n target_stop = int(target_stop)\n\n values = [int(journey_id),\n int(origin_stop),\n target_stop,\n int(label.departure_time),\n int(label.arrival_time_target),\n int(label.n_boardings)]\n\n journey_list.append(values)\n journey_id += 1\n print(\"Inserting journeys without route into database\")\n insert_journeys_stmt = '''INSERT INTO journeys(\n journey_id,\n from_stop_I,\n to_stop_I,\n departure_time,\n arrival_time_target,\n n_boardings) VALUES (%s) ''' % (\", \".join([\"?\" for x in range(6)]))\n #self.conn.executemany(insert_journeys_stmt, journey_list)\n self._executemany_exclusive(insert_journeys_stmt, journey_list)\n self.conn.commit()", "response": "Insert journeys that are not part of a route."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate journey labels for all journeys in the source stop.", "response": "def _journey_label_generator(self, destination_stop_Is=None, origin_stop_Is=None):\n \"\"\"\n Parameters\n ----------\n destination_stop_Is: list-like\n origin_stop_Is: list-like\n\n Yields\n ------\n (origin_stop_I, destination_stop_I, journey_labels) : tuple\n \"\"\"\n conn = self.conn\n conn.row_factory = sqlite3.Row\n if destination_stop_Is is None:\n destination_stop_Is = self.get_targets_having_journeys()\n if origin_stop_Is is None:\n origin_stop_Is = self.get_origins_having_journeys()\n\n for destination_stop_I in destination_stop_Is:\n if self.track_route:\n label_features = \"journey_id, from_stop_I, to_stop_I, n_boardings, movement_duration, \" \\\n \"journey_duration, in_vehicle_duration, transfer_wait_duration, walking_duration, \" \\\n \"departure_time, arrival_time_target\"\"\"\n else:\n label_features = \"journey_id, from_stop_I, to_stop_I, n_boardings, departure_time, \" \\\n \"arrival_time_target\"\n sql = \"SELECT \" + label_features + \" FROM journeys WHERE to_stop_I = %s\" % destination_stop_I\n\n df = pd.read_sql_query(sql, self.conn)\n for origin_stop_I in origin_stop_Is:\n selection = df.loc[df['from_stop_I'] == origin_stop_I]\n journey_labels = []\n for journey in selection.to_dict(orient='records'):\n journey[\"pre_journey_wait_fp\"] = -1\n try:\n journey_labels.append(LabelGeneric(journey))\n except Exception as e:\n print(journey)\n raise e\n yield origin_stop_I, destination_stop_I, journey_labels"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _insert_travel_impedance_data_to_db(self, travel_impedance_measure_name, data):\n f = float\n data_tuple = [(x[\"from_stop_I\"], x[\"to_stop_I\"], f(x[\"min\"]), f(x[\"max\"]), f(x[\"median\"]), f(x[\"mean\"])) for x in data]\n insert_stmt = '''INSERT OR REPLACE INTO ''' + travel_impedance_measure_name + ''' (\n from_stop_I,\n to_stop_I,\n min,\n max,\n median,\n mean) VALUES (?, ?, ?, ?, ?, ?) '''\n self.conn.executemany(insert_stmt, data_tuple)\n self.conn.commit()", "response": "Inserts the given data to the database."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplots the trip counts per day of the given G.", "response": "def plot_trip_counts_per_day(G, ax=None, highlight_dates=None, highlight_date_labels=None, show=False):\n \"\"\"\n Parameters\n ----------\n G: gtfspy.GTFS\n ax: maptlotlib.Axes, optional\n highlight_dates: list[str|datetime.datetime]\n The values of highlight dates should represent dates, and or datetime objects.\n highlight_date_labels: list\n The labels for each highlight dates.\n show: bool, optional\n whether or not to immediately show the results\n\n Returns\n -------\n ax: maptlotlib.Axes object\n \"\"\"\n daily_trip_counts = G.get_trip_counts_per_day()\n if ax is None:\n _fig, ax = plt.subplots()\n daily_trip_counts[\"datetime\"] = pandas.to_datetime(daily_trip_counts[\"date_str\"])\n daily_trip_counts.plot(\"datetime\", \"trip_counts\", kind=\"line\", ax=ax, marker=\"o\", color=\"C0\", ls=\":\",\n label=\"Trip counts\")\n ax.set_xlabel(\"Date\")\n ax.set_ylabel(\"Trip counts per day\")\n if highlight_dates is not None:\n assert isinstance(highlight_dates, list)\n if highlight_date_labels is not None:\n assert isinstance(highlight_date_labels, list)\n assert len(highlight_dates) == len(highlight_date_labels), \"Number of highlight date labels do not match\"\n else:\n highlight_date_labels = [None] * len(highlight_dates)\n for i, (highlight_date, label) in enumerate(zip(highlight_dates, highlight_date_labels)):\n color = \"C\" + str(int(i % 8 + 1))\n highlight_date = pandas.to_datetime(highlight_date)\n ax.axvline(highlight_date, color=color, label=label)\n ax.legend(loc=\"best\")\n ax.grid()\n if show:\n plt.show()\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncreating all the views for the current date and time range.", "response": "def make_views(cls, conn):\n \"\"\"Create day_trips and day_stop_times views.\n\n day_trips: day_trips2 x trips = days x trips\n day_stop_times: day_trips2 x trips x stop_times = days x trips x stop_times\n \"\"\"\n conn.execute('DROP VIEW IF EXISTS main.day_trips')\n conn.execute('CREATE VIEW day_trips AS '\n 'SELECT day_trips2.*, trips.* '\n #'days.day_start_ut+trips.start_time_ds AS start_time_ut, '\n #'days.day_start_ut+trips.end_time_ds AS end_time_ut '\n 'FROM day_trips2 JOIN trips USING (trip_I);')\n conn.commit()\n\n conn.execute('DROP VIEW IF EXISTS main.day_stop_times')\n conn.execute('CREATE VIEW day_stop_times AS '\n 'SELECT day_trips2.*, trips.*, stop_times.*, '\n #'days.day_start_ut+trips.start_time_ds AS start_time_ut, '\n #'days.day_start_ut+trips.end_time_ds AS end_time_ut, '\n 'day_trips2.day_start_ut+stop_times.arr_time_ds AS arr_time_ut, '\n 'day_trips2.day_start_ut+stop_times.dep_time_ds AS dep_time_ut '\n 'FROM day_trips2 '\n 'JOIN trips USING (trip_I) '\n 'JOIN stop_times USING (trip_I)')\n conn.commit()"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a colourbar with limits of lwr and upr", "response": "def createcolorbar(cmap, norm):\n \"\"\"Create a colourbar with limits of lwr and upr\"\"\"\n cax, kw = matplotlib.colorbar.make_axes(matplotlib.pyplot.gca())\n c = matplotlib.colorbar.ColorbarBase(cax, cmap=cmap, norm=norm)\n return c"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write_walk_transfer_edges(gtfs, output_file_name):\n transfers = gtfs.get_table(\"stop_distances\")\n transfers.drop([u\"min_transfer_time\", u\"timed_transfer\"], 1, inplace=True)\n with util.create_file(output_file_name, tmpdir=True, keepext=True) as tmpfile:\n transfers.to_csv(tmpfile, encoding='utf-8', index=False)", "response": "Writes the walk transfer edges to a file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting the nodes of the current node to a file.", "response": "def write_nodes(gtfs, output, fields=None):\n \"\"\"\n Parameters\n ----------\n gtfs: gtfspy.GTFS\n output: str\n Path to the output file\n fields: list, optional\n which pieces of information to provide\n \"\"\"\n nodes = gtfs.get_table(\"stops\")\n if fields is not None:\n nodes = nodes[fields]\n with util.create_file(output, tmpdir=True, keepext=True) as tmpfile:\n nodes.to_csv(tmpfile, encoding='utf-8', index=False, sep=\";\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting a dictionary of stops to a file.", "response": "def write_stops_geojson(gtfs, out_file, fields=None):\n \"\"\"\n Parameters\n ----------\n gtfs: gtfspy.GTFS\n out_file: file-like or path to file\n fields: dict\n simultaneously map each original_name to the new_name\n Returns\n -------\n \"\"\"\n geojson = create_stops_geojson_dict(gtfs, fields)\n if hasattr(out_file, \"write\"):\n out_file.write(json.dumps(geojson))\n else:\n with util.create_file(out_file, tmpdir=True, keepext=True) as tmpfile_path:\n tmpfile = open(tmpfile_path, 'w')\n tmpfile.write(json.dumps(geojson))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef write_combined_transit_stop_to_stop_network(gtfs, output_path, fmt=None):\n if fmt is None:\n fmt = \"edg\"\n multi_di_graph = combined_stop_to_stop_transit_network(gtfs)\n _write_stop_to_stop_network_edges(multi_di_graph, output_path, fmt=fmt)", "response": "Writes a combined stop to stop network."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef write_static_networks(gtfs, output_dir, fmt=None):\n if fmt is None:\n fmt = \"edg\"\n single_layer_networks = stop_to_stop_networks_by_type(gtfs)\n util.makedirs(output_dir)\n for route_type, net in single_layer_networks.items():\n tag = route_types.ROUTE_TYPE_TO_LOWERCASE_TAG[route_type]\n file_name = os.path.join(output_dir, \"network_\" + tag + \".\" + fmt)\n if len(net.edges()) > 0:\n _write_stop_to_stop_network_edges(net, file_name, fmt=fmt)", "response": "Writes static networks to a directory."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef write_temporal_networks_by_route_type(gtfs, extract_output_dir):\n util.makedirs(extract_output_dir)\n for route_type in route_types.TRANSIT_ROUTE_TYPES:\n pandas_data_frame = temporal_network(gtfs, start_time_ut=None, end_time_ut=None, route_type=route_type)\n tag = route_types.ROUTE_TYPE_TO_LOWERCASE_TAG[route_type]\n out_file_name = os.path.join(extract_output_dir, tag + \".tnet\")\n pandas_data_frame.to_csv(out_file_name, encoding='utf-8', index=False)", "response": "Writes temporal networks to disk."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrite a temporal network to a file.", "response": "def write_temporal_network(gtfs, output_filename, start_time_ut=None, end_time_ut=None):\n \"\"\"\n Parameters\n ----------\n gtfs : gtfspy.GTFS\n output_filename : str\n path to the directory where to store the extracts\n start_time_ut: int | None\n start time of the extract in unixtime (seconds after epoch)\n end_time_ut: int | None\n end time of the extract in unixtime (seconds after epoch)\n \"\"\"\n util.makedirs(os.path.dirname(os.path.abspath(output_filename)))\n pandas_data_frame = temporal_network(gtfs, start_time_ut=start_time_ut, end_time_ut=end_time_ut)\n pandas_data_frame.to_csv(output_filename, encoding='utf-8', index=False)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _write_stop_to_stop_network_edges(net, file_name, data=True, fmt=None):\n if fmt is None:\n fmt = \"edg\"\n\n if fmt == \"edg\":\n if data:\n networkx.write_edgelist(net, file_name, data=True)\n else:\n networkx.write_edgelist(net, file_name)\n elif fmt == \"csv\":\n with open(file_name, 'w') as f:\n # writing out the header\n edge_iter = net.edges_iter(data=True)\n _, _, edg_data = next(edge_iter)\n edg_data_keys = list(sorted(edg_data.keys()))\n header = \";\".join([\"from_stop_I\", \"to_stop_I\"] + edg_data_keys)\n f.write(header)\n for from_node_I, to_node_I, data in net.edges_iter(data=True):\n f.write(\"\\n\")\n values = [str(from_node_I), str(to_node_I)]\n data_values = []\n for key in edg_data_keys:\n if key == \"route_I_counts\":\n route_I_counts_string = str(data[key]).replace(\" \", \"\")[1:-1]\n data_values.append(route_I_counts_string)\n else:\n data_values.append(str(data[key]))\n all_values = values + data_values\n f.write(\";\".join(all_values))", "response": "Writes out the stop to stop network edges."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nwrite out the GTFS files to a file.", "response": "def write_gtfs(gtfs, output):\n \"\"\"\n Write out the database according to the GTFS format.\n\n Parameters\n ----------\n gtfs: gtfspy.GTFS\n output: str\n Path where to put the GTFS files\n if output ends with \".zip\" a ZIP-file is created instead.\n\n Returns\n -------\n None\n \"\"\"\n output = os.path.abspath(output)\n uuid_str = \"tmp_\" + str(uuid.uuid1())\n if output[-4:] == '.zip':\n zip = True\n out_basepath = os.path.dirname(os.path.abspath(output))\n if not os.path.exists(out_basepath):\n raise IOError(out_basepath + \" does not exist, cannot write gtfs as a zip\")\n tmp_dir = os.path.join(out_basepath, str(uuid_str))\n # zip_file_na,e = ../out_basedir + \".zip\n else:\n zip = False\n out_basepath = output\n tmp_dir = os.path.join(out_basepath + \"_\" + str(uuid_str))\n\n os.makedirs(tmp_dir, exist_ok=True)\n\n gtfs_table_to_writer = {\n \"agency\": _write_gtfs_agencies,\n \"calendar\": _write_gtfs_calendar,\n \"calendar_dates\": _write_gtfs_calendar_dates,\n # fare attributes and fare_rules omitted (seldomly used)\n \"feed_info\": _write_gtfs_feed_info,\n # \"frequencies\": not written, as they are incorporated into trips and routes,\n # Frequencies table is expanded into other tables on initial import. -> Thus frequencies.txt is not created\n \"routes\": _write_gtfs_routes,\n \"shapes\": _write_gtfs_shapes,\n \"stops\": _write_gtfs_stops,\n \"stop_times\": _write_gtfs_stop_times,\n \"transfers\": _write_gtfs_transfers,\n \"trips\": _write_gtfs_trips,\n }\n\n for table, writer in gtfs_table_to_writer.items():\n fname_to_write = os.path.join(tmp_dir, table + '.txt')\n print(fname_to_write)\n writer(gtfs, open(os.path.join(tmp_dir, table + '.txt'), 'w'))\n\n if zip:\n shutil.make_archive(output[:-4], 'zip', tmp_dir)\n shutil.rmtree(tmp_dir)\n else:\n print(\"moving \" + str(tmp_dir) + \" to \" + out_basepath)\n os.rename(tmp_dir, out_basepath)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves all columns ending with I from a pandas. DataFrame df", "response": "def _remove_I_columns(df):\n \"\"\"\n Remove columns ending with I from a pandas.DataFrame\n\n Parameters\n ----------\n df: dataFrame\n\n Returns\n -------\n None\n \"\"\"\n all_columns = list(filter(lambda el: el[-2:] == \"_I\", df.columns))\n for column in all_columns:\n del df[column]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _scan_footpaths_to_departure_stop(self, connection_dep_stop, connection_dep_time, arrival_time_target):\n for _, neighbor, data in self._walk_network.edges_iter(nbunch=[connection_dep_stop],\n data=True):\n d_walk = data['d_walk']\n neighbor_dep_time = connection_dep_time - d_walk / self._walk_speed\n pt = LabelTimeSimple(departure_time=neighbor_dep_time, arrival_time_target=arrival_time_target)\n self._stop_profiles[neighbor].update_pareto_optimal_tuples(pt)", "response": "Scan the footpaths to the departure stop."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplots a route network from a GTFS object.", "response": "def plot_route_network_from_gtfs(g, ax=None, spatial_bounds=None, map_alpha=0.8, scalebar=True, legend=True,\n return_smopy_map=False, map_style=None):\n \"\"\"\n Parameters\n ----------\n g: A gtfspy.gtfs.GTFS object\n Where to get the data from?\n ax: matplotlib.Axes object, optional\n If None, a new figure and an axis is created\n spatial_bounds: dict, optional\n with str keys: lon_min, lon_max, lat_min, lat_max\n return_smopy_map: bool, optional\n defaulting to false\n\n Returns\n -------\n ax: matplotlib.axes.Axes\n\n \"\"\"\n assert(isinstance(g, GTFS))\n route_shapes = g.get_all_route_shapes()\n\n if spatial_bounds is None:\n spatial_bounds = get_spatial_bounds(g, as_dict=True)\n if ax is not None:\n bbox = ax.get_window_extent().transformed(ax.figure.dpi_scale_trans.inverted())\n width, height = bbox.width, bbox.height\n spatial_bounds = _expand_spatial_bounds_to_fit_axes(spatial_bounds, width, height)\n return plot_as_routes(route_shapes,\n ax=ax,\n spatial_bounds=spatial_bounds,\n map_alpha=map_alpha,\n plot_scalebar=scalebar,\n legend=legend,\n return_smopy_map=return_smopy_map,\n map_style=map_style)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef plot_as_routes(route_shapes, ax=None, spatial_bounds=None, map_alpha=0.8, plot_scalebar=True, legend=True,\n return_smopy_map=False, line_width_attribute=None, line_width_scale=1.0, map_style=None):\n \"\"\"\n Parameters\n ----------\n route_shapes: list of dicts that should have the following keys\n name, type, agency, lats, lons\n with types\n list, list, str, list, list\n ax: axis object\n spatial_bounds: dict\n map_alpha:\n plot_scalebar: bool\n legend:\n return_smopy_map:\n line_width_attribute:\n line_width_scale:\n\n Returns\n -------\n ax: matplotlib.axes object\n \"\"\"\n lon_min = spatial_bounds['lon_min']\n lon_max = spatial_bounds['lon_max']\n lat_min = spatial_bounds['lat_min']\n lat_max = spatial_bounds['lat_max']\n if ax is None:\n fig = plt.figure()\n ax = fig.add_subplot(111)\n\n smopy_map = get_smopy_map(lon_min, lon_max, lat_min, lat_max, map_style=map_style)\n ax = smopy_map.show_mpl(figsize=None, ax=ax, alpha=map_alpha)\n bound_pixel_xs, bound_pixel_ys = smopy_map.to_pixels(numpy.array([lat_min, lat_max]),\n numpy.array([lon_min, lon_max]))\n\n route_types_to_lines = {}\n for shape in route_shapes:\n route_type = ROUTE_TYPE_CONVERSION[shape['type']]\n lats = numpy.array(shape['lats'])\n lons = numpy.array(shape['lons'])\n if line_width_attribute:\n line_width = line_width_scale * shape[line_width_attribute]\n else:\n line_width = 1\n xs, ys = smopy_map.to_pixels(lats, lons)\n line, = ax.plot(xs, ys, linewidth=line_width, color=ROUTE_TYPE_TO_COLOR[route_type], zorder=ROUTE_TYPE_TO_ZORDER[route_type])\n route_types_to_lines[route_type] = line\n\n if legend:\n lines = list(route_types_to_lines.values())\n labels = [ROUTE_TYPE_TO_SHORT_DESCRIPTION[route_type] for route_type in route_types_to_lines.keys()]\n ax.legend(lines, labels, loc=\"upper left\")\n\n if plot_scalebar:\n _add_scale_bar(ax, lat_max, lon_min, lon_max, bound_pixel_xs.max() - bound_pixel_xs.min())\n\n ax.set_xticks([])\n ax.set_yticks([])\n\n ax.set_xlim(bound_pixel_xs.min(), bound_pixel_xs.max())\n ax.set_ylim(bound_pixel_ys.max(), bound_pixel_ys.min())\n if return_smopy_map:\n return ax, smopy_map\n else:\n return ax", "response": "Plots a route as a single line."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _expand_spatial_bounds_to_fit_axes(bounds, ax_width, ax_height):\n b = bounds\n height_meters = util.wgs84_distance(b['lat_min'], b['lon_min'], b['lat_max'], b['lon_min'])\n width_meters = util.wgs84_distance(b['lat_min'], b['lon_min'], b['lat_min'], b['lon_max'])\n x_per_y_meters = width_meters / height_meters\n x_per_y_axes = ax_width / ax_height\n if x_per_y_axes > x_per_y_meters: # x-axis\n # axis x_axis has slack -> the spatial longitude bounds need to be extended\n width_meters_new = (height_meters * x_per_y_axes)\n d_lon_new = ((b['lon_max'] - b['lon_min']) / width_meters) * width_meters_new\n mean_lon = (b['lon_min'] + b['lon_max'])/2.\n lon_min = mean_lon - d_lon_new / 2.\n lon_max = mean_lon + d_lon_new / 2.\n spatial_bounds = {\n \"lon_min\": lon_min,\n \"lon_max\": lon_max,\n \"lat_min\": b['lat_min'],\n \"lat_max\": b['lat_max']\n }\n else:\n # axis y_axis has slack -> the spatial latitude bounds need to be extended\n height_meters_new = (width_meters / x_per_y_axes)\n d_lat_new = ((b['lat_max'] - b['lat_min']) / height_meters) * height_meters_new\n mean_lat = (b['lat_min'] + b['lat_max']) / 2.\n lat_min = mean_lat - d_lat_new / 2.\n lat_max = mean_lat + d_lat_new / 2.\n spatial_bounds = {\n \"lon_min\": b['lon_min'],\n \"lon_max\": b['lon_max'],\n \"lat_min\": lat_min,\n \"lat_max\": lat_max\n }\n return spatial_bounds", "response": "Expand the spatial bounds to fit axes."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nplotting all stops of a single object.", "response": "def plot_all_stops(g, ax=None, scalebar=False):\n \"\"\"\n Parameters\n ----------\n g: A gtfspy.gtfs.GTFS object\n ax: matplotlib.Axes object, optional\n If None, a new figure and an axis is created, otherwise results are plotted on the axis.\n scalebar: bool, optional\n Whether to include a scalebar to the plot.\n\n Returns\n -------\n ax: matplotlib.Axes\n\n \"\"\"\n assert(isinstance(g, GTFS))\n lon_min, lon_max, lat_min, lat_max = get_spatial_bounds(g)\n smopy_map = get_smopy_map(lon_min, lon_max, lat_min, lat_max)\n if ax is None:\n fig = plt.figure()\n ax = fig.add_subplot(111)\n ax = smopy_map.show_mpl(figsize=None, ax=ax, alpha=0.8)\n\n stops = g.stops()\n lats = numpy.array(stops['lat'])\n lons = numpy.array(stops['lon'])\n\n xs, ys = smopy_map.to_pixels(lats, lons)\n ax.scatter(xs, ys, color=\"red\", s=10)\n\n ax.set_xlim(min(xs), max(xs))\n ax.set_ylim(max(ys), min(ys))\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef set_process_timezone(TZ):\n try:\n prev_timezone = os.environ['TZ']\n except KeyError:\n prev_timezone = None\n os.environ['TZ'] = TZ\n time.tzset() # Cause C-library functions to notice the update.\n return prev_timezone", "response": "Sets the process timezone."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef wgs84_distance(lat1, lon1, lat2, lon2):\n dLat = math.radians(lat2 - lat1)\n dLon = math.radians(lon2 - lon1)\n a = (math.sin(dLat / 2) * math.sin(dLat / 2) +\n math.cos(math.radians(lat1)) * math.cos(math.radians(lat2)) *\n math.sin(dLon / 2) * math.sin(dLon / 2))\n c = 2 * math.atan2(math.sqrt(a), math.sqrt(1 - a))\n d = EARTH_RADIUS * c\n return d", "response": "Distance between two points in WGS84 coord system."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create_file(fname=None, fname_tmp=None, tmpdir=None,\n save_tmpfile=False, keepext=False):\n \"\"\"Context manager for making files with possibility of failure.\n\n If you are creating a file, it is possible that the code will fail\n and leave a corrupt intermediate file. This is especially damaging\n if this is used as automatic input to another process. This context\n manager helps by creating a temporary filename, your code runs and\n creates that temporary file, and then if no exceptions are raised,\n the context manager will move the temporary file to the original\n filename you intended to open.\n\n Parameters\n ----------\n fname : str\n Target filename, this file will be created if all goes well\n fname_tmp : str\n If given, this is used as the temporary filename.\n tmpdir : str or bool\n If given, put temporary files in this directory. If `True`,\n then find a good tmpdir that is not on local filesystem.\n save_tmpfile : bool\n If true, the temporary file is not deleteted if an exception\n is raised.\n keepext : bool, default False\n If true, have tmpfile have same extension as final file.\n\n Returns (as context manager value)\n ----------------------------------\n fname_tmp: str\n Temporary filename to be used. Same as `fname_tmp`\n if given as an argument.\n\n Raises\n ------\n Re-raises any except occuring during the context block.\n \"\"\"\n # Do nothing if requesting sqlite memory DB.\n if fname == ':memory:':\n yield fname\n return\n if fname_tmp is None:\n # no tmpfile name given - compute some basic info\n basename = os.path.basename(fname)\n root, ext = os.path.splitext(basename)\n dir_ = this_dir = os.path.dirname(fname)\n # Remove filename extension, in case this matters for\n # automatic things itself.\n if not keepext:\n root = root + ext\n ext = ''\n if tmpdir:\n # we should use a different temporary directory\n if tmpdir is True:\n # Find a directory ourself, searching some common\n # places.\n for dir__ in possible_tmpdirs:\n if os.access(dir__, os.F_OK):\n dir_ = dir__\n break\n # Make the actual tmpfile, with our chosen tmpdir, directory,\n # extension. Set it to not delete automatically, since on\n # success we will move it to elsewhere.\n tmpfile = tempfile.NamedTemporaryFile(\n prefix='tmp-' + root + '-', suffix=ext, dir=dir_, delete=False)\n fname_tmp = tmpfile.name\n try:\n yield fname_tmp\n except Exception as e:\n if save_tmpfile:\n print(\"Temporary file is '%s'\" % fname_tmp)\n else:\n os.unlink(fname_tmp)\n raise\n # Move the file back to the original location.\n try:\n os.rename(fname_tmp, fname)\n # We have to manually set permissions. tempfile does not use\n # umask, for obvious reasons.\n os.chmod(fname, 0o777 & ~current_umask)\n # 'Invalid cross-device link' - you can't rename files across\n # filesystems. So, we have to fallback to moving it. But, we\n # want to move it using tmpfiles also, so that the final file\n # appearing is atomic. We use... tmpfiles.\n except OSError as e:\n # New temporary file in same directory\n tmpfile2 = tempfile.NamedTemporaryFile(\n prefix='tmp-' + root + '-', suffix=ext, dir=this_dir, delete=False)\n # Copy contents over\n shutil.copy(fname_tmp, tmpfile2.name)\n # Rename new tmpfile, unlink old one on other filesystem.\n os.rename(tmpfile2.name, fname)\n os.chmod(fname, 0o666 & ~current_umask)\n os.unlink(fname_tmp)", "response": "Context manager for creating a new file in a context block."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef execute(cur, *args):\n stmt = args[0]\n if len(args) > 1:\n stmt = stmt.replace('%', '%%').replace('?', '%r')\n print(stmt % (args[1]))\n return cur.execute(*args)", "response": "Utility function to print sqlite queries before executing."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef str_time_to_day_seconds(time):\n t = str(time).split(':')\n seconds = int(t[0]) * 3600 + int(t[1]) * 60 + int(t[2])\n return seconds", "response": "Converts time strings to integer seconds"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate directories if they do not exist.", "response": "def makedirs(path):\n \"\"\"\n Create directories if they do not exist, otherwise do nothing.\n\n Return path for convenience\n \"\"\"\n if not os.path.isdir(path):\n os.makedirs(path)\n return path"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef source_csv_to_pandas(path, table, read_csv_args=None):\n if '.txt' not in table:\n table += '.txt'\n\n if isinstance(path, dict):\n data_obj = path[table]\n f = data_obj.split(\"\\n\")\n else:\n if os.path.isdir(path):\n f = open(os.path.join(path, table))\n\n else:\n\n z = zipfile.ZipFile(path)\n for path in z.namelist():\n if table in path:\n table = path\n break\n try:\n f = zip_open(z, table)\n except KeyError as e:\n return pd.DataFrame()\n\n if read_csv_args:\n df = pd.read_csv(**read_csv_args)\n else:\n df = pd.read_csv(f)\n return df", "response": "Reads a CSV file and returns a pandas DataFrame."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef write_shapefile(data, shapefile_path):\n from numpy import int64\n \"\"\"\n :param data: list of dicts where dictionary contains the keys lons and lats\n :param shapefile_path: path where shapefile is saved\n :return:\n \"\"\"\n\n w = shp.Writer(shp.POLYLINE) # shapeType=3)\n\n fields = []\n encode_strings = []\n\n # This makes sure every geom has all the attributes\n w.autoBalance = 1\n # Create all attribute fields except for lats and lons. In addition the field names are saved for the\n # datastoring phase. Encode_strings stores .encode methods as strings for all fields that are strings\n if not fields:\n for key, value in data[0].items():\n if key != u'lats' and key != u'lons':\n fields.append(key)\n\n if type(value) == float:\n w.field(key.encode('ascii'), fieldType='N', size=11, decimal=3)\n print(\"float\", type(value))\n elif type(value) == int or type(value) == int64:\n print(\"int\", type(value))\n\n # encode_strings.append(\".encode('ascii')\")\n w.field(key.encode('ascii'), fieldType='N', size=6, decimal=0)\n else:\n print(\"other type\", type(value))\n\n w.field(key.encode('ascii'))\n\n for dict_item in data:\n line = []\n lineparts = []\n records = []\n records_string = ''\n for lat, lon in zip(dict_item[u'lats'], dict_item[u'lons']):\n line.append([float(lon), float(lat)])\n lineparts.append(line)\n w.line(parts=lineparts)\n\n # The shapefile records command is built up as strings to allow a differing number of columns\n for field in fields:\n if records_string:\n records_string += \", dict_item['\" + field + \"']\"\n else:\n records_string += \"dict_item['\" + field + \"']\"\n method_string = \"w.record(\" + records_string + \")\"\n\n # w.record(dict_item['name'], dict_item['agency'], dict_item['agency_name'], dict_item['type'], dict_item['lons'])\n print(method_string)\n eval(method_string)\n w.save(shapefile_path)", "response": "Write a shapefile to a shapefile_path."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef draw_net_using_node_coords(net):\n import matplotlib.pyplot as plt\n fig = plt.figure()\n node_coords = {}\n for node, data in net.nodes(data=True):\n node_coords[node] = (data['lon'], data['lat'])\n ax = fig.add_subplot(111)\n networkx.draw(net, pos=node_coords, ax=ax, node_size=50)\n return fig", "response": "Plot a networkx. Graph by using the lat and lon attributes of nodes."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef difference_of_pandas_dfs(df_self, df_other, col_names=None):\n df = pd.concat([df_self, df_other])\n df = df.reset_index(drop=True)\n df_gpby = df.groupby(col_names)\n idx = [x[0] for x in list(df_gpby.groups.values()) if len(x) == 1]\n df_sym_diff = df.reindex(idx)\n df_diff = pd.concat([df_other, df_sym_diff])\n df_diff = df_diff.reset_index(drop=True)\n df_gpby = df_diff.groupby(col_names)\n idx = [x[0] for x in list(df_gpby.groups.values()) if len(x) == 2]\n df_diff = df_diff.reindex(idx)\n return df_diff", "response": "Returns a dataframe with all of df_self that are not in df_other when considering the columns specified in col_names."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _finalize_profiles(self):\n for stop, stop_profile in self._stop_profiles.items():\n assert (isinstance(stop_profile, NodeProfileMultiObjective))\n neighbor_label_bags = []\n walk_durations_to_neighbors = []\n departure_arrival_stop_pairs = []\n if stop_profile.get_walk_to_target_duration() != 0 and stop in self._walk_network.node:\n neighbors = networkx.all_neighbors(self._walk_network, stop)\n for neighbor in neighbors:\n neighbor_profile = self._stop_profiles[neighbor]\n assert (isinstance(neighbor_profile, NodeProfileMultiObjective))\n neighbor_real_connection_labels = neighbor_profile.get_labels_for_real_connections()\n neighbor_label_bags.append(neighbor_real_connection_labels)\n walk_durations_to_neighbors.append(int(self._walk_network.get_edge_data(stop, neighbor)[\"d_walk\"] /\n self._walk_speed))\n departure_arrival_stop_pairs.append((stop, neighbor))\n stop_profile.finalize(neighbor_label_bags, walk_durations_to_neighbors, departure_arrival_stop_pairs)", "response": "Finalize the profiles of the first walks."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nimports a GTFS database into a new database.", "response": "def import_gtfs(gtfs_sources, output, preserve_connection=False,\n print_progress=True, location_name=None, **kwargs):\n \"\"\"Import a GTFS database\n\n gtfs_sources: str, dict, list\n Paths to the gtfs zip file or to the directory containing the GTFS data.\n Alternatively, a dict can be provide that maps gtfs filenames\n (like 'stops.txt' and 'agencies.txt') to their string presentations.\n\n output: str or sqlite3.Connection\n path to the new database to be created, or an existing\n sqlite3 connection\n preserve_connection: bool, optional\n Whether to close the connection in the end, or not.\n print_progress: bool, optional\n Whether to print progress output\n location_name: str, optional\n set the location of this database\n \"\"\"\n if isinstance(output, sqlite3.Connection):\n conn = output\n else:\n # if os.path.isfile(output):\n # raise RuntimeError('File already exists')\n conn = sqlite3.connect(output)\n if not isinstance(gtfs_sources, list):\n gtfs_sources = [gtfs_sources]\n cur = conn.cursor()\n time_import_start = time.time()\n\n # These are a bit unsafe, but make importing much faster,\n # especially on scratch.\n cur.execute('PRAGMA page_size = 4096;')\n cur.execute('PRAGMA mmap_size = 1073741824;')\n cur.execute('PRAGMA cache_size = -2000000;')\n cur.execute('PRAGMA temp_store=2;')\n # Changes of isolation level are python3.6 workarounds -\n # eventually will probably be fixed and this can be removed.\n conn.isolation_level = None # change to autocommit mode (former default)\n cur.execute('PRAGMA journal_mode = OFF;')\n #cur.execute('PRAGMA journal_mode = WAL;')\n cur.execute('PRAGMA synchronous = OFF;')\n conn.isolation_level = '' # change back to python default.\n # end python3.6 workaround\n\n # Do the actual importing.\n loaders = [L(gtfssource=gtfs_sources, print_progress=print_progress, **kwargs) for L in Loaders]\n\n for loader in loaders:\n loader.assert_exists_if_required()\n\n # Do initial import. This consists of making tables, raw insert\n # of the CSVs, and then indexing.\n\n for loader in loaders:\n loader.import_(conn)\n\n # Do any operations that require all tables present.\n for Loader in loaders:\n Loader.post_import_round2(conn)\n\n # Make any views\n for Loader in loaders:\n Loader.make_views(conn)\n\n # Make any views\n for F in postprocessors:\n F(conn)\n\n # Set up same basic metadata.\n from gtfspy import gtfs as mod_gtfs\n G = mod_gtfs.GTFS(output)\n G.meta['gen_time_ut'] = time.time()\n G.meta['gen_time'] = time.ctime()\n G.meta['import_seconds'] = time.time() - time_import_start\n G.meta['download_date'] = ''\n G.meta['location_name'] = ''\n G.meta['n_gtfs_sources'] = len(gtfs_sources)\n\n # Extract things from GTFS\n download_date_strs = []\n for i, source in enumerate(gtfs_sources):\n if len(gtfs_sources) == 1:\n prefix = \"\"\n else:\n prefix = \"feed_\" + str(i) + \"_\"\n if isinstance(source, string_types):\n G.meta[prefix + 'original_gtfs'] = decode_six(source) if source else None\n # Extract GTFS date. Last date pattern in filename.\n filename_date_list = re.findall(r'\\d{4}-\\d{2}-\\d{2}', source)\n if filename_date_list:\n date_str = filename_date_list[-1]\n G.meta[prefix + 'download_date'] = date_str\n download_date_strs.append(date_str)\n if location_name:\n G.meta['location_name'] = location_name\n else:\n location_name_list = re.findall(r'/([^/]+)/\\d{4}-\\d{2}-\\d{2}', source)\n if location_name_list:\n G.meta[prefix + 'location_name'] = location_name_list[-1]\n else:\n try:\n G.meta[prefix + 'location_name'] = source.split(\"/\")[-4]\n except:\n G.meta[prefix + 'location_name'] = source\n\n if G.meta['download_date'] == \"\":\n unique_download_dates = list(set(download_date_strs))\n if len(unique_download_dates) == 1:\n G.meta['download_date'] = unique_download_dates[0]\n\n G.meta['timezone'] = cur.execute('SELECT timezone FROM agencies LIMIT 1').fetchone()[0]\n stats.update_stats(G)\n del G\n\n if print_progress:\n print(\"Vacuuming...\")\n # Next 3 lines are python 3.6 work-arounds again.\n conn.isolation_level = None # former default of autocommit mode\n cur.execute('VACUUM;')\n conn.isolation_level = '' # back to python default\n # end python3.6 workaround\n if print_progress:\n print(\"Analyzing...\")\n cur.execute('ANALYZE')\n if not (preserve_connection is True):\n conn.close()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef validate_day_start_ut(conn):\n G = GTFS(conn)\n cur = conn.execute('SELECT date, day_start_ut FROM days')\n for date, day_start_ut in cur:\n #print date, day_start_ut\n assert day_start_ut == G.get_day_start_ut(date)", "response": "This validates the day_start_ut of the days table."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef main_make_views(gtfs_fname):\n print(\"creating views\")\n conn = GTFS(fname_or_conn=gtfs_fname).conn\n for L in Loaders:\n L(None).make_views(conn)\n conn.commit()", "response": "Re - create all views."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks row counts and compares them with the GTFS object", "response": "def _validate_table_row_counts(self):\n \"\"\"\n Imports source .txt files, checks row counts and then compares the rowcounts with the gtfsobject\n :return:\n \"\"\"\n for db_table_name in DB_TABLE_NAME_TO_SOURCE_FILE.keys():\n table_name_source_file = DB_TABLE_NAME_TO_SOURCE_FILE[db_table_name]\n row_warning_str = DB_TABLE_NAME_TO_ROWS_MISSING_WARNING[db_table_name]\n\n # Row count in GTFS object:\n database_row_count = self.gtfs.get_row_count(db_table_name)\n\n # Row counts in source files:\n source_row_count = 0\n for gtfs_source in self.gtfs_sources:\n frequencies_in_source = source_csv_to_pandas(gtfs_source, 'frequencies.txt')\n try:\n if table_name_source_file == 'trips' and not frequencies_in_source.empty:\n source_row_count += self._frequency_generated_trips_rows(gtfs_source)\n\n elif table_name_source_file == 'stop_times' and not frequencies_in_source.empty:\n source_row_count += self._compute_number_of_frequency_generated_stop_times(gtfs_source)\n else:\n df = source_csv_to_pandas(gtfs_source, table_name_source_file)\n\n source_row_count += len(df.index)\n except IOError as e:\n if hasattr(e, \"filename\") and db_table_name in e.filename:\n pass\n else:\n raise e\n\n\n if source_row_count == database_row_count and self.verbose:\n print(\"Row counts match for \" + table_name_source_file + \" between the source and database (\"\n + str(database_row_count) + \")\")\n else:\n difference = database_row_count - source_row_count\n ('Row counts do not match for ' + str(table_name_source_file) + ': (source=' + str(source_row_count) +\n ', database=' + str(database_row_count) + \")\")\n if table_name_source_file == \"calendar\" and difference > 0:\n query = \"SELECT count(*) FROM (SELECT * FROM calendar ORDER BY service_I DESC LIMIT \" \\\n + str(int(difference)) + \\\n \") WHERE start_date=end_date AND m=0 AND t=0 AND w=0 AND th=0 AND f=0 AND s=0 AND su=0\"\n number_of_entries_added_by_calendar_dates_loader = self.gtfs.execute_custom_query(query).fetchone()[\n 0]\n if number_of_entries_added_by_calendar_dates_loader == difference and self.verbose:\n print(\" But don't worry, the extra entries seem to just dummy entries due to calendar_dates\")\n else:\n if self.verbose:\n print(\" Reason for this is unknown.\")\n self.warnings_container.add_warning(row_warning_str, self.location, difference)\n else:\n self.warnings_container.add_warning(row_warning_str, self.location, difference)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _validate_no_null_values(self):\n for table in DB_TABLE_NAMES:\n null_not_ok_warning = \"Null values in must-have columns in table {table}\".format(table=table)\n null_warn_warning = \"Null values in good-to-have columns in table {table}\".format(table=table)\n null_not_ok_fields = DB_TABLE_NAME_TO_FIELDS_WHERE_NULL_NOT_OK[table]\n null_warn_fields = DB_TABLE_NAME_TO_FIELDS_WHERE_NULL_OK_BUT_WARN[table]\n\n # CW, TODO: make this validation source by source\n df = self.gtfs.get_table(table)\n\n for warning, fields in zip([null_not_ok_warning, null_warn_warning], [null_not_ok_fields, null_warn_fields]):\n null_unwanted_df = df[fields]\n rows_having_null = null_unwanted_df.isnull().any(1)\n if sum(rows_having_null) > 0:\n rows_having_unwanted_null = df[rows_having_null.values]\n self.warnings_container.add_warning(warning, rows_having_unwanted_null, len(rows_having_unwanted_null))", "response": "Validate that no null values are present in the database."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _validate_danglers(self):\n for query, warning in zip(DANGLER_QUERIES, DANGLER_WARNINGS):\n dangler_count = self.gtfs.execute_custom_query(query).fetchone()[0]\n if dangler_count > 0:\n if self.verbose:\n print(str(dangler_count) + \" \" + warning)\n self.warnings_container.add_warning(warning, self.location, count=dangler_count)", "response": "Checks for rows that are not referenced in the tables that should be linked\n "} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncomputes the number of stop times generated by the given GTFS source.", "response": "def _compute_number_of_frequency_generated_stop_times(self, gtfs_source_path):\n \"\"\"\n Parameters\n ----------\n Same as for \"_frequency_generated_trips_rows\" but for stop times table\n gtfs_source_path:\n table_name:\n\n Return\n ------\n \"\"\"\n df_freq = self._frequency_generated_trips_rows(gtfs_source_path, return_df_freq=True)\n df_stop_times = source_csv_to_pandas(gtfs_source_path, \"stop_times\")\n df_stop_freq = pd.merge(df_freq, df_stop_times, how='outer', on='trip_id')\n return int(df_stop_freq['n_trips'].fillna(1).sum(axis=0))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating the set of pareto optimal tuples.", "response": "def update_pareto_optimal_tuples(self, new_label):\n \"\"\"\n Parameters\n ----------\n new_label: LabelTime\n\n Returns\n -------\n updated: bool\n \"\"\"\n assert (isinstance(new_label, LabelTime))\n if self._labels:\n assert (new_label.departure_time <= self._labels[-1].departure_time)\n best_later_departing_arrival_time = self._labels[-1].arrival_time_target\n else:\n best_later_departing_arrival_time = float('inf')\n\n walk_to_target_arrival_time = new_label.departure_time + self._walk_to_target_duration\n\n best_arrival_time = min(walk_to_target_arrival_time,\n best_later_departing_arrival_time,\n new_label.arrival_time_target)\n # this should be changed to get constant time insertions / additions\n # (with time-indexing)\n if (new_label.arrival_time_target < walk_to_target_arrival_time and\n new_label.arrival_time_target < best_later_departing_arrival_time):\n self._labels.append(LabelTime(new_label.departure_time, best_arrival_time))\n return True\n else:\n return False"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef print_coords(rows, prefix=''):\n lat = [row['lat'] for row in rows]\n lon = [row['lon'] for row in rows]\n print('COORDS'+'-' * 5)\n print(\"%slat, %slon = %r, %r\" % (prefix, prefix, lat, lon))\n print('-'*5)", "response": "Print the coordinates within a sequence."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds the segments between each stop and the shape point.", "response": "def find_segments(stops, shape):\n \"\"\"Find corresponding shape points for a list of stops and create shape break points.\n\n Parameters\n ----------\n stops: stop-sequence (list)\n List of stop points\n shape: list of shape points\n shape-sequence of shape points\n\n Returns\n -------\n break_points: list[int]\n stops[i] corresponds to shape[break_points[i]]. This list can\n be used to partition the shape points into segments between\n one stop and the next.\n badness: float\n Lower indicates better fit to the shape. This is the sum of\n distances (in meters) between every each stop and its closest\n shape point. This is not needed in normal use, but in the\n cases where you must determine the best-fitting shape for a\n stop-sequence, use this.\n \"\"\"\n if not shape:\n return [], 0\n break_points = []\n last_i = 0\n cumul_d = 0\n badness = 0\n d_last_stop = float('inf')\n lstlat, lstlon = None, None\n break_shape_points = []\n for stop in stops:\n stlat, stlon = stop['lat'], stop['lon']\n best_d = float('inf')\n # print stop\n if badness > 500 and badness > 30 * len(break_points):\n return [], badness\n for i in range(last_i, len(shape)):\n d = wgs84_distance(stlat, stlon, shape[i]['lat'], shape[i]['lon'])\n if lstlat:\n d_last_stop = wgs84_distance(lstlat, lstlon, shape[i]['lat'], shape[i]['lon'])\n # If we are getting closer to next stop, record this as\n # the best stop so far.continue\n if d < best_d:\n best_d = d\n best_i = i\n # print best_d, i, last_i, len(shape)\n cumul_d += d\n # We have to be very careful about our stop condition.\n # This is trial and error, basically.\n if (d_last_stop < d) or (d > 500) or (i < best_i + 100):\n continue\n # We have decided our best stop, stop looking and continue\n # the outer loop.\n else:\n badness += best_d\n break_points.append(best_i)\n last_i = best_i\n lstlat, lstlon = stlat, stlon\n break_shape_points.append(shape[best_i])\n break\n else:\n # Executed if we did *not* break the inner loop\n badness += best_d\n break_points.append(best_i)\n last_i = best_i\n lstlat, lstlon = stlat, stlon\n break_shape_points.append(shape[best_i])\n pass\n # print \"Badness:\", badness\n # print_coords(stops, 'stop')\n # print_coords(shape, 'shape')\n # print_coords(break_shape_points, 'break')\n return break_points, badness"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds the best shape_id for a stop - sequence.", "response": "def find_best_segments(cur, stops, shape_ids, route_id=None,\n breakpoints_cache=None):\n \"\"\"Finds the best shape_id for a stop-sequence.\n\n This is used in cases like when you have GPS data with a route\n name, but you don't know the route direction. It tries shapes\n going both directions and returns the shape that best matches.\n Could be used in other cases as well.\n\n Parameters\n ----------\n cur : sqlite3.Cursor\n database cursor\n stops : list\n shape_ids : list of shape_id:s\n route_id : route_id to search for stops\n breakpoints_cache : dict\n If given, use this to cache results from this function.\n \"\"\"\n cache_key = None\n if breakpoints_cache is not None:\n # Calculate a cache key for this sequence. If shape_id and\n # all stop_Is are the same, then we assume that it is the same\n # route and re-use existing breakpoints.\n cache_key = (route_id, tuple(x['stop_I'] for x in stops))\n if cache_key in breakpoints_cache:\n print('found in cache')\n return breakpoints_cache[cache_key]\n\n if route_id is not None:\n cur.execute('''SELECT DISTINCT shape_id\n FROM routes\n LEFT JOIN trips\n USING (route_I)\n WHERE route_id=?''',\n (route_id,))\n data = cur.fetchall()\n # If not data, then route_id didn't match anything, or there\n # were no shapes defined. We have to exit in this case.\n if not data:\n print(\"No data for route_id=%s\" % route_id)\n return [], None, None, None\n #\n shape_ids = zip(*data)[0]\n # print 'find_best_segments:', shape_ids\n results = []\n for shape_id in shape_ids:\n shape = get_shape_points(cur, shape_id)\n breakpoints, badness = find_segments(stops, shape)\n results.append([badness, breakpoints, shape, shape_id])\n if len(stops) > 5 and badness < 5*(len(stops)):\n break\n\n best = np.argmin(zip(*results)[0])\n # print 'best', best\n badness = results[best][0]\n breakpoints = results[best][1]\n shape = results[best][2]\n shape_id = results[best][3]\n if breakpoints_cache is not None:\n print(\"storing in cache\", cache_key[0], hash(cache_key[1:]))\n breakpoints_cache[cache_key] = breakpoints, badness, shape, shape_id\n return breakpoints, badness, shape, shape_id"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef return_segments(shape, break_points):\n # print 'xxx'\n # print stops\n # print shape\n # print break_points\n # assert len(stops) == len(break_points)\n segs = []\n bp = 0 # not used\n bp2 = 0\n for i in range(len(break_points)-1):\n bp = break_points[i] if break_points[i] is not None else bp2\n bp2 = break_points[i+1] if break_points[i+1] is not None else bp\n segs.append(shape[bp:bp2+1])\n segs.append([])\n return segs", "response": "Break a shape into segments between stops using break_points."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating a cumulative distance between each stop and shape - sequence.", "response": "def gen_cumulative_distances(stops):\n \"\"\"\n Add a 'd' key for distances to a stop/shape-sequence.\n\n This takes a shape-sequence or stop-sequence, and adds an extra\n 'd' key that is cumulative, geographic distances between each\n point. This uses `wgs84_distance` from the util module. The\n distances are in meters. Distances are rounded to the nearest\n integer, because otherwise JSON size increases greatly.\n\n Parameters\n ----------\n stops: list\n elements are dicts with 'lat' and 'lon' keys\n and the function adds the 'd' key ('d' stands for distance)\n to the dictionaries\n \"\"\"\n stops[0]['d'] = 0.0\n for i in range(1, len(stops)):\n stops[i]['d'] = stops[i-1]['d'] + wgs84_distance(\n stops[i-1]['lat'], stops[i-1]['lon'],\n stops[i]['lat'], stops[i]['lon'],\n )\n for stop in stops:\n stop['d'] = int(stop['d'])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving a shape_id return its shape - sequence.", "response": "def get_shape_points(cur, shape_id):\n \"\"\"\n Given a shape_id, return its shape-sequence.\n\n Parameters\n ----------\n cur: sqlite3.Cursor\n cursor to a GTFS database\n shape_id: str\n id of the route\n\n Returns\n -------\n shape_points: list\n elements are dictionaries containing the 'seq', 'lat', and 'lon' of the shape\n \"\"\"\n cur.execute('''SELECT seq, lat, lon, d FROM shapes where shape_id=?\n ORDER BY seq''', (shape_id,))\n shape_points = [dict(seq=row[0], lat=row[1], lon=row[2], d=row[3])\n for row in cur]\n return shape_points"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_shape_points2(cur, shape_id):\n cur.execute('''SELECT seq, lat, lon, d FROM shapes where shape_id=?\n ORDER BY seq''', (shape_id,))\n shape_points = {'seqs': [], 'lats': [], 'lons': [], 'd': []}\n for row in cur:\n shape_points['seqs'].append(row[0])\n shape_points['lats'].append(row[1])\n shape_points['lons'].append(row[2])\n shape_points['d'].append(row[3])\n return shape_points", "response": "Given a shape_id return its shape - sequence as a dict of lists."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_route_shape_segments(cur, route_id):\n cur.execute('''SELECT seq, lat, lon\n FROM (\n SELECT shape_id\n FROM route\n LEFT JOIN trips\n USING (route_I)\n WHERE route_id=? limit 1\n )\n JOIN shapes\n USING (shape_id)\n ORDER BY seq''', (route_id,))\n shape_points = [dict(seq=row[0], lat=row[1], lon=row[2]) for row in cur]\n return shape_points", "response": "Given a route_id return its stop - sequence."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_shape_between_stops(cur, trip_I, seq_stop1=None, seq_stop2=None, shape_breaks=None):\n\n assert (seq_stop1 and seq_stop2) or shape_breaks\n if not shape_breaks:\n shape_breaks = []\n for seq_stop in [seq_stop1, seq_stop2]:\n query = \"\"\"SELECT shape_break FROM stop_times\n WHERE trip_I=%d AND seq=%d\n \"\"\" % (trip_I, seq_stop)\n for row in cur.execute(query):\n shape_breaks.append(row[0])\n assert len(shape_breaks) == 2\n\n query = \"\"\"SELECT seq, lat, lon\n FROM (SELECT shape_id FROM trips WHERE trip_I=%d)\n JOIN shapes USING (shape_id)\n WHERE seq>=%d AND seq <= %d;\n \"\"\" % (trip_I, shape_breaks[0], shape_breaks[1])\n shapedict = {'lat': [], 'lon': [], 'seq': []}\n for row in cur.execute(query):\n shapedict['seq'].append(row[0])\n shapedict['lat'].append(row[1])\n shapedict['lon'].append(row[2])\n return shapedict", "response": "Given a trip_I return the shape points between two stops."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting all scheduled stops on a particular route_id. Given a route_id, return the trip-stop-list with latitude/longitudes. This is a bit more tricky than it seems, because we have to go from table route->trips->stop_times. This functions finds an arbitrary trip (in trip table) with this route ID and, and then returns all stop points for that trip. Parameters ---------- cur : sqlite3.Cursor cursor to sqlite3 DB containing GTFS route_id : string or any route_id to get stop points of offset : int LIMIT offset if you don't want the first trip returned. tripid_glob : string If given, allows you to limit tripids which can be selected. Mainly useful in debugging. Returns ------- stop-list List of stops in stop-seq format.", "response": "def get_trip_points(cur, route_id, offset=0, tripid_glob=''):\n \"\"\"Get all scheduled stops on a particular route_id.\n\n Given a route_id, return the trip-stop-list with\n latitude/longitudes. This is a bit more tricky than it seems,\n because we have to go from table route->trips->stop_times. This\n functions finds an arbitrary trip (in trip table) with this route ID\n and, and then returns all stop points for that trip.\n\n Parameters\n ----------\n cur : sqlite3.Cursor\n cursor to sqlite3 DB containing GTFS\n route_id : string or any\n route_id to get stop points of\n offset : int\n LIMIT offset if you don't want the first trip returned.\n tripid_glob : string\n If given, allows you to limit tripids which can be selected.\n Mainly useful in debugging.\n\n Returns\n -------\n stop-list\n List of stops in stop-seq format.\n \"\"\"\n extra_where = ''\n if tripid_glob:\n extra_where = \"AND trip_id GLOB '%s'\" % tripid_glob\n cur.execute('SELECT seq, lat, lon '\n 'FROM (select trip_I from route '\n ' LEFT JOIN trips USING (route_I) '\n ' WHERE route_id=? %s limit 1 offset ? ) '\n 'JOIN stop_times USING (trip_I) '\n 'LEFT JOIN stop USING (stop_id) '\n 'ORDER BY seq' % extra_where, (route_id, offset))\n stop_points = [dict(seq=row[0], lat=row[1], lon=row[2]) for row in cur]\n return stop_points"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ninterpolating passage times for a given set of shape points.", "response": "def interpolate_shape_times(shape_distances, shape_breaks, stop_times):\n \"\"\"\n Interpolate passage times for shape points.\n\n Parameters\n ----------\n shape_distances: list\n list of cumulative distances along the shape\n shape_breaks: list\n list of shape_breaks\n stop_times: list\n list of stop_times\n\n Returns\n -------\n shape_times: list of ints (seconds) / numpy array\n interpolated shape passage times\n\n The values of stop times before the first shape-break are given the first\n stopping time, and the any shape points after the last break point are\n given the value of the last shape point.\n \"\"\"\n shape_times = np.zeros(len(shape_distances))\n shape_times[:shape_breaks[0]] = stop_times[0]\n for i in range(len(shape_breaks)-1):\n cur_break = shape_breaks[i]\n cur_time = stop_times[i]\n next_break = shape_breaks[i+1]\n next_time = stop_times[i+1]\n if cur_break == next_break:\n shape_times[cur_break] = stop_times[i]\n else:\n cur_distances = shape_distances[cur_break:next_break+1]\n norm_distances = ((np.array(cur_distances)-float(cur_distances[0])) /\n float(cur_distances[-1] - cur_distances[0]))\n times = (1.-norm_distances)*cur_time+norm_distances*next_time\n shape_times[cur_break:next_break] = times[:-1]\n # deal final ones separately:\n shape_times[shape_breaks[-1]:] = stop_times[-1]\n return list(shape_times)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef evaluate_earliest_arrival_time_at_target(self, dep_time, transfer_margin):\n minimum = dep_time + self._walk_to_target_duration\n dep_time_plus_transfer_margin = dep_time + transfer_margin\n for label in self._labels:\n if label.departure_time >= dep_time_plus_transfer_margin and label.arrival_time_target < minimum:\n minimum = label.arrival_time_target\n return float(minimum)", "response": "Evaluate the earliest arrival time at the target given a departure time and transfer margin."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nruns the actual simulation.", "response": "def _run(self):\n \"\"\"\n Run the actual simulation.\n \"\"\"\n if self._has_run:\n raise RuntimeError(\"This spreader instance has already been run: \"\n \"create a new Spreader object for a new run.\")\n i = 1\n while self.event_heap.size() > 0 and len(self._uninfected_stops) > 0:\n event = self.event_heap.pop_next_event()\n this_stop = self._stop_I_to_spreading_stop[event.from_stop_I]\n\n if event.arr_time_ut > self.start_time_ut + self.max_duration_ut:\n break\n\n if this_stop.can_infect(event):\n\n target_stop = self._stop_I_to_spreading_stop[event.to_stop_I]\n already_visited = target_stop.has_been_visited()\n target_stop.visit(event)\n\n if not already_visited:\n self._uninfected_stops.remove(event.to_stop_I)\n print(i, self.event_heap.size())\n transfer_distances = self.gtfs.get_straight_line_transfer_distances(event.to_stop_I)\n self.event_heap.add_walk_events_to_heap(transfer_distances, event, self.start_time_ut,\n self.walk_speed, self._uninfected_stops,\n self.max_duration_ut)\n i += 1\n self._has_run = True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the walk paths between stops and updates these to the database.", "response": "def add_walk_distances_to_db_python(gtfs, osm_path, cutoff_distance_m=1000):\n \"\"\"\n Computes the walk paths between stops, and updates these to the gtfs database.\n\n Parameters\n ----------\n gtfs: gtfspy.GTFS or str\n A GTFS object or a string representation.\n osm_path: str\n path to the OpenStreetMap file\n cutoff_distance_m: number\n maximum allowed distance in meters\n\n Returns\n -------\n None\n\n See Also\n --------\n gtfspy.calc_transfers\n compute_walk_paths_java\n \"\"\"\n if isinstance(gtfs, str):\n gtfs = GTFS(gtfs)\n assert (isinstance(gtfs, GTFS))\n print(\"Reading in walk network\")\n walk_network = create_walk_network_from_osm(osm_path)\n print(\"Matching stops to the OSM network\")\n stop_I_to_nearest_osm_node, stop_I_to_nearest_osm_node_distance = match_stops_to_nodes(gtfs, walk_network)\n\n transfers = gtfs.get_straight_line_transfer_distances()\n\n from_I_to_to_stop_Is = {stop_I: set() for stop_I in stop_I_to_nearest_osm_node}\n for transfer_tuple in transfers.itertuples():\n from_I = transfer_tuple.from_stop_I\n to_I = transfer_tuple.to_stop_I\n from_I_to_to_stop_Is[from_I].add(to_I)\n\n print(\"Computing walking distances\")\n for from_I, to_stop_Is in from_I_to_to_stop_Is.items():\n from_node = stop_I_to_nearest_osm_node[from_I]\n from_dist = stop_I_to_nearest_osm_node_distance[from_I]\n shortest_paths = networkx.single_source_dijkstra_path_length(walk_network,\n from_node,\n cutoff=cutoff_distance_m - from_dist,\n weight=\"distance\")\n for to_I in to_stop_Is:\n to_distance = stop_I_to_nearest_osm_node_distance[to_I]\n to_node = stop_I_to_nearest_osm_node[to_I]\n osm_distance = shortest_paths.get(to_node, float('inf'))\n total_distance = from_dist + osm_distance + to_distance\n from_stop_I_transfers = transfers[transfers['from_stop_I'] == from_I]\n straigth_distance = from_stop_I_transfers[from_stop_I_transfers[\"to_stop_I\"] == to_I][\"d\"].values[0]\n assert (straigth_distance < total_distance + 2) # allow for a maximum of 2 meters in calculations\n if total_distance <= cutoff_distance_m:\n gtfs.conn.execute(\"UPDATE stop_distances \"\n \"SET d_walk = \" + str(int(total_distance)) +\n \" WHERE from_stop_I=\" + str(from_I) + \" AND to_stop_I=\" + str(to_I))\n\n gtfs.conn.commit()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmatch stops to nodes in a GTFS object.", "response": "def match_stops_to_nodes(gtfs, walk_network):\n \"\"\"\n Parameters\n ----------\n gtfs : a GTFS object\n walk_network : networkx.Graph\n\n Returns\n -------\n stop_I_to_node: dict\n maps stop_I to closest walk_network node\n stop_I_to_dist: dict\n maps stop_I to the distance to the closest walk_network node\n \"\"\"\n network_nodes = walk_network.nodes(data=\"true\")\n\n stop_Is = set(gtfs.get_straight_line_transfer_distances()['from_stop_I'])\n stops_df = gtfs.stops()\n\n geo_index = GeoGridIndex(precision=6)\n for net_node, data in network_nodes:\n geo_index.add_point(GeoPoint(data['lat'], data['lon'], ref=net_node))\n stop_I_to_node = {}\n stop_I_to_dist = {}\n for stop_I in stop_Is:\n stop_lat = float(stops_df[stops_df.stop_I == stop_I].lat)\n stop_lon = float(stops_df[stops_df.stop_I == stop_I].lon)\n geo_point = GeoPoint(stop_lat, stop_lon)\n min_dist = float('inf')\n min_dist_node = None\n search_distances_m = [0.100, 0.500]\n for search_distance_m in search_distances_m:\n for point, distance in geo_index.get_nearest_points(geo_point, search_distance_m, \"km\"):\n if distance < min_dist:\n min_dist = distance * 1000\n min_dist_node = point.ref\n if min_dist_node is not None:\n break\n if min_dist_node is None:\n warn(\"No OSM node found for stop: \" + str(stops_df[stops_df.stop_I == stop_I]))\n stop_I_to_node[stop_I] = min_dist_node\n stop_I_to_dist[stop_I] = min_dist\n return stop_I_to_node, stop_I_to_dist"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconstructing the walk network for a stop transfer.", "response": "def walk_transfer_stop_to_stop_network(gtfs, max_link_distance=None):\n \"\"\"\n Construct the walk network.\n If OpenStreetMap-based walking distances have been computed, then those are used as the distance.\n Otherwise, the great circle distances (\"d\") is used.\n\n Parameters\n ----------\n gtfs: gtfspy.GTFS\n max_link_distance: int, optional\n If given, all walking transfers with great circle distance longer\n than this limit (expressed in meters) will be omitted.\n\n Returns\n -------\n net: networkx.DiGraph\n edges have attributes\n d:\n straight-line distance between stops\n d_walk:\n distance along the road/tracks/..\n \"\"\"\n if max_link_distance is None:\n max_link_distance = 1000\n net = networkx.Graph()\n _add_stops_to_net(net, gtfs.get_table(\"stops\"))\n stop_distances = gtfs.get_table(\"stop_distances\")\n if stop_distances[\"d_walk\"][0] is None:\n osm_distances_available = False\n warn(\"Warning: OpenStreetMap-based walking distances have not been computed, using euclidean distances instead.\"\n \"Ignore this warning if running unit tests.\")\n else:\n osm_distances_available = True\n\n for stop_distance_tuple in stop_distances.itertuples():\n from_node = stop_distance_tuple.from_stop_I\n to_node = stop_distance_tuple.to_stop_I\n\n if osm_distances_available:\n if stop_distance_tuple.d_walk > max_link_distance or isnan(stop_distance_tuple.d_walk):\n continue\n data = {'d': stop_distance_tuple.d, 'd_walk': stop_distance_tuple.d_walk}\n else:\n if stop_distance_tuple.d > max_link_distance:\n continue\n data = {'d': stop_distance_tuple.d}\n net.add_edge(from_node, to_node, data)\n return net"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef stop_to_stop_network_for_route_type(gtfs,\n route_type,\n link_attributes=None,\n start_time_ut=None,\n end_time_ut=None):\n \"\"\"\n Get a stop-to-stop network describing a single mode of travel.\n\n Parameters\n ----------\n gtfs : gtfspy.GTFS\n route_type : int\n See gtfspy.route_types.TRANSIT_ROUTE_TYPES for the list of possible types.\n link_attributes: list[str], optional\n defaulting to use the following link attributes:\n \"n_vehicles\" : Number of vehicles passed\n \"duration_min\" : minimum travel time between stops\n \"duration_max\" : maximum travel time between stops\n \"duration_median\" : median travel time between stops\n \"duration_avg\" : average travel time between stops\n \"d\" : distance along straight line (wgs84_distance)\n \"distance_shape\" : minimum distance along shape\n \"capacity_estimate\" : approximate capacity passed through the stop\n \"route_I_counts\" : dict from route_I to counts\n start_time_ut: int\n start time of the time span (in unix time)\n end_time_ut: int\n end time of the time span (in unix time)\n\n Returns\n -------\n net: networkx.DiGraph\n A directed graph Directed graph\n \"\"\"\n if link_attributes is None:\n link_attributes = DEFAULT_STOP_TO_STOP_LINK_ATTRIBUTES\n assert(route_type in route_types.TRANSIT_ROUTE_TYPES)\n\n stops_dataframe = gtfs.get_stops_for_route_type(route_type)\n net = networkx.DiGraph()\n _add_stops_to_net(net, stops_dataframe)\n\n events_df = gtfs.get_transit_events(start_time_ut=start_time_ut,\n end_time_ut=end_time_ut,\n route_type=route_type)\n if len(net.nodes()) < 2:\n assert events_df.shape[0] == 0\n\n # group events by links, and loop over them (i.e. each link):\n link_event_groups = events_df.groupby(['from_stop_I', 'to_stop_I'], sort=False)\n for key, link_events in link_event_groups:\n from_stop_I, to_stop_I = key\n assert isinstance(link_events, pd.DataFrame)\n # 'dep_time_ut' 'arr_time_ut' 'shape_id' 'route_type' 'trip_I' 'duration' 'from_seq' 'to_seq'\n if link_attributes is None:\n net.add_edge(from_stop_I, to_stop_I)\n else:\n link_data = {}\n if \"duration_min\" in link_attributes:\n link_data['duration_min'] = float(link_events['duration'].min())\n if \"duration_max\" in link_attributes:\n link_data['duration_max'] = float(link_events['duration'].max())\n if \"duration_median\" in link_attributes:\n link_data['duration_median'] = float(link_events['duration'].median())\n if \"duration_avg\" in link_attributes:\n link_data['duration_avg'] = float(link_events['duration'].mean())\n # statistics on numbers of vehicles:\n if \"n_vehicles\" in link_attributes:\n link_data['n_vehicles'] = int(link_events.shape[0])\n if \"capacity_estimate\" in link_attributes:\n link_data['capacity_estimate'] = route_types.ROUTE_TYPE_TO_APPROXIMATE_CAPACITY[route_type] \\\n * int(link_events.shape[0])\n if \"d\" in link_attributes:\n from_lat = net.node[from_stop_I]['lat']\n from_lon = net.node[from_stop_I]['lon']\n to_lat = net.node[to_stop_I]['lat']\n to_lon = net.node[to_stop_I]['lon']\n distance = wgs84_distance(from_lat, from_lon, to_lat, to_lon)\n link_data['d'] = int(distance)\n if \"distance_shape\" in link_attributes:\n assert \"shape_id\" in link_events.columns.values\n found = None\n for i, shape_id in enumerate(link_events[\"shape_id\"].values):\n if shape_id is not None:\n found = i\n break\n if found is None:\n link_data[\"distance_shape\"] = None\n else:\n link_event = link_events.iloc[found]\n distance = gtfs.get_shape_distance_between_stops(\n link_event[\"trip_I\"],\n int(link_event[\"from_seq\"]),\n int(link_event[\"to_seq\"])\n )\n link_data['distance_shape'] = distance\n if \"route_I_counts\" in link_attributes:\n link_data[\"route_I_counts\"] = link_events.groupby(\"route_I\").size().to_dict()\n net.add_edge(from_stop_I, to_stop_I, attr_dict=link_data)\n return net", "response": "Returns a stop - to - stop network describing a single mode of travel."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef stop_to_stop_networks_by_type(gtfs):\n route_type_to_network = dict()\n for route_type in route_types.ALL_ROUTE_TYPES:\n if route_type == route_types.WALK:\n net = walk_transfer_stop_to_stop_network(gtfs)\n else:\n net = stop_to_stop_network_for_route_type(gtfs, route_type)\n route_type_to_network[route_type] = net\n assert len(route_type_to_network) == len(route_types.ALL_ROUTE_TYPES)\n return route_type_to_network", "response": "Compute stop - to - stop networks for all travel modes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef combined_stop_to_stop_transit_network(gtfs, start_time_ut=None, end_time_ut=None):\n multi_di_graph = networkx.MultiDiGraph()\n for route_type in route_types.TRANSIT_ROUTE_TYPES:\n graph = stop_to_stop_network_for_route_type(gtfs, route_type,\n start_time_ut=start_time_ut, end_time_ut=end_time_ut)\n for from_node, to_node, data in graph.edges(data=True):\n data['route_type'] = route_type\n multi_di_graph.add_edges_from(graph.edges(data=True))\n multi_di_graph.add_nodes_from(graph.nodes(data=True))\n return multi_di_graph", "response": "Compute stop - to - stop networks for all travel modes and combine them into a single network."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nadd nodes from the pandas dataframe describing a part of the stops table in the GTFS database.", "response": "def _add_stops_to_net(net, stops):\n \"\"\"\n Add nodes to the network from the pandas dataframe describing (a part of the) stops table in the GTFS database.\n\n Parameters\n ----------\n net: networkx.Graph\n stops: pandas.DataFrame\n \"\"\"\n for stop in stops.itertuples():\n data = {\n \"lat\": stop.lat,\n \"lon\": stop.lon,\n \"name\": stop.name\n }\n net.add_node(stop.stop_I, data)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing the temporal network of the data and return it as a pandas. DataFrame.", "response": "def temporal_network(gtfs,\n start_time_ut=None,\n end_time_ut=None,\n route_type=None):\n \"\"\"\n Compute the temporal network of the data, and return it as a pandas.DataFrame\n\n Parameters\n ----------\n gtfs : gtfspy.GTFS\n start_time_ut: int | None\n start time of the time span (in unix time)\n end_time_ut: int | None\n end time of the time span (in unix time)\n route_type: int | None\n Specifies which mode of public transport are included, or whether all modes should be included.\n The int should be one of the standard GTFS route_types:\n (see also gtfspy.route_types.TRANSIT_ROUTE_TYPES )\n If route_type is not specified, all modes are included.\n\n Returns\n -------\n events_df: pandas.DataFrame\n Columns: departure_stop, arrival_stop, departure_time_ut, arrival_time_ut, route_type, route_I, trip_I\n \"\"\"\n events_df = gtfs.get_transit_events(start_time_ut=start_time_ut,\n end_time_ut=end_time_ut,\n route_type=route_type)\n events_df.drop('to_seq', 1, inplace=True)\n events_df.drop('shape_id', 1, inplace=True)\n events_df.drop('duration', 1, inplace=True)\n events_df.drop('route_id', 1, inplace=True)\n events_df.rename(\n columns={\n 'from_seq': \"seq\"\n },\n inplace=True\n )\n return events_df"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef route_to_route_network(gtfs, walking_threshold, start_time, end_time):\n graph = networkx.Graph()\n routes = gtfs.get_table(\"routes\")\n\n for i in routes.itertuples():\n graph.add_node(i.route_id, attr_dict={\"type\": i.type, \"color\": route_types.ROUTE_TYPE_TO_COLOR[i.type]})\n\n\n query = \"\"\"SELECT stop1.route_id AS route_id1, stop1.type, stop2.route_id AS route_id2, stop2.type FROM\n (SELECT * FROM stop_distances WHERE d_walk < %s) sd,\n (SELECT * FROM stop_times, trips, routes \n WHERE stop_times.trip_I=trips.trip_I AND trips.route_I=routes.route_I \n AND stop_times.dep_time_ds > %s AND stop_times.dep_time_ds < %s) stop1,\n (SELECT * FROM stop_times, trips, routes \n WHERE stop_times.trip_I=trips.trip_I AND trips.route_I=routes.route_I \n AND stop_times.dep_time_ds > %s AND stop_times.dep_time_ds < %s) stop2\n WHERE sd.from_stop_I = stop1.stop_I AND sd.to_stop_I = stop2.stop_I AND stop1.route_id != stop2.route_id\n GROUP BY stop1.route_id, stop2.route_id\"\"\" % (walking_threshold, start_time, end_time, start_time,\n end_time)\n df = gtfs.execute_custom_query_pandas(query)\n\n for items in df.itertuples():\n graph.add_edge(items.route_id1, items.route_id2)\n graph.remove_nodes_from(networkx.isolates(graph))\n return graph", "response": "This function creates a networkx graph where the nodes are bus routes and edges are edges between the routes."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mean_temporal_distance(self):\n total_width = self.end_time_dep - self.start_time_dep\n total_area = sum([block.area() for block in self._profile_blocks])\n return total_area / total_width", "response": "Get mean temporal distance to the target."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nplot the temporal distance cumulative density function.", "response": "def plot_temporal_distance_cdf(self):\n \"\"\"\n Plot the temporal distance cumulative density function.\n\n Returns\n -------\n fig: matplotlib.Figure\n \"\"\"\n xvalues, cdf = self.profile_block_analyzer._temporal_distance_cdf()\n fig = plt.figure()\n ax = fig.add_subplot(111)\n xvalues = numpy.array(xvalues) / 60.0\n ax.plot(xvalues, cdf, \"-k\")\n ax.fill_between(xvalues, cdf, color=\"red\", alpha=0.2)\n ax.set_ylabel(\"CDF(t)\")\n ax.set_xlabel(\"Temporal distance t (min)\")\n return fig"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nplotting the temporal distance probability density function.", "response": "def plot_temporal_distance_pdf(self, use_minutes=True, color=\"green\", ax=None):\n \"\"\"\n Plot the temporal distance probability density function.\n\n Returns\n -------\n fig: matplotlib.Figure\n \"\"\"\n from matplotlib import pyplot as plt\n plt.rc('text', usetex=True)\n temporal_distance_split_points_ordered, densities, delta_peaks = self._temporal_distance_pdf()\n xs = []\n for i, x in enumerate(temporal_distance_split_points_ordered):\n xs.append(x)\n xs.append(x)\n xs = numpy.array(xs)\n ys = [0]\n for y in densities:\n ys.append(y)\n ys.append(y)\n ys.append(0)\n ys = numpy.array(ys)\n # convert data to minutes:\n xlabel = \"Temporal distance (s)\"\n ylabel = \"Probability density (t)\"\n if use_minutes:\n xs /= 60.0\n ys *= 60.0\n xlabel = \"Temporal distance (min)\"\n delta_peaks = {peak / 60.0: mass for peak, mass in delta_peaks.items()}\n\n if ax is None:\n fig = plt.figure()\n ax = fig.add_subplot(111)\n ax.plot(xs, ys, \"k-\")\n ax.fill_between(xs, ys, color=\"green\", alpha=0.2)\n\n if delta_peaks:\n peak_height = max(ys) * 1.4\n max_x = max(xs)\n min_x = min(xs)\n now_max_x = max(xs) + 0.3 * (max_x - min_x)\n now_min_x = min_x - 0.1 * (max_x - min_x)\n\n text_x_offset = 0.1 * (now_max_x - max_x)\n\n for loc, mass in delta_peaks.items():\n ax.plot([loc, loc], [0, peak_height], color=\"green\", lw=5)\n ax.text(loc + text_x_offset, peak_height * 0.99, \"$P(\\\\mathrm{walk}) = %.2f$\" % (mass), color=\"green\")\n ax.set_xlim(now_min_x, now_max_x)\n\n tot_delta_peak_mass = sum(delta_peaks.values())\n transit_text_x = (min_x + max_x) / 2\n transit_text_y = min(ys[ys > 0]) / 2.\n ax.text(transit_text_x,\n transit_text_y,\n \"$P(mathrm{PT}) = %.2f$\" % (1 - tot_delta_peak_mass),\n color=\"green\",\n va=\"center\",\n ha=\"center\")\n\n ax.set_xlabel(xlabel)\n ax.set_ylabel(ylabel)\n ax.set_ylim(bottom=0)\n return ax.figure"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nplots the temporal distance probability density function.", "response": "def plot_temporal_distance_pdf_horizontal(self, use_minutes=True,\n color=\"green\",\n ax=None,\n duration_divider=60.0,\n legend_font_size=None,\n legend_loc=None):\n \"\"\"\n Plot the temporal distance probability density function.\n\n Returns\n -------\n fig: matplotlib.Figure\n \"\"\"\n from matplotlib import pyplot as plt\n plt.rc('text', usetex=True)\n\n if ax is None:\n fig = plt.figure()\n ax = fig.add_subplot(111)\n\n temporal_distance_split_points_ordered, densities, delta_peaks = self._temporal_distance_pdf()\n xs = []\n for i, x in enumerate(temporal_distance_split_points_ordered):\n xs.append(x)\n xs.append(x)\n xs = numpy.array(xs)\n ys = [0]\n for y in densities:\n ys.append(y)\n ys.append(y)\n ys.append(0)\n ys = numpy.array(ys)\n # convert data to minutes:\n xlabel = \"Temporal distance (s)\"\n ylabel = \"Probability density $P(\\\\tau)$\"\n if use_minutes:\n xs /= duration_divider\n ys *= duration_divider\n xlabel = \"Temporal distance (min)\"\n delta_peaks = {peak / 60.0: mass for peak, mass in delta_peaks.items()}\n\n if delta_peaks:\n peak_height = max(ys) * 1.4\n max_x = max(xs)\n min_x = min(xs)\n now_max_x = max(xs) + 0.3 * (max_x - min_x)\n now_min_x = min_x - 0.1 * (max_x - min_x)\n\n text_x_offset = 0.1 * (now_max_x - max_x)\n\n for loc, mass in delta_peaks.items():\n text = \"$P(\\\\mathrm{walk}) = \" + (\"%.2f$\" % (mass))\n ax.plot([0, peak_height], [loc, loc], color=color, lw=5, label=text)\n\n ax.plot(ys, xs, \"k-\")\n if delta_peaks:\n tot_delta_peak_mass = sum(delta_peaks.values())\n fill_label = \"$P(\\\\mathrm{PT}) = %.2f$\" % (1-tot_delta_peak_mass)\n else:\n fill_label = None\n ax.fill_betweenx(xs, ys, color=color, alpha=0.2, label=fill_label)\n\n ax.set_ylabel(xlabel)\n ax.set_xlabel(ylabel)\n ax.set_xlim(left=0, right=max(ys) * 1.2)\n if delta_peaks:\n if legend_font_size is None:\n legend_font_size = 12\n if legend_loc is None:\n legend_loc = \"best\"\n ax.legend(loc=legend_loc, prop={'size': legend_font_size})\n\n\n if True:\n line_tyles = [\"-.\", \"--\", \"-\"][::-1]\n to_plot_funcs = [self.max_temporal_distance, self.mean_temporal_distance, self.min_temporal_distance]\n\n xmin, xmax = ax.get_xlim()\n for to_plot_func, ls in zip(to_plot_funcs, line_tyles):\n y = to_plot_func() / duration_divider\n assert y < float('inf')\n # factor of 10 just to be safe that the lines cover the whole region.\n ax.plot([xmin, xmax*10], [y, y], color=\"black\", ls=ls, lw=1)\n\n return ax.figure"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nplotting the temporal distance profile of the current state of the object.", "response": "def plot_temporal_distance_profile(self,\n timezone=None,\n color=\"black\",\n alpha=0.15,\n ax=None,\n lw=2,\n label=\"\",\n plot_tdist_stats=False,\n plot_trip_stats=False,\n format_string=\"%Y-%m-%d %H:%M:%S\",\n plot_journeys=False,\n duration_divider=60.0,\n fill_color=\"green\",\n journey_letters=None,\n return_letters=False):\n \"\"\"\n Parameters\n ----------\n timezone: str\n color: color\n format_string: str, None\n if None, the original values are used\n plot_journeys: bool, optional\n if True, small dots are plotted at the departure times\n \"\"\"\n if ax is None:\n fig = plt.figure()\n ax = fig.add_subplot(111)\n\n if timezone is None:\n warnings.warn(\"Warning: No timezone specified, defaulting to UTC\")\n timezone = pytz.timezone(\"Etc/UTC\")\n\n def _ut_to_unloc_datetime(ut):\n dt = datetime.datetime.fromtimestamp(ut, timezone)\n return dt.replace(tzinfo=None)\n\n if format_string:\n x_axis_formatter = md.DateFormatter(format_string)\n ax.xaxis.set_major_formatter(x_axis_formatter)\n else:\n _ut_to_unloc_datetime = lambda x: x\n\n ax.set_xlim(\n _ut_to_unloc_datetime(self.start_time_dep),\n _ut_to_unloc_datetime(self.end_time_dep)\n )\n\n if plot_tdist_stats:\n line_tyles = [\"-.\", \"--\", \"-\"][::-1]\n # to_plot_labels = [\"maximum temporal distance\", \"mean temporal distance\", \"minimum temporal distance\"]\n to_plot_labels = [\"$\\\\tau_\\\\mathrm{max} \\\\;$ = \", \"$\\\\tau_\\\\mathrm{mean}$ = \", \"$\\\\tau_\\\\mathrm{min} \\\\:\\\\:$ = \"]\n to_plot_funcs = [self.max_temporal_distance, self.mean_temporal_distance, self.min_temporal_distance]\n\n xmin, xmax = ax.get_xlim()\n for to_plot_label, to_plot_func, ls in zip(to_plot_labels, to_plot_funcs, line_tyles):\n y = to_plot_func() / duration_divider\n assert y < float('inf'), to_plot_label\n to_plot_label = to_plot_label + \"%.1f min\" % (y)\n ax.plot([xmin, xmax], [y, y], color=\"black\", ls=ls, lw=1, label=to_plot_label)\n\n if plot_trip_stats:\n assert (not plot_tdist_stats)\n line_tyles = [\"-\", \"-.\", \"--\"]\n to_plot_labels = [\"min journey duration\", \"max journey duration\", \"mean journey duration\"]\n to_plot_funcs = [self.min_trip_duration, self.max_trip_duration, self.mean_trip_duration]\n\n xmin, xmax = ax.get_xlim()\n for to_plot_label, to_plot_func, ls in zip(to_plot_labels, to_plot_funcs, line_tyles):\n y = to_plot_func() / duration_divider\n if not numpy.math.isnan(y):\n ax.plot([xmin, xmax], [y, y], color=\"red\", ls=ls, lw=2)\n txt = to_plot_label + \"\\n = %.1f min\" % y\n ax.text(xmax + 0.01 * (xmax - xmin), y, txt, color=\"red\", va=\"center\", ha=\"left\")\n\n old_xmax = xmax\n xmax += (xmax - xmin) * 0.3\n ymin, ymax = ax.get_ylim()\n ax.fill_between([old_xmax, xmax], ymin, ymax, color=\"gray\", alpha=0.1)\n ax.set_xlim(xmin, xmax)\n\n # plot the actual profile\n vertical_lines, slopes = self.profile_block_analyzer.get_vlines_and_slopes_for_plotting()\n for i, line in enumerate(slopes):\n xs = [_ut_to_unloc_datetime(x) for x in line['x']]\n if i is 0:\n label = u\"profile\"\n else:\n label = None\n ax.plot(xs, numpy.array(line['y']) / duration_divider, \"-\", color=color, lw=lw, label=label)\n\n for line in vertical_lines:\n xs = [_ut_to_unloc_datetime(x) for x in line['x']]\n ax.plot(xs, numpy.array(line['y']) / duration_divider, \":\", color=color) # , lw=lw)\n\n assert (isinstance(ax, plt.Axes))\n\n if plot_journeys:\n xs = [_ut_to_unloc_datetime(x) for x in self.trip_departure_times]\n ys = self.trip_durations\n ax.plot(xs, numpy.array(ys) / duration_divider, \"o\", color=\"black\", ms=8, label=\"journeys\")\n if journey_letters is None:\n journey_letters = \"ABCDEFGHIJKLMNOPQRSTUVWXYZ\"\n\n def cycle_journey_letters(journey_letters):\n # cycle('ABCD') --> A B C D A B C D A B C D ...\n saved = []\n for element in journey_letters:\n yield element\n saved.append(element)\n count = 1\n while saved:\n for element in saved:\n yield element + str(count)\n count += 1\n journey_letters_iterator = cycle_journey_letters(journey_letters)\n time_letters = {int(time): letter for letter, time in zip(journey_letters_iterator, self.trip_departure_times)}\n for x, y, letter in zip(xs, ys, journey_letters_iterator):\n walking = - self._walk_time_to_target / 30 if numpy.isfinite(self._walk_time_to_target) else 0\n ax.text(x + datetime.timedelta(seconds=(self.end_time_dep - self.start_time_dep) / 60),\n (y + walking) / duration_divider, letter, va=\"top\", ha=\"left\")\n\n fill_between_x = []\n fill_between_y = []\n for line in slopes:\n xs = [_ut_to_unloc_datetime(x) for x in line['x']]\n fill_between_x.extend(xs)\n fill_between_y.extend(numpy.array(line[\"y\"]) / duration_divider)\n\n ax.fill_between(fill_between_x, y1=fill_between_y, color=fill_color, alpha=alpha, label=label)\n\n ax.set_ylim(bottom=0)\n ax.set_ylim(ax.get_ylim()[0], ax.get_ylim()[1] * 1.05)\n\n if rcParams['text.usetex']:\n ax.set_xlabel(r\"Departure time $t_{\\mathrm{dep}}$\")\n else:\n ax.set_xlabel(\"Departure time\")\n\n ax.set_ylabel(r\"Temporal distance $\\tau$ (min)\")\n if plot_journeys and return_letters:\n return ax, time_letters\n else:\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_leg(self, leg):\n assert(isinstance(leg, Connection))\n if not self.legs:\n self.departure_time = leg.departure_time\n self.arrival_time = leg.arrival_time\n if leg.trip_id and (not self.legs or (leg.trip_id != self.legs[-1].trip_id)):\n self.n_boardings += 1\n self.arrival_time = leg.arrival_time\n self.legs.append(leg)", "response": "Adds a new entry to the list of active legs."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets stop pairs through which transfers take place Returns ------- list", "response": "def get_transfer_stop_pairs(self):\n \"\"\"\n Get stop pairs through which transfers take place\n\n Returns\n -------\n transfer_stop_pairs: list\n \"\"\"\n transfer_stop_pairs = []\n previous_arrival_stop = None\n current_trip_id = None\n for leg in self.legs:\n if leg.trip_id is not None and leg.trip_id != current_trip_id and previous_arrival_stop is not None:\n transfer_stop_pair = (previous_arrival_stop, leg.departure_stop)\n transfer_stop_pairs.append(transfer_stop_pair)\n previous_arrival_stop = leg.arrival_stop\n current_trip_id = leg.trip_id\n return transfer_stop_pairs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _truncate_colormap(cmap, minval=0.0, maxval=1.0, n=100):\n new_cmap = LinearSegmentedColormap.from_list(\n 'trunc({n},{a:.2f},{b:.2f})'.format(n=cmap.name, a=minval, b=maxval),\n cmap(numpy.linspace(minval, maxval, n))\n )\n return new_cmap", "response": "Truncates a colormap to use."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a NodeProfileAnalyzerTime object for the given max_n_boardings.", "response": "def get_time_profile_analyzer(self, max_n_boardings=None):\n \"\"\"\n Parameters\n ----------\n max_n_boardings: int\n The maximum number of boardings allowed for the labels used to construct the \"temporal distance profile\"\n\n Returns\n -------\n analyzer: NodeProfileAnalyzerTime\n \"\"\"\n if max_n_boardings is None:\n max_n_boardings = self.max_trip_n_boardings()\n # compute only if not yet computed\n if not max_n_boardings in self._n_boardings_to_simple_time_analyzers:\n if max_n_boardings == 0:\n valids = []\n else:\n candidate_labels = [LabelTimeSimple(label.departure_time, label.arrival_time_target)\n for label in self._node_profile_final_labels if\n ((self.start_time_dep <= label.departure_time)\n and label.n_boardings <= max_n_boardings)]\n valids = compute_pareto_front(candidate_labels)\n valids.sort(key=lambda label: -label.departure_time)\n profile = NodeProfileSimple(self._walk_to_target_duration)\n for valid in valids:\n profile.update_pareto_optimal_tuples(valid)\n npat = NodeProfileAnalyzerTime.from_profile(profile, self.start_time_dep, self.end_time_dep)\n self._n_boardings_to_simple_time_analyzers[max_n_boardings] = npat\n return self._n_boardings_to_simple_time_analyzers[max_n_boardings]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the median temporal distance of vehicle legs in the list.", "response": "def median_temporal_distances(self, min_n_boardings=None, max_n_boardings=None):\n \"\"\"\n Returns\n -------\n mean_temporal_distances: list\n list indices encode the number of vehicle legs each element\n in the list tells gets the mean temporal distance\n \"\"\"\n if min_n_boardings is None:\n min_n_boardings = 0\n\n if max_n_boardings is None:\n max_n_boardings = self.max_trip_n_boardings()\n if max_n_boardings is None:\n max_n_boardings = 0\n\n median_temporal_distances = [float('inf') for _ in range(min_n_boardings, max_n_boardings + 1)]\n for n_boardings in range(min_n_boardings, max_n_boardings + 1):\n simple_analyzer = self.get_time_profile_analyzer(n_boardings)\n median_temporal_distances[n_boardings] = simple_analyzer.median_temporal_distance()\n return median_temporal_distances"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_directory_as_inmemory_db(cls, gtfs_directory):\n # this import is here to avoid circular imports (which turned out to be a problem)\n from gtfspy.import_gtfs import import_gtfs\n conn = sqlite3.connect(\":memory:\")\n import_gtfs(gtfs_directory,\n conn,\n preserve_connection=True,\n print_progress=False)\n return cls(conn)", "response": "Instantiate a GTFS object by computing\n from a directory in memory."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the path to the main database", "response": "def get_main_database_path(self):\n \"\"\"\n Should return the path to the database\n\n Returns\n -------\n path : unicode\n path to the database, empty string for in-memory databases\n \"\"\"\n cur = self.conn.cursor()\n cur.execute(\"PRAGMA database_list\")\n rows = cur.fetchall()\n for row in rows:\n if row[1] == str(\"main\"):\n return row[2]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the distance along a shape between stops.", "response": "def get_shape_distance_between_stops(self, trip_I, from_stop_seq, to_stop_seq):\n \"\"\"\n Get the distance along a shape between stops\n\n Parameters\n ----------\n trip_I : int\n trip_ID along which we travel\n from_stop_seq : int\n the sequence number of the 'origin' stop\n to_stop_seq : int\n the sequence number of the 'destination' stop\n\n Returns\n -------\n distance : float, None\n If the shape calculation succeeded, return a float, otherwise return None\n (i.e. in the case where the shapes table is empty)\n \"\"\"\n\n query_template = \"SELECT shape_break FROM stop_times WHERE trip_I={trip_I} AND seq={seq} \"\n stop_seqs = [from_stop_seq, to_stop_seq]\n shape_breaks = []\n for seq in stop_seqs:\n q = query_template.format(seq=seq, trip_I=trip_I)\n shape_breaks.append(self.conn.execute(q).fetchone())\n query_template = \"SELECT max(d) - min(d) \" \\\n \"FROM shapes JOIN trips ON(trips.shape_id=shapes.shape_id) \" \\\n \"WHERE trip_I={trip_I} AND shapes.seq>={from_stop_seq} AND shapes.seq<={to_stop_seq};\"\n distance_query = query_template.format(trip_I=trip_I, from_stop_seq=from_stop_seq, to_stop_seq=to_stop_seq)\n return self.conn.execute(distance_query).fetchone()[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_directly_accessible_stops_within_distance(self, stop, distance):\n query = \"\"\"SELECT stop.* FROM\n (SELECT st2.* FROM \n (SELECT * FROM stop_distances\n WHERE from_stop_I = %s) sd,\n (SELECT * FROM stop_times) st1,\n (SELECT * FROM stop_times) st2\n WHERE sd.d < %s AND sd.to_stop_I = st1.stop_I AND st1.trip_I = st2.trip_I \n GROUP BY st2.stop_I) sq,\n (SELECT * FROM stops) stop\n WHERE sq.stop_I = stop.stop_I\"\"\" % (stop, distance)\n return pd.read_sql_query(query, self.conn)", "response": "Returns the stops that are accessible without transfer from the stops that are within a specific walking distance."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the name of the GTFS timezone", "response": "def get_timezone_name(self):\n \"\"\"\n Get name of the GTFS timezone\n\n Returns\n -------\n timezone_name : str\n name of the time zone, e.g. \"Europe/Helsinki\"\n \"\"\"\n tz_name = self.conn.execute('SELECT timezone FROM agencies LIMIT 1').fetchone()\n if tz_name is None:\n raise ValueError(\"This database does not have a timezone defined.\")\n return tz_name[0]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_timezone_string(self, dt=None):\n if dt is None:\n download_date = self.meta.get('download_date')\n if download_date:\n dt = datetime.datetime.strptime(download_date, '%Y-%m-%d')\n else:\n dt = datetime.datetime.today()\n loc_dt = self._timezone.localize(dt)\n # get the timezone\n timezone_string = loc_dt.strftime(\"%z\")\n return timezone_string", "response": "Returns the timezone string of the GTFS database object as a string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef unlocalized_datetime_to_ut_seconds(self, unlocalized_datetime):\n loc_dt = self._timezone.localize(unlocalized_datetime)\n unixtime_seconds = calendar.timegm(loc_dt.utctimetuple())\n return unixtime_seconds", "response": "Convert a datetime object to unixtime."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_day_start_ut(self, date):\n if isinstance(date, string_types):\n date = datetime.datetime.strptime(date, '%Y-%m-%d')\n\n date_noon = datetime.datetime(date.year, date.month, date.day, 12, 0, 0)\n ut_noon = self.unlocalized_datetime_to_ut_seconds(date_noon)\n return ut_noon - 12 * 60 * 60", "response": "Get day start time as unix time in seconds"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting all trip trajectories within a given time span.", "response": "def get_trip_trajectories_within_timespan(self, start, end, use_shapes=True, filter_name=None):\n \"\"\"\n Get complete trip data for visualizing public transport operation based on gtfs.\n\n Parameters\n ----------\n start: number\n Earliest position data to return (in unix time)\n end: number\n Latest position data to return (in unix time)\n use_shapes: bool, optional\n Whether or not shapes should be included\n filter_name: str\n Pick only routes having this name.\n\n Returns\n -------\n trips: dict\n trips['trips'] is a list whose each element (e.g. el = trips['trips'][0])\n is a dict with the following properties:\n el['lats'] -- list of latitudes\n el['lons'] -- list of longitudes\n el['times'] -- list of passage_times\n el['route_type'] -- type of vehicle as specified by GTFS\n el['name'] -- name of the route\n \"\"\"\n trips = []\n trip_df = self.get_tripIs_active_in_range(start, end)\n print(\"gtfs_viz.py: fetched \" + str(len(trip_df)) + \" trip ids\")\n shape_cache = {}\n\n # loop over all trips:\n for row in trip_df.itertuples():\n trip_I = row.trip_I\n day_start_ut = row.day_start_ut\n shape_id = row.shape_id\n\n trip = {}\n\n name, route_type = self.get_route_name_and_type_of_tripI(trip_I)\n trip['route_type'] = int(route_type)\n trip['name'] = str(name)\n\n if filter_name and (name != filter_name):\n continue\n\n stop_lats = []\n stop_lons = []\n stop_dep_times = []\n shape_breaks = []\n stop_seqs = []\n\n # get stop_data and store it:\n stop_time_df = self.get_trip_stop_time_data(trip_I, day_start_ut)\n for stop_row in stop_time_df.itertuples():\n stop_lats.append(float(stop_row.lat))\n stop_lons.append(float(stop_row.lon))\n stop_dep_times.append(float(stop_row.dep_time_ut))\n try:\n stop_seqs.append(int(stop_row.seq))\n except TypeError:\n stop_seqs.append(None)\n if use_shapes:\n try:\n shape_breaks.append(int(stop_row.shape_break))\n except (TypeError, ValueError):\n shape_breaks.append(None)\n\n if use_shapes:\n # get shape data (from cache, if possible)\n if shape_id not in shape_cache:\n shape_cache[shape_id] = shapes.get_shape_points2(self.conn.cursor(), shape_id)\n shape_data = shape_cache[shape_id]\n # noinspection PyBroadException\n try:\n trip['times'] = shapes.interpolate_shape_times(shape_data['d'], shape_breaks, stop_dep_times)\n trip['lats'] = shape_data['lats']\n trip['lons'] = shape_data['lons']\n start_break = shape_breaks[0]\n end_break = shape_breaks[-1]\n trip['times'] = trip['times'][start_break:end_break + 1]\n trip['lats'] = trip['lats'][start_break:end_break + 1]\n trip['lons'] = trip['lons'][start_break:end_break + 1]\n except:\n # In case interpolation fails:\n trip['times'] = stop_dep_times\n trip['lats'] = stop_lats\n trip['lons'] = stop_lons\n else:\n trip['times'] = stop_dep_times\n trip['lats'] = stop_lats\n trip['lons'] = stop_lons\n trips.append(trip)\n return {\"trips\": trips}"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_stop_count_data(self, start_ut, end_ut):\n # TODO! this function could perhaps be made a single sql query now with the new tables?\n trips_df = self.get_tripIs_active_in_range(start_ut, end_ut)\n # stop_I -> count, lat, lon, name\n stop_counts = Counter()\n\n # loop over all trips:\n for row in trips_df.itertuples():\n # get stop_data and store it:\n stops_seq = self.get_trip_stop_time_data(row.trip_I, row.day_start_ut)\n for stop_time_row in stops_seq.itertuples(index=False):\n if (stop_time_row.dep_time_ut >= start_ut) and (stop_time_row.dep_time_ut <= end_ut):\n stop_counts[stop_time_row.stop_I] += 1\n\n all_stop_data = self.stops()\n counts = [stop_counts[stop_I] for stop_I in all_stop_data[\"stop_I\"].values]\n\n all_stop_data.loc[:, \"count\"] = pd.Series(counts, index=all_stop_data.index)\n return all_stop_data", "response": "Get stop count data."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_segment_count_data(self, start, end, use_shapes=True):\n cur = self.conn.cursor()\n # get all possible trip_ids that take place between start and end\n trips_df = self.get_tripIs_active_in_range(start, end)\n # stop_I -> count, lat, lon, name\n segment_counts = Counter()\n seg_to_info = {}\n # tripI_to_seq = \"inverted segToShapeData\"\n tripI_to_seq = defaultdict(list)\n\n # loop over all trips:\n for row in trips_df.itertuples():\n # get stop_data and store it:\n stops_df = self.get_trip_stop_time_data(row.trip_I, row.day_start_ut)\n for i in range(len(stops_df) - 1):\n (stop_I, dep_time_ut, s_lat, s_lon, s_seq, shape_break) = stops_df.iloc[i]\n (stop_I_n, dep_time_ut_n, s_lat_n, s_lon_n, s_seq_n, shape_break_n) = stops_df.iloc[i + 1]\n # test if _contained_ in the interval\n # overlap would read:\n # (dep_time_ut <= end) and (start <= dep_time_ut_n)\n if (dep_time_ut >= start) and (dep_time_ut_n <= end):\n seg = (stop_I, stop_I_n)\n segment_counts[seg] += 1\n if seg not in seg_to_info:\n seg_to_info[seg] = {\n u\"trip_I\": row.trip_I,\n u\"lats\": [s_lat, s_lat_n],\n u\"lons\": [s_lon, s_lon_n],\n u\"shape_id\": row.shape_id,\n u\"stop_seqs\": [s_seq, s_seq_n],\n u\"shape_breaks\": [shape_break, shape_break_n]\n }\n tripI_to_seq[row.trip_I].append(seg)\n\n stop_names = {}\n for (stop_I, stop_J) in segment_counts.keys():\n for s in [stop_I, stop_J]:\n if s not in stop_names:\n stop_names[s] = self.stop(s)[u'name'].values[0]\n\n seg_data = []\n for seg, count in segment_counts.items():\n segInfo = seg_to_info[seg]\n shape_breaks = segInfo[u\"shape_breaks\"]\n seg_el = {}\n if use_shapes and shape_breaks and shape_breaks[0] and shape_breaks[1]:\n shape = shapes.get_shape_between_stops(\n cur,\n segInfo[u'trip_I'],\n shape_breaks=shape_breaks\n )\n seg_el[u'lats'] = segInfo[u'lats'][:1] + shape[u'lat'] + segInfo[u'lats'][1:]\n seg_el[u'lons'] = segInfo[u'lons'][:1] + shape[u'lon'] + segInfo[u'lons'][1:]\n else:\n seg_el[u'lats'] = segInfo[u'lats']\n seg_el[u'lons'] = segInfo[u'lons']\n seg_el[u'name'] = stop_names[seg[0]] + u\"-\" + stop_names[seg[1]]\n seg_el[u'count'] = count\n seg_data.append(seg_el)\n return seg_data", "response": "Get the count of vehicle segments that are fully contained within the interval start and end."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngets the shapes of all routes.", "response": "def get_all_route_shapes(self, use_shapes=True):\n \"\"\"\n Get the shapes of all routes.\n\n Parameters\n ----------\n use_shapes : bool, optional\n by default True (i.e. use shapes as the name of the function indicates)\n if False (fall back to lats and longitudes)\n\n Returns\n -------\n routeShapes: list of dicts that should have the following keys\n name, type, agency, lats, lons\n with types\n list, list, str, list, list\n \"\"\"\n cur = self.conn.cursor()\n\n # all shape_id:s corresponding to a route_I:\n # query = \"SELECT DISTINCT name, shape_id, trips.route_I, route_type\n # FROM trips LEFT JOIN routes USING(route_I)\"\n # data1 = pd.read_sql_query(query, self.conn)\n # one (arbitrary) shape_id per route_I (\"one direction\") -> less than half of the routes\n query = \"SELECT routes.name as name, shape_id, route_I, trip_I, routes.type, \" \\\n \" agency_id, agencies.name as agency_name, max(end_time_ds-start_time_ds) as trip_duration \" \\\n \"FROM trips \" \\\n \"LEFT JOIN routes \" \\\n \"USING(route_I) \" \\\n \"LEFT JOIN agencies \" \\\n \"USING(agency_I) \" \\\n \"GROUP BY routes.route_I\"\n data = pd.read_sql_query(query, self.conn)\n\n routeShapes = []\n for i, row in enumerate(data.itertuples()):\n datum = {\"name\": str(row.name), \"type\": int(row.type), \"route_I\": row.route_I, \"agency\": str(row.agency_id),\n \"agency_name\": str(row.agency_name)}\n # this function should be made also non-shape friendly (at this point)\n if use_shapes and row.shape_id:\n shape = shapes.get_shape_points2(cur, row.shape_id)\n lats = shape['lats']\n lons = shape['lons']\n else:\n stop_shape = self.get_trip_stop_coordinates(row.trip_I)\n lats = list(stop_shape['lat'])\n lons = list(stop_shape['lon'])\n datum['lats'] = [float(lat) for lat in lats]\n datum['lons'] = [float(lon) for lon in lons]\n routeShapes.append(datum)\n return routeShapes"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the list of trip_IDs that are active between start and end times.", "response": "def get_tripIs_active_in_range(self, start, end):\n \"\"\"\n Obtain from the (standard) GTFS database, list of trip_IDs (and other trip_related info)\n that are active between given 'start' and 'end' times.\n\n The start time of a trip is determined by the departure time at the last stop of the trip.\n The end time of a trip is determined by the arrival time at the last stop of the trip.\n\n Parameters\n ----------\n start, end : int\n the start and end of the time interval in unix time seconds\n\n Returns\n -------\n active_trips : pandas.DataFrame with columns\n trip_I, day_start_ut, start_time_ut, end_time_ut, shape_id\n \"\"\"\n to_select = \"trip_I, day_start_ut, start_time_ut, end_time_ut, shape_id \"\n query = \"SELECT \" + to_select + \\\n \"FROM day_trips \" \\\n \"WHERE \" \\\n \"(end_time_ut > {start_ut} AND start_time_ut < {end_ut})\".format(start_ut=start, end_ut=end)\n return pd.read_sql_query(query, self.conn)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_trip_counts_per_day(self):\n query = \"SELECT date, count(*) AS number_of_trips FROM day_trips GROUP BY date\"\n # this yields the actual data\n trip_counts_per_day = pd.read_sql_query(query, self.conn, index_col=\"date\")\n # the rest is simply code for filling out \"gaps\" in the time span\n # (necessary for some visualizations)\n max_day = trip_counts_per_day.index.max()\n min_day = trip_counts_per_day.index.min()\n min_date = datetime.datetime.strptime(min_day, '%Y-%m-%d')\n max_date = datetime.datetime.strptime(max_day, '%Y-%m-%d')\n num_days = (max_date - min_date).days\n dates = [min_date + datetime.timedelta(days=x) for x in range(num_days + 1)]\n trip_counts = []\n date_strings = []\n for date in dates:\n date_string = date.strftime(\"%Y-%m-%d\")\n date_strings.append(date_string)\n try:\n value = trip_counts_per_day.loc[date_string, 'number_of_trips']\n except KeyError:\n # set value to 0 if dsut is not present, i.e. when no trips\n # take place on that day\n value = 0\n trip_counts.append(value)\n # check that all date_strings are included (move this to tests?)\n for date_string in trip_counts_per_day.index:\n assert date_string in date_strings\n data = {\"date\": dates, \"date_str\": date_strings, \"trip_counts\": trip_counts}\n return pd.DataFrame(data)", "response": "Get the trip counts per day between the start and end day of the feed."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_suitable_date_for_daily_extract(self, date=None, ut=False):\n daily_trips = self.get_trip_counts_per_day()\n max_daily_trips = daily_trips[u'trip_counts'].max(axis=0)\n if date in daily_trips[u'date_str']:\n start_index = daily_trips[daily_trips[u'date_str'] == date].index.tolist()[0]\n daily_trips[u'old_index'] = daily_trips.index\n daily_trips[u'date_dist'] = abs(start_index - daily_trips.index)\n daily_trips = daily_trips.sort_values(by=[u'date_dist', u'old_index']).reindex()\n for row in daily_trips.itertuples():\n if row.trip_counts >= 0.9 * max_daily_trips:\n if ut:\n return self.get_day_start_ut(row.date_str)\n else:\n return row.date_str", "response": "Returns the suitable date for daily extract."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_weekly_extract_start_date(self, ut=False, weekdays_at_least_of_max=0.9,\n verbose=False, download_date_override=None):\n \"\"\"\n Find a suitable weekly extract start date (monday).\n The goal is to obtain as 'usual' week as possible.\n The weekdays of the weekly extract week should contain\n at least 0.9 of the total maximum of trips.\n\n Parameters\n ----------\n ut: return unixtime?\n weekdays_at_least_of_max: float\n\n download_date_override: str, semi-optional\n Download-date in format %Y-%m-%d, weeks close to this.\n Overrides the (possibly) recorded downloaded date in the database\n\n Returns\n -------\n date: int or str\n\n Raises\n ------\n error: RuntimeError\n If no download date could be found.\n \"\"\"\n daily_trip_counts = self.get_trip_counts_per_day()\n if isinstance(download_date_override, str):\n search_start_date = datetime.datetime.strptime(download_date_override, \"%Y-%m-%d\")\n elif isinstance(download_date_override, datetime.datetime):\n search_start_date = download_date_override\n else:\n assert download_date_override is None\n download_date_str = self.meta['download_date']\n if download_date_str == \"\":\n warnings.warn(\"Download date is not speficied in the database. \"\n \"Download date used in GTFS.\" + self.get_weekly_extract_start_date.__name__ +\n \"() defaults to the smallest date when any operations take place.\")\n search_start_date = daily_trip_counts['date'].min()\n else:\n search_start_date = datetime.datetime.strptime(download_date_str, \"%Y-%m-%d\")\n\n feed_min_date = daily_trip_counts['date'].min()\n feed_max_date = daily_trip_counts['date'].max()\n assert (feed_max_date - feed_min_date >= datetime.timedelta(days=7)), \\\n \"Dataset is not long enough for providing week long extracts\"\n\n # get first a valid monday where the search for the week can be started:\n next_monday_from_search_start_date = search_start_date + timedelta(days=(7 - search_start_date.weekday()))\n if not (feed_min_date <= next_monday_from_search_start_date <= feed_max_date):\n warnings.warn(\"The next monday after the (possibly user) specified download date is not present in the database.\"\n \"Resorting to first monday after the beginning of operations instead.\")\n next_monday_from_search_start_date = feed_min_date + timedelta(days=(7 - feed_min_date.weekday()))\n\n max_trip_count = daily_trip_counts['trip_counts'].quantile(0.95)\n # Take 95th percentile to omit special days, if any exist.\n\n threshold = weekdays_at_least_of_max * max_trip_count\n threshold_fulfilling_days = daily_trip_counts['trip_counts'] > threshold\n\n # look forward first\n # get the index of the trip:\n search_start_monday_index = daily_trip_counts[daily_trip_counts['date'] == next_monday_from_search_start_date].index[0]\n\n # get starting point\n while_loop_monday_index = search_start_monday_index\n while len(daily_trip_counts.index) >= while_loop_monday_index + 7:\n if all(threshold_fulfilling_days[while_loop_monday_index:while_loop_monday_index + 5]):\n row = daily_trip_counts.iloc[while_loop_monday_index]\n if ut:\n return self.get_day_start_ut(row.date_str)\n else:\n return row['date']\n while_loop_monday_index += 7\n\n while_loop_monday_index = search_start_monday_index - 7\n # then backwards\n while while_loop_monday_index >= 0:\n if all(threshold_fulfilling_days[while_loop_monday_index:while_loop_monday_index + 5]):\n row = daily_trip_counts.iloc[while_loop_monday_index]\n if ut:\n return self.get_day_start_ut(row.date_str)\n else:\n return row['date']\n while_loop_monday_index -= 7\n\n raise RuntimeError(\"No suitable weekly extract start date could be determined!\")", "response": "Returns a date that can be used to obtain the start date of the weekly extract week."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_closest_stop(self, lat, lon):\n cur = self.conn.cursor()\n min_dist = float(\"inf\")\n min_stop_I = None\n rows = cur.execute(\"SELECT stop_I, lat, lon FROM stops\")\n for stop_I, lat_s, lon_s in rows:\n dist_now = wgs84_distance(lat, lon, lat_s, lon_s)\n if dist_now < min_dist:\n min_dist = dist_now\n min_stop_I = stop_I\n return min_stop_I", "response": "Get the closest stop to a given location."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_route_name_and_type_of_tripI(self, trip_I):\n cur = self.conn.cursor()\n results = cur.execute(\"SELECT name, type FROM routes JOIN trips USING(route_I) WHERE trip_I={trip_I}\"\n .format(trip_I=trip_I))\n name, rtype = results.fetchone()\n return u\"%s\" % str(name), int(rtype)", "response": "Get route short name and type of a given trip."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget route short name and type of a route.", "response": "def get_route_name_and_type(self, route_I):\n \"\"\"\n Get route short name and type\n\n Parameters\n ----------\n route_I: int\n route index (database specific)\n\n Returns\n -------\n name: str\n short name of the route, eg. 195N\n type: int\n route_type according to the GTFS standard\n \"\"\"\n cur = self.conn.cursor()\n results = cur.execute(\"SELECT name, type FROM routes WHERE route_I=(?)\", (route_I,))\n name, rtype = results.fetchone()\n return name, int(rtype)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_trip_stop_coordinates(self, trip_I):\n query = \"\"\"SELECT lat, lon\n FROM stop_times\n JOIN stops\n USING(stop_I)\n WHERE trip_I={trip_I}\n ORDER BY stop_times.seq\"\"\".format(trip_I=trip_I)\n stop_coords = pd.read_sql(query, self.conn)\n return stop_coords", "response": "Get the stop coordinates for a given trip_I"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_trip_stop_time_data(self, trip_I, day_start_ut):\n to_select = \"stop_I, \" + str(day_start_ut) + \"+dep_time_ds AS dep_time_ut, lat, lon, seq, shape_break\"\n str_to_run = \"SELECT \" + to_select + \"\"\"\n FROM stop_times JOIN stops USING(stop_I)\n WHERE (trip_I ={trip_I}) ORDER BY seq\n \"\"\"\n str_to_run = str_to_run.format(trip_I=trip_I)\n return pd.read_sql_query(str_to_run, self.conn)", "response": "Get the data for a particular trip stop"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the events that are in the given trip and time range.", "response": "def get_events_by_tripI_and_dsut(self, trip_I, day_start_ut,\n start_ut=None, end_ut=None):\n \"\"\"\n Get trip data as a list of events (i.e. dicts).\n\n Parameters\n ----------\n trip_I : int\n shorthand index of the trip.\n day_start_ut : int\n the start time of the day in unix time (seconds)\n start_ut : int, optional\n consider only events that start after this time\n If not specified, this filtering is not applied.\n end_ut : int, optional\n Consider only events that end before this time\n If not specified, this filtering is not applied.\n\n Returns\n -------\n events: list of dicts\n each element contains the following data:\n from_stop: int (stop_I)\n to_stop: int (stop_I)\n dep_time_ut: int (in unix time)\n arr_time_ut: int (in unix time)\n \"\"\"\n # for checking input:\n assert day_start_ut <= start_ut\n assert day_start_ut <= end_ut\n assert start_ut <= end_ut\n events = []\n # check that trip takes place on that day:\n if not self.tripI_takes_place_on_dsut(trip_I, day_start_ut):\n return events\n\n query = \"\"\"SELECT stop_I, arr_time_ds+?, dep_time_ds+?\n FROM stop_times JOIN stops USING(stop_I)\n WHERE\n (trip_I = ?)\n \"\"\"\n params = [day_start_ut, day_start_ut,\n trip_I]\n if start_ut:\n query += \"AND (dep_time_ds > ?-?)\"\n params += [start_ut, day_start_ut]\n if end_ut:\n query += \"AND (arr_time_ds < ?-?)\"\n params += [end_ut, day_start_ut]\n query += \"ORDER BY arr_time_ds\"\n cur = self.conn.cursor()\n rows = cur.execute(query, params)\n stop_data = list(rows)\n for i in range(len(stop_data) - 1):\n event = {\n \"from_stop\": stop_data[i][0],\n \"to_stop\": stop_data[i + 1][0],\n \"dep_time_ut\": stop_data[i][2],\n \"arr_time_ut\": stop_data[i + 1][1]\n }\n events.append(event)\n return events"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nchecks that a trip takes place during a given day", "response": "def tripI_takes_place_on_dsut(self, trip_I, day_start_ut):\n \"\"\"\n Check that a trip takes place during a day\n\n Parameters\n ----------\n trip_I : int\n index of the trip in the gtfs data base\n day_start_ut : int\n the starting time of the day in unix time (seconds)\n\n Returns\n -------\n takes_place: bool\n boolean value describing whether the trip takes place during\n the given day or not\n \"\"\"\n query = \"SELECT * FROM days WHERE trip_I=? AND day_start_ut=?\"\n params = (trip_I, day_start_ut)\n cur = self.conn.cursor()\n rows = list(cur.execute(query, params))\n if len(rows) == 0:\n return False\n else:\n assert len(rows) == 1, 'On a day, a trip_I should be present at most once'\n return True"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef day_start_ut(self, ut):\n # set timezone to the one of gtfs\n old_tz = self.set_current_process_time_zone()\n ut = time.mktime(time.localtime(ut)[:3] + (12, 00, 0, 0, 0, -1)) - 43200\n set_process_timezone(old_tz)\n return ut", "response": "Convert unixtime to unixtime on GTFS start - of - day."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nincrement the GTFS - definition of day start.", "response": "def increment_day_start_ut(self, day_start_ut, n_days=1):\n \"\"\"Increment the GTFS-definition of \"day start\".\n\n Parameters\n ----------\n day_start_ut : int\n unixtime of the previous start of day. If this time is between\n 12:00 or greater, there *will* be bugs. To solve this, run the\n input through day_start_ut first.\n n_days: int\n number of days to increment\n \"\"\"\n old_tz = self.set_current_process_time_zone()\n day0 = time.localtime(day_start_ut + 43200) # time of noon\n dayN = time.mktime(day0[:2] + # YYYY, MM\n (day0[2] + n_days,) + # DD\n (12, 00, 0, 0, 0, -1)) - 43200 # HHMM, etc. Minus 12 hours.\n set_process_timezone(old_tz)\n return dayN"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_possible_day_starts(self, start_ut, end_ut, max_time_overnight=None):\n if max_time_overnight is None:\n # 7 hours:\n max_time_overnight = 7 * 60 * 60\n\n # sanity checks for the timezone parameter\n # assert timezone < 14\n # assert timezone > -14\n # tz_seconds = int(timezone*3600)\n assert start_ut < end_ut\n start_day_ut = self.day_start_ut(start_ut)\n # start_day_ds = int(start_ut+tz_seconds) % seconds_in_a_day #??? needed?\n start_day_ds = start_ut - start_day_ut\n # assert (start_day_ut+tz_seconds) % seconds_in_a_day == 0\n end_day_ut = self.day_start_ut(end_ut)\n # end_day_ds = int(end_ut+tz_seconds) % seconds_in_a_day #??? needed?\n # end_day_ds = end_ut - end_day_ut\n # assert (end_day_ut+tz_seconds) % seconds_in_a_day == 0\n\n # If we are early enough in a day that we might have trips from\n # the previous day still running, decrement the start day.\n if start_day_ds < max_time_overnight:\n start_day_ut = self.increment_day_start_ut(start_day_ut, n_days=-1)\n\n # day_start_times_ut = range(start_day_ut, end_day_ut+seconds_in_a_day, seconds_in_a_day)\n\n # Create a list of all possible day start times. This is roughly\n # range(day_start_ut, day_end_ut+1day, 1day).\n day_start_times_ut = [start_day_ut]\n while day_start_times_ut[-1] < end_day_ut:\n day_start_times_ut.append(self.increment_day_start_ut(day_start_times_ut[-1]))\n\n start_times_ds = []\n end_times_ds = []\n # For every possible day start:\n for dsut in day_start_times_ut:\n # start day_seconds starts at either zero, or time - daystart\n day_start_ut = max(0, start_ut - dsut)\n start_times_ds.append(day_start_ut)\n # end day_seconds is time-day_start\n day_end_ut = end_ut - dsut\n end_times_ds.append(day_end_ut)\n # Return three tuples which can be zip:ped together.\n return day_start_times_ut, start_times_ds, end_times_ds", "response": "Get all possible day start times between start_ut and end_ut"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_tripIs_within_range_by_dsut(self,\n start_time_ut,\n end_time_ut):\n \"\"\"\n Obtain a list of trip_Is that take place during a time interval.\n The trip needs to be only partially overlapping with the given time interval.\n The grouping by dsut (day_start_ut) is required as same trip_I could\n take place on multiple days.\n\n Parameters\n ----------\n start_time_ut : int\n start of the time interval in unix time (seconds)\n end_time_ut: int\n end of the time interval in unix time (seconds)\n\n Returns\n -------\n trip_I_dict: dict\n keys: day_start_times to list of integers (trip_Is)\n \"\"\"\n cur = self.conn.cursor()\n assert start_time_ut <= end_time_ut\n dst_ut, st_ds, et_ds = \\\n self._get_possible_day_starts(start_time_ut, end_time_ut, 7)\n # noinspection PyTypeChecker\n assert len(dst_ut) >= 0\n trip_I_dict = {}\n for day_start_ut, start_ds, end_ds in \\\n zip(dst_ut, st_ds, et_ds):\n query = \"\"\"\n SELECT distinct(trip_I)\n FROM days\n JOIN trips\n USING(trip_I)\n WHERE\n (days.day_start_ut == ?)\n AND (\n (trips.start_time_ds <= ?)\n AND\n (trips.end_time_ds >= ?)\n )\n \"\"\"\n params = (day_start_ut, end_ds, start_ds)\n trip_Is = [el[0] for el in cur.execute(query, params)]\n if len(trip_Is) > 0:\n trip_I_dict[day_start_ut] = trip_Is\n return trip_I_dict", "response": "Returns a dict of trip_Is that take place during a given time interval."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting all stop data as a pandas DataFrame for all stops or an individual stop", "response": "def stop(self, stop_I):\n \"\"\"\n Get all stop data as a pandas DataFrame for all stops, or an individual stop'\n\n Parameters\n ----------\n stop_I : int\n stop index\n\n Returns\n -------\n stop: pandas.DataFrame\n \"\"\"\n return pd.read_sql_query(\"SELECT * FROM stops WHERE stop_I={stop_I}\".format(stop_I=stop_I), self.conn)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a DataFrame containing all stops for a given route type.", "response": "def get_stops_for_route_type(self, route_type):\n \"\"\"\n Parameters\n ----------\n route_type: int\n\n Returns\n -------\n stops: pandas.DataFrame\n\n \"\"\"\n if route_type is WALK:\n return self.stops()\n else:\n return pd.read_sql_query(\"SELECT DISTINCT stops.* \"\n \"FROM stops JOIN stop_times ON stops.stop_I == stop_times.stop_I \"\n \" JOIN trips ON stop_times.trip_I = trips.trip_I\"\n \" JOIN routes ON trips.route_I == routes.route_I \"\n \"WHERE routes.type=(?)\", self.conn, params=(route_type,))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef generate_routable_transit_events(self, start_time_ut=None, end_time_ut=None, route_type=None):\n from gtfspy.networks import temporal_network\n df = temporal_network(self, start_time_ut=start_time_ut, end_time_ut=end_time_ut, route_type=route_type)\n df.sort_values(\"dep_time_ut\", ascending=False, inplace=True)\n\n for row in df.itertuples():\n yield row", "response": "Generate a list of tuples that define a transition between start_time_ut and end_time_ut."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_transit_events(self, start_time_ut=None, end_time_ut=None, route_type=None):\n table_name = self._get_day_trips_table_name()\n event_query = \"SELECT stop_I, seq, trip_I, route_I, routes.route_id AS route_id, routes.type AS route_type, \" \\\n \"shape_id, day_start_ut+dep_time_ds AS dep_time_ut, day_start_ut+arr_time_ds AS arr_time_ut \" \\\n \"FROM \" + table_name + \" \" \\\n \"JOIN trips USING(trip_I) \" \\\n \"JOIN routes USING(route_I) \" \\\n \"JOIN stop_times USING(trip_I)\"\n\n where_clauses = []\n if end_time_ut:\n where_clauses.append(table_name + \".start_time_ut< {end_time_ut}\".format(end_time_ut=end_time_ut))\n where_clauses.append(\"dep_time_ut <={end_time_ut}\".format(end_time_ut=end_time_ut))\n if start_time_ut:\n where_clauses.append(table_name + \".end_time_ut > {start_time_ut}\".format(start_time_ut=start_time_ut))\n where_clauses.append(\"arr_time_ut >={start_time_ut}\".format(start_time_ut=start_time_ut))\n if route_type is not None:\n assert route_type in ALL_ROUTE_TYPES\n where_clauses.append(\"routes.type={route_type}\".format(route_type=route_type))\n if len(where_clauses) > 0:\n event_query += \" WHERE \"\n for i, where_clause in enumerate(where_clauses):\n if i is not 0:\n event_query += \" AND \"\n event_query += where_clause\n # ordering is required for later stages\n event_query += \" ORDER BY trip_I, day_start_ut+dep_time_ds;\"\n events_result = pd.read_sql_query(event_query, self.conn)\n # 'filter' results so that only real \"events\" are taken into account\n from_indices = numpy.nonzero(\n (events_result['trip_I'][:-1].values == events_result['trip_I'][1:].values) *\n (events_result['seq'][:-1].values < events_result['seq'][1:].values)\n )[0]\n to_indices = from_indices + 1\n # these should have same trip_ids\n assert (events_result['trip_I'][from_indices].values == events_result['trip_I'][to_indices].values).all()\n trip_Is = events_result['trip_I'][from_indices]\n from_stops = events_result['stop_I'][from_indices]\n to_stops = events_result['stop_I'][to_indices]\n shape_ids = events_result['shape_id'][from_indices]\n dep_times = events_result['dep_time_ut'][from_indices]\n arr_times = events_result['arr_time_ut'][to_indices]\n route_types = events_result['route_type'][from_indices]\n route_ids = events_result['route_id'][from_indices]\n route_Is = events_result['route_I'][from_indices]\n durations = arr_times.values - dep_times.values\n assert (durations >= 0).all()\n from_seqs = events_result['seq'][from_indices]\n to_seqs = events_result['seq'][to_indices]\n data_tuples = zip(from_stops, to_stops, dep_times, arr_times,\n shape_ids, route_types, route_ids, trip_Is,\n durations, from_seqs, to_seqs, route_Is)\n columns = [\"from_stop_I\", \"to_stop_I\", \"dep_time_ut\", \"arr_time_ut\",\n \"shape_id\", \"route_type\", \"route_id\", \"trip_I\",\n \"duration\", \"from_seq\", \"to_seq\", \"route_I\"]\n df = pd.DataFrame.from_records(data_tuples, columns=columns)\n return df", "response": "Returns a pandas. DataFrame containing all events that take place during a given time interval."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the difference between the routes in this db and the ones in the other db.", "response": "def get_route_difference_with_other_db(self, other_gtfs, start_time, end_time, uniqueness_threshold=None,\n uniqueness_ratio=None):\n \"\"\"\n Compares the routes based on stops in the schedule with the routes in another db and returns the ones without match.\n Uniqueness thresholds or ratio can be used to allow small differences\n :param uniqueness_threshold:\n :param uniqueness_ratio:\n :return:\n \"\"\"\n from gtfspy.stats import frequencies_by_generated_route\n\n this_df = frequencies_by_generated_route(self, start_time, end_time)\n other_df = frequencies_by_generated_route(other_gtfs, start_time, end_time)\n this_routes = {x: set(x.split(',')) for x in this_df[\"route\"]}\n other_routes = {x: set(x.split(',')) for x in other_df[\"route\"]}\n # this_df[\"route_set\"] = this_df.apply(lambda x: set(x.route.split(',')), axis=1)\n # other_df[\"route_set\"] = other_df.apply(lambda x: set(x.route.split(',')), axis=1)\n\n this_uniques = list(this_routes.keys())\n other_uniques = list(other_routes.keys())\n print(\"initial routes A:\", len(this_uniques))\n print(\"initial routes B:\", len(other_uniques))\n for i_key, i in this_routes.items():\n for j_key, j in other_routes.items():\n union = i | j\n intersection = i & j\n symmetric_difference = i ^ j\n if uniqueness_ratio:\n if len(intersection) / len(union) >= uniqueness_ratio:\n try:\n this_uniques.remove(i_key)\n this_df = this_df[this_df[\"route\"] != i_key]\n except ValueError:\n pass\n try:\n other_uniques.remove(j_key)\n other_df = other_df[other_df[\"route\"] != j_key]\n except ValueError:\n pass\n\n print(\"unique routes A\", len(this_df))\n print(\"unique routes B\", len(other_df))\n return this_df, other_df"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngetting straight line transfer distances to stations that can be transferred to.", "response": "def get_straight_line_transfer_distances(self, stop_I=None):\n \"\"\"\n Get (straight line) distances to stations that can be transferred to.\n\n Parameters\n ----------\n stop_I : int, optional\n If not specified return all possible transfer distances\n\n Returns\n -------\n distances: pandas.DataFrame\n each row has the following items\n from_stop_I: int\n to_stop_I: int\n d: float or int #distance in meters\n \"\"\"\n if stop_I is not None:\n query = u\"\"\" SELECT from_stop_I, to_stop_I, d\n FROM stop_distances\n WHERE\n from_stop_I=?\n \"\"\"\n params = (u\"{stop_I}\".format(stop_I=stop_I),)\n else:\n query = \"\"\" SELECT from_stop_I, to_stop_I, d\n FROM stop_distances\n \"\"\"\n params = None\n stop_data_df = pd.read_sql_query(query, self.conn, params=params)\n return stop_data_df"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the first and last day_start_ut span of the last day_start_ut.", "response": "def get_day_start_ut_span(self):\n \"\"\"\n Return the first and last day_start_ut\n\n Returns\n -------\n first_day_start_ut: int\n last_day_start_ut: int\n \"\"\"\n cur = self.conn.cursor()\n first_day_start_ut, last_day_start_ut = \\\n cur.execute(\"SELECT min(day_start_ut), max(day_start_ut) FROM days;\").fetchone()\n return first_day_start_ut, last_day_start_ut"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef homogenize_stops_table_with_other_db(self, source):\n cur = self.conn.cursor()\n self.attach_gtfs_database(source)\n\n query_inner_join = \"\"\"SELECT t1.*\n FROM stops t1\n INNER JOIN other.stops t2\n ON t1.stop_id=t2.stop_id\n AND find_distance(t1.lon, t1.lat, t2.lon, t2.lat) <= 50\"\"\"\n df_inner_join = self.execute_custom_query_pandas(query_inner_join)\n print(\"number of common stops: \", len(df_inner_join.index))\n df_not_in_other = self.execute_custom_query_pandas(\"SELECT * FROM stops EXCEPT \" + query_inner_join)\n print(\"number of stops missing in second feed: \", len(df_not_in_other.index))\n df_not_in_self = self.execute_custom_query_pandas(\"SELECT * FROM other.stops EXCEPT \" +\n query_inner_join.replace(\"t1.*\", \"t2.*\"))\n print(\"number of stops missing in first feed: \", len(df_not_in_self.index))\n try:\n self.execute_custom_query(\"\"\"ALTER TABLE stops ADD COLUMN stop_pair_I INT \"\"\")\n\n self.execute_custom_query(\"\"\"ALTER TABLE other.stops ADD COLUMN stop_pair_I INT \"\"\")\n except sqlite3.OperationalError:\n pass\n stop_id_stub = \"added_stop_\"\n counter = 0\n rows_to_update_self = []\n rows_to_update_other = []\n rows_to_add_to_self = []\n rows_to_add_to_other = []\n\n for items in df_inner_join.itertuples(index=False):\n rows_to_update_self.append((counter, items[1]))\n rows_to_update_other.append((counter, items[1]))\n counter += 1\n\n for items in df_not_in_other.itertuples(index=False):\n rows_to_update_self.append((counter, items[1]))\n rows_to_add_to_other.append((stop_id_stub + str(counter),) + tuple(items[x] for x in [2, 3, 4, 5, 6, 8, 9])\n + (counter,))\n counter += 1\n\n for items in df_not_in_self.itertuples(index=False):\n rows_to_update_other.append((counter, items[1]))\n rows_to_add_to_self.append((stop_id_stub + str(counter),) + tuple(items[x] for x in [2, 3, 4, 5, 6, 8, 9])\n + (counter,))\n counter += 1\n\n query_add_row = \"\"\"INSERT INTO stops(\n stop_id,\n code,\n name,\n desc,\n lat,\n lon,\n location_type,\n wheelchair_boarding,\n stop_pair_I) VALUES (%s) \"\"\" % (\", \".join([\"?\" for x in range(9)]))\n\n query_update_row = \"\"\"UPDATE stops SET stop_pair_I=? WHERE stop_id=?\"\"\"\n print(\"adding rows to databases\")\n cur.executemany(query_add_row, rows_to_add_to_self)\n cur.executemany(query_update_row, rows_to_update_self)\n cur.executemany(query_add_row.replace(\"stops\", \"other.stops\"), rows_to_add_to_other)\n cur.executemany(query_update_row.replace(\"stops\", \"other.stops\"), rows_to_update_other)\n self.conn.commit()\n print(\"finished\")", "response": "This function checks the external database for common stops and adds the missing stops to the common stops table."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_data_as_dataframe(self,\n travel_impedance_measure,\n from_stop_I=None,\n to_stop_I=None,\n statistic=None):\n \"\"\"\n Recover pre-computed travel_impedance between od-pairs from the database.\n\n Returns\n -------\n values: number | Pandas DataFrame\n \"\"\"\n to_select = []\n where_clauses = []\n to_select.append(\"from_stop_I\")\n to_select.append(\"to_stop_I\")\n if from_stop_I is not None:\n where_clauses.append(\"from_stop_I=\" + str(int(from_stop_I)))\n if to_stop_I is not None:\n where_clauses.append(\"to_stop_I=\" + str(int(to_stop_I)))\n where_clause = \"\"\n if len(where_clauses) > 0:\n where_clause = \" WHERE \" + \" AND \".join(where_clauses)\n if not statistic:\n to_select.extend([\"min\", \"mean\", \"median\", \"max\"])\n else:\n to_select.append(statistic)\n to_select_clause = \",\".join(to_select)\n if not to_select_clause:\n to_select_clause = \"*\"\n sql = \"SELECT \" + to_select_clause + \" FROM \" + travel_impedance_measure + where_clause + \";\"\n df = pd.read_sql(sql, self.conn)\n return df", "response": "Reads the data from the database into a Pandas DataFrame."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef insert_data(self, travel_impedance_measure_name, data):\n f = float\n data_tuple = [(int(x[\"from_stop_I\"]), int(x[\"to_stop_I\"]), f(x[\"min\"]), f(x[\"max\"]), f(x[\"median\"]), f(x[\"mean\"])) for\n x in data]\n insert_stmt = '''INSERT OR REPLACE INTO ''' + travel_impedance_measure_name + ''' (\n from_stop_I,\n to_stop_I,\n min,\n max,\n median,\n mean) VALUES (?, ?, ?, ?, ?, ?) '''\n self.conn.executemany(insert_stmt, data_tuple)\n self.conn.commit()", "response": "Inserts data into the table with the given name."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef add_event(self, event):\n assert event.dep_time_ut <= event.arr_time_ut\n heappush(self.heap, event)", "response": "Adds an event to the heap"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nadds events from a given distance table to the heap.", "response": "def add_walk_events_to_heap(self, transfer_distances, e, start_time_ut, walk_speed, uninfected_stops, max_duration_ut):\n \"\"\"\n Parameters\n ----------\n transfer_distances:\n e : Event\n start_time_ut : int\n walk_speed : float\n uninfected_stops : list\n max_duration_ut : int\n \"\"\"\n n = len(transfer_distances)\n dists_values = transfer_distances.values\n to_stop_I_index = np.nonzero(transfer_distances.columns == 'to_stop_I')[0][0]\n d_index = np.nonzero(transfer_distances.columns == 'd')[0][0]\n for i in range(n):\n transfer_to_stop_I = dists_values[i, to_stop_I_index]\n if transfer_to_stop_I in uninfected_stops:\n d = dists_values[i, d_index]\n transfer_arr_time = e.arr_time_ut + int(d/float(walk_speed))\n if transfer_arr_time > start_time_ut+max_duration_ut:\n continue\n te = Event(transfer_arr_time, e.arr_time_ut, e.to_stop_I, transfer_to_stop_I, WALK)\n self.add_event(te)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _check_dep_time_is_valid(self, dep_time):\n assert dep_time <= self._min_dep_time, \"Labels should be entered in decreasing order of departure time.\"\n dep_time_index = self.dep_times_to_index[dep_time]\n if self._min_dep_time < float('inf'):\n min_dep_index = self.dep_times_to_index[self._min_dep_time]\n assert min_dep_index == dep_time_index or (min_dep_index == dep_time_index - 1), \\\n \"dep times should be ordered sequentially\"\n else:\n assert dep_time_index is 0, \"first dep_time index should be zero (ensuring that all connections are properly handled)\"\n self._min_dep_time = dep_time", "response": "A simple checker that the dep_time is valid."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef update(self, new_labels, departure_time_backup=None):\n if self._closed:\n raise RuntimeError(\"Profile is closed, no updates can be made\")\n try:\n departure_time = next(iter(new_labels)).departure_time\n except StopIteration:\n departure_time = departure_time_backup\n self._check_dep_time_is_valid(departure_time)\n\n for new_label in new_labels:\n assert (new_label.departure_time == departure_time)\n dep_time_index = self.dep_times_to_index[departure_time]\n\n if dep_time_index > 0:\n # Departure time is modified in order to not pass on labels which are not Pareto-optimal when departure time is ignored.\n mod_prev_labels = [label.get_copy_with_specified_departure_time(departure_time) for label\n in self._label_bags[dep_time_index - 1]]\n else:\n mod_prev_labels = list()\n mod_prev_labels += self._label_bags[dep_time_index]\n\n walk_label = self._get_label_to_target(departure_time)\n if walk_label:\n new_labels = new_labels + [walk_label]\n new_frontier = merge_pareto_frontiers(new_labels, mod_prev_labels)\n\n self._label_bags[dep_time_index] = new_frontier\n return True", "response": "Update the profile with the new labels."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nevaluating the pareto optimal set of labels given a departure time.", "response": "def evaluate(self, dep_time, first_leg_can_be_walk=True, connection_arrival_time=None):\n\n \"\"\"\n Get the pareto_optimal set of Labels, given a departure time.\n\n Parameters\n ----------\n dep_time : float, int\n time in unix seconds\n first_leg_can_be_walk : bool, optional\n whether to allow walking to target to be included into the profile\n (I.e. whether this function is called when scanning a pseudo-connection:\n \"double\" walks are not allowed.)\n connection_arrival_time: float, int, optional\n used for computing the walking label if dep_time, i.e., connection.arrival_stop_next_departure_time, is infinity)\n connection: connection object\n\n Returns\n -------\n pareto_optimal_labels : set\n Set of Labels\n \"\"\"\n walk_labels = list()\n # walk label towards target\n if first_leg_can_be_walk and self._walk_to_target_duration != float('inf'):\n # add walk_label\n if connection_arrival_time is not None:\n walk_labels.append(self._get_label_to_target(connection_arrival_time))\n else:\n walk_labels.append(self._get_label_to_target(dep_time))\n\n # if dep time is larger than the largest dep time -> only walk labels are possible\n if dep_time in self.dep_times_to_index:\n assert (dep_time != float('inf'))\n index = self.dep_times_to_index[dep_time]\n labels = self._label_bags[index]\n pareto_optimal_labels = merge_pareto_frontiers(labels, walk_labels)\n else:\n pareto_optimal_labels = walk_labels\n\n if not first_leg_can_be_walk:\n pareto_optimal_labels = [label for label in pareto_optimal_labels if not label.first_leg_is_walk]\n return pareto_optimal_labels"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef finalize(self, neighbor_label_bags=None, walk_durations=None, departure_arrival_stop_pairs=None):\n assert (not self._finalized)\n if self._final_pareto_optimal_labels is None:\n self._compute_real_connection_labels()\n if neighbor_label_bags is not None:\n assert (len(walk_durations) == len(neighbor_label_bags))\n self._compute_final_pareto_optimal_labels(neighbor_label_bags,\n walk_durations,\n departure_arrival_stop_pairs)\n else:\n self._final_pareto_optimal_labels = self._real_connection_labels\n self._finalized = True\n self._closed = True", "response": "Finalizes the internal state of the internal state of the internal state of the internal state."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndo this GTFS contain this file? ( file specified by the class )", "response": "def exists_by_source(self):\n \"\"\"Does this GTFS contain this file? (file specified by the class)\"\"\"\n exists_list = []\n for source in self.gtfs_sources:\n if isinstance(source, dict):\n # source can now be either a dict or a zipfile\n if self.fname in source:\n if source[self.fname]:\n exists_list.append(True)\n continue\n # Handle zipfiles specially\n if \"zipfile\" in source:\n try:\n Z = zipfile.ZipFile(source['zipfile'], mode='r')\n Z.getinfo(os.path.join(source['zip_commonprefix'], self.fname))\n exists_list.append(True)\n continue\n # File does not exist in the zip archive\n except KeyError:\n print(self.fname, ' missing in ', source)\n exists_list.append(False)\n continue\n # Normal filename\n elif isinstance(source, string_types):\n if os.path.exists(os.path.join(source, self.fname)):\n exists_list.append(True)\n continue\n exists_list.append(False)\n # the \"file\" was not found in any of the sources, return false\n return exists_list"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_table(self, conn):\n # Make cursor\n cur = conn.cursor()\n # Drop table if it already exists, to be recreated. This\n # could in the future abort if table already exists, and not\n # recreate it from scratch.\n #cur.execute('''DROP TABLE IF EXISTS %s'''%self.table)\n #conn.commit()\n if self.tabledef is None:\n return\n if not self.tabledef.startswith('CREATE'):\n # \"normal\" table creation.\n cur.execute('CREATE TABLE IF NOT EXISTS %s %s'\n % (self.table, self.tabledef)\n )\n else:\n # When tabledef contains the full CREATE statement (for\n # virtual tables).\n cur.execute(self.tabledef)\n conn.commit()", "response": "Create the table if it does not exist."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading data from GTFS file into database", "response": "def insert_data(self, conn):\n \"\"\"Load data from GTFS file into database\"\"\"\n cur = conn.cursor()\n # This is a bit hackish. It is annoying to have to write the\n # INSERT statement yourself and keep it up to date with the\n # table rows. This gets the first row, figures out the field\n # names from that, and then makes an INSERT statement like\n # \"INSERT INTO table (col1, col2, ...) VALUES (:col1, :col2,\n # ...)\". The \":col1\" is sqlite syntax for named value.\n\n csv_reader_generators, prefixes = self._get_csv_reader_generators()\n for csv_reader, prefix in zip(csv_reader_generators, prefixes):\n try:\n row = next(iter(self.gen_rows([csv_reader], [prefix])))\n fields = row.keys()\n except StopIteration:\n # The file has *only* a header and no data.\n # next(iter()) yields StopIteration and we can't\n # proceed. Since there is nothing to import, just continue the loop\n print(\"Not importing %s into %s for %s\" % (self.fname, self.table, prefix))\n continue\n stmt = '''INSERT INTO %s (%s) VALUES (%s)''' % (\n self.table,\n (', '.join([x for x in fields if x[0] != '_'] + self.extra_keys)),\n (', '.join([\":\" + x for x in fields if x[0] != '_'] + self.extra_values))\n )\n\n # This does the actual insertions. Passed the INSERT\n # statement and then an iterator over dictionaries. Each\n # dictionary is inserted.\n if self.print_progress:\n print('Importing %s into %s for %s' % (self.fname, self.table, prefix))\n # the first row was consumed by fetching the fields\n # (this could be optimized)\n from itertools import chain\n rows = chain([row], self.gen_rows([csv_reader], [prefix]))\n cur.executemany(stmt, rows)\n conn.commit()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef import_(self, conn):\n if self.print_progress:\n print('Beginning', self.__class__.__name__)\n # what is this mystical self._conn ?\n self._conn = conn\n\n self.create_table(conn)\n # This does insertions\n if self.mode in ('all', 'import') and self.fname and self.exists() and self.table not in ignore_tables:\n self.insert_data(conn)\n # This makes indexes in the DB.\n if self.mode in ('all', 'index') and hasattr(self, 'index'):\n self.create_index(conn)\n # Any post-processing to be done after the full import.\n if self.mode in ('all', 'import') and hasattr(self, 'post_import'):\n self.run_post_import(conn)\n # Commit it all\n conn.commit()", "response": "Copy data and store in the connection object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef copy(cls, conn, **where):\n cur = conn.cursor()\n if where and cls.copy_where:\n copy_where = cls.copy_where.format(**where)\n # print(copy_where)\n else:\n copy_where = ''\n cur.execute('INSERT INTO %s '\n 'SELECT * FROM source.%s %s' % (cls.table, cls.table, copy_where))", "response": "Copy data from one table to another while filtering data at the same time."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_journey_legs_to_target(self, target, fastest_path=True, min_boardings=False, all_leg_sections=True,\n ignore_walk=False, diff_threshold=None, diff_path=None):\n \"\"\"\n Returns a dataframe of aggregated sections from source nodes to target. The returned sections are either\n transfer point to transfer point or stop to stop. In a before after setting, the results can be filtered based\n on values in a difference db.\n :param target:\n :param fastest_path:\n :param min_boardings:\n :param all_leg_sections:\n :param ignore_walk:\n :param diff_threshold:\n :param diff_path:\n :return:\n \"\"\"\n assert not (fastest_path and min_boardings)\n if min_boardings:\n raise NotImplementedError\n if all_leg_sections and diff_threshold:\n raise NotImplementedError\n\n added_constraints = \"\"\n add_diff = \"\"\n if fastest_path:\n added_constraints += \" AND journeys.pre_journey_wait_fp>=0\"\n if ignore_walk:\n added_constraints += \" AND legs.trip_I >= 0\"\n if diff_path and diff_threshold:\n self.conn = attach_database(self.conn, diff_path, name=\"diff\")\n add_diff = \", diff.diff_temporal_distance\"\n added_constraints += \" AND abs(diff_temporal_distance.diff_mean) >= %s \" \\\n \"AND diff_temporal_distance.from_stop_I = journeys.from_stop_I \" \\\n \"AND diff_temporal_distance.to_stop_I = journeys.to_stop_I\" % (diff_threshold,)\n\n if all_leg_sections:\n df = self._get_journey_legs_to_target_with_all_sections(target, added_constraints)\n else:\n query = \"\"\"SELECT from_stop_I, to_stop_I, coalesce(type, -1) AS type,\n count(*) AS n_trips\n FROM\n (SELECT legs.* FROM legs, journeys %s\n WHERE journeys.journey_id = legs.journey_id AND journeys.to_stop_I = %s %s) q1\n LEFT JOIN (SELECT * FROM other.trips, other.routes WHERE trips.route_I = routes.route_I) q2\n ON q1.trip_I = q2.trip_I\n GROUP BY from_stop_I, to_stop_I, type\"\"\" % (add_diff, str(target), added_constraints)\n df = read_sql_query(query, self.conn)\n\n return df", "response": "Get all journey legs from source nodes to target."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_upstream_stops_ratio(self, target, trough_stops, ratio):\n if isinstance(trough_stops, list):\n trough_stops = \",\".join(trough_stops)\n query = \"\"\"SELECT stops.* FROM other.stops, \n (SELECT q2.from_stop_I AS stop_I FROM \n (SELECT journeys.from_stop_I, count(*) AS n_total FROM journeys\n WHERE journeys.to_stop_I = {target} \n GROUP BY from_stop_I) q1,\n (SELECT journeys.from_stop_I, count(*) AS n_trough FROM journeys, legs \n WHERE journeys.journey_id=legs.journey_id AND legs.from_stop_I IN ({trough_stops}) AND journeys.to_stop_I = {target}\n GROUP BY journeys.from_stop_I) q2\n WHERE q1.from_stop_I = q2.from_stop_I AND n_trough/(n_total*1.0) >= {ratio}) q1\n WHERE stops.stop_I = q1.stop_I\"\"\".format(target=target, trough_stops=trough_stops, ratio=ratio)\n df = read_sql_query(query, self.conn)\n return df", "response": "Returns the set of upstream stops for which the ratio or higher proportion of trips to the target passes trough a set of trough stops."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_spatial_bounds(gtfs, as_dict=False):\n stats = get_stats(gtfs)\n lon_min = stats['lon_min']\n lon_max = stats['lon_max']\n lat_min = stats['lat_min']\n lat_max = stats['lat_max']\n if as_dict:\n return {'lon_min': lon_min, 'lon_max': lon_max, 'lat_min': lat_min, 'lat_max': lat_max}\n else:\n return lon_min, lon_max, lat_min, lat_max", "response": "Returns the minimum and maximum longitude and latitude of the node in the specified GTFS."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets median latitude AND longitude of stops in GTFS", "response": "def get_median_lat_lon_of_stops(gtfs):\n \"\"\"\n Get median latitude AND longitude of stops\n\n Parameters\n ----------\n gtfs: GTFS\n\n Returns\n -------\n median_lat : float\n median_lon : float\n \"\"\"\n stops = gtfs.get_table(\"stops\")\n median_lat = numpy.percentile(stops['lat'].values, 50)\n median_lon = numpy.percentile(stops['lon'].values, 50)\n return median_lat, median_lon"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_centroid_of_stops(gtfs):\n stops = gtfs.get_table(\"stops\")\n mean_lat = numpy.mean(stops['lat'].values)\n mean_lon = numpy.mean(stops['lon'].values)\n return mean_lat, mean_lon", "response": "Get mean latitude AND longitude of stops in GTFS"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting data from get_stats to csv file.", "response": "def write_stats_as_csv(gtfs, path_to_csv, re_write=False):\n \"\"\"\n Writes data from get_stats to csv file\n\n Parameters\n ----------\n gtfs: GTFS\n path_to_csv: str\n filepath to the csv file to be generated\n re_write:\n insted of appending, create a new one.\n \"\"\"\n stats_dict = get_stats(gtfs)\n # check if file exist\n if re_write:\n os.remove(path_to_csv)\n \n #if not os.path.isfile(path_to_csv):\n # is_new = True\n #else:\n # is_new = False\n \n is_new = True\n mode = 'r' if os.path.exists(path_to_csv) else 'w+'\n with open(path_to_csv, mode) as csvfile:\n for line in csvfile:\n if line:\n is_new = False\n else:\n is_new = True\n\n with open(path_to_csv, 'a') as csvfile:\n if (sys.version_info > (3, 0)):\n delimiter = u\",\"\n else:\n delimiter = b\",\"\n statswriter = csv.writer(csvfile, delimiter=delimiter)\n # write column names if\n if is_new:\n statswriter.writerow([key for key in sorted(stats_dict.keys())])\n\n row_to_write = []\n # write stats row sorted by column name\n for key in sorted(stats_dict.keys()):\n row_to_write.append(stats_dict[key])\n statswriter.writerow(row_to_write)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget basic statistics of the GTFS data.", "response": "def get_stats(gtfs):\n \"\"\"\n Get basic statistics of the GTFS data.\n\n Parameters\n ----------\n gtfs: GTFS\n\n Returns\n -------\n stats: dict\n A dictionary of various statistics.\n Keys should be strings, values should be inputtable to a database (int, date, str, ...)\n (but not a list)\n \"\"\"\n stats = {}\n # Basic table counts\n for table in ['agencies', 'routes', 'stops', 'stop_times', 'trips', 'calendar', 'shapes', 'calendar_dates',\n 'days', 'stop_distances', 'frequencies', 'feed_info', 'transfers']:\n stats[\"n_\" + table] = gtfs.get_row_count(table)\n\n # Agency names\n agencies = gtfs.get_table(\"agencies\")\n stats[\"agencies\"] = \"_\".join(agencies['name'].values)\n\n # Stop lat/lon range\n stops = gtfs.get_table(\"stops\")\n lats = stops['lat'].values\n lons = stops['lon'].values\n percentiles = [0, 10, 50, 90, 100]\n\n try:\n lat_percentiles = numpy.percentile(lats, percentiles)\n except IndexError:\n lat_percentiles = [None] * 5\n lat_min, lat_10, lat_median, lat_90, lat_max = lat_percentiles\n stats[\"lat_min\"] = lat_min\n stats[\"lat_10\"] = lat_10\n stats[\"lat_median\"] = lat_median\n stats[\"lat_90\"] = lat_90\n stats[\"lat_max\"] = lat_max\n\n try:\n lon_percentiles = numpy.percentile(lons, percentiles)\n except IndexError:\n lon_percentiles = [None] * 5\n lon_min, lon_10, lon_median, lon_90, lon_max = lon_percentiles\n stats[\"lon_min\"] = lon_min\n stats[\"lon_10\"] = lon_10\n stats[\"lon_median\"] = lon_median\n stats[\"lon_90\"] = lon_90\n stats[\"lon_max\"] = lon_max\n\n if len(lats) > 0:\n stats[\"height_km\"] = wgs84_distance(lat_min, lon_median, lat_max, lon_median) / 1000.\n stats[\"width_km\"] = wgs84_distance(lon_min, lat_median, lon_max, lat_median) / 1000.\n else:\n stats[\"height_km\"] = None\n stats[\"width_km\"] = None\n\n first_day_start_ut, last_day_start_ut = gtfs.get_day_start_ut_span()\n stats[\"start_time_ut\"] = first_day_start_ut\n if last_day_start_ut is None:\n stats[\"end_time_ut\"] = None\n else:\n # 28 (instead of 24) comes from the GTFS stANDard\n stats[\"end_time_ut\"] = last_day_start_ut + 28 * 3600\n\n stats[\"start_date\"] = gtfs.get_min_date()\n stats[\"end_date\"] = gtfs.get_max_date()\n\n # Maximum activity day\n max_activity_date = gtfs.execute_custom_query(\n 'SELECT count(*), date '\n 'FROM days '\n 'GROUP BY date '\n 'ORDER BY count(*) DESC, date '\n 'LIMIT 1;').fetchone()\n if max_activity_date:\n stats[\"max_activity_date\"] = max_activity_date[1]\n max_activity_hour = gtfs.get_cursor().execute(\n 'SELECT count(*), arr_time_hour FROM day_stop_times '\n 'WHERE date=? GROUP BY arr_time_hour '\n 'ORDER BY count(*) DESC;', (stats[\"max_activity_date\"],)).fetchone()\n if max_activity_hour:\n stats[\"max_activity_hour\"] = max_activity_hour[1]\n else:\n stats[\"max_activity_hour\"] = None\n\n # Fleet size estimate: considering each line separately\n if max_activity_date and max_activity_hour:\n fleet_size_estimates = _fleet_size_estimate(gtfs, stats['max_activity_hour'], stats['max_activity_date'])\n stats.update(fleet_size_estimates)\n\n # Compute simple distributions of various columns that have a finite range of values.\n # Commented lines refer to values that are not imported yet, ?\n\n stats['routes__type__dist'] = _distribution(gtfs, 'routes', 'type')\n # stats['stop_times__pickup_type__dist'] = _distribution(gtfs, 'stop_times', 'pickup_type')\n # stats['stop_times__drop_off_type__dist'] = _distribution(gtfs, 'stop_times', 'drop_off_type')\n # stats['stop_times__timepoint__dist'] = _distribution(gtfs, 'stop_times', 'timepoint')\n stats['calendar_dates__exception_type__dist'] = _distribution(gtfs, 'calendar_dates', 'exception_type')\n stats['frequencies__exact_times__dist'] = _distribution(gtfs, 'frequencies', 'exact_times')\n stats['transfers__transfer_type__dist'] = _distribution(gtfs, 'transfers', 'transfer_type')\n stats['agencies__lang__dist'] = _distribution(gtfs, 'agencies', 'lang')\n stats['stops__location_type__dist'] = _distribution(gtfs, 'stops', 'location_type')\n # stats['stops__wheelchair_boarding__dist'] = _distribution(gtfs, 'stops', 'wheelchair_boarding')\n # stats['trips__wheelchair_accessible__dist'] = _distribution(gtfs, 'trips', 'wheelchair_accessible')\n # stats['trips__bikes_allowed__dist'] = _distribution(gtfs, 'trips', 'bikes_allowed')\n # stats[''] = _distribution(gtfs, '', '')\n stats = _feed_calendar_span(gtfs, stats)\n\n return stats"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _distribution(gtfs, table, column):\n cur = gtfs.conn.cursor()\n cur.execute('SELECT {column}, count(*) '\n 'FROM {table} GROUP BY {column} '\n 'ORDER BY {column}'.format(column=column, table=table))\n return ' '.join('%s:%s' % (t, c) for t, c in cur)", "response": "Count occurrences of values AND return it as a string."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalculating the maximum number of vehicles in a single terminal and returns a list of the maximum number of vehicles in a single terminal.", "response": "def _fleet_size_estimate(gtfs, hour, date):\n \"\"\"\n Calculates fleet size estimates by two separate formula:\n 1. Considering all routes separately with no interlining and doing a deficit calculation at every terminal\n 2. By looking at the maximum number of vehicles in simultaneous movement\n\n Parameters\n ----------\n gtfs: GTFS\n hour: int\n date: ?\n\n Returns\n -------\n results: dict\n a dict with keys:\n fleet_size_route_based\n fleet_size_max_movement\n\n \"\"\"\n results = {}\n\n fleet_size_list = []\n cur = gtfs.conn.cursor()\n rows = cur.execute(\n 'SELECT type, max(vehicles) '\n 'FROM ('\n 'SELECT type, direction_id, sum(vehicles) as vehicles '\n 'FROM '\n '('\n 'SELECT trips.route_I, trips.direction_id, routes.route_id, name, type, count(*) as vehicles, cycle_time_min '\n 'FROM trips, routes, days, '\n '('\n 'SELECT first_trip.route_I, first_trip.direction_id, first_trip_start_time, first_trip_end_time, '\n 'MIN(start_time_ds) as return_trip_start_time, end_time_ds as return_trip_end_time, '\n '(end_time_ds - first_trip_start_time)/60 as cycle_time_min '\n 'FROM '\n 'trips, '\n '(SELECT route_I, direction_id, MIN(start_time_ds) as first_trip_start_time, '\n 'end_time_ds as first_trip_end_time '\n 'FROM trips, days '\n 'WHERE trips.trip_I=days.trip_I AND start_time_ds >= ? * 3600 '\n 'AND start_time_ds <= (? + 1) * 3600 AND date = ? '\n 'GROUP BY route_I, direction_id) first_trip '\n 'WHERE first_trip.route_I = trips.route_I '\n 'AND first_trip.direction_id != trips.direction_id '\n 'AND start_time_ds >= first_trip_end_time '\n 'GROUP BY trips.route_I, trips.direction_id'\n ') return_trip '\n 'WHERE trips.trip_I=days.trip_I AND trips.route_I= routes.route_I '\n 'AND date = ? AND trips.route_I = return_trip.route_I '\n 'AND trips.direction_id = return_trip.direction_id '\n 'AND start_time_ds >= first_trip_start_time '\n 'AND start_time_ds < return_trip_end_time '\n 'GROUP BY trips.route_I, trips.direction_id '\n 'ORDER BY type, name, vehicles desc'\n ') cycle_times '\n 'GROUP BY direction_id, type'\n ') vehicles_type '\n 'GROUP BY type;', (hour, hour, date, date))\n for row in rows:\n fleet_size_list.append(str(row[0]) + ':' + str(row[1]))\n results['fleet_size_route_based'] = \" \".join(fleet_size_list)\n\n # Fleet size estimate: maximum number of vehicles in movement\n fleet_size_list = []\n fleet_size_dict = {}\n if hour:\n for minute in range(hour * 3600, (hour + 1) * 3600, 60):\n rows = gtfs.conn.cursor().execute(\n 'SELECT type, count(*) '\n 'FROM trips, routes, days '\n 'WHERE trips.route_I = routes.route_I '\n 'AND trips.trip_I=days.trip_I '\n 'AND start_time_ds <= ? '\n 'AND end_time_ds > ? + 60 '\n 'AND date = ? '\n 'GROUP BY type;',\n (minute, minute, date))\n\n for row in rows:\n if fleet_size_dict.get(row[0], 0) < row[1]:\n fleet_size_dict[row[0]] = row[1]\n\n for key in fleet_size_dict.keys():\n fleet_size_list.append(str(key) + ':' + str(fleet_size_dict[key]))\n results[\"fleet_size_max_movement\"] = ' '.join(fleet_size_list)\n return results"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the temporal coverage of each source feed in a given GTFS object.", "response": "def _feed_calendar_span(gtfs, stats):\n \"\"\"\n Computes the temporal coverage of each source feed\n\n Parameters\n ----------\n gtfs: gtfspy.GTFS object\n stats: dict\n where to append the stats\n\n Returns\n -------\n stats: dict\n \"\"\"\n n_feeds = _n_gtfs_sources(gtfs)[0]\n max_start = None\n min_end = None\n if n_feeds > 1:\n for i in range(n_feeds):\n feed_key = \"feed_\" + str(i) + \"_\"\n start_key = feed_key + \"calendar_start\"\n end_key = feed_key + \"calendar_end\"\n calendar_span = gtfs.conn.cursor().execute(\n 'SELECT min(date), max(date) FROM trips, days '\n 'WHERE trips.trip_I = days.trip_I AND trip_id LIKE ?;', (feed_key + '%',)).fetchone()\n\n stats[start_key] = calendar_span[0]\n stats[end_key] = calendar_span[1]\n if calendar_span[0] is not None and calendar_span[1] is not None:\n if not max_start and not min_end:\n max_start = calendar_span[0]\n min_end = calendar_span[1]\n else:\n if gtfs.get_day_start_ut(calendar_span[0]) > gtfs.get_day_start_ut(max_start):\n max_start = calendar_span[0]\n if gtfs.get_day_start_ut(calendar_span[1]) < gtfs.get_day_start_ut(min_end):\n min_end = calendar_span[1]\n stats[\"latest_feed_start_date\"] = max_start\n stats[\"earliest_feed_end_date\"] = min_end\n else:\n stats[\"latest_feed_start_date\"] = stats[\"start_date\"]\n stats[\"earliest_feed_end_date\"] = stats[\"end_date\"]\n return stats"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the frequency of all types of routes per day.", "response": "def route_frequencies(gtfs, results_by_mode=False):\n \"\"\"\n Return the frequency of all types of routes per day.\n\n Parameters\n -----------\n gtfs: GTFS\n\n Returns\n -------\n pandas.DataFrame with columns\n route_I, type, frequency\n \"\"\"\n day = gtfs.get_suitable_date_for_daily_extract()\n query = (\n \" SELECT f.route_I, type, frequency FROM routes as r\"\n \" JOIN\"\n \" (SELECT route_I, COUNT(route_I) as frequency\"\n \" FROM\"\n \" (SELECT date, route_I, trip_I\"\n \" FROM day_stop_times\"\n \" WHERE date = '{day}'\"\n \" GROUP by route_I, trip_I)\"\n \" GROUP BY route_I) as f\"\n \" ON f.route_I = r.route_I\"\n \" ORDER BY frequency DESC\".format(day=day))\n \n return pd.DataFrame(gtfs.execute_custom_query_pandas(query))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns all the number of vehicles that pass hourly through a stop in a time frame.", "response": "def hourly_frequencies(gtfs, st, et, route_type):\n \"\"\"\n Return all the number of vehicles (i.e. busses,trams,etc) that pass hourly through a stop in a time frame.\n\n Parameters\n ----------\n gtfs: GTFS\n st : int\n start time of the time framein unix time\n et : int\n end time of the time frame in unix time\n route_type: int\n\n Returns\n -------\n numeric pandas.DataFrame with columns\n stop_I, lat, lon, frequency\n \"\"\" \n timeframe = et-st\n hours = timeframe/ 3600\n day = gtfs.get_suitable_date_for_daily_extract()\n stops = gtfs.get_stops_for_route_type(route_type).T.drop_duplicates().T\n query = (\"SELECT * FROM stops as x\"\n \" JOIN\"\n \" (SELECT * , COUNT(*)/{h} as frequency\"\n \" FROM stop_times, days\"\n \" WHERE stop_times.trip_I = days.trip_I\"\n \" AND dep_time_ds > {st}\"\n \" AND dep_time_ds < {et}\"\n \" AND date = '{day}'\"\n \" GROUP BY stop_I) as y\"\n \" ON y.stop_I = x.stop_I\".format(h=hours, st=st, et=et, day=day))\n try:\n trips_frequency = gtfs.execute_custom_query_pandas(query).T.drop_duplicates().T\n df = pd.merge(stops[['stop_I', 'lat', 'lon']], trips_frequency[['stop_I', 'frequency']],\n on='stop_I', how='inner')\n return df.apply(pd.to_numeric)\n except:\n raise ValueError(\"Maybe too short time frame!\")"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the sum of vehicle hours in a particular day by route type.", "response": "def get_vehicle_hours_by_type(gtfs, route_type):\n \"\"\"\n Return the sum of vehicle hours in a particular day by route type.\n \"\"\"\n\n day = gtfs.get_suitable_date_for_daily_extract()\n query = (\" SELECT * , SUM(end_time_ds - start_time_ds)/3600 as vehicle_hours_type\"\n \" FROM\"\n \" (SELECT * FROM day_trips as q1\"\n \" INNER JOIN\"\n \" (SELECT route_I, type FROM routes) as q2\"\n \" ON q1.route_I = q2.route_I\"\n \" WHERE type = {route_type}\"\n \" AND date = '{day}')\".format(day=day, route_type=route_type))\n df = gtfs.execute_custom_query_pandas(query)\n return df['vehicle_hours_type'].item()"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nscans the footpaths originating from stop_id and update the stop label with the arrival time.", "response": "def _scan_footpaths(self, stop_id, walk_departure_time):\n \"\"\"\n Scan the footpaths originating from stop_id\n\n Parameters\n ----------\n stop_id: int\n \"\"\"\n for _, neighbor, data in self._walk_network.edges_iter(nbunch=[stop_id], data=True):\n d_walk = data[\"d_walk\"]\n arrival_time = walk_departure_time + d_walk / self._walk_speed\n self._update_stop_label(neighbor, arrival_time)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef timeit(method):\n def timed(*args, **kw):\n time_start = time.time()\n result = method(*args, **kw)\n time_end = time.time()\n print('timeit: %r %2.2f sec (%r, %r) ' % (method.__name__, time_end-time_start, str(args)[:20], kw))\n return result\n\n return timed", "response": "A decorator for printing out the execution time for a function."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nvalidating and return the set of warnings that are found in the GTFS feed.", "response": "def validate_and_get_warnings(self):\n \"\"\"\n Validates/checks a given GTFS feed with respect to a number of different issues.\n\n The set of warnings that are checked for, can be found in the gtfs_validator.ALL_WARNINGS\n\n Returns\n -------\n warnings: WarningsContainer\n \"\"\"\n self.warnings_container.clear()\n self._validate_stops_with_same_stop_time()\n self._validate_speeds_and_trip_times()\n self._validate_stop_spacings()\n self._validate_stop_sequence()\n self._validate_misplaced_stops()\n return self.warnings_container"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef clean_password(self):\n value = self.cleaned_data.get('password')\n if value not in self.valid_passwords:\n raise forms.ValidationError('Incorrect password.')\n return value", "response": "Check that the password is valid."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef clean(self):\n cleaned_data = super(AuthForm, self).clean()\n user = self.get_user()\n if self.staff_only and (not user or not user.is_staff):\n raise forms.ValidationError('Sorry, only staff are allowed.')\n if self.superusers_only and (not user or not user.is_superuser):\n raise forms.ValidationError('Sorry, only superusers are allowed.')\n return cleaned_data", "response": "When receiving the filled out form check for valid access."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef authenticate(self, token_value):\n try:\n backend_path, user_id = token_value.split(':', 1)\n except (ValueError, AttributeError):\n return False\n backend = auth.load_backend(backend_path)\n return bool(backend.get_user(user_id))", "response": "Check that the password is valid."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_lockdown_form(form_path):\n if not form_path:\n raise ImproperlyConfigured('No LOCKDOWN_FORM specified.')\n form_path_list = form_path.split(\".\")\n new_module = \".\".join(form_path_list[:-1])\n attr = form_path_list[-1]\n try:\n mod = import_module(new_module)\n except (ImportError, ValueError):\n raise ImproperlyConfigured('Module configured in LOCKDOWN_FORM (%s) to'\n ' contain the form class couldn\\'t be '\n 'found.' % new_module)\n try:\n form = getattr(mod, attr)\n except AttributeError:\n raise ImproperlyConfigured('The module configured in LOCKDOWN_FORM '\n ' (%s) doesn\\'t define a \"%s\" form.'\n % (new_module, attr))\n return form", "response": "Return a lockdown form class for a given string pointing to a lockdown form."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks if each request is allowed to access the current resource.", "response": "def process_request(self, request):\n \"\"\"Check if each request is allowed to access the current resource.\"\"\"\n try:\n session = request.session\n except AttributeError:\n raise ImproperlyConfigured('django-lockdown requires the Django '\n 'sessions framework')\n\n # Don't lock down if django-lockdown is disabled altogether.\n if settings.ENABLED is False:\n return None\n\n # Don't lock down if the client REMOTE_ADDR matched and is part of the\n # exception list.\n if self.remote_addr_exceptions:\n remote_addr_exceptions = self.remote_addr_exceptions\n else:\n remote_addr_exceptions = settings.REMOTE_ADDR_EXCEPTIONS\n\n if remote_addr_exceptions:\n # If forwarding proxies are used they must be listed as trusted\n trusted_proxies = self.trusted_proxies or settings.TRUSTED_PROXIES\n\n remote_addr = request.META.get('REMOTE_ADDR')\n if remote_addr in remote_addr_exceptions:\n return None\n if remote_addr in trusted_proxies:\n # If REMOTE_ADDR is a trusted proxy check x-forwarded-for\n x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')\n if x_forwarded_for:\n remote_addr = x_forwarded_for.split(',')[-1].strip()\n if remote_addr in remote_addr_exceptions:\n return None\n\n # Don't lock down if the URL matches an exception pattern.\n if self.url_exceptions:\n url_exceptions = compile_url_exceptions(self.url_exceptions)\n else:\n url_exceptions = compile_url_exceptions(settings.URL_EXCEPTIONS)\n for pattern in url_exceptions:\n if pattern.search(request.path):\n return None\n\n # Don't lock down if the URL resolves to a whitelisted view.\n try:\n resolved_path = resolve(request.path)\n except Resolver404:\n pass\n else:\n if resolved_path.func in settings.VIEW_EXCEPTIONS:\n return None\n\n # Don't lock down if outside of the lockdown dates.\n if self.until_date:\n until_date = self.until_date\n else:\n until_date = settings.UNTIL_DATE\n\n if self.after_date:\n after_date = self.after_date\n else:\n after_date = settings.AFTER_DATE\n\n if until_date or after_date:\n locked_date = False\n if until_date and datetime.datetime.now() < until_date:\n locked_date = True\n if after_date and datetime.datetime.now() > after_date:\n locked_date = True\n if not locked_date:\n return None\n\n form_data = request.POST if request.method == 'POST' else None\n if self.form:\n form_class = self.form\n else:\n form_class = get_lockdown_form(settings.FORM)\n form = form_class(data=form_data, **self.form_kwargs)\n\n authorized = False\n token = session.get(self.session_key)\n if hasattr(form, 'authenticate'):\n if form.authenticate(token):\n authorized = True\n elif token is True:\n authorized = True\n\n if authorized and self.logout_key and self.logout_key in request.GET:\n if self.session_key in session:\n del session[self.session_key]\n querystring = request.GET.copy()\n del querystring[self.logout_key]\n return self.redirect(request)\n\n # Don't lock down if the user is already authorized for previewing.\n if authorized:\n return None\n\n if form.is_valid():\n if hasattr(form, 'generate_token'):\n token = form.generate_token()\n else:\n token = True\n session[self.session_key] = token\n return self.redirect(request)\n\n page_data = {'until_date': until_date, 'after_date': after_date}\n if not hasattr(form, 'show_form') or form.show_form():\n page_data['form'] = form\n\n if self.extra_context:\n page_data.update(self.extra_context)\n\n return render(request, 'lockdown/form.html', page_data)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ninfers a single node from a pattern.", "response": "def infer(pattern, base_path=None):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#infer\n \"\"\"\n package = Package({}, base_path=base_path)\n descriptor = package.infer(pattern)\n return descriptor"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the profile with the received ID as a dict", "response": "def get(self, profile_id):\n '''Returns the profile with the received ID as a dict\n\n If a local copy of the profile exists, it'll be returned. If not, it'll\n be downloaded from the web. The results are cached, so any subsequent\n calls won't hit the filesystem or the web.\n\n Args:\n profile_id (str): The ID of the profile you want.\n\n Raises:\n RegistryError: If there was some problem opening the profile file\n or its format was incorrect.\n '''\n if profile_id not in self._profiles:\n try:\n self._profiles[profile_id] = self._get_profile(profile_id)\n except (ValueError,\n IOError) as e:\n six.raise_from(RegistryError(e), e)\n return self._profiles[profile_id]"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_profile(self, profile_id):\n '''dict: Return the profile with the received ID as a dict (None if it\n doesn't exist).'''\n profile_metadata = self._registry.get(profile_id)\n if not profile_metadata:\n return\n\n path = self._get_absolute_path(profile_metadata.get('schema_path'))\n url = profile_metadata.get('schema')\n if path:\n try:\n return self._load_json_file(path)\n except IOError as local_exc:\n if not url:\n raise local_exc\n\n try:\n return self._load_json_url(url)\n except IOError:\n msg = (\n 'Error loading profile locally at \"{path}\" '\n 'and remotely at \"{url}\".'\n ).format(path=path, url=url)\n six.raise_from(IOError(msg), local_exc)\n elif url:\n return self._load_json_url(url)", "response": "Return the profile with the received ID as a dict."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the registry as dict with profiles keyed by id.", "response": "def _get_registry(self, registry_path_or_url):\n '''dict: Return the registry as dict with profiles keyed by id.'''\n if registry_path_or_url.startswith('http'):\n profiles = self._load_json_url(registry_path_or_url)\n else:\n profiles = self._load_json_file(registry_path_or_url)\n try:\n registry = {}\n for profile in profiles:\n registry[profile['id']] = profile\n return registry\n except KeyError as e:\n msg = (\n 'Registry at \"{path}\" has no \"id\" column.'\n ).format(path=registry_path_or_url)\n six.raise_from(ValueError(msg), e)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _load_json_url(self, url):\n '''dict: Return the JSON at the local path or URL as a dict.'''\n res = requests.get(url)\n res.raise_for_status()\n\n return res.json()", "response": "Return the JSON at the local path or URL as a dict."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngetting base path of the descriptor if string or return None.", "response": "def get_descriptor_base_path(descriptor):\n \"\"\"Get descriptor base path if string or return None.\n \"\"\"\n\n # Infer from path/url\n if isinstance(descriptor, six.string_types):\n if os.path.exists(descriptor):\n base_path = os.path.dirname(os.path.abspath(descriptor))\n else:\n # suppose descriptor is a URL\n base_path = os.path.dirname(descriptor)\n\n # Current dir by default\n else:\n base_path = '.'\n\n return base_path"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dereference_package_descriptor(descriptor, base_path):\n for resource in descriptor.get('resources', []):\n dereference_resource_descriptor(resource, base_path, descriptor)\n return descriptor", "response": "Dereference a data package descriptor."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef dereference_resource_descriptor(descriptor, base_path, base_descriptor=None):\n PROPERTIES = ['schema', 'dialect']\n if base_descriptor is None:\n base_descriptor = descriptor\n for property in PROPERTIES:\n value = descriptor.get(property)\n\n # URI -> No\n if not isinstance(value, six.string_types):\n continue\n\n # URI -> Pointer\n if value.startswith('#'):\n try:\n pointer = jsonpointer.JsonPointer(value[1:])\n descriptor[property] = pointer.resolve(base_descriptor)\n except Exception as error:\n message = 'Not resolved Pointer URI \"%s\" for resource.%s' % (value, property)\n six.raise_from(\n exceptions.DataPackageException(message),\n error\n )\n\n # URI -> Remote\n elif value.startswith('http'):\n try:\n response = requests.get(value)\n response.raise_for_status()\n descriptor[property] = response.json()\n except Exception as error:\n message = 'Not resolved Remote URI \"%s\" for resource.%s' % (value, property)\n six.raise_from(\n exceptions.DataPackageException(message),\n error\n )\n\n # URI -> Local\n else:\n if not is_safe_path(value):\n raise exceptions.DataPackageException(\n 'Not safe path in Local URI \"%s\" '\n 'for resource.%s' % (value, property))\n if not base_path:\n raise exceptions.DataPackageException(\n 'Local URI \"%s\" requires base path '\n 'for resource.%s' % (value, property))\n fullpath = os.path.join(base_path, value)\n try:\n with io.open(fullpath, encoding='utf-8') as file:\n descriptor[property] = json.load(file)\n except Exception as error:\n message = 'Not resolved Local URI \"%s\" for resource.%s' % (value, property)\n six.raise_from(\n exceptions.DataPackageException(message),\n error\n )\n\n return descriptor", "response": "Dereference resource descriptor (IN-PLACE FOR NOW)."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nexpand the descriptor to include all resources in the data package.", "response": "def expand_package_descriptor(descriptor):\n \"\"\"Apply defaults to data package descriptor (IN-PLACE FOR NOW).\n \"\"\"\n descriptor.setdefault('profile', config.DEFAULT_DATA_PACKAGE_PROFILE)\n for resource in descriptor.get('resources', []):\n expand_resource_descriptor(resource)\n return descriptor"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef expand_resource_descriptor(descriptor):\n descriptor.setdefault('profile', config.DEFAULT_RESOURCE_PROFILE)\n if descriptor['profile'] == 'tabular-data-resource':\n\n # Schema\n schema = descriptor.get('schema')\n if schema is not None:\n for field in schema.get('fields', []):\n field.setdefault('type', config.DEFAULT_FIELD_TYPE)\n field.setdefault('format', config.DEFAULT_FIELD_FORMAT)\n schema.setdefault('missingValues', config.DEFAULT_MISSING_VALUES)\n\n # Dialect\n dialect = descriptor.get('dialect')\n if dialect is not None:\n for key, value in config.DEFAULT_DIALECT.items():\n dialect.setdefault(key, value)\n\n return descriptor", "response": "Expand the resource descriptor to include default values."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_safe_path(path):\n contains_windows_var = lambda val: re.match(r'%.+%', val)\n contains_posix_var = lambda val: re.match(r'\\$.+', val)\n\n unsafeness_conditions = [\n os.path.isabs(path),\n ('..%s' % os.path.sep) in path,\n path.startswith('~'),\n os.path.expandvars(path) != path,\n contains_windows_var(path),\n contains_posix_var(path),\n ]\n\n return not any(unsafeness_conditions)", "response": "Check if path is safe and allowed."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _extract_zip_if_possible(descriptor):\n tempdir = None\n result = descriptor\n try:\n if isinstance(descriptor, six.string_types):\n res = requests.get(descriptor)\n res.raise_for_status()\n result = res.content\n except (IOError,\n ValueError,\n requests.exceptions.RequestException):\n pass\n try:\n the_zip = result\n if isinstance(the_zip, bytes):\n try:\n os.path.isfile(the_zip)\n except (TypeError, ValueError):\n # the_zip contains the zip file contents\n the_zip = io.BytesIO(the_zip)\n if zipfile.is_zipfile(the_zip):\n with zipfile.ZipFile(the_zip, 'r') as z:\n _validate_zip(z)\n descriptor_path = [\n f for f in z.namelist() if f.endswith('datapackage.json')][0]\n tempdir = tempfile.mkdtemp('-datapackage')\n z.extractall(tempdir)\n result = os.path.join(tempdir, descriptor_path)\n else:\n result = descriptor\n except (TypeError,\n zipfile.BadZipfile):\n pass\n if hasattr(descriptor, 'seek'):\n # Rewind descriptor if it's a file, as we read it for testing if it's\n # a zip file\n descriptor.seek(0)\n return (tempdir, result)", "response": "Extract the archive if possible."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _validate_zip(the_zip):\n datapackage_jsons = [f for f in the_zip.namelist() if f.endswith('datapackage.json')]\n if len(datapackage_jsons) != 1:\n msg = 'DataPackage must have only one \"datapackage.json\" (had {n})'\n raise exceptions.DataPackageException(msg.format(n=len(datapackage_jsons)))", "response": "Validate zipped data package"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_resource(self, name):\n for resource in self.resources:\n if resource.name == name:\n return resource\n return None", "response": "Get a resource by its name."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding a resource to the current resource set.", "response": "def add_resource(self, descriptor):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#package\n \"\"\"\n self.__current_descriptor.setdefault('resources', [])\n self.__current_descriptor['resources'].append(descriptor)\n self.__build()\n return self.__resources[-1]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nremoving a resource from the resource list.", "response": "def remove_resource(self, name):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#package\n \"\"\"\n resource = self.get_resource(name)\n if resource:\n predicat = lambda resource: resource.get('name') != name\n self.__current_descriptor['resources'] = list(filter(\n predicat, self.__current_descriptor['resources']))\n self.__build()\n return resource"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef infer(self, pattern=False):\n\n # Files\n if pattern:\n\n # No base path\n if not self.__base_path:\n message = 'Base path is required for pattern infer'\n raise exceptions.DataPackageException(message)\n\n # Add resources\n options = {'recursive': True} if '**' in pattern else {}\n for path in glob.glob(os.path.join(self.__base_path, pattern), **options):\n self.add_resource({'path': os.path.relpath(path, self.__base_path)})\n\n # Resources\n for index, resource in enumerate(self.resources):\n descriptor = resource.infer()\n self.__current_descriptor['resources'][index] = descriptor\n self.__build()\n\n # Profile\n if self.__next_descriptor['profile'] == config.DEFAULT_DATA_PACKAGE_PROFILE:\n if self.resources and all(map(lambda resource: resource.tabular, self.resources)):\n self.__current_descriptor['profile'] = 'tabular-data-package'\n self.__build()\n\n return self.__current_descriptor", "response": "Infer the current state of the data package."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsaves the current state of the object to a file or zip file.", "response": "def save(self, target=None, storage=None, **options):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#package\n \"\"\"\n\n # Save package to storage\n if storage is not None:\n if not isinstance(storage, Storage):\n storage = Storage.connect(storage, **options)\n buckets = []\n schemas = []\n for resource in self.resources:\n if resource.tabular:\n resource.infer()\n buckets.append(_slugify_resource_name(resource.name))\n schemas.append(resource.schema.descriptor)\n schemas = list(map(_slugify_foreign_key, schemas))\n storage.create(buckets, schemas, force=True)\n for bucket in storage.buckets:\n resource = self.resources[buckets.index(bucket)]\n storage.write(bucket, resource.iter())\n\n # Save descriptor to json\n elif str(target).endswith('.json'):\n mode = 'w'\n encoding = 'utf-8'\n if six.PY2:\n mode = 'wb'\n encoding = None\n helpers.ensure_dir(target)\n with io.open(target, mode=mode, encoding=encoding) as file:\n json.dump(self.__current_descriptor, file, indent=4)\n\n # Save package to zip\n else:\n try:\n with zipfile.ZipFile(target, 'w') as z:\n descriptor = json.loads(json.dumps(self.__current_descriptor))\n for index, resource in enumerate(self.resources):\n if not resource.name:\n continue\n if not resource.local:\n continue\n path = os.path.abspath(resource.source)\n basename = resource.descriptor.get('name')\n resource_format = resource.descriptor.get('format')\n if resource_format:\n basename = '.'.join([basename, resource_format.lower()])\n path_inside_dp = os.path.join('data', basename)\n z.write(path, path_inside_dp)\n descriptor['resources'][index]['path'] = path_inside_dp\n z.writestr('datapackage.json', json.dumps(descriptor))\n except (IOError, zipfile.BadZipfile, zipfile.LargeZipFile) as exception:\n six.raise_from(exceptions.DataPackageException(exception), exception)\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef attributes(self):\n\n # Deprecate\n warnings.warn(\n 'Property \"package.attributes\" is deprecated.',\n UserWarning)\n\n # Get attributes\n attributes = set(self.to_dict().keys())\n try:\n attributes.update(self.profile.properties.keys())\n except AttributeError:\n pass\n\n return tuple(attributes)", "response": "Returns a tuple of attributes defined in the schema and the data package."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a tuple of the schema s required attributed.", "response": "def required_attributes(self):\n \"\"\"tuple: The schema's required attributed.\n \"\"\"\n\n # Deprecate\n warnings.warn(\n 'Property \"package.required_attributes\" is deprecated.',\n UserWarning)\n required = ()\n\n # Get required\n try:\n if self.profile.required is not None:\n required = tuple(self.profile.required)\n except AttributeError:\n pass\n\n return required"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate(self):\n\n # Deprecate\n warnings.warn(\n 'Property \"package.validate\" is deprecated.',\n UserWarning)\n\n descriptor = self.to_dict()\n self.profile.validate(descriptor)", "response": "Validate this Data Package.\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nvalidate the given descriptor.", "response": "def validate(self, descriptor):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#schema\n \"\"\"\n\n # Collect errors\n errors = []\n for error in self._validator.iter_errors(descriptor):\n if isinstance(error, jsonschema.exceptions.ValidationError):\n message = str(error.message)\n if six.PY2:\n message = message.replace('u\\'', '\\'')\n descriptor_path = '/'.join(map(str, error.path))\n profile_path = '/'.join(map(str, error.schema_path))\n error = exceptions.ValidationError(\n 'Descriptor validation error: %s '\n 'at \"%s\" in descriptor and '\n 'at \"%s\" in profile'\n % (message, descriptor_path, profile_path))\n errors.append(error)\n\n # Raise error\n if errors:\n message = 'There are %s validation errors (see exception.errors)' % len(errors)\n raise exceptions.ValidationError(message, errors=errors)\n\n return True"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef tabular(self):\n if self.__current_descriptor.get('profile') == 'tabular-data-resource':\n return True\n if not self.__strict:\n if self.__current_descriptor.get('format') in config.TABULAR_FORMATS:\n return True\n if self.__source_inspection.get('tabular', False):\n return True\n return False", "response": "Return True if the resource is tabular."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef iter(self, relations=False, **options):\n\n # Error for non tabular\n if not self.tabular:\n message = 'Methods iter/read are not supported for non tabular data'\n raise exceptions.DataPackageException(message)\n\n # Get relations\n if relations:\n relations = self.__get_relations()\n\n return self.__get_table().iter(relations=relations, **options)", "response": "Get an iterator over the resource."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef raw_iter(self, stream=False):\n\n # Error for inline\n if self.inline:\n message = 'Methods raw_iter/raw_read are not supported for inline data'\n raise exceptions.DataPackageException(message)\n\n # Get filelike\n if self.multipart:\n filelike = _MultipartSource(self.source, remote=self.remote)\n elif self.remote:\n if self.__table_options.get('http_session'):\n http_session = self.__table_options['http_session']\n else:\n http_session = requests.Session()\n http_session.headers = config.HTTP_HEADERS\n res = http_session.get(self.source, stream=True)\n filelike = res.raw\n else:\n filelike = io.open(self.source, 'rb')\n\n return filelike", "response": "Get the raw content of the resource in a file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nread the raw data from the resource table.", "response": "def raw_read(self):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#resource\n \"\"\"\n contents = b''\n with self.raw_iter() as filelike:\n for chunk in filelike:\n contents += chunk\n return contents"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ninfer the resource from the source file.", "response": "def infer(self, **options):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#resource\n \"\"\"\n descriptor = deepcopy(self.__current_descriptor)\n\n # Blank -> Stop\n if self.__source_inspection.get('blank'):\n return descriptor\n\n # Name\n if not descriptor.get('name'):\n descriptor['name'] = self.__source_inspection['name']\n\n # Only for non inline/storage\n if not self.inline and not self.__storage:\n\n # Format\n if not descriptor.get('format'):\n descriptor['format'] = self.__source_inspection['format']\n\n # Mediatype\n if not descriptor.get('mediatype'):\n descriptor['mediatype'] = 'text/%s' % descriptor['format']\n\n # Encoding\n if not descriptor.get('encoding'):\n contents = b''\n with self.raw_iter(stream=True) as stream:\n for chunk in stream:\n contents += chunk\n if len(contents) > 1000: break\n encoding = cchardet.detect(contents)['encoding']\n if encoding is not None:\n encoding = encoding.lower()\n descriptor['encoding'] = 'utf-8' if encoding == 'ascii' else encoding\n\n # Schema\n if not descriptor.get('schema'):\n if self.tabular:\n descriptor['schema'] = self.__get_table().infer(**options)\n\n # Profile\n if descriptor.get('profile') == config.DEFAULT_RESOURCE_PROFILE:\n if self.tabular:\n descriptor['profile'] = 'tabular-data-resource'\n\n # Save descriptor\n self.__current_descriptor = descriptor\n self.__build()\n\n return descriptor"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncommits the current state of the resource.", "response": "def commit(self, strict=None):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#resource\n \"\"\"\n if strict is not None:\n self.__strict = strict\n elif self.__current_descriptor == self.__next_descriptor:\n return False\n self.__current_descriptor = deepcopy(self.__next_descriptor)\n self.__table = None\n self.__build()\n return True"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsaving the current resource to a file.", "response": "def save(self, target, storage=None, **options):\n \"\"\"https://github.com/frictionlessdata/datapackage-py#resource\n \"\"\"\n\n # Save resource to storage\n if storage is not None:\n if self.tabular:\n self.infer()\n storage.create(target, self.schema.descriptor, force=True)\n storage.write(target, self.iter())\n\n # Save descriptor to json\n else:\n mode = 'w'\n encoding = 'utf-8'\n if six.PY2:\n mode = 'wb'\n encoding = None\n helpers.ensure_dir(target)\n with io.open(target, mode=mode, encoding=encoding) as file:\n json.dump(self.__current_descriptor, file, indent=4)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\npush a new data package to the storage.", "response": "def push_datapackage(descriptor, backend, **backend_options):\n \"\"\"Push Data Package to storage.\n\n All parameters should be used as keyword arguments.\n\n Args:\n descriptor (str): path to descriptor\n backend (str): backend name like `sql` or `bigquery`\n backend_options (dict): backend options mentioned in backend docs\n\n \"\"\"\n\n # Deprecated\n warnings.warn(\n 'Functions \"push/pull_datapackage\" are deprecated. '\n 'Please use \"Package\" class',\n UserWarning)\n\n # Init maps\n tables = []\n schemas = []\n datamap = {}\n mapping = {}\n\n # Init model\n model = Package(descriptor)\n\n # Get storage\n plugin = import_module('jsontableschema.plugins.%s' % backend)\n storage = plugin.Storage(**backend_options)\n\n # Collect tables/schemas/data\n for resource in model.resources:\n if not resource.tabular:\n continue\n name = resource.descriptor.get('name', None)\n table = _convert_path(resource.descriptor['path'], name)\n schema = resource.descriptor['schema']\n data = resource.table.iter(keyed=True)\n # TODO: review\n def values(schema, data):\n for item in data:\n row = []\n for field in schema['fields']:\n row.append(item.get(field['name'], None))\n yield tuple(row)\n tables.append(table)\n schemas.append(schema)\n datamap[table] = values(schema, data)\n if name is not None:\n mapping[name] = table\n schemas = _convert_schemas(mapping, schemas)\n\n # Create tables\n for table in tables:\n if table in storage.buckets:\n storage.delete(table)\n storage.create(tables, schemas)\n\n # Write data to tables\n for table in storage.buckets:\n if table in datamap:\n storage.write(table, datamap[table])\n return storage"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\npull a datapackage from the storage.", "response": "def pull_datapackage(descriptor, name, backend, **backend_options):\n \"\"\"Pull Data Package from storage.\n\n All parameters should be used as keyword arguments.\n\n Args:\n descriptor (str): path where to store descriptor\n name (str): name of the pulled datapackage\n backend (str): backend name like `sql` or `bigquery`\n backend_options (dict): backend options mentioned in backend docs\n\n \"\"\"\n\n # Deprecated\n warnings.warn(\n 'Functions \"push/pull_datapackage\" are deprecated. '\n 'Please use \"Package\" class',\n UserWarning)\n\n # Save datapackage name\n datapackage_name = name\n\n # Get storage\n plugin = import_module('jsontableschema.plugins.%s' % backend)\n storage = plugin.Storage(**backend_options)\n\n # Iterate over tables\n resources = []\n for table in storage.buckets:\n\n # Prepare\n schema = storage.describe(table)\n base = os.path.dirname(descriptor)\n path, name = _restore_path(table)\n fullpath = os.path.join(base, path)\n\n # Write data\n helpers.ensure_dir(fullpath)\n with io.open(fullpath, 'wb') as file:\n model = Schema(deepcopy(schema))\n data = storage.iter(table)\n writer = csv.writer(file, encoding='utf-8')\n writer.writerow(model.headers)\n for row in data:\n writer.writerow(row)\n\n # Add resource\n resource = {'schema': schema, 'path': path}\n if name is not None:\n resource['name'] = name\n resources.append(resource)\n\n # Write descriptor\n mode = 'w'\n encoding = 'utf-8'\n if six.PY2:\n mode = 'wb'\n encoding = None\n resources = _restore_resources(resources)\n helpers.ensure_dir(descriptor)\n with io.open(descriptor,\n mode=mode,\n encoding=encoding) as file:\n descriptor = {\n 'name': datapackage_name,\n 'resources': resources,\n }\n json.dump(descriptor, file, indent=4)\n return storage"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _convert_path(path, name):\n table = os.path.splitext(path)[0]\n table = table.replace(os.path.sep, '__')\n if name is not None:\n table = '___'.join([table, name])\n table = re.sub('[^0-9a-zA-Z_]+', '_', table)\n table = table.lower()\n return table", "response": "Convert resource s path and name to storage s table name."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nrestores resource s path and name from storage s table.", "response": "def _restore_path(table):\n \"\"\"Restore resource's path and name from storage's table.\n\n Args:\n table (str): table name\n\n Returns:\n (str, str): resource path and name\n\n \"\"\"\n name = None\n splited = table.split('___')\n path = splited[0]\n if len(splited) == 2:\n name = splited[1]\n path = path.replace('__', os.path.sep)\n path += '.csv'\n return path, name"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts schemas to be compatible with storage schemas.", "response": "def _convert_schemas(mapping, schemas):\n \"\"\"Convert schemas to be compatible with storage schemas.\n\n Foreign keys related operations.\n\n Args:\n mapping (dict): mapping between resource name and table name\n schemas (list): schemas\n\n Raises:\n ValueError: if there is no resource\n for some foreign key in given mapping\n\n Returns:\n list: converted schemas\n\n \"\"\"\n schemas = deepcopy(schemas)\n for schema in schemas:\n for fk in schema.get('foreignKeys', []):\n resource = fk['reference']['resource']\n if resource != 'self':\n if resource not in mapping:\n message = 'Not resource \"%s\" for foreign key \"%s\"'\n message = message % (resource, fk)\n raise ValueError(message)\n fk['reference']['resource'] = mapping[resource]\n return schemas"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nrestore schemas from being compatible with storage schemas.", "response": "def _restore_resources(resources):\n \"\"\"Restore schemas from being compatible with storage schemas.\n\n Foreign keys related operations.\n\n Args:\n list: resources from storage\n\n Returns:\n list: restored resources\n\n \"\"\"\n resources = deepcopy(resources)\n for resource in resources:\n schema = resource['schema']\n for fk in schema.get('foreignKeys', []):\n _, name = _restore_path(fk['reference']['resource'])\n fk['reference']['resource'] = name\n return resources"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _make_non_blocking(file_obj):\n\n if USING_WINDOWS:\n LPDWORD = POINTER(DWORD)\n PIPE_NOWAIT = wintypes.DWORD(0x00000001)\n\n SetNamedPipeHandleState = windll.kernel32.SetNamedPipeHandleState\n SetNamedPipeHandleState.argtypes = [HANDLE, LPDWORD, LPDWORD, LPDWORD]\n SetNamedPipeHandleState.restype = BOOL\n\n h = msvcrt.get_osfhandle(file_obj.fileno())\n\n res = windll.kernel32.SetNamedPipeHandleState(h, byref(PIPE_NOWAIT), None, None)\n if res == 0:\n raise ValueError(WinError())\n\n else:\n # Set the file status flag (F_SETFL) on the pipes to be non-blocking\n # so we can attempt to read from a pipe with no new data without locking\n # the program up\n fcntl.fcntl(file_obj, fcntl.F_SETFL, os.O_NONBLOCK)", "response": "make file object non - blocking"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nspawn a new gdb subprocess with the arguments supplied to the object during initialization.", "response": "def spawn_new_gdb_subprocess(self):\n \"\"\"Spawn a new gdb subprocess with the arguments supplied to the object\n during initialization. If gdb subprocess already exists, terminate it before\n spanwing a new one.\n Return int: gdb process id\n \"\"\"\n if self.gdb_process:\n self.logger.debug(\n \"Killing current gdb subprocess (pid %d)\" % self.gdb_process.pid\n )\n self.exit()\n\n self.logger.debug('Launching gdb: \"%s\"' % \" \".join(self.cmd))\n\n # Use pipes to the standard streams\n self.gdb_process = subprocess.Popen(\n self.cmd,\n shell=False,\n stdout=subprocess.PIPE,\n stdin=subprocess.PIPE,\n stderr=subprocess.PIPE,\n bufsize=0,\n )\n\n _make_non_blocking(self.gdb_process.stdout)\n _make_non_blocking(self.gdb_process.stderr)\n\n # save file numbers for use later\n self.stdout_fileno = self.gdb_process.stdout.fileno()\n self.stderr_fileno = self.gdb_process.stderr.fileno()\n self.stdin_fileno = self.gdb_process.stdin.fileno()\n\n self.read_list = [self.stdout_fileno, self.stderr_fileno]\n self.write_list = [self.stdin_fileno]\n\n # string buffers for unifinished gdb output\n self._incomplete_output = {\"stdout\": None, \"stderr\": None}\n return self.gdb_process.pid"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nverifying that the gdb process is still running. Raise NoGdbProcessError if either of the above are not true.", "response": "def verify_valid_gdb_subprocess(self):\n \"\"\"Verify there is a process object, and that it is still running.\n Raise NoGdbProcessError if either of the above are not true.\"\"\"\n if not self.gdb_process:\n raise NoGdbProcessError(\"gdb process is not attached\")\n\n elif self.gdb_process.poll() is not None:\n raise NoGdbProcessError(\n \"gdb process has already finished with return code: %s\"\n % str(self.gdb_process.poll())\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites to gdb process.", "response": "def write(\n self,\n mi_cmd_to_write,\n timeout_sec=DEFAULT_GDB_TIMEOUT_SEC,\n raise_error_on_timeout=True,\n read_response=True,\n ):\n \"\"\"Write to gdb process. Block while parsing responses from gdb for a maximum of timeout_sec.\n\n Args:\n mi_cmd_to_write (str or list): String to write to gdb. If list, it is joined by newlines.\n timeout_sec (float): Maximum number of seconds to wait for response before exiting. Must be >= 0.\n raise_error_on_timeout (bool): If read_response is True, raise error if no response is received\n read_response (bool): Block and read response. If there is a separate thread running,\n this can be false, and the reading thread read the output.\n Returns:\n List of parsed gdb responses if read_response is True, otherwise []\n Raises:\n NoGdbProcessError if there is no gdb subprocess running\n TypeError if mi_cmd_to_write is not valid\n \"\"\"\n self.verify_valid_gdb_subprocess()\n if timeout_sec < 0:\n self.logger.warning(\"timeout_sec was negative, replacing with 0\")\n timeout_sec = 0\n\n # Ensure proper type of the mi command\n if type(mi_cmd_to_write) in [str, unicode]:\n pass\n elif type(mi_cmd_to_write) == list:\n mi_cmd_to_write = \"\\n\".join(mi_cmd_to_write)\n else:\n raise TypeError(\n \"The gdb mi command must a be str or list. Got \"\n + str(type(mi_cmd_to_write))\n )\n\n self.logger.debug(\"writing: %s\", mi_cmd_to_write)\n\n if not mi_cmd_to_write.endswith(\"\\n\"):\n mi_cmd_to_write_nl = mi_cmd_to_write + \"\\n\"\n else:\n mi_cmd_to_write_nl = mi_cmd_to_write\n\n if USING_WINDOWS:\n # select not implemented in windows for pipes\n # assume it's always ready\n outputready = [self.stdin_fileno]\n else:\n _, outputready, _ = select.select([], self.write_list, [], timeout_sec)\n for fileno in outputready:\n if fileno == self.stdin_fileno:\n # ready to write\n self.gdb_process.stdin.write(mi_cmd_to_write_nl.encode())\n # don't forget to flush for Python3, otherwise gdb won't realize there is data\n # to evaluate, and we won't get a response\n self.gdb_process.stdin.flush()\n else:\n self.logger.error(\"got unexpected fileno %d\" % fileno)\n\n if read_response is True:\n return self.get_gdb_response(\n timeout_sec=timeout_sec, raise_error_on_timeout=raise_error_on_timeout\n )\n\n else:\n return []"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_gdb_response(\n self, timeout_sec=DEFAULT_GDB_TIMEOUT_SEC, raise_error_on_timeout=True\n ):\n \"\"\"Get response from GDB, and block while doing so. If GDB does not have any response ready to be read\n by timeout_sec, an exception is raised.\n\n Args:\n timeout_sec (float): Maximum time to wait for reponse. Must be >= 0. Will return after\n raise_error_on_timeout (bool): Whether an exception should be raised if no response was found\n after timeout_sec\n\n Returns:\n List of parsed GDB responses, returned from gdbmiparser.parse_response, with the\n additional key 'stream' which is either 'stdout' or 'stderr'\n\n Raises:\n GdbTimeoutError if response is not received within timeout_sec\n ValueError if select returned unexpected file number\n NoGdbProcessError if there is no gdb subprocess running\n \"\"\"\n\n self.verify_valid_gdb_subprocess()\n if timeout_sec < 0:\n self.logger.warning(\"timeout_sec was negative, replacing with 0\")\n timeout_sec = 0\n\n if USING_WINDOWS:\n retval = self._get_responses_windows(timeout_sec)\n else:\n retval = self._get_responses_unix(timeout_sec)\n\n if not retval and raise_error_on_timeout:\n raise GdbTimeoutError(\n \"Did not get response from gdb after %s seconds\" % timeout_sec\n )\n\n else:\n return retval", "response": "Get response from GDB and block while doing so."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget responses on windows.", "response": "def _get_responses_windows(self, timeout_sec):\n \"\"\"Get responses on windows. Assume no support for select and use a while loop.\"\"\"\n timeout_time_sec = time.time() + timeout_sec\n responses = []\n while True:\n try:\n self.gdb_process.stdout.flush()\n if PYTHON3:\n raw_output = self.gdb_process.stdout.readline().replace(\n b\"\\r\", b\"\\n\"\n )\n else:\n raw_output = self.gdb_process.stdout.read().replace(b\"\\r\", b\"\\n\")\n responses += self._get_responses_list(raw_output, \"stdout\")\n except IOError:\n pass\n\n try:\n self.gdb_process.stderr.flush()\n if PYTHON3:\n raw_output = self.gdb_process.stderr.readline().replace(\n b\"\\r\", b\"\\n\"\n )\n else:\n raw_output = self.gdb_process.stderr.read().replace(b\"\\r\", b\"\\n\")\n responses += self._get_responses_list(raw_output, \"stderr\")\n except IOError:\n pass\n\n if time.time() > timeout_time_sec:\n break\n\n return responses"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_responses_unix(self, timeout_sec):\n timeout_time_sec = time.time() + timeout_sec\n responses = []\n while True:\n select_timeout = timeout_time_sec - time.time()\n # I prefer to not pass a negative value to select\n if select_timeout <= 0:\n select_timeout = 0\n events, _, _ = select.select(self.read_list, [], [], select_timeout)\n responses_list = None # to avoid infinite loop if using Python 2\n try:\n for fileno in events:\n # new data is ready to read\n if fileno == self.stdout_fileno:\n self.gdb_process.stdout.flush()\n raw_output = self.gdb_process.stdout.read()\n stream = \"stdout\"\n\n elif fileno == self.stderr_fileno:\n self.gdb_process.stderr.flush()\n raw_output = self.gdb_process.stderr.read()\n stream = \"stderr\"\n\n else:\n raise ValueError(\n \"Developer error. Got unexpected file number %d\" % fileno\n )\n\n responses_list = self._get_responses_list(raw_output, stream)\n responses += responses_list\n\n except IOError: # only occurs in python 2.7\n pass\n\n if timeout_sec == 0: # just exit immediately\n break\n\n elif responses_list and self._allow_overwrite_timeout_times:\n # update timeout time to potentially be closer to now to avoid lengthy wait times when nothing is being output by gdb\n timeout_time_sec = min(\n time.time() + self.time_to_check_for_additional_output_sec,\n timeout_time_sec,\n )\n\n elif time.time() > timeout_time_sec:\n break\n\n return responses", "response": "Get the list of responses on unix - like system. Use select to wait for output."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting parsed response list from string output", "response": "def _get_responses_list(self, raw_output, stream):\n \"\"\"Get parsed response list from string output\n Args:\n raw_output (unicode): gdb output to parse\n stream (str): either stdout or stderr\n \"\"\"\n responses = []\n\n raw_output, self._incomplete_output[stream] = _buffer_incomplete_responses(\n raw_output, self._incomplete_output.get(stream)\n )\n\n if not raw_output:\n return responses\n\n response_list = list(\n filter(lambda x: x, raw_output.decode(errors=\"replace\").split(\"\\n\"))\n ) # remove blank lines\n\n # parse each response from gdb into a dict, and store in a list\n for response in response_list:\n if gdbmiparser.response_is_finished(response):\n pass\n else:\n parsed_response = gdbmiparser.parse_response(response)\n parsed_response[\"stream\"] = stream\n\n self.logger.debug(\"%s\", pformat(parsed_response))\n\n responses.append(parsed_response)\n\n return responses"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef send_signal_to_gdb(self, signal_input):\n try:\n signal = int(signal_input)\n except Exception:\n signal = SIGNAL_NAME_TO_NUM.get(signal_input.upper())\n\n if not signal:\n raise ValueError(\n 'Could not find signal corresponding to \"%s\"' % str(signal)\n )\n\n if self.gdb_process:\n os.kill(self.gdb_process.pid, signal)\n else:\n raise NoGdbProcessError(\n \"Cannot send signal to gdb process because no process exists.\"\n )", "response": "Send a signal to the gdb process."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef exit(self):\n if self.gdb_process:\n self.gdb_process.terminate()\n self.gdb_process.communicate()\n self.gdb_process = None\n return None", "response": "Terminate gdb process\n Returns None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding and debug an application programatically For a list of GDB MI commands, see https://www.sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html", "response": "def main(verbose=True):\n \"\"\"Build and debug an application programatically\n\n For a list of GDB MI commands, see https://www.sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI.html\n \"\"\"\n\n # Build C program\n find_executable(MAKE_CMD)\n if not find_executable(MAKE_CMD):\n print(\n 'Could not find executable \"%s\". Ensure it is installed and on your $PATH.'\n % MAKE_CMD\n )\n exit(1)\n subprocess.check_output([MAKE_CMD, \"-C\", SAMPLE_C_CODE_DIR, \"--quiet\"])\n\n # Initialize object that manages gdb subprocess\n gdbmi = GdbController(verbose=verbose)\n\n # Send gdb commands. Gdb machine interface commands are easier to script around,\n # hence the name \"machine interface\".\n # Responses are automatically printed as they are received if verbose is True.\n # Responses are returned after writing, by default.\n\n # Load the file\n responses = gdbmi.write(\"-file-exec-and-symbols %s\" % SAMPLE_C_BINARY)\n # Get list of source files used to compile the binary\n responses = gdbmi.write(\"-file-list-exec-source-files\")\n # Add breakpoint\n responses = gdbmi.write(\"-break-insert main\")\n # Run\n responses = gdbmi.write(\"-exec-run\")\n responses = gdbmi.write(\"-exec-next\")\n responses = gdbmi.write(\"-exec-next\")\n responses = gdbmi.write(\"-exec-continue\") # noqa: F841\n\n # gdbmi.gdb_process will be None because the gdb subprocess (and its inferior\n # program) will be terminated\n gdbmi.exit()"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read(self, count):\n new_index = self.index + count\n if new_index > self.len:\n buf = self.raw_text[self.index :] # return to the end, don't fail\n else:\n buf = self.raw_text[self.index : new_index]\n self.index = new_index\n\n return buf", "response": "Read count characters starting at self. index and return those characters as a string\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadvance the index past specific chars", "response": "def advance_past_chars(self, chars):\n \"\"\"Advance the index past specific chars\n Args chars (list): list of characters to advance past\n\n Return substring that was advanced past\n \"\"\"\n start_index = self.index\n while True:\n current_char = self.raw_text[self.index]\n self.index += 1\n if current_char in chars:\n break\n\n elif self.index == self.len:\n break\n\n return self.raw_text[start_index : self.index - 1]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadvancing to the next string that has gdb escapes.", "response": "def advance_past_string_with_gdb_escapes(self, chars_to_remove_gdb_escape=None):\n \"\"\"characters that gdb escapes that should not be\n escaped by this parser\n \"\"\"\n\n if chars_to_remove_gdb_escape is None:\n chars_to_remove_gdb_escape = ['\"']\n\n buf = \"\"\n while True:\n c = self.raw_text[self.index]\n self.index += 1\n logging.debug(\"%s\", fmt_cyan(c))\n\n if c == \"\\\\\":\n # We are on a backslash and there is another character after the backslash\n # to parse. Handle this case specially since gdb escaped it for us\n\n # Get the next char that is being escaped\n c2 = self.raw_text[self.index]\n self.index += 1\n # only store the escaped character in the buffer; don't store the backslash\n # (don't leave it escaped)\n buf += c2\n\n elif c == '\"':\n # Quote is closed. Exit (and don't include the end quote).\n break\n\n else:\n # capture this character, and keep capturing\n buf += c\n return buf"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_response(gdb_mi_text):\n stream = StringStream(gdb_mi_text, debug=_DEBUG)\n\n if _GDB_MI_NOTIFY_RE.match(gdb_mi_text):\n token, message, payload = _get_notify_msg_and_payload(gdb_mi_text, stream)\n return {\n \"type\": \"notify\",\n \"message\": message,\n \"payload\": payload,\n \"token\": token,\n }\n\n elif _GDB_MI_RESULT_RE.match(gdb_mi_text):\n token, message, payload = _get_result_msg_and_payload(gdb_mi_text, stream)\n return {\n \"type\": \"result\",\n \"message\": message,\n \"payload\": payload,\n \"token\": token,\n }\n\n elif _GDB_MI_CONSOLE_RE.match(gdb_mi_text):\n return {\n \"type\": \"console\",\n \"message\": None,\n \"payload\": _GDB_MI_CONSOLE_RE.match(gdb_mi_text).groups()[0],\n }\n\n elif _GDB_MI_LOG_RE.match(gdb_mi_text):\n return {\n \"type\": \"log\",\n \"message\": None,\n \"payload\": _GDB_MI_LOG_RE.match(gdb_mi_text).groups()[0],\n }\n\n elif _GDB_MI_TARGET_OUTPUT_RE.match(gdb_mi_text):\n return {\n \"type\": \"target\",\n \"message\": None,\n \"payload\": _GDB_MI_TARGET_OUTPUT_RE.match(gdb_mi_text).groups()[0],\n }\n\n elif response_is_finished(gdb_mi_text):\n return {\"type\": \"done\", \"message\": None, \"payload\": None}\n\n else:\n # This was not gdb mi output, so it must have just been printed by\n # the inferior program that's being debugged\n return {\"type\": \"output\", \"message\": None, \"payload\": gdb_mi_text}", "response": "Parse the gdb mi output into a dictionary."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef assert_match(actual_char_or_str, expected_char_or_str):\n if expected_char_or_str != actual_char_or_str:\n print(\"Expected\")\n pprint(expected_char_or_str)\n print(\"\")\n print(\"Got\")\n pprint(actual_char_or_str)\n raise ValueError()", "response": "Assert that the character or string of the current node matches the expected character."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_notify_msg_and_payload(result, stream):\n token = stream.advance_past_chars([\"=\", \"*\"])\n token = int(token) if token != \"\" else None\n logger.debug(\"%s\", fmt_green(\"parsing message\"))\n message = stream.advance_past_chars([\",\"])\n\n logger.debug(\"parsed message\")\n logger.debug(\"%s\", fmt_green(message))\n\n payload = _parse_dict(stream)\n return token, message.strip(), payload", "response": "Get notify message and payload dict from stream."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting result message and payload dict", "response": "def _get_result_msg_and_payload(result, stream):\n \"\"\"Get result message and payload dict\"\"\"\n\n groups = _GDB_MI_RESULT_RE.match(result).groups()\n token = int(groups[0]) if groups[0] != \"\" else None\n message = groups[1]\n\n if groups[2] is None:\n payload = None\n else:\n stream.advance_past_chars([\",\"])\n payload = _parse_dict(stream)\n\n return token, message, payload"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nparsing a dictionary of key value pairs.", "response": "def _parse_dict(stream):\n \"\"\"Parse dictionary, with optional starting character '{'\n return (tuple):\n Number of characters parsed from to_parse\n Parsed dictionary\n \"\"\"\n obj = {}\n\n logger.debug(\"%s\", fmt_green(\"parsing dict\"))\n\n while True:\n c = stream.read(1)\n if c in _WHITESPACE:\n pass\n elif c in [\"{\", \",\"]:\n pass\n elif c in [\"}\", \"\"]:\n # end of object, exit loop\n break\n\n else:\n stream.seek(-1)\n key, val = _parse_key_val(stream)\n if key in obj:\n # This is a gdb bug. We should never get repeated keys in a dict!\n # See https://sourceware.org/bugzilla/show_bug.cgi?id=22217\n # and https://github.com/cs01/pygdbmi/issues/19\n # Example:\n # thread-ids={thread-id=\"1\",thread-id=\"2\"}\n # Results in:\n # thread-ids: {{'thread-id': ['1', '2']}}\n # Rather than the lossy\n # thread-ids: {'thread-id': 2} # '1' got overwritten!\n if isinstance(obj[key], list):\n obj[key].append(val)\n else:\n obj[key] = [obj[key], val]\n else:\n obj[key] = val\n\n look_ahead_for_garbage = True\n c = stream.read(1)\n while look_ahead_for_garbage:\n if c in [\"}\", \",\", \"\"]:\n look_ahead_for_garbage = False\n else:\n # got some garbage text, skip it. for example:\n # name=\"gdb\"gargage # skip over 'garbage'\n # name=\"gdb\"\\n # skip over '\\n'\n logger.debug(\"skipping unexpected charcter: \" + c)\n c = stream.read(1)\n stream.seek(-1)\n\n logger.debug(\"parsed dict\")\n logger.debug(\"%s\", fmt_green(obj))\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses key and value combination", "response": "def _parse_key_val(stream):\n \"\"\"Parse key, value combination\n return (tuple):\n Parsed key (string)\n Parsed value (either a string, array, or dict)\n \"\"\"\n\n logger.debug(\"parsing key/val\")\n key = _parse_key(stream)\n val = _parse_val(stream)\n\n logger.debug(\"parsed key/val\")\n logger.debug(\"%s\", fmt_green(key))\n logger.debug(\"%s\", fmt_green(val))\n\n return key, val"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _parse_key(stream):\n logger.debug(\"parsing key\")\n\n key = stream.advance_past_chars([\"=\"])\n\n logger.debug(\"parsed key:\")\n logger.debug(\"%s\", fmt_green(key))\n return key", "response": "Parse a key value combination returning a Parsed key"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _parse_val(stream):\n\n logger.debug(\"parsing value\")\n\n while True:\n c = stream.read(1)\n\n if c == \"{\":\n # Start object\n val = _parse_dict(stream)\n break\n\n elif c == \"[\":\n # Start of an array\n val = _parse_array(stream)\n break\n\n elif c == '\"':\n # Start of a string\n val = stream.advance_past_string_with_gdb_escapes()\n break\n\n elif _DEBUG:\n raise ValueError(\"unexpected character: %s\" % c)\n\n else:\n print(\n 'pygdbmi warning: encountered unexpected character: \"%s\". Continuing.'\n % c\n )\n val = \"\" # this will be overwritten if there are more characters to be read\n\n logger.debug(\"parsed value:\")\n logger.debug(\"%s\", fmt_green(val))\n\n return val", "response": "Parse a value from string\n returns a parsed value"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nparses an array of items from a GDBMI stream.", "response": "def _parse_array(stream):\n \"\"\"Parse an array, stream should be passed the initial [\n returns:\n Parsed array\n \"\"\"\n\n logger.debug(\"parsing array\")\n arr = []\n while True:\n c = stream.read(1)\n\n if c in _GDB_MI_VALUE_START_CHARS:\n stream.seek(-1)\n val = _parse_val(stream)\n arr.append(val)\n elif c in _WHITESPACE:\n pass\n elif c == \",\":\n pass\n elif c == \"]\":\n # Stop when this array has finished. Note\n # that elements of this array can be also be arrays.\n break\n\n logger.debug(\"parsed array:\")\n logger.debug(\"%s\", fmt_green(arr))\n return arr"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef as_parameters(*parameters, variables=None):\n s = json.dumps(parameters)\n s = s[1:-1]\n if variables:\n for v in variables:\n if v in parameters:\n s = s.replace('\"' + v + '\"', v)\n return s", "response": "Dump python list as the parameter of javascript function\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef generate_local_url(self, js_name):\n host = self._settings['local_host'].format(**self._host_context).rstrip('/')\n return '{}/{}.js'.format(host, js_name)", "response": "Generate the local url for a js file."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a single item from an iterable with the given key.", "response": "def ifetch_single(iterable, key, default=EMPTY, getter=None):\n \"\"\"\n getter() g(item, key):pass\n \"\"\"\n\n def _getter(item):\n if getter:\n custom_getter = partial(getter, key=key)\n return custom_getter(item)\n else:\n try:\n attrgetter = operator.attrgetter(key)\n return attrgetter(item)\n except AttributeError:\n pass\n\n try:\n itemgetter = operator.itemgetter(key)\n return itemgetter(item)\n except KeyError:\n pass\n\n if default is not EMPTY:\n return default\n\n raise ValueError('Item %r has no attr or key for %r' % (item, key))\n\n return map(_getter, iterable)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_or_create_subscription(self):\n topic_path = self._get_topic_path()\n subscription_name = '{}-{}-{}-worker'.format(\n queue.PUBSUB_OBJECT_PREFIX, self.name, uuid4().hex)\n subscription_path = self.subscriber_client.subscription_path(\n self.project, subscription_name)\n\n try:\n self.subscriber_client.get_subscription(subscription_path)\n except google.cloud.exceptions.NotFound:\n logger.info(\"Creating worker subscription {}\".format(\n subscription_name))\n self.subscriber_client.create_subscription(\n subscription_path, topic_path)\n\n return subscription_path", "response": "In a broadcast queue workers have a unique subscription ensuring that every worker recieves a copy of every task."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef cleanup(self):\n if self.subscription:\n logger.info(\"Deleting worker subscription...\")\n self.subscriber_client.delete_subscription(self.subscription)", "response": "Deletes this worker s subscription."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_or_create_subscription(self):\n topic_path = self._get_topic_path()\n subscription_name = '{}-{}-shared'.format(\n PUBSUB_OBJECT_PREFIX, self.name)\n subscription_path = self.subscriber_client.subscription_path(\n self.project, subscription_name)\n\n try:\n self.subscriber_client.get_subscription(subscription_path)\n except google.cloud.exceptions.NotFound:\n logger.info(\"Creating shared subscription {}\".format(\n subscription_name))\n try:\n self.subscriber_client.create_subscription(\n subscription_path, topic=topic_path)\n except google.cloud.exceptions.Conflict:\n # Another worker created the subscription before us, ignore.\n pass\n\n return subscription_path", "response": "Get or create a shared subscription."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef enqueue_task(self, task):\n data = dumps(task)\n\n if self._async:\n self.publisher_client.publish(self.topic_path, data=data)\n logger.info('Task {} queued.'.format(task.id))\n else:\n unpickled_task = unpickle(data)\n logger.info(\n 'Executing task {} synchronously.'.format(unpickled_task.id)\n )\n with measure_time() as summary, self.queue_context():\n unpickled_task.execute(queue=self)\n summary(unpickled_task.summary())\n\n return TaskResult(task.id, self)", "response": "Enqueues a task asynchronously."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unpickle(pickled_string):\n try:\n obj = loads(pickled_string)\n except Exception as e:\n raise UnpickleError('Could not unpickle', pickled_string, e)\n return obj", "response": "Unpickles a string and raises a unified UnpickleError in case anything fails."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the result of the task.", "response": "def result(self, timeout=None):\n \"\"\"Gets the result of the task.\n\n Arguments:\n timeout: Maximum seconds to wait for a result before raising a\n TimeoutError. If set to None, this will wait forever. If the\n queue doesn't store results and timeout is None, this call will\n never return.\n \"\"\"\n start = time.time()\n while True:\n task = self.get_task()\n if not task or task.status not in (FINISHED, FAILED):\n if not timeout:\n continue\n elif time.time() - start < timeout:\n continue\n else:\n raise TimeoutError()\n\n if task.status == FAILED:\n raise task.result\n\n return task.result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nlaunching a Process, return his pid", "response": "def service_start(service=None, param=None):\n \"\"\"\n Launch a Process, return his pid\n \"\"\"\n if service is not None:\n to_run = [\"python\", service]\n if param is not None:\n to_run += param\n return subprocess.Popen(to_run)\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nupdate the running processes list and return the list of the running processes", "response": "def update_running_pids(old_procs):\n \"\"\"\n Update the list of the running process and return the list\n \"\"\"\n new_procs = []\n for proc in old_procs:\n if proc.poll() is None and check_pid(proc.pid):\n publisher.debug(str(proc.pid) + ' is alive')\n new_procs.append(proc)\n else:\n try:\n publisher.debug(str(proc.pid) + ' is gone')\n os.kill(proc.pid, signal.SIGKILL)\n except:\n # the process is just already gone\n pass\n return new_procs"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nrunning the split_processing function for the RIPE process_name and filenames.", "response": "def run_splitted_processing(max_simultaneous_processes, process_name,\n filenames):\n \"\"\"\n Run processes which push the routing dump of the RIPE in a redis\n database.\n The dump has been splitted in multiple files and each process run\n on one of this files.\n \"\"\"\n pids = []\n while len(filenames) > 0:\n while len(filenames) > 0 and len(pids) < max_simultaneous_processes:\n filename = filenames.pop()\n pids.append(service_start(service=process_name,\n param=['-f', filename, '-d',\n imported_day]))\n while len(pids) == max_simultaneous_processes:\n time.sleep(sleep_timer)\n pids = update_running_pids(pids)\n while len(pids) > 0:\n # Wait until all the processes are finished\n time.sleep(sleep_timer)\n pids = update_running_pids(pids)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsplits the file into several files.", "response": "def fsplit(file_to_split):\n \"\"\"\n Split the file and return the list of filenames.\n \"\"\"\n dirname = file_to_split + '_splitted'\n if not os.path.exists(dirname):\n os.mkdir(dirname)\n part_file_size = os.path.getsize(file_to_split) / number_of_files + 1\n splitted_files = []\n with open(file_to_split, \"r\") as f:\n number = 0\n actual = 0\n while 1:\n prec = actual\n # Jump of \"size\" from the current place in the file\n f.seek(part_file_size, os.SEEK_CUR)\n\n # find the next separator or EOF\n s = f.readline()\n if len(s) == 0:\n s = f.readline()\n while len(s) != 0 and s != separator:\n s = f.readline()\n\n # Get the current place\n actual = f.tell()\n new_file = os.path.join(dirname, str(number))\n\n # Create the new file\n with open(file_to_split, \"r\") as temp:\n temp.seek(prec)\n # Get the text we want to put in the new file\n copy = temp.read(actual - prec)\n # Write the new file\n open(new_file, 'w').write(copy)\n splitted_files.append(new_file)\n number += 1\n\n # End of file\n if len(s) == 0:\n break\n return splitted_files"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __entry_point():\n ip = request.remote_addr\n ua = request.headers.get('User-Agent', 'Empty User-Agent')\n method = request.json.get('method')\n if method is None:\n __query_logging(ip, ua, method, level='warning')\n return json.dumps({'error': 'No method provided.'})\n if method not in authorized_methods:\n # unauthorized query\n __query_logging(ip, ua, method, level='warning')\n return json.dumps({'error': 'Unauthorized method.'})\n fct = globals().get(method)\n if fct is None:\n # unknown method, the method is authorized, but does not exists...\n __query_logging(ip, ua, method, level='warning')\n return json.dumps({'error': 'Unknown method.'})\n if request.json.get('ip') is None:\n __query_logging(ip, ua, method, level='warning')\n return json.dumps({'error': 'No IP provided, not going to work.'})\n try:\n result = fct(request.json)\n __query_logging(ip, ua, method, request.json.get('ip'),\n request.json.get('announce_date'), request.json.get('days_limit'))\n return result\n except Exception:\n __query_logging(ip, ua, method, request.json.get('ip'), level='error')\n return json.dumps({'error': 'Something went wrong.'})", "response": "This function is called by the entry point of the API. It is called by the API when a request is made on the URL."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef asn(self, ip, announce_date=None):\n assignations, announce_date, _ = self.run(ip, announce_date)\n return next((assign for assign in assignations if assign is not None), None), announce_date", "response": "Get the ASN. nacute entry for an IP."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef date_asn_block(self, ip, announce_date=None):\n assignations, announce_date, keys = self.run(ip, announce_date)\n pos = next((i for i, j in enumerate(assignations) if j is not None), None)\n if pos is not None:\n block = keys[pos]\n if block != '0.0.0.0/0':\n return announce_date, assignations[pos], block\n return None", "response": "Returns the ASN and the IP Block announcing the IP at a specific date."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef history(self, ip, days_limit=None):\n all_dates = sorted(self.routing_db.smembers('imported_dates'), reverse=True)\n if days_limit is not None:\n all_dates = all_dates[:days_limit]\n return [self.date_asn_block(ip, date) for date in all_dates]", "response": "Get the full history of an IP. It takes time."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef aggregate_history(self, ip, days_limit=None):\n first_date = None\n last_date = None\n prec_asn = None\n prec_block = None\n for entry in self.history(ip, days_limit):\n if entry is None:\n continue\n date, asn, block = entry\n if first_date is None:\n last_date = date\n first_date = date\n prec_asn = asn\n prec_block = block\n elif prec_asn == asn and prec_block == block:\n first_date = date\n else:\n yield first_date, last_date, prec_asn, prec_block\n last_date = date\n first_date = date\n prec_asn = asn\n prec_block = block\n if first_date is not None:\n yield first_date, last_date, prec_asn, prec_block", "response": "Get the full history of an IP and aggregate it for one day per day."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndownloads a URL to a file in the bview directory.", "response": "def downloadURL(url, filename):\n \"\"\"\n Inconditianilly download the URL in a temporary directory.\n When finished, the file is moved in the real directory.\n Like this an other process will not attempt to extract an inclomplete file.\n \"\"\"\n path_temp_bviewfile = os.path.join(c.raw_data, c.bview_dir, 'tmp', filename)\n path_bviewfile = os.path.join(c.raw_data, c.bview_dir, filename)\n try:\n f = urlopen(url)\n except:\n return False\n if f.getcode() != 200:\n publisher.warning('{} unavailable, code: {}'.format(url, f.getcode()))\n return False\n try:\n with open(path_temp_bviewfile, 'w') as outfile:\n outfile.write(f.read())\n os.rename(path_temp_bviewfile, path_bviewfile)\n except:\n os.remove(path_temp_bviewfile)\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nverifying that the file has not already been downloaded.", "response": "def already_downloaded(filename):\n \"\"\"\n Verify that the file has not already been downloaded.\n \"\"\"\n cur_file = os.path.join(c.bview_dir, filename)\n old_file = os.path.join(c.bview_dir, 'old', filename)\n if not os.path.exists(cur_file) and not os.path.exists(old_file):\n return False\n return True"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nbuilds interval of urls to download.", "response": "def to_download():\n \"\"\"\n Build interval of urls to download.\n We always get the first file of the next day.\n Ex: 2013-01-01 => 2013-01-02.0000\n \"\"\"\n first_day = parse(interval_first)\n last_day = parse(interval_last)\n format_change = parse('2010-06-14')\n one_day = datetime.timedelta(1)\n cur_day = first_day\n url_list = []\n while cur_day < last_day:\n fname = filename.format(day=cur_day.strftime(\"%Y%m%d\"))\n if cur_day > format_change:\n cur_day += one_day\n url = base_url.format(year_month=cur_day.strftime(\"%Y.%m\"),\n file_day=cur_day.strftime(\"%Y%m%d\"))\n else:\n url = base_url_old.format(year_month=cur_day.strftime(\"%Y.%m\"),\n file_day=cur_day.strftime(\"%Y%m%d\"))\n cur_day += one_day\n url_list.append((fname, url))\n return sorted(url_list, key=lambda tup: tup[0], reverse=True)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_page_url(page_num, current_app, url_view_name, url_extra_args, url_extra_kwargs, url_param_name, url_get_params, url_anchor):\n if url_view_name is not None:\n # Add page param to the kwargs list. Overrides any previously set parameter of the same name.\n url_extra_kwargs[url_param_name] = page_num\n\n try:\n url = reverse(url_view_name, args=url_extra_args, kwargs=url_extra_kwargs, current_app=current_app)\n except NoReverseMatch as e: # Attempt to load view from application root, allowing the use of non-namespaced view names if your view is defined in the root application\n if settings.SETTINGS_MODULE:\n\n if django.VERSION < (1, 9, 0):\n separator = '.'\n else:\n separator = ':' # Namespace separator changed to colon after 1.8\n\n project_name = settings.SETTINGS_MODULE.split('.')[0]\n try:\n url = reverse(project_name + separator + url_view_name, args=url_extra_args, kwargs=url_extra_kwargs, current_app=current_app)\n except NoReverseMatch:\n raise e # Raise the original exception so the error message doesn't confusingly include something the Developer didn't add to the view name themselves\n else:\n raise e # We can't determine the project name so just re-throw the exception\n\n else:\n url = ''\n url_get_params = url_get_params or QueryDict(url)\n url_get_params = url_get_params.copy()\n url_get_params[url_param_name] = str(page_num)\n\n if len(url_get_params) > 0:\n if not isinstance(url_get_params, QueryDict):\n tmp = QueryDict(mutable=True)\n tmp.update(url_get_params)\n url_get_params = tmp\n url += '?' + url_get_params.urlencode()\n\n if (url_anchor is not None):\n url += '#' + url_anchor\n\n return url", "response": "Helper function to return a valid URL string given the template tag parameters and the parameters of the page."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef bootstrap_paginate(parser, token):\n bits = token.split_contents()\n if len(bits) < 2:\n raise TemplateSyntaxError(\"'%s' takes at least one argument\"\n \" (Page object reference)\" % bits[0])\n page = parser.compile_filter(bits[1])\n kwargs = {}\n bits = bits[2:]\n\n kwarg_re = re.compile(r'(\\w+)=(.+)')\n\n if len(bits):\n for bit in bits:\n match = kwarg_re.match(bit)\n if not match:\n raise TemplateSyntaxError(\"Malformed arguments to bootstrap_pagination paginate tag\")\n name, value = match.groups()\n kwargs[name] = parser.compile_filter(value)\n\n return BootstrapPaginationNode(page, kwargs)", "response": "Renders a page object as a Twitter Bootstrap styled pagination bar."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconfigures all Jenkins devel jobs.", "response": "def configure_devel_jobs(\n config_url, rosdistro_name, source_build_name, groovy_script=None,\n dry_run=False, whitelist_repository_names=None):\n \"\"\"\n Configure all Jenkins devel jobs.\n\n L{configure_release_job} will be invoked for source repository and target\n which matches the build file criteria.\n \"\"\"\n config = get_config_index(config_url)\n build_files = get_source_build_files(config, rosdistro_name)\n build_file = build_files[source_build_name]\n\n index = get_index(config.rosdistro_index_url)\n\n dist_cache = None\n if build_file.notify_maintainers:\n dist_cache = get_distribution_cache(index, rosdistro_name)\n\n # get targets\n targets = []\n for os_name in build_file.targets.keys():\n for os_code_name in build_file.targets[os_name].keys():\n for arch in build_file.targets[os_name][os_code_name]:\n targets.append((os_name, os_code_name, arch))\n print('The build file contains the following targets:')\n for os_name, os_code_name, arch in targets:\n print(' -', os_name, os_code_name, arch)\n\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n print('No distribution file matches the build file')\n return\n\n devel_view_name = get_devel_view_name(\n rosdistro_name, source_build_name, pull_request=False)\n pull_request_view_name = get_devel_view_name(\n rosdistro_name, source_build_name, pull_request=True)\n\n # all further configuration will be handled by either the Jenkins API\n # or by a generated groovy script\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url) if groovy_script is None else False\n\n view_configs = {}\n views = {}\n if build_file.test_commits_force is not False:\n views[devel_view_name] = configure_devel_view(\n jenkins, devel_view_name, dry_run=dry_run)\n if build_file.test_pull_requests_force is not False:\n views[pull_request_view_name] = configure_devel_view(\n jenkins, pull_request_view_name, dry_run=dry_run)\n if not jenkins:\n view_configs.update(views)\n groovy_data = {\n 'dry_run': dry_run,\n 'expected_num_views': len(view_configs),\n }\n\n repo_names = dist_file.repositories.keys()\n filtered_repo_names = build_file.filter_repositories(repo_names)\n\n devel_job_names = []\n pull_request_job_names = []\n job_configs = OrderedDict()\n for repo_name in sorted(repo_names):\n if whitelist_repository_names:\n if repo_name not in whitelist_repository_names:\n print(\n \"Skipping repository '%s' not in explicitly passed list\" %\n repo_name, file=sys.stderr)\n continue\n\n is_disabled = repo_name not in filtered_repo_names\n if is_disabled and build_file.skip_ignored_repositories:\n print(\"Skipping ignored repository '%s'\" % repo_name,\n file=sys.stderr)\n continue\n\n repo = dist_file.repositories[repo_name]\n if not repo.source_repository:\n print(\"Skipping repository '%s': no source section\" % repo_name)\n continue\n if not repo.source_repository.version:\n print(\"Skipping repository '%s': no source version\" % repo_name)\n continue\n\n job_types = []\n # check for testing commits\n if build_file.test_commits_force is False:\n print((\"Skipping repository '%s': 'test_commits' is forced to \" +\n \"false in the build file\") % repo_name)\n elif repo.source_repository.test_commits is False:\n print((\"Skipping repository '%s': 'test_commits' of the \" +\n \"repository set to false\") % repo_name)\n elif repo.source_repository.test_commits is None and \\\n not build_file.test_commits_default:\n print((\"Skipping repository '%s': 'test_commits' defaults to \" +\n \"false in the build file\") % repo_name)\n else:\n job_types.append('commit')\n\n if not is_disabled:\n # check for testing pull requests\n if build_file.test_pull_requests_force is False:\n # print((\"Skipping repository '%s': 'test_pull_requests' \" +\n # \"is forced to false in the build file\") % repo_name)\n pass\n elif repo.source_repository.test_pull_requests is False:\n # print((\"Skipping repository '%s': 'test_pull_requests' of \" +\n # \"the repository set to false\") % repo_name)\n pass\n elif repo.source_repository.test_pull_requests is None and \\\n not build_file.test_pull_requests_default:\n # print((\"Skipping repository '%s': 'test_pull_requests' \" +\n # \"defaults to false in the build file\") % repo_name)\n pass\n else:\n print(\"Pull request job for repository '%s'\" % repo_name)\n job_types.append('pull_request')\n\n for job_type in job_types:\n pull_request = job_type == 'pull_request'\n for os_name, os_code_name, arch in targets:\n try:\n job_name, job_config = configure_devel_job(\n config_url, rosdistro_name, source_build_name,\n repo_name, os_name, os_code_name, arch, pull_request,\n config=config, build_file=build_file,\n index=index, dist_file=dist_file,\n dist_cache=dist_cache, jenkins=jenkins, views=views,\n is_disabled=is_disabled,\n groovy_script=groovy_script,\n dry_run=dry_run)\n if not pull_request:\n devel_job_names.append(job_name)\n else:\n pull_request_job_names.append(job_name)\n if groovy_script is not None:\n print(\"Configuration for job '%s'\" % job_name)\n job_configs[job_name] = job_config\n except JobValidationError as e:\n print(e.message, file=sys.stderr)\n\n groovy_data['expected_num_jobs'] = len(job_configs)\n groovy_data['job_prefixes_and_names'] = {}\n\n devel_job_prefix = '%s__' % devel_view_name\n pull_request_job_prefix = '%s__' % pull_request_view_name\n if not whitelist_repository_names:\n groovy_data['job_prefixes_and_names']['devel'] = \\\n (devel_job_prefix, devel_job_names)\n groovy_data['job_prefixes_and_names']['pull_request'] = \\\n (pull_request_job_prefix, pull_request_job_names)\n\n if groovy_script is None:\n # delete obsolete jobs in these views\n from ros_buildfarm.jenkins import remove_jobs\n print('Removing obsolete devel jobs')\n remove_jobs(\n jenkins, devel_job_prefix, devel_job_names, dry_run=dry_run)\n print('Removing obsolete pull request jobs')\n remove_jobs(\n jenkins, pull_request_job_prefix, pull_request_job_names,\n dry_run=dry_run)\n if groovy_script is not None:\n print(\n \"Writing groovy script '%s' to reconfigure %d views and %d jobs\" %\n (groovy_script, len(view_configs), len(job_configs)))\n content = expand_template(\n 'snippet/reconfigure_jobs.groovy.em', groovy_data)\n write_groovy_script_and_configs(\n groovy_script, content, job_configs, view_configs=view_configs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconfigures a single Jenkins devel job.", "response": "def configure_devel_job(\n config_url, rosdistro_name, source_build_name,\n repo_name, os_name, os_code_name, arch,\n pull_request=False,\n config=None, build_file=None,\n index=None, dist_file=None, dist_cache=None,\n jenkins=None, views=None,\n is_disabled=False,\n groovy_script=None,\n source_repository=None,\n build_targets=None,\n dry_run=False):\n \"\"\"\n Configure a single Jenkins devel job.\n\n This includes the following steps:\n - clone the source repository to use\n - clone the ros_buildfarm repository\n - write the distribution repository keys into files\n - invoke the release/run_devel_job.py script\n \"\"\"\n if config is None:\n config = get_config_index(config_url)\n if build_file is None:\n build_files = get_source_build_files(config, rosdistro_name)\n build_file = build_files[source_build_name]\n # Overwrite build_file.targets if build_targets is specified\n if build_targets is not None:\n build_file.targets = build_targets\n\n if index is None:\n index = get_index(config.rosdistro_index_url)\n if dist_file is None:\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n raise JobValidationError(\n 'No distribution file matches the build file')\n\n repo_names = dist_file.repositories.keys()\n\n if repo_name is not None:\n if repo_name not in repo_names:\n raise JobValidationError(\n \"Invalid repository name '%s' \" % repo_name +\n 'choose one of the following: %s' %\n ', '.join(sorted(repo_names)))\n\n repo = dist_file.repositories[repo_name]\n if not repo.source_repository:\n raise JobValidationError(\n \"Repository '%s' has no source section\" % repo_name)\n if not repo.source_repository.version:\n raise JobValidationError(\n \"Repository '%s' has no source version\" % repo_name)\n source_repository = repo.source_repository\n\n if os_name not in build_file.targets.keys():\n raise JobValidationError(\n \"Invalid OS name '%s' \" % os_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets.keys())))\n if os_code_name not in build_file.targets[os_name].keys():\n raise JobValidationError(\n \"Invalid OS code name '%s' \" % os_code_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets[os_name].keys())))\n if arch not in build_file.targets[os_name][os_code_name]:\n raise JobValidationError(\n \"Invalid architecture '%s' \" % arch +\n 'choose one of the following: %s' % ', '.join(sorted(\n build_file.targets[os_name][os_code_name])))\n\n if dist_cache is None and build_file.notify_maintainers:\n dist_cache = get_distribution_cache(index, rosdistro_name)\n if jenkins is None:\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url)\n if views is None:\n view_name = get_devel_view_name(\n rosdistro_name, source_build_name, pull_request=pull_request)\n configure_devel_view(jenkins, view_name, dry_run=dry_run)\n\n job_name = get_devel_job_name(\n rosdistro_name, source_build_name,\n repo_name, os_name, os_code_name, arch, pull_request)\n\n job_config = _get_devel_job_config(\n index, config, rosdistro_name, source_build_name,\n build_file, os_name, os_code_name, arch, source_repository,\n repo_name, pull_request, job_name, dist_cache=dist_cache,\n is_disabled=is_disabled)\n # jenkinsapi.jenkins.Jenkins evaluates to false if job count is zero\n if isinstance(jenkins, object) and jenkins is not False:\n from ros_buildfarm.jenkins import configure_job\n configure_job(jenkins, job_name, job_config, dry_run=dry_run)\n\n return job_name, job_config"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef configure_release_jobs(\n config_url, rosdistro_name, release_build_name, groovy_script=None,\n dry_run=False, whitelist_package_names=None):\n \"\"\"\n Configure all Jenkins release jobs.\n\n L{configure_release_job} will be invoked for every released package and\n target which matches the build file criteria.\n\n Additionally a job to import Debian packages into the Debian repository is\n created.\n \"\"\"\n config = get_config_index(config_url)\n build_files = get_release_build_files(config, rosdistro_name)\n build_file = build_files[release_build_name]\n\n index = get_index(config.rosdistro_index_url)\n\n # get targets\n platforms = []\n for os_name in build_file.targets.keys():\n for os_code_name in build_file.targets[os_name].keys():\n platforms.append((os_name, os_code_name))\n print('The build file contains the following targets:')\n for os_name, os_code_name in platforms:\n print(' - %s %s: %s' % (os_name, os_code_name, ', '.join(\n build_file.targets[os_name][os_code_name])))\n\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n print('No distribution file matches the build file')\n return\n\n pkg_names = dist_file.release_packages.keys()\n filtered_pkg_names = build_file.filter_packages(pkg_names)\n explicitly_ignored_pkg_names = set(pkg_names) - set(filtered_pkg_names)\n if explicitly_ignored_pkg_names:\n print(('The following packages are being %s because of ' +\n 'white-/blacklisting:') %\n ('ignored' if build_file.skip_ignored_packages else 'disabled'))\n for pkg_name in sorted(explicitly_ignored_pkg_names):\n print(' -', pkg_name)\n\n dist_cache = get_distribution_cache(index, rosdistro_name)\n\n if explicitly_ignored_pkg_names:\n # get direct dependencies from distro cache for each package\n direct_dependencies = {}\n for pkg_name in pkg_names:\n direct_dependencies[pkg_name] = _get_direct_dependencies(\n pkg_name, dist_cache, pkg_names) or set([])\n\n # find recursive downstream deps for all explicitly ignored packages\n ignored_pkg_names = set(explicitly_ignored_pkg_names)\n while True:\n implicitly_ignored_pkg_names = _get_downstream_package_names(\n ignored_pkg_names, direct_dependencies)\n if implicitly_ignored_pkg_names - ignored_pkg_names:\n ignored_pkg_names |= implicitly_ignored_pkg_names\n continue\n break\n implicitly_ignored_pkg_names = \\\n ignored_pkg_names - explicitly_ignored_pkg_names\n\n if implicitly_ignored_pkg_names:\n print(('The following packages are being %s because their ' +\n 'dependencies are being ignored:') % ('ignored'\n if build_file.skip_ignored_packages else 'disabled'))\n for pkg_name in sorted(implicitly_ignored_pkg_names):\n print(' -', pkg_name)\n filtered_pkg_names = \\\n set(filtered_pkg_names) - implicitly_ignored_pkg_names\n\n # all further configuration will be handled by either the Jenkins API\n # or by a generated groovy script\n jenkins = False\n if groovy_script is None:\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url)\n\n all_view_configs = {}\n all_job_configs = OrderedDict()\n\n job_name, job_config = configure_import_package_job(\n config_url, rosdistro_name, release_build_name,\n config=config, build_file=build_file, jenkins=jenkins, dry_run=dry_run)\n if not jenkins:\n all_job_configs[job_name] = job_config\n\n job_name, job_config = configure_sync_packages_to_main_job(\n config_url, rosdistro_name, release_build_name,\n config=config, build_file=build_file, jenkins=jenkins, dry_run=dry_run)\n if not jenkins:\n all_job_configs[job_name] = job_config\n\n for os_name, os_code_name in platforms:\n for arch in sorted(build_file.targets[os_name][os_code_name]):\n job_name, job_config = configure_sync_packages_to_testing_job(\n config_url, rosdistro_name, release_build_name,\n os_code_name, arch,\n config=config, build_file=build_file, jenkins=jenkins,\n dry_run=dry_run)\n if not jenkins:\n all_job_configs[job_name] = job_config\n\n targets = []\n for os_name, os_code_name in platforms:\n targets.append((os_name, os_code_name, 'source'))\n for arch in build_file.targets[os_name][os_code_name]:\n targets.append((os_name, os_code_name, arch))\n views = configure_release_views(\n jenkins, rosdistro_name, release_build_name, targets,\n dry_run=dry_run)\n if not jenkins:\n all_view_configs.update(views)\n groovy_data = {\n 'dry_run': dry_run,\n 'expected_num_views': len(views),\n }\n\n # binary jobs must be generated in topological order\n from catkin_pkg.package import parse_package_string\n from ros_buildfarm.common import topological_order_packages\n pkgs = {}\n for pkg_name in pkg_names:\n if pkg_name not in dist_cache.release_package_xmls:\n print(\"Skipping package '%s': no released package.xml in cache\" %\n (pkg_name), file=sys.stderr)\n continue\n pkg_xml = dist_cache.release_package_xmls[pkg_name]\n pkg = parse_package_string(pkg_xml)\n pkgs[pkg_name] = pkg\n ordered_pkg_tuples = topological_order_packages(pkgs)\n\n other_build_files = [v for k, v in build_files.items() if k != release_build_name]\n\n all_source_job_names = []\n all_binary_job_names = []\n for pkg_name in [p.name for _, p in ordered_pkg_tuples]:\n if whitelist_package_names:\n if pkg_name not in whitelist_package_names:\n print(\"Skipping package '%s' not in the explicitly passed list\" %\n pkg_name, file=sys.stderr)\n continue\n\n pkg = dist_file.release_packages[pkg_name]\n repo_name = pkg.repository_name\n repo = dist_file.repositories[repo_name]\n is_disabled = pkg_name not in filtered_pkg_names\n if is_disabled and build_file.skip_ignored_packages:\n print(\"Skipping ignored package '%s' in repository '%s'\" %\n (pkg_name, repo_name), file=sys.stderr)\n continue\n if not repo.release_repository:\n print((\"Skipping package '%s' in repository '%s': no release \" +\n \"section\") % (pkg_name, repo_name), file=sys.stderr)\n continue\n if not repo.release_repository.version:\n print((\"Skipping package '%s' in repository '%s': no release \" +\n \"version\") % (pkg_name, repo_name), file=sys.stderr)\n continue\n\n for os_name, os_code_name in platforms:\n other_build_files_same_platform = []\n for other_build_file in other_build_files:\n if os_name not in other_build_file.targets:\n continue\n if os_code_name not in other_build_file.targets[os_name]:\n continue\n other_build_files_same_platform.append(other_build_file)\n\n try:\n source_job_names, binary_job_names, job_configs = \\\n configure_release_job(\n config_url, rosdistro_name, release_build_name,\n pkg_name, os_name, os_code_name,\n config=config, build_file=build_file,\n index=index, dist_file=dist_file,\n dist_cache=dist_cache,\n jenkins=jenkins, views=views,\n generate_import_package_job=False,\n generate_sync_packages_jobs=False,\n is_disabled=is_disabled,\n other_build_files_same_platform=other_build_files_same_platform,\n groovy_script=groovy_script,\n dry_run=dry_run)\n all_source_job_names += source_job_names\n all_binary_job_names += binary_job_names\n if groovy_script is not None:\n print('Configuration for jobs: ' +\n ', '.join(source_job_names + binary_job_names))\n for source_job_name in source_job_names:\n all_job_configs[source_job_name] = job_configs[source_job_name]\n for binary_job_name in binary_job_names:\n all_job_configs[binary_job_name] = job_configs[binary_job_name]\n except JobValidationError as e:\n print(e.message, file=sys.stderr)\n\n groovy_data['expected_num_jobs'] = len(all_job_configs)\n groovy_data['job_prefixes_and_names'] = {}\n\n # with an explicit list of packages we don't delete obsolete jobs\n if not whitelist_package_names:\n # delete obsolete binary jobs\n for os_name, os_code_name in platforms:\n for arch in build_file.targets[os_name][os_code_name]:\n binary_view = get_release_binary_view_name(\n rosdistro_name, release_build_name,\n os_name, os_code_name, arch)\n binary_job_prefix = '%s__' % binary_view\n\n excluded_job_names = set([\n j for j in all_binary_job_names\n if j.startswith(binary_job_prefix)])\n if groovy_script is None:\n print(\"Removing obsolete binary jobs with prefix '%s'\" %\n binary_job_prefix)\n from ros_buildfarm.jenkins import remove_jobs\n remove_jobs(\n jenkins, binary_job_prefix, excluded_job_names,\n dry_run=dry_run)\n else:\n binary_key = 'binary_%s_%s_%s' % \\\n (os_name, os_code_name, arch)\n groovy_data['job_prefixes_and_names'][binary_key] = \\\n (binary_job_prefix, excluded_job_names)\n\n # delete obsolete source jobs\n # requires knowledge about all other release build files\n for os_name, os_code_name in platforms:\n other_source_job_names = []\n # get source job names for all other release build files\n for other_release_build_name in [\n k for k in build_files.keys() if k != release_build_name]:\n other_build_file = build_files[other_release_build_name]\n other_dist_file = get_distribution_file(\n index, rosdistro_name, other_build_file)\n if not other_dist_file:\n continue\n\n if os_name not in other_build_file.targets or \\\n os_code_name not in other_build_file.targets[os_name]:\n continue\n\n if other_build_file.skip_ignored_packages:\n filtered_pkg_names = other_build_file.filter_packages(\n pkg_names)\n else:\n filtered_pkg_names = pkg_names\n for pkg_name in sorted(filtered_pkg_names):\n pkg = other_dist_file.release_packages[pkg_name]\n repo_name = pkg.repository_name\n repo = other_dist_file.repositories[repo_name]\n if not repo.release_repository:\n continue\n if not repo.release_repository.version:\n continue\n\n other_job_name = get_sourcedeb_job_name(\n rosdistro_name, other_release_build_name,\n pkg_name, os_name, os_code_name)\n other_source_job_names.append(other_job_name)\n\n source_view_prefix = get_release_source_view_name(\n rosdistro_name, os_name, os_code_name)\n source_job_prefix = '%s__' % source_view_prefix\n excluded_job_names = set([\n j for j in (all_source_job_names + other_source_job_names)\n if j.startswith(source_job_prefix)])\n if groovy_script is None:\n print(\"Removing obsolete source jobs with prefix '%s'\" %\n source_job_prefix)\n from ros_buildfarm.jenkins import remove_jobs\n remove_jobs(\n jenkins, source_job_prefix, excluded_job_names,\n dry_run=dry_run)\n else:\n source_key = 'source_%s_%s' % (os_name, os_code_name)\n groovy_data['job_prefixes_and_names'][source_key] = (\n source_job_prefix, excluded_job_names)\n\n if groovy_script is not None:\n print(\n \"Writing groovy script '%s' to reconfigure %d views and %d jobs\" %\n (groovy_script, len(all_view_configs), len(all_job_configs)))\n content = expand_template(\n 'snippet/reconfigure_jobs.groovy.em', groovy_data)\n write_groovy_script_and_configs(\n groovy_script, content, all_job_configs,\n view_configs=all_view_configs)", "response": "Configure all Jenkins release jobs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef configure_release_job(\n config_url, rosdistro_name, release_build_name,\n pkg_name, os_name, os_code_name,\n config=None, build_file=None,\n index=None, dist_file=None, dist_cache=None,\n jenkins=None, views=None,\n generate_import_package_job=True,\n generate_sync_packages_jobs=True,\n is_disabled=False, other_build_files_same_platform=None,\n groovy_script=None,\n filter_arches=None,\n dry_run=False):\n \"\"\"\n Configure a Jenkins release job.\n\n The following jobs are created for each package:\n - M source jobs, one for each OS node name\n - M * N binary jobs, one for each combination of OS code name and arch\n \"\"\"\n if config is None:\n config = get_config_index(config_url)\n if build_file is None:\n build_files = get_release_build_files(config, rosdistro_name)\n build_file = build_files[release_build_name]\n\n if index is None:\n index = get_index(config.rosdistro_index_url)\n if dist_file is None:\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n raise JobValidationError(\n 'No distribution file matches the build file')\n\n pkg_names = dist_file.release_packages.keys()\n\n if pkg_name not in pkg_names:\n raise JobValidationError(\n \"Invalid package name '%s' \" % pkg_name +\n 'choose one of the following: ' + ', '.join(sorted(pkg_names)))\n\n pkg = dist_file.release_packages[pkg_name]\n repo_name = pkg.repository_name\n repo = dist_file.repositories[repo_name]\n\n if not repo.release_repository:\n raise JobValidationError(\n \"Repository '%s' has no release section\" % repo_name)\n\n if not repo.release_repository.version:\n raise JobValidationError(\n \"Repository '%s' has no release version\" % repo_name)\n\n if os_name not in build_file.targets.keys():\n raise JobValidationError(\n \"Invalid OS name '%s' \" % os_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets.keys())))\n\n if os_code_name not in build_file.targets[os_name].keys():\n raise JobValidationError(\n \"Invalid OS code name '%s' \" % os_code_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets[os_name].keys())))\n\n if dist_cache is None and \\\n (build_file.notify_maintainers or\n build_file.abi_incompatibility_assumed):\n dist_cache = get_distribution_cache(index, rosdistro_name)\n if jenkins is None:\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url)\n if views is None:\n targets = []\n targets.append((os_name, os_code_name, 'source'))\n for arch in build_file.targets[os_name][os_code_name]:\n targets.append((os_name, os_code_name, arch))\n configure_release_views(\n jenkins, rosdistro_name, release_build_name, targets,\n dry_run=dry_run)\n\n if generate_import_package_job:\n configure_import_package_job(\n config_url, rosdistro_name, release_build_name,\n config=config, build_file=build_file, jenkins=jenkins,\n dry_run=dry_run)\n\n if generate_sync_packages_jobs:\n configure_sync_packages_to_main_job(\n config_url, rosdistro_name, release_build_name,\n config=config, build_file=build_file, jenkins=jenkins,\n dry_run=dry_run)\n for arch in build_file.targets[os_name][os_code_name]:\n configure_sync_packages_to_testing_job(\n config_url, rosdistro_name, release_build_name,\n os_code_name, arch,\n config=config, build_file=build_file, jenkins=jenkins,\n dry_run=dry_run)\n\n source_job_names = []\n binary_job_names = []\n job_configs = {}\n\n # sourcedeb job\n # since sourcedeb jobs are potentially being shared across multiple build\n # files the configuration has to take all of them into account in order to\n # generate a job which all build files agree on\n source_job_name = get_sourcedeb_job_name(\n rosdistro_name, release_build_name,\n pkg_name, os_name, os_code_name)\n\n # while the package is disabled in the current build file\n # it might be used by sibling build files\n is_source_disabled = is_disabled\n if is_source_disabled and other_build_files_same_platform:\n # check if sourcedeb job is used by any other build file with the same platform\n for other_build_file in other_build_files_same_platform:\n if other_build_file.filter_packages([pkg_name]):\n is_source_disabled = False\n break\n\n job_config = _get_sourcedeb_job_config(\n config_url, rosdistro_name, release_build_name,\n config, build_file, os_name, os_code_name,\n pkg_name, repo_name, repo.release_repository, dist_cache=dist_cache,\n is_disabled=is_source_disabled,\n other_build_files_same_platform=other_build_files_same_platform)\n # jenkinsapi.jenkins.Jenkins evaluates to false if job count is zero\n if isinstance(jenkins, object) and jenkins is not False:\n from ros_buildfarm.jenkins import configure_job\n configure_job(jenkins, source_job_name, job_config, dry_run=dry_run)\n source_job_names.append(source_job_name)\n job_configs[source_job_name] = job_config\n\n dependency_names = []\n if build_file.abi_incompatibility_assumed:\n dependency_names = _get_direct_dependencies(\n pkg_name, dist_cache, pkg_names)\n # if dependencies are not yet available in rosdistro cache\n # skip binary jobs\n if dependency_names is None:\n print((\"Skipping binary jobs for package '%s' because it is not \" +\n \"yet in the rosdistro cache\") % pkg_name, file=sys.stderr)\n return source_job_names, binary_job_names, job_configs\n\n # binarydeb jobs\n for arch in build_file.targets[os_name][os_code_name]:\n if filter_arches and arch not in filter_arches:\n continue\n\n job_name = get_binarydeb_job_name(\n rosdistro_name, release_build_name,\n pkg_name, os_name, os_code_name, arch)\n\n upstream_job_names = [source_job_name] + [\n get_binarydeb_job_name(\n rosdistro_name, release_build_name,\n dependency_name, os_name, os_code_name, arch)\n for dependency_name in dependency_names]\n\n job_config = _get_binarydeb_job_config(\n config_url, rosdistro_name, release_build_name,\n config, build_file, os_name, os_code_name, arch,\n pkg_name, repo_name, repo.release_repository,\n dist_cache=dist_cache, upstream_job_names=upstream_job_names,\n is_disabled=is_disabled)\n # jenkinsapi.jenkins.Jenkins evaluates to false if job count is zero\n if isinstance(jenkins, object) and jenkins is not False:\n configure_job(jenkins, job_name, job_config, dry_run=dry_run)\n binary_job_names.append(job_name)\n job_configs[job_name] = job_config\n\n return source_job_names, binary_job_names, job_configs", "response": "Configure a Jenkins release job."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconfigures all Jenkins doc jobs for the given doc build.", "response": "def configure_doc_jobs(\n config_url, rosdistro_name, doc_build_name, groovy_script=None,\n dry_run=False, whitelist_repository_names=None):\n \"\"\"\n Configure all Jenkins doc jobs.\n\n L{configure_doc_job} will be invoked for doc repository and target\n which matches the build file criteria.\n \"\"\"\n config = get_config_index(config_url)\n build_files = get_doc_build_files(config, rosdistro_name)\n build_file = build_files[doc_build_name]\n\n index = get_index(config.rosdistro_index_url)\n\n dist_cache = None\n if build_file.notify_maintainers:\n dist_cache = get_distribution_cache(index, rosdistro_name)\n\n # get targets\n targets = []\n for os_name in build_file.targets.keys():\n for os_code_name in build_file.targets[os_name].keys():\n for arch in build_file.targets[os_name][os_code_name]:\n targets.append((os_name, os_code_name, arch))\n print('The build file contains the following targets:')\n for os_name, os_code_name, arch in targets:\n print(' -', os_name, os_code_name, arch)\n\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n print('No distribution file matches the build file')\n return\n\n doc_view_name = get_doc_view_name(rosdistro_name, doc_build_name)\n\n # all further configuration will be handled by either the Jenkins API\n # or by a generated groovy script\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url) if groovy_script is None else False\n\n view_configs = {}\n views = {}\n views[doc_view_name] = configure_doc_view(\n jenkins, doc_view_name, dry_run=dry_run)\n if not jenkins:\n view_configs.update(views)\n groovy_data = {\n 'dry_run': dry_run,\n 'expected_num_views': len(view_configs),\n }\n\n repo_names = dist_file.repositories.keys()\n filtered_repo_names = build_file.filter_repositories(repo_names)\n\n job_names = []\n job_configs = OrderedDict()\n for repo_name in sorted(repo_names):\n if whitelist_repository_names:\n if repo_name not in whitelist_repository_names:\n print(\n \"Skipping repository '%s' not in explicitly passed list\" %\n repo_name, file=sys.stderr)\n continue\n is_disabled = repo_name not in filtered_repo_names\n if is_disabled and build_file.skip_ignored_repositories:\n print(\"Skipping ignored repository '%s'\" % repo_name,\n file=sys.stderr)\n continue\n\n repo = dist_file.repositories[repo_name]\n if not repo.doc_repository:\n print(\"Skipping repository '%s': no doc section\" % repo_name)\n continue\n if not repo.doc_repository.version:\n print(\"Skipping repository '%s': no doc version\" % repo_name)\n continue\n\n for os_name, os_code_name, arch in targets:\n try:\n job_name, job_config = configure_doc_job(\n config_url, rosdistro_name, doc_build_name,\n repo_name, os_name, os_code_name, arch,\n config=config, build_file=build_file,\n index=index, dist_file=dist_file,\n dist_cache=dist_cache, jenkins=jenkins, views=views,\n is_disabled=is_disabled,\n groovy_script=groovy_script,\n dry_run=dry_run)\n job_names.append(job_name)\n if groovy_script is not None:\n print(\"Configuration for job '%s'\" % job_name)\n job_configs[job_name] = job_config\n except JobValidationError as e:\n print(e.message, file=sys.stderr)\n\n groovy_data['expected_num_jobs'] = len(job_configs)\n groovy_data['job_prefixes_and_names'] = {}\n\n job_prefix = '%s__' % doc_view_name\n if not whitelist_repository_names:\n groovy_data['job_prefixes_and_names']['doc'] = (job_prefix, job_names)\n\n if groovy_script is None:\n # delete obsolete jobs in this view\n from ros_buildfarm.jenkins import remove_jobs\n print('Removing obsolete doc jobs')\n remove_jobs(jenkins, job_prefix, job_names, dry_run=dry_run)\n if groovy_script is not None:\n print(\n \"Writing groovy script '%s' to reconfigure %d views and %d jobs\" %\n (groovy_script, len(view_configs), len(job_configs)))\n content = expand_template(\n 'snippet/reconfigure_jobs.groovy.em', groovy_data)\n write_groovy_script_and_configs(\n groovy_script, content, job_configs, view_configs=view_configs)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef configure_doc_job(\n config_url, rosdistro_name, doc_build_name,\n repo_name, os_name, os_code_name, arch,\n config=None, build_file=None,\n index=None, dist_file=None, dist_cache=None,\n jenkins=None, views=None,\n is_disabled=False,\n groovy_script=None,\n doc_repository=None,\n dry_run=False):\n \"\"\"\n Configure a single Jenkins doc job.\n\n This includes the following steps:\n - clone the doc repository to use\n - clone the ros_buildfarm repository\n - write the distribution repository keys into files\n - invoke the run_doc_job.py script\n \"\"\"\n if config is None:\n config = get_config_index(config_url)\n if build_file is None:\n build_files = get_doc_build_files(config, rosdistro_name)\n build_file = build_files[doc_build_name]\n\n if index is None:\n index = get_index(config.rosdistro_index_url)\n if dist_file is None:\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n raise JobValidationError(\n 'No distribution file matches the build file')\n\n repo_names = dist_file.repositories.keys()\n\n if repo_name is not None:\n if repo_name not in repo_names:\n raise JobValidationError(\n \"Invalid repository name '%s' \" % repo_name +\n 'choose one of the following: %s' %\n ', '.join(sorted(repo_names)))\n\n repo = dist_file.repositories[repo_name]\n if not repo.doc_repository:\n raise JobValidationError(\n \"Repository '%s' has no doc section\" % repo_name)\n if not repo.doc_repository.version:\n raise JobValidationError(\n \"Repository '%s' has no doc version\" % repo_name)\n doc_repository = repo.doc_repository\n\n if os_name not in build_file.targets.keys():\n raise JobValidationError(\n \"Invalid OS name '%s' \" % os_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets.keys())))\n if os_code_name not in build_file.targets[os_name].keys():\n raise JobValidationError(\n \"Invalid OS code name '%s' \" % os_code_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets[os_name].keys())))\n if arch not in build_file.targets[os_name][os_code_name]:\n raise JobValidationError(\n \"Invalid architecture '%s' \" % arch +\n 'choose one of the following: %s' % ', '.join(sorted(\n build_file.targets[os_name][os_code_name])))\n\n if dist_cache is None and build_file.notify_maintainers:\n dist_cache = get_distribution_cache(index, rosdistro_name)\n if jenkins is None:\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url)\n if views is None:\n view_name = get_doc_view_name(\n rosdistro_name, doc_build_name)\n configure_doc_view(jenkins, view_name, dry_run=dry_run)\n\n job_name = get_doc_job_name(\n rosdistro_name, doc_build_name,\n repo_name, os_name, os_code_name, arch)\n\n job_config = _get_doc_job_config(\n config, config_url, rosdistro_name, doc_build_name,\n build_file, os_name, os_code_name, arch, doc_repository,\n repo_name, dist_cache=dist_cache, is_disabled=is_disabled)\n # jenkinsapi.jenkins.Jenkins evaluates to false if job count is zero\n if isinstance(jenkins, object) and jenkins is not False:\n from ros_buildfarm.jenkins import configure_job\n configure_job(jenkins, job_name, job_config, dry_run=dry_run)\n\n return job_name, job_config", "response": "Configure a single Jenkins doc job."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_affected_by_sync(\n package_descriptors, targets,\n testing_repo_data, main_repo_data):\n \"\"\"\n For each package and target check if it is affected by a sync.\n\n This is the case when the package version in the testing repo is different\n from the version in the main repo.\n\n :return: a dict indexed by package names containing\n dicts indexed by targets containing a boolean flag\n \"\"\"\n affected_by_sync = {}\n for package_descriptor in package_descriptors.values():\n pkg_name = package_descriptor.pkg_name\n debian_pkg_name = package_descriptor.debian_pkg_name\n\n affected_by_sync[pkg_name] = {}\n for target in targets:\n testing_version = _strip_version_suffix(\n testing_repo_data.get(target, {}).get(debian_pkg_name, None))\n main_version = _strip_version_suffix(\n main_repo_data.get(target, {}).get(debian_pkg_name, None))\n\n affected_by_sync[pkg_name][target] = \\\n testing_version != main_version\n return affected_by_sync", "response": "Returns a dict indexed by package names containing a boolean flag indicating if the version of the package in testing repo is different from the version in main repo."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_regressions(\n package_descriptors, targets,\n building_repo_data, testing_repo_data, main_repo_data):\n \"\"\"\n For each package and target check if it is a regression.\n\n This is the case if the main repo contains a package version which is\n higher then in any of the other repos or if any of the other repos does not\n contain that package at all.\n\n :return: a dict indexed by package names containing\n dicts indexed by targets containing a boolean flag\n \"\"\"\n regressions = {}\n for package_descriptor in package_descriptors.values():\n pkg_name = package_descriptor.pkg_name\n debian_pkg_name = package_descriptor.debian_pkg_name\n\n regressions[pkg_name] = {}\n for target in targets:\n regressions[pkg_name][target] = False\n main_version = \\\n main_repo_data.get(target, {}).get(debian_pkg_name, None)\n if main_version is not None:\n main_ver_loose = LooseVersion(main_version)\n for repo_data in [building_repo_data, testing_repo_data]:\n version = \\\n repo_data.get(target, {}).get(debian_pkg_name, None)\n if not version or main_ver_loose > LooseVersion(version):\n regressions[pkg_name][target] = True\n return regressions", "response": "Returns a dictionary indexed by package names containing a boolean flag indicating if the main repo contains a version that is higher than the main version of the package."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_version_status(\n package_descriptors, targets, repos_data,\n strip_version=False, strip_os_code_name=False):\n \"\"\"\n For each package and target check if it is affected by a sync.\n\n This is the case when the package version in the testing repo is different\n from the version in the main repo.\n\n :return: a dict indexed by package names containing\n dicts indexed by targets containing\n a list of status strings (one for each repo)\n \"\"\"\n status = {}\n for package_descriptor in package_descriptors.values():\n pkg_name = package_descriptor.pkg_name\n debian_pkg_name = package_descriptor.debian_pkg_name\n ref_version = package_descriptor.version\n if strip_version:\n ref_version = _strip_version_suffix(ref_version)\n\n status[pkg_name] = {}\n for target in targets:\n statuses = []\n for repo_data in repos_data:\n version = repo_data.get(target, {}).get(debian_pkg_name, None)\n if strip_version:\n version = _strip_version_suffix(version)\n if strip_os_code_name:\n version = _strip_os_code_name_suffix(\n version, target.os_code_name)\n\n if ref_version:\n if not version:\n statuses.append('missing')\n elif version.startswith(ref_version): # including equal\n statuses.append('equal')\n else:\n if _version_is_gt_other(version, ref_version):\n statuses.append('higher')\n else:\n statuses.append('lower')\n else:\n if not version:\n statuses.append('ignore')\n else:\n statuses.append('obsolete')\n status[pkg_name][target] = statuses\n return status", "response": "Get the status of a set of packages and targets."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nstripping trailing junk from the version number.", "response": "def _strip_version_suffix(version):\n \"\"\"\n Remove trailing junk from the version number.\n\n >>> strip_version_suffix('')\n ''\n >>> strip_version_suffix('None')\n 'None'\n >>> strip_version_suffix('1.2.3-4trusty-20140131-1359-+0000')\n '1.2.3-4'\n >>> strip_version_suffix('1.2.3-foo')\n '1.2.3'\n \"\"\"\n global version_regex\n if not version:\n return version\n match = version_regex.search(version)\n return match.group(0) if match else version"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning a dict indexed by package names containing a boolean flag indicating if the version in one repo is equal for all targets.", "response": "def get_homogeneous(package_descriptors, targets, repos_data):\n \"\"\"\n For each package check if the version in one repo is equal for all targets.\n\n The version could be different in different repos though.\n\n :return: a dict indexed by package names containing a boolean flag\n \"\"\"\n homogeneous = {}\n for package_descriptor in package_descriptors.values():\n pkg_name = package_descriptor.pkg_name\n debian_pkg_name = package_descriptor.debian_pkg_name\n\n versions = []\n for repo_data in repos_data:\n versions.append(set([]))\n for target in targets:\n version = _strip_version_suffix(\n repo_data.get(target, {}).get(debian_pkg_name, None))\n versions[-1].add(version)\n homogeneous[pkg_name] = max([len(v) for v in versions]) == 1\n return homogeneous"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the number of packages per target and repository.", "response": "def get_package_counts(package_descriptors, targets, repos_data):\n \"\"\"\n Get the number of packages per target and repository.\n\n :return: a dict indexed by targets containing\n a list of integer values (one for each repo)\n \"\"\"\n counts = {}\n for target in targets:\n counts[target] = [0] * len(repos_data)\n for package_descriptor in package_descriptors.values():\n debian_pkg_name = package_descriptor.debian_pkg_name\n\n for target in targets:\n for i, repo_data in enumerate(repos_data):\n version = repo_data.get(target, {}).get(debian_pkg_name, None)\n if version:\n counts[target][i] += 1\n return counts"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_jenkins_job_urls(\n rosdistro_name, jenkins_url, release_build_name, targets):\n \"\"\"\n Get the Jenkins job urls for each target.\n\n The placeholder {pkg} needs to be replaced with the ROS package name.\n\n :return: a dict indexed by targets containing a string\n \"\"\"\n urls = {}\n for target in targets:\n view_name = get_release_view_name(\n rosdistro_name, release_build_name,\n target.os_name, target.os_code_name, target.arch)\n base_url = jenkins_url + '/view/%s/job/%s__{pkg}__' % \\\n (view_name, view_name)\n if target.arch == 'source':\n urls[target] = base_url + '%s_%s__source' % \\\n (target.os_name, target.os_code_name)\n else:\n urls[target] = base_url + '%s_%s_%s__binary' % \\\n (target.os_name, target.os_code_name, target.arch)\n return urls", "response": "Get the Jenkins job urls for each target."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef configure_ci_jobs(\n config_url, rosdistro_name, ci_build_name,\n groovy_script=None, dry_run=False):\n \"\"\"Configure all Jenkins CI jobs.\"\"\"\n config = get_config_index(config_url)\n build_files = get_ci_build_files(config, rosdistro_name)\n build_file = build_files[ci_build_name]\n\n index = get_index(config.rosdistro_index_url)\n\n # get targets\n targets = []\n for os_name in build_file.targets.keys():\n for os_code_name in build_file.targets[os_name].keys():\n for arch in build_file.targets[os_name][os_code_name]:\n targets.append((os_name, os_code_name, arch))\n print('The build file contains the following targets:')\n for os_name, os_code_name, arch in targets:\n print(' -', os_name, os_code_name, arch)\n\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n print('No distribution file matches the build file')\n return\n\n ci_view_name = get_ci_view_name(rosdistro_name)\n\n # all further configuration will be handled by either the Jenkins API\n # or by a generated groovy script\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url) if groovy_script is None else False\n\n view_configs = {}\n views = {\n ci_view_name: configure_ci_view(\n jenkins, ci_view_name, dry_run=dry_run)\n }\n if not jenkins:\n view_configs.update(views)\n groovy_data = {\n 'dry_run': dry_run,\n 'expected_num_views': len(view_configs),\n }\n\n ci_job_names = []\n job_configs = OrderedDict()\n\n is_disabled = False\n\n for os_name, os_code_name, arch in targets:\n try:\n job_name, job_config = configure_ci_job(\n config_url, rosdistro_name, ci_build_name,\n os_name, os_code_name, arch,\n config=config, build_file=build_file,\n index=index, dist_file=dist_file,\n jenkins=jenkins, views=views,\n is_disabled=is_disabled,\n groovy_script=groovy_script,\n dry_run=dry_run,\n trigger_timer=build_file.jenkins_job_schedule)\n ci_job_names.append(job_name)\n if groovy_script is not None:\n print(\"Configuration for job '%s'\" % job_name)\n job_configs[job_name] = job_config\n except JobValidationError as e:\n print(e.message, file=sys.stderr)\n\n groovy_data['expected_num_jobs'] = len(job_configs)\n groovy_data['job_prefixes_and_names'] = {}\n\n if groovy_script is not None:\n print(\n \"Writing groovy script '%s' to reconfigure %d jobs\" %\n (groovy_script, len(job_configs)))\n content = expand_template(\n 'snippet/reconfigure_jobs.groovy.em', groovy_data)\n write_groovy_script_and_configs(\n groovy_script, content, job_configs, view_configs)", "response": "Configure all Jenkins CI jobs."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef configure_ci_job(\n config_url, rosdistro_name, ci_build_name,\n os_name, os_code_name, arch,\n config=None, build_file=None,\n index=None, dist_file=None,\n jenkins=None, views=None,\n is_disabled=False,\n groovy_script=None,\n build_targets=None,\n dry_run=False,\n underlay_source_paths=None,\n trigger_timer=None):\n \"\"\"\n Configure a single Jenkins CI job.\n\n This includes the following steps:\n - clone the ros_buildfarm repository\n - write the distribution repository keys into files\n - invoke the ci/run_ci_job.py script\n \"\"\"\n if config is None:\n config = get_config_index(config_url)\n if build_file is None:\n build_files = get_ci_build_files(config, rosdistro_name)\n build_file = build_files[ci_build_name]\n # Overwrite build_file.targets if build_targets is specified\n if build_targets is not None:\n build_file.targets = build_targets\n\n if index is None:\n index = get_index(config.rosdistro_index_url)\n if dist_file is None:\n dist_file = get_distribution_file(index, rosdistro_name, build_file)\n if not dist_file:\n raise JobValidationError(\n 'No distribution file matches the build file')\n\n if os_name not in build_file.targets.keys():\n raise JobValidationError(\n \"Invalid OS name '%s' \" % os_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets.keys())))\n if os_code_name not in build_file.targets[os_name].keys():\n raise JobValidationError(\n \"Invalid OS code name '%s' \" % os_code_name +\n 'choose one of the following: ' +\n ', '.join(sorted(build_file.targets[os_name].keys())))\n if arch not in build_file.targets[os_name][os_code_name]:\n raise JobValidationError(\n \"Invalid architecture '%s' \" % arch +\n 'choose one of the following: %s' % ', '.join(sorted(\n build_file.targets[os_name][os_code_name])))\n\n if len(build_file.underlay_from_ci_jobs) > 1:\n raise JobValidationError(\n 'Only a single underlay job is currently supported, but the ' +\n 'build file lists %d.' % len(build_file.underlay_from_ci_jobs))\n\n underlay_source_job = None\n if build_file.underlay_from_ci_jobs:\n underlay_source_job = get_ci_job_name(\n rosdistro_name, os_name, os_code_name, arch,\n build_file.underlay_from_ci_jobs[0])\n underlay_source_paths = (underlay_source_paths or []) + \\\n ['$UNDERLAY_JOB_SPACE']\n\n if jenkins is None:\n from ros_buildfarm.jenkins import connect\n jenkins = connect(config.jenkins_url)\n if views is None:\n view_name = get_ci_view_name(rosdistro_name)\n configure_ci_view(jenkins, view_name, dry_run=dry_run)\n\n job_name = get_ci_job_name(\n rosdistro_name, os_name, os_code_name, arch, ci_build_name)\n\n job_config = _get_ci_job_config(\n index, rosdistro_name, build_file, os_name,\n os_code_name, arch,\n build_file.repos_files,\n underlay_source_job,\n underlay_source_paths,\n trigger_timer,\n is_disabled=is_disabled)\n # jenkinsapi.jenkins.Jenkins evaluates to false if job count is zero\n if isinstance(jenkins, object) and jenkins is not False:\n from ros_buildfarm.jenkins import configure_job\n configure_job(jenkins, job_name, job_config, dry_run=dry_run)\n\n return job_name, job_config", "response": "Configure a single CI job."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrite out the groovy script and the job_configs to file.", "response": "def write_groovy_script_and_configs(\n filename, content, job_configs, view_configs=None):\n \"\"\"Write out the groovy script and configs to file.\n\n This writes the reconfigure script to the file location\n and places the expanded configs in subdirectories 'view_configs' /\n 'job_configs' that the script can then access when run.\n \"\"\"\n with open(filename, 'w') as h:\n h.write(content)\n\n if view_configs:\n view_config_dir = os.path.join(os.path.dirname(filename), 'view_configs')\n if not os.path.isdir(view_config_dir):\n os.makedirs(view_config_dir)\n for config_name, config_body in view_configs.items():\n config_filename = os.path.join(view_config_dir, config_name)\n with open(config_filename, 'w') as config_fh:\n config_fh.write(config_body)\n\n job_config_dir = os.path.join(os.path.dirname(filename), 'job_configs')\n if not os.path.isdir(job_config_dir):\n os.makedirs(job_config_dir)\n # prefix each config file with a serial number to maintain order\n format_str = '%0' + str(len(str(len(job_configs)))) + 'd'\n i = 0\n for config_name, config_body in job_configs.items():\n i += 1\n config_filename = os.path.join(\n job_config_dir,\n format_str % i + ' ' + config_name)\n with open(config_filename, 'w') as config_fh:\n config_fh.write(config_body)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a list of tuples containing the relative path and a list of packages which have message generators and then .", "response": "def topological_order_packages(packages):\n \"\"\"\n Order packages topologically.\n\n First returning packages which have message generators and then\n the rest based on all direct depends and indirect recursive run_depends.\n\n :param packages: A dict mapping relative paths to ``Package`` objects ``dict``\n :returns: A list of tuples containing the relative path and a ``Package`` object, ``list``\n \"\"\"\n from catkin_pkg.topological_order import _PackageDecorator\n from catkin_pkg.topological_order import _sort_decorated_packages\n\n decorators_by_name = {}\n for path, package in packages.items():\n decorators_by_name[package.name] = _PackageDecorator(package, path)\n\n # calculate transitive dependencies\n for decorator in decorators_by_name.values():\n decorator.depends_for_topological_order = set([])\n all_depends = \\\n decorator.package.build_depends + decorator.package.buildtool_depends + \\\n decorator.package.run_depends + decorator.package.test_depends\n # skip external dependencies, meaning names that are not known packages\n unique_depend_names = set([\n d.name for d in all_depends if d.name in decorators_by_name.keys()])\n for name in unique_depend_names:\n if name in decorator.depends_for_topological_order:\n # avoid function call to improve performance\n # check within the loop since the set changes every cycle\n continue\n decorators_by_name[name]._add_recursive_run_depends(\n decorators_by_name, decorator.depends_for_topological_order)\n\n ordered_pkg_tuples = _sort_decorated_packages(decorators_by_name)\n for pkg_path, pkg in ordered_pkg_tuples:\n if pkg_path is None:\n raise RuntimeError('Circular dependency in: %s' % pkg)\n return ordered_pkg_tuples"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef parse_public(data):\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n key_type = None\n\n # Appears to be PEM formatted\n if data[0:5] == b'-----':\n key_type, algo, data = _unarmor_pem(data)\n\n if key_type == 'private key':\n raise ValueError(pretty_message(\n '''\n The data specified does not appear to be a public key or\n certificate, but rather a private key\n '''\n ))\n\n # When a public key returning from _unarmor_pem has a known algorithm\n # of RSA, that means the DER structure is of the type RSAPublicKey, so\n # we need to wrap it in the PublicKeyInfo structure.\n if algo == 'rsa':\n return keys.PublicKeyInfo.wrap(data, 'rsa')\n\n if key_type is None or key_type == 'public key':\n try:\n pki = keys.PublicKeyInfo.load(data)\n # Call .native to fully parse since asn1crypto is lazy\n pki.native\n return pki\n except (ValueError):\n pass # Data was not PublicKeyInfo\n\n try:\n rpk = keys.RSAPublicKey.load(data)\n # Call .native to fully parse since asn1crypto is lazy\n rpk.native\n return keys.PublicKeyInfo.wrap(rpk, 'rsa')\n except (ValueError):\n pass # Data was not an RSAPublicKey\n\n if key_type is None or key_type == 'certificate':\n try:\n parsed_cert = x509.Certificate.load(data)\n key_info = parsed_cert['tbs_certificate']['subject_public_key_info']\n return key_info\n except (ValueError):\n pass # Data was not a cert\n\n raise ValueError('The data specified does not appear to be a known public key or certificate format')", "response": "Loads a public key from a DER or PEM - formatted file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef parse_certificate(data):\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n key_type = None\n\n # Appears to be PEM formatted\n if data[0:5] == b'-----':\n key_type, _, data = _unarmor_pem(data)\n\n if key_type == 'private key':\n raise ValueError(pretty_message(\n '''\n The data specified does not appear to be a certificate, but\n rather a private key\n '''\n ))\n\n if key_type == 'public key':\n raise ValueError(pretty_message(\n '''\n The data specified does not appear to be a certificate, but\n rather a public key\n '''\n ))\n\n if key_type is None or key_type == 'certificate':\n try:\n return x509.Certificate.load(data)\n except (ValueError):\n pass # Data was not a Certificate\n\n raise ValueError(pretty_message(\n '''\n The data specified does not appear to be a known certificate format\n '''\n ))", "response": "Parses a certificate from a DER or PEM - formatted file. Supports X.509\n certificates only."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef parse_private(data, password=None):\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if password is not None:\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n else:\n password = b''\n\n # Appears to be PEM formatted\n if data[0:5] == b'-----':\n key_type, _, data = _unarmor_pem(data, password)\n\n if key_type == 'public key':\n raise ValueError(pretty_message(\n '''\n The data specified does not appear to be a private key, but\n rather a public key\n '''\n ))\n\n if key_type == 'certificate':\n raise ValueError(pretty_message(\n '''\n The data specified does not appear to be a private key, but\n rather a certificate\n '''\n ))\n\n try:\n pki = keys.PrivateKeyInfo.load(data)\n # Call .native to fully parse since asn1crypto is lazy\n pki.native\n return pki\n except (ValueError):\n pass # Data was not PrivateKeyInfo\n\n try:\n parsed_wrapper = keys.EncryptedPrivateKeyInfo.load(data)\n encryption_algorithm_info = parsed_wrapper['encryption_algorithm']\n encrypted_data = parsed_wrapper['encrypted_data'].native\n decrypted_data = _decrypt_encrypted_data(encryption_algorithm_info, encrypted_data, password)\n pki = keys.PrivateKeyInfo.load(decrypted_data)\n # Call .native to fully parse since asn1crypto is lazy\n pki.native\n return pki\n except (ValueError):\n pass # Data was not EncryptedPrivateKeyInfo\n\n try:\n parsed = keys.RSAPrivateKey.load(data)\n # Call .native to fully parse since asn1crypto is lazy\n parsed.native\n return keys.PrivateKeyInfo.wrap(parsed, 'rsa')\n except (ValueError):\n pass # Data was not an RSAPrivateKey\n\n try:\n parsed = keys.DSAPrivateKey.load(data)\n # Call .native to fully parse since asn1crypto is lazy\n parsed.native\n return keys.PrivateKeyInfo.wrap(parsed, 'dsa')\n except (ValueError):\n pass # Data was not a DSAPrivateKey\n\n try:\n parsed = keys.ECPrivateKey.load(data)\n # Call .native to fully parse since asn1crypto is lazy\n parsed.native\n return keys.PrivateKeyInfo.wrap(parsed, 'ec')\n except (ValueError):\n pass # Data was not an ECPrivateKey\n\n raise ValueError(pretty_message(\n '''\n The data specified does not appear to be a known private key format\n '''\n ))", "response": "Parses a private key from a DER or PEM - formatted file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _unarmor_pem(data, password=None):\n\n object_type, headers, der_bytes = pem.unarmor(data)\n\n type_regex = '^((DSA|EC|RSA) PRIVATE KEY|ENCRYPTED PRIVATE KEY|PRIVATE KEY|PUBLIC KEY|RSA PUBLIC KEY|CERTIFICATE)'\n armor_type = re.match(type_regex, object_type)\n if not armor_type:\n raise ValueError(pretty_message(\n '''\n data does not seem to contain a PEM-encoded certificate, private\n key or public key\n '''\n ))\n\n pem_header = armor_type.group(1)\n\n data = data.strip()\n\n # RSA private keys are encrypted after being DER-encoded, but before base64\n # encoding, so they need to be hanlded specially\n if pem_header in set(['RSA PRIVATE KEY', 'DSA PRIVATE KEY', 'EC PRIVATE KEY']):\n algo = armor_type.group(2).lower()\n return ('private key', algo, _unarmor_pem_openssl_private(headers, der_bytes, password))\n\n key_type = pem_header.lower()\n algo = None\n if key_type == 'encrypted private key':\n key_type = 'private key'\n elif key_type == 'rsa public key':\n key_type = 'public key'\n algo = 'rsa'\n\n return (key_type, algo, der_bytes)", "response": "Unarmor PEM - encoded data into a 3 - element tuple."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nparse a PKCS#1 private key, or encrypted private key :param headers: A dict of \"Name: Value\" lines from right after the PEM header :param data: A byte string of the DER-encoded PKCS#1 private key :param password: A byte string of the password to use if the private key is encrypted :return: A byte string of the DER-encoded private key", "response": "def _unarmor_pem_openssl_private(headers, data, password):\n \"\"\"\n Parses a PKCS#1 private key, or encrypted private key\n\n :param headers:\n A dict of \"Name: Value\" lines from right after the PEM header\n\n :param data:\n A byte string of the DER-encoded PKCS#1 private key\n\n :param password:\n A byte string of the password to use if the private key is encrypted\n\n :return:\n A byte string of the DER-encoded private key\n \"\"\"\n\n enc_algo = None\n enc_iv_hex = None\n enc_iv = None\n\n if 'DEK-Info' in headers:\n params = headers['DEK-Info']\n if params.find(',') != -1:\n enc_algo, enc_iv_hex = params.strip().split(',')\n else:\n enc_algo = 'RC4'\n\n if not enc_algo:\n return data\n\n if enc_iv_hex:\n enc_iv = binascii.unhexlify(enc_iv_hex.encode('ascii'))\n enc_algo = enc_algo.lower()\n\n enc_key_length = {\n 'aes-128-cbc': 16,\n 'aes-128': 16,\n 'aes-192-cbc': 24,\n 'aes-192': 24,\n 'aes-256-cbc': 32,\n 'aes-256': 32,\n 'rc4': 16,\n 'rc4-64': 8,\n 'rc4-40': 5,\n 'rc2-64-cbc': 8,\n 'rc2-40-cbc': 5,\n 'rc2-cbc': 16,\n 'rc2': 16,\n 'des-ede3-cbc': 24,\n 'des-ede3': 24,\n 'des3': 24,\n 'des-ede-cbc': 16,\n 'des-cbc': 8,\n 'des': 8,\n }[enc_algo]\n\n enc_key = hashlib.md5(password + enc_iv[0:8]).digest()\n while enc_key_length > len(enc_key):\n enc_key += hashlib.md5(enc_key + password + enc_iv[0:8]).digest()\n enc_key = enc_key[0:enc_key_length]\n\n enc_algo_name = {\n 'aes-128-cbc': 'aes',\n 'aes-128': 'aes',\n 'aes-192-cbc': 'aes',\n 'aes-192': 'aes',\n 'aes-256-cbc': 'aes',\n 'aes-256': 'aes',\n 'rc4': 'rc4',\n 'rc4-64': 'rc4',\n 'rc4-40': 'rc4',\n 'rc2-64-cbc': 'rc2',\n 'rc2-40-cbc': 'rc2',\n 'rc2-cbc': 'rc2',\n 'rc2': 'rc2',\n 'des-ede3-cbc': 'tripledes',\n 'des-ede3': 'tripledes',\n 'des3': 'tripledes',\n 'des-ede-cbc': 'tripledes',\n 'des-cbc': 'des',\n 'des': 'des',\n }[enc_algo]\n decrypt_func = crypto_funcs[enc_algo_name]\n\n if enc_algo_name == 'rc4':\n return decrypt_func(enc_key, data)\n\n return decrypt_func(enc_key, data, enc_iv)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse a PKCS12 ANS. 1 DER - encoded structure and extracts certs and keys from the cert chain.", "response": "def parse_pkcs12(data, password=None):\n \"\"\"\n Parses a PKCS#12 ANS.1 DER-encoded structure and extracts certs and keys\n\n :param data:\n A byte string of a DER-encoded PKCS#12 file\n\n :param password:\n A byte string of the password to any encrypted data\n\n :raises:\n ValueError - when any of the parameters are of the wrong type or value\n OSError - when an error is returned by one of the OS decryption functions\n\n :return:\n A three-element tuple of:\n 1. An asn1crypto.keys.PrivateKeyInfo object\n 2. An asn1crypto.x509.Certificate object\n 3. A list of zero or more asn1crypto.x509.Certificate objects that are\n \"extra\" certificates, possibly intermediates from the cert chain\n \"\"\"\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if password is not None:\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n else:\n password = b''\n\n certs = {}\n private_keys = {}\n\n pfx = pkcs12.Pfx.load(data)\n\n auth_safe = pfx['auth_safe']\n if auth_safe['content_type'].native != 'data':\n raise ValueError(pretty_message(\n '''\n Only password-protected PKCS12 files are currently supported\n '''\n ))\n authenticated_safe = pfx.authenticated_safe\n\n mac_data = pfx['mac_data']\n if mac_data:\n mac_algo = mac_data['mac']['digest_algorithm']['algorithm'].native\n key_length = {\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64,\n 'sha512_224': 28,\n 'sha512_256': 32,\n }[mac_algo]\n mac_key = pkcs12_kdf(\n mac_algo,\n password,\n mac_data['mac_salt'].native,\n mac_data['iterations'].native,\n key_length,\n 3 # ID 3 is for generating an HMAC key\n )\n hash_mod = getattr(hashlib, mac_algo)\n computed_hmac = hmac.new(mac_key, auth_safe['content'].contents, hash_mod).digest()\n stored_hmac = mac_data['mac']['digest'].native\n if not constant_compare(computed_hmac, stored_hmac):\n raise ValueError('Password provided is invalid')\n\n for content_info in authenticated_safe:\n content = content_info['content']\n\n if isinstance(content, core.OctetString):\n _parse_safe_contents(content.native, certs, private_keys, password)\n\n elif isinstance(content, cms.EncryptedData):\n encrypted_content_info = content['encrypted_content_info']\n\n encryption_algorithm_info = encrypted_content_info['content_encryption_algorithm']\n encrypted_content = encrypted_content_info['encrypted_content'].native\n decrypted_content = _decrypt_encrypted_data(encryption_algorithm_info, encrypted_content, password)\n\n _parse_safe_contents(decrypted_content, certs, private_keys, password)\n\n else:\n raise ValueError(pretty_message(\n '''\n Public-key-based PKCS12 files are not currently supported\n '''\n ))\n\n key_fingerprints = set(private_keys.keys())\n cert_fingerprints = set(certs.keys())\n\n common_fingerprints = sorted(list(key_fingerprints & cert_fingerprints))\n\n key = None\n cert = None\n other_certs = []\n\n if len(common_fingerprints) >= 1:\n fingerprint = common_fingerprints[0]\n key = private_keys[fingerprint]\n cert = certs[fingerprint]\n other_certs = [certs[f] for f in certs if f != fingerprint]\n return (key, cert, other_certs)\n\n if len(private_keys) > 0:\n first_key = sorted(list(private_keys.keys()))[0]\n key = private_keys[first_key]\n\n if len(certs) > 0:\n first_key = sorted(list(certs.keys()))[0]\n cert = certs[first_key]\n del certs[first_key]\n\n if len(certs) > 0:\n other_certs = sorted(list(certs.values()))\n\n return (key, cert, other_certs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _parse_safe_contents(safe_contents, certs, private_keys, password):\n\n if isinstance(safe_contents, byte_cls):\n safe_contents = pkcs12.SafeContents.load(safe_contents)\n\n for safe_bag in safe_contents:\n bag_value = safe_bag['bag_value']\n\n if isinstance(bag_value, pkcs12.CertBag):\n if bag_value['cert_id'].native == 'x509':\n cert = bag_value['cert_value'].parsed\n public_key_info = cert['tbs_certificate']['subject_public_key_info']\n certs[public_key_info.fingerprint] = bag_value['cert_value'].parsed\n\n elif isinstance(bag_value, keys.PrivateKeyInfo):\n private_keys[bag_value.fingerprint] = bag_value\n\n elif isinstance(bag_value, keys.EncryptedPrivateKeyInfo):\n encryption_algorithm_info = bag_value['encryption_algorithm']\n encrypted_key_bytes = bag_value['encrypted_data'].native\n decrypted_key_bytes = _decrypt_encrypted_data(encryption_algorithm_info, encrypted_key_bytes, password)\n private_key = keys.PrivateKeyInfo.load(decrypted_key_bytes)\n private_keys[private_key.fingerprint] = private_key\n\n elif isinstance(bag_value, pkcs12.SafeContents):\n _parse_safe_contents(bag_value, certs, private_keys, password)\n\n else:\n # We don't care about CRL bags or secret bags\n pass", "response": "Parses a asn1crypto. ANS. A. SafeContents and stores the certificate and private key information for the certificate and key pairs in the dict."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _decrypt_encrypted_data(encryption_algorithm_info, encrypted_content, password):\n\n decrypt_func = crypto_funcs[encryption_algorithm_info.encryption_cipher]\n\n # Modern, PKCS#5 PBES2-based encryption\n if encryption_algorithm_info.kdf == 'pbkdf2':\n\n if encryption_algorithm_info.encryption_cipher == 'rc5':\n raise ValueError(pretty_message(\n '''\n PBES2 encryption scheme utilizing RC5 encryption is not supported\n '''\n ))\n\n enc_key = pbkdf2(\n encryption_algorithm_info.kdf_hmac,\n password,\n encryption_algorithm_info.kdf_salt,\n encryption_algorithm_info.kdf_iterations,\n encryption_algorithm_info.key_length\n )\n enc_iv = encryption_algorithm_info.encryption_iv\n\n plaintext = decrypt_func(enc_key, encrypted_content, enc_iv)\n\n elif encryption_algorithm_info.kdf == 'pbkdf1':\n derived_output = pbkdf1(\n encryption_algorithm_info.kdf_hmac,\n password,\n encryption_algorithm_info.kdf_salt,\n encryption_algorithm_info.kdf_iterations,\n encryption_algorithm_info.key_length + 8\n )\n enc_key = derived_output[0:8]\n enc_iv = derived_output[8:16]\n\n plaintext = decrypt_func(enc_key, encrypted_content, enc_iv)\n\n elif encryption_algorithm_info.kdf == 'pkcs12_kdf':\n enc_key = pkcs12_kdf(\n encryption_algorithm_info.kdf_hmac,\n password,\n encryption_algorithm_info.kdf_salt,\n encryption_algorithm_info.kdf_iterations,\n encryption_algorithm_info.key_length,\n 1 # ID 1 is for generating a key\n )\n\n # Since RC4 is a stream cipher, we don't use an IV\n if encryption_algorithm_info.encryption_cipher == 'rc4':\n plaintext = decrypt_func(enc_key, encrypted_content)\n\n else:\n enc_iv = pkcs12_kdf(\n encryption_algorithm_info.kdf_hmac,\n password,\n encryption_algorithm_info.kdf_salt,\n encryption_algorithm_info.kdf_iterations,\n encryption_algorithm_info.encryption_block_size,\n 2 # ID 2 is for generating an IV\n )\n plaintext = decrypt_func(enc_key, encrypted_content, enc_iv)\n\n return plaintext", "response": "Decrypts the ASN. 1 data and returns the decrypted ASN. 1 data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef aes_cbc_pkcs7_encrypt(key, data, iv):\n\n cipher = _calculate_aes_cipher(key)\n\n if not iv:\n iv = rand_bytes(16)\n elif len(iv) != 16:\n raise ValueError(pretty_message(\n '''\n iv must be 16 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return (iv, _encrypt(cipher, key, data, iv, True))", "response": "Encrypts plaintext using AES with a 128 192 or 256 bit key and PKCS#7 padding."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef aes_cbc_pkcs7_decrypt(key, data, iv):\n\n cipher = _calculate_aes_cipher(key)\n\n if len(iv) != 16:\n raise ValueError(pretty_message(\n '''\n iv must be 16 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return _decrypt(cipher, key, data, iv, True)", "response": "Decrypts a data block of data using a 128 192 or 256 bit key."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _calculate_aes_cipher(key):\n\n if len(key) not in [16, 24, 32]:\n raise ValueError(pretty_message(\n '''\n key must be either 16, 24 or 32 bytes (128, 192 or 256 bits)\n long - is %s\n ''',\n len(key)\n ))\n\n if len(key) == 16:\n cipher = 'aes128'\n elif len(key) == 24:\n cipher = 'aes192'\n elif len(key) == 32:\n cipher = 'aes256'\n\n return cipher", "response": "Calculates the AES cipher for the given key."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nencrypt plaintext using RC4 with a 40 - 128 bit key", "response": "def rc4_encrypt(key, data):\n \"\"\"\n Encrypts plaintext using RC4 with a 40-128 bit key\n\n :param key:\n The encryption key - a byte string 5-16 bytes long\n\n :param data:\n The plaintext - a byte string\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by OpenSSL\n\n :return:\n A byte string of the ciphertext\n \"\"\"\n\n if len(key) < 5 or len(key) > 16:\n raise ValueError(pretty_message(\n '''\n key must be 5 to 16 bytes (40 to 128 bits) long - is %s\n ''',\n len(key)\n ))\n\n return _encrypt('rc4', key, data, None, None)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef tripledes_cbc_pkcs5_decrypt(key, data, iv):\n\n if len(key) != 16 and len(key) != 24:\n raise ValueError(pretty_message(\n '''\n key must be 16 bytes (2 key) or 24 bytes (3 key) long - is %s\n ''',\n len(key)\n ))\n\n if len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n cipher = 'tripledes_3key'\n # Expand 2-key to actual 24 byte byte string used by cipher\n if len(key) == 16:\n key = key + key[0:8]\n cipher = 'tripledes_2key'\n\n return _decrypt(cipher, key, data, iv, True)", "response": "Decrypts the ciphertext in CBC mode using either the 2 or 3 key variant and PKCS#5 padding."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef des_cbc_pkcs5_decrypt(key, data, iv):\n\n if len(key) != 8:\n raise ValueError(pretty_message(\n '''\n key must be 8 bytes (56 bits + 8 parity bits) long - is %s\n ''',\n len(key)\n ))\n\n if len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return _decrypt('des', key, data, iv, True)", "response": "Decrypts DES ciphertext using a 56 bit key and PKCS#5 padding."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nencrypt a plaintext object with the specified cipher and data and returns the ciphertext.", "response": "def _encrypt(cipher, key, data, iv, padding):\n \"\"\"\n Encrypts plaintext\n\n :param cipher:\n A unicode string of \"aes128\", \"aes192\", \"aes256\", \"des\",\n \"tripledes_2key\", \"tripledes_3key\", \"rc2\", \"rc4\"\n\n :param key:\n The encryption key - a byte string 5-32 bytes long\n\n :param data:\n The plaintext - a byte string\n\n :param iv:\n The initialization vector - a byte string - unused for RC4\n\n :param padding:\n Boolean, if padding should be used - unused for RC4\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by OpenSSL\n\n :return:\n A byte string of the ciphertext\n \"\"\"\n\n if not isinstance(key, byte_cls):\n raise TypeError(pretty_message(\n '''\n key must be a byte string, not %s\n ''',\n type_name(key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if cipher != 'rc4' and not isinstance(iv, byte_cls):\n raise TypeError(pretty_message(\n '''\n iv must be a byte string, not %s\n ''',\n type_name(iv)\n ))\n\n if cipher != 'rc4' and not padding:\n raise ValueError('padding must be specified')\n\n evp_cipher_ctx = None\n\n try:\n evp_cipher_ctx = libcrypto.EVP_CIPHER_CTX_new()\n if is_null(evp_cipher_ctx):\n handle_openssl_error(0)\n\n evp_cipher, buffer_size = _setup_evp_encrypt_decrypt(cipher, data)\n\n if iv is None:\n iv = null()\n\n if cipher in set(['rc2', 'rc4']):\n res = libcrypto.EVP_EncryptInit_ex(evp_cipher_ctx, evp_cipher, null(), null(), null())\n handle_openssl_error(res)\n res = libcrypto.EVP_CIPHER_CTX_set_key_length(evp_cipher_ctx, len(key))\n handle_openssl_error(res)\n if cipher == 'rc2':\n res = libcrypto.EVP_CIPHER_CTX_ctrl(\n evp_cipher_ctx,\n LibcryptoConst.EVP_CTRL_SET_RC2_KEY_BITS,\n len(key) * 8,\n null()\n )\n handle_openssl_error(res)\n evp_cipher = null()\n\n res = libcrypto.EVP_EncryptInit_ex(evp_cipher_ctx, evp_cipher, null(), key, iv)\n handle_openssl_error(res)\n\n if padding is not None:\n res = libcrypto.EVP_CIPHER_CTX_set_padding(evp_cipher_ctx, int(padding))\n handle_openssl_error(res)\n\n buffer = buffer_from_bytes(buffer_size)\n output_length = new(libcrypto, 'int *')\n\n res = libcrypto.EVP_EncryptUpdate(evp_cipher_ctx, buffer, output_length, data, len(data))\n handle_openssl_error(res)\n\n output = bytes_from_buffer(buffer, deref(output_length))\n\n res = libcrypto.EVP_EncryptFinal_ex(evp_cipher_ctx, buffer, output_length)\n handle_openssl_error(res)\n\n output += bytes_from_buffer(buffer, deref(output_length))\n\n return output\n\n finally:\n if evp_cipher_ctx:\n libcrypto.EVP_CIPHER_CTX_free(evp_cipher_ctx)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _setup_evp_encrypt_decrypt(cipher, data):\n\n evp_cipher = {\n 'aes128': libcrypto.EVP_aes_128_cbc,\n 'aes192': libcrypto.EVP_aes_192_cbc,\n 'aes256': libcrypto.EVP_aes_256_cbc,\n 'rc2': libcrypto.EVP_rc2_cbc,\n 'rc4': libcrypto.EVP_rc4,\n 'des': libcrypto.EVP_des_cbc,\n 'tripledes_2key': libcrypto.EVP_des_ede_cbc,\n 'tripledes_3key': libcrypto.EVP_des_ede3_cbc,\n }[cipher]()\n\n if cipher == 'rc4':\n buffer_size = len(data)\n else:\n block_size = {\n 'aes128': 16,\n 'aes192': 16,\n 'aes256': 16,\n 'rc2': 8,\n 'des': 8,\n 'tripledes_2key': 8,\n 'tripledes_3key': 8,\n }[cipher]\n buffer_size = block_size * int(math.ceil(len(data) / block_size))\n\n return (evp_cipher, buffer_size)", "response": "This function creates an EVP_CIPHER pointer object and determines the buffer size needed for the parameter specified."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef handle_error(result):\n\n if result:\n return\n\n _, error_string = get_error()\n\n if not isinstance(error_string, str_cls):\n error_string = _try_decode(error_string)\n\n raise OSError(error_string)", "response": "Handles Windows error messages from the last Windows error message."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nhandling the Windows error and returns the last error message.", "response": "def handle_error(result, exception_class=None):\n \"\"\"\n Extracts the last Windows error message into a python unicode string\n\n :param result:\n A function result, 0 or None indicates failure\n\n :param exception_class:\n The exception class to use for the exception if an error occurred\n\n :return:\n A unicode string error message\n \"\"\"\n\n if result == 0:\n return\n\n if result == Secur32Const.SEC_E_OUT_OF_SEQUENCE:\n raise TLSError('A packet was received out of order')\n\n if result == Secur32Const.SEC_E_MESSAGE_ALTERED:\n raise TLSError('A packet was received altered')\n\n if result == Secur32Const.SEC_E_CONTEXT_EXPIRED:\n raise TLSError('The TLS session expired')\n\n _, error_string = get_error()\n\n if not isinstance(error_string, str_cls):\n error_string = _try_decode(error_string)\n\n if exception_class is None:\n exception_class = OSError\n\n raise exception_class(('SECURITY_STATUS error 0x%0.2X: ' % result) + error_string)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates a public and private key pair for the specified algorithm and bit size.", "response": "def generate_pair(algorithm, bit_size=None, curve=None):\n \"\"\"\n Generates a public/private key pair\n\n :param algorithm:\n The key algorithm - \"rsa\", \"dsa\" or \"ec\"\n\n :param bit_size:\n An integer - used for \"rsa\" and \"dsa\". For \"rsa\" the value maye be 1024,\n 2048, 3072 or 4096. For \"dsa\" the value may be 1024, plus 2048 or 3072\n if OpenSSL 1.0.0 or newer is available.\n\n :param curve:\n A unicode string - used for \"ec\" keys. Valid values include \"secp256r1\",\n \"secp384r1\" and \"secp521r1\".\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A 2-element tuple of (PublicKey, PrivateKey). The contents of each key\n may be saved by calling .asn1.dump().\n \"\"\"\n\n if algorithm not in set(['rsa', 'dsa', 'ec']):\n raise ValueError(pretty_message(\n '''\n algorithm must be one of \"rsa\", \"dsa\", \"ec\", not %s\n ''',\n repr(algorithm)\n ))\n\n if algorithm == 'rsa':\n if bit_size not in set([1024, 2048, 3072, 4096]):\n raise ValueError(pretty_message(\n '''\n bit_size must be one of 1024, 2048, 3072, 4096, not %s\n ''',\n repr(bit_size)\n ))\n\n elif algorithm == 'dsa':\n if libcrypto_version_info < (1,):\n if bit_size != 1024:\n raise ValueError(pretty_message(\n '''\n bit_size must be 1024, not %s\n ''',\n repr(bit_size)\n ))\n else:\n if bit_size not in set([1024, 2048, 3072]):\n raise ValueError(pretty_message(\n '''\n bit_size must be one of 1024, 2048, 3072, not %s\n ''',\n repr(bit_size)\n ))\n\n elif algorithm == 'ec':\n if curve not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise ValueError(pretty_message(\n '''\n curve must be one of \"secp256r1\", \"secp384r1\", \"secp521r1\",\n not %s\n ''',\n repr(curve)\n ))\n\n if algorithm == 'rsa':\n rsa = None\n exponent = None\n\n try:\n rsa = libcrypto.RSA_new()\n if is_null(rsa):\n handle_openssl_error(0)\n\n exponent_pointer = new(libcrypto, 'BIGNUM **')\n result = libcrypto.BN_dec2bn(exponent_pointer, b'65537')\n handle_openssl_error(result)\n exponent = unwrap(exponent_pointer)\n\n result = libcrypto.RSA_generate_key_ex(rsa, bit_size, exponent, null())\n handle_openssl_error(result)\n\n buffer_length = libcrypto.i2d_RSAPublicKey(rsa, null())\n if buffer_length < 0:\n handle_openssl_error(buffer_length)\n buffer = buffer_from_bytes(buffer_length)\n result = libcrypto.i2d_RSAPublicKey(rsa, buffer_pointer(buffer))\n if result < 0:\n handle_openssl_error(result)\n public_key_bytes = bytes_from_buffer(buffer, buffer_length)\n\n buffer_length = libcrypto.i2d_RSAPrivateKey(rsa, null())\n if buffer_length < 0:\n handle_openssl_error(buffer_length)\n buffer = buffer_from_bytes(buffer_length)\n result = libcrypto.i2d_RSAPrivateKey(rsa, buffer_pointer(buffer))\n if result < 0:\n handle_openssl_error(result)\n private_key_bytes = bytes_from_buffer(buffer, buffer_length)\n\n finally:\n if rsa:\n libcrypto.RSA_free(rsa)\n if exponent:\n libcrypto.BN_free(exponent)\n\n elif algorithm == 'dsa':\n dsa = None\n\n try:\n dsa = libcrypto.DSA_new()\n if is_null(dsa):\n handle_openssl_error(0)\n\n result = libcrypto.DSA_generate_parameters_ex(dsa, bit_size, null(), 0, null(), null(), null())\n handle_openssl_error(result)\n\n result = libcrypto.DSA_generate_key(dsa)\n handle_openssl_error(result)\n\n buffer_length = libcrypto.i2d_DSA_PUBKEY(dsa, null())\n if buffer_length < 0:\n handle_openssl_error(buffer_length)\n buffer = buffer_from_bytes(buffer_length)\n result = libcrypto.i2d_DSA_PUBKEY(dsa, buffer_pointer(buffer))\n if result < 0:\n handle_openssl_error(result)\n public_key_bytes = bytes_from_buffer(buffer, buffer_length)\n\n buffer_length = libcrypto.i2d_DSAPrivateKey(dsa, null())\n if buffer_length < 0:\n handle_openssl_error(buffer_length)\n buffer = buffer_from_bytes(buffer_length)\n result = libcrypto.i2d_DSAPrivateKey(dsa, buffer_pointer(buffer))\n if result < 0:\n handle_openssl_error(result)\n private_key_bytes = bytes_from_buffer(buffer, buffer_length)\n\n finally:\n if dsa:\n libcrypto.DSA_free(dsa)\n\n elif algorithm == 'ec':\n ec_key = None\n\n try:\n curve_id = {\n 'secp256r1': LibcryptoConst.NID_X9_62_prime256v1,\n 'secp384r1': LibcryptoConst.NID_secp384r1,\n 'secp521r1': LibcryptoConst.NID_secp521r1,\n }[curve]\n\n ec_key = libcrypto.EC_KEY_new_by_curve_name(curve_id)\n if is_null(ec_key):\n handle_openssl_error(0)\n\n result = libcrypto.EC_KEY_generate_key(ec_key)\n handle_openssl_error(result)\n\n libcrypto.EC_KEY_set_asn1_flag(ec_key, LibcryptoConst.OPENSSL_EC_NAMED_CURVE)\n\n buffer_length = libcrypto.i2o_ECPublicKey(ec_key, null())\n if buffer_length < 0:\n handle_openssl_error(buffer_length)\n buffer = buffer_from_bytes(buffer_length)\n result = libcrypto.i2o_ECPublicKey(ec_key, buffer_pointer(buffer))\n if result < 0:\n handle_openssl_error(result)\n public_key_point_bytes = bytes_from_buffer(buffer, buffer_length)\n\n # i2o_ECPublicKey only returns the ECPoint bytes, so we have to\n # manually wrap it in a PublicKeyInfo structure to get it to parse\n public_key = keys.PublicKeyInfo({\n 'algorithm': keys.PublicKeyAlgorithm({\n 'algorithm': 'ec',\n 'parameters': keys.ECDomainParameters(\n name='named',\n value=curve\n )\n }),\n 'public_key': public_key_point_bytes\n })\n public_key_bytes = public_key.dump()\n\n buffer_length = libcrypto.i2d_ECPrivateKey(ec_key, null())\n if buffer_length < 0:\n handle_openssl_error(buffer_length)\n buffer = buffer_from_bytes(buffer_length)\n result = libcrypto.i2d_ECPrivateKey(ec_key, buffer_pointer(buffer))\n if result < 0:\n handle_openssl_error(result)\n private_key_bytes = bytes_from_buffer(buffer, buffer_length)\n\n finally:\n if ec_key:\n libcrypto.EC_KEY_free(ec_key)\n\n return (load_public_key(public_key_bytes), load_private_key(private_key_bytes))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef generate_dh_parameters(bit_size):\n\n if not isinstance(bit_size, int_types):\n raise TypeError(pretty_message(\n '''\n bit_size must be an integer, not %s\n ''',\n type_name(bit_size)\n ))\n\n if bit_size < 512:\n raise ValueError('bit_size must be greater than or equal to 512')\n\n if bit_size > 4096:\n raise ValueError('bit_size must be less than or equal to 4096')\n\n if bit_size % 64 != 0:\n raise ValueError('bit_size must be a multiple of 64')\n\n dh = None\n\n try:\n dh = libcrypto.DH_new()\n if is_null(dh):\n handle_openssl_error(0)\n\n result = libcrypto.DH_generate_parameters_ex(dh, bit_size, LibcryptoConst.DH_GENERATOR_2, null())\n handle_openssl_error(result)\n\n buffer_length = libcrypto.i2d_DHparams(dh, null())\n if buffer_length < 0:\n handle_openssl_error(buffer_length)\n buffer = buffer_from_bytes(buffer_length)\n result = libcrypto.i2d_DHparams(dh, buffer_pointer(buffer))\n if result < 0:\n handle_openssl_error(result)\n dh_params_bytes = bytes_from_buffer(buffer, buffer_length)\n\n return algos.DHParameters.load(dh_params_bytes)\n\n finally:\n if dh:\n libcrypto.DH_free(dh)", "response": "Generates DH parameters for use with Diffie - Hellman key exchange."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload an x509 certificate into a Certificate object", "response": "def load_certificate(source):\n \"\"\"\n Loads an x509 certificate into a Certificate object\n\n :param source:\n A byte string of file contents, a unicode string filename or an\n asn1crypto.x509.Certificate object\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A Certificate object\n \"\"\"\n\n if isinstance(source, asn1x509.Certificate):\n certificate = source\n\n elif isinstance(source, byte_cls):\n certificate = parse_certificate(source)\n\n elif isinstance(source, str_cls):\n with open(source, 'rb') as f:\n certificate = parse_certificate(f.read())\n\n else:\n raise TypeError(pretty_message(\n '''\n source must be a byte string, unicode string or\n asn1crypto.x509.Certificate object, not %s\n ''',\n type_name(source)\n ))\n\n return _load_x509(certificate)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload an ASN. 1 object of an x509 certificate into a Certificate object.", "response": "def _load_x509(certificate):\n \"\"\"\n Loads an ASN.1 object of an x509 certificate into a Certificate object\n\n :param certificate:\n An asn1crypto.x509.Certificate object\n\n :return:\n A Certificate object\n \"\"\"\n\n source = certificate.dump()\n\n buffer = buffer_from_bytes(source)\n evp_pkey = libcrypto.d2i_X509(null(), buffer_pointer(buffer), len(source))\n if is_null(evp_pkey):\n handle_openssl_error(0)\n return Certificate(evp_pkey, certificate)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_private_key(source, password=None):\n\n if isinstance(source, keys.PrivateKeyInfo):\n private_object = source\n\n else:\n if password is not None:\n if isinstance(password, str_cls):\n password = password.encode('utf-8')\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n\n if isinstance(source, str_cls):\n with open(source, 'rb') as f:\n source = f.read()\n\n elif not isinstance(source, byte_cls):\n raise TypeError(pretty_message(\n '''\n source must be a byte string, unicode string or\n asn1crypto.keys.PrivateKeyInfo object, not %s\n ''',\n type_name(source)\n ))\n\n private_object = parse_private(source, password)\n\n return _load_key(private_object)", "response": "Loads a private key into a private key object."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_public_key(source):\n\n if isinstance(source, keys.PublicKeyInfo):\n public_key = source\n\n elif isinstance(source, byte_cls):\n public_key = parse_public(source)\n\n elif isinstance(source, str_cls):\n with open(source, 'rb') as f:\n public_key = parse_public(f.read())\n\n else:\n raise TypeError(pretty_message(\n '''\n source must be a byte string, unicode string or\n asn1crypto.keys.PublicKeyInfo object, not %s\n ''',\n type_name(public_key)\n ))\n\n if public_key.algorithm == 'dsa':\n if libcrypto_version_info < (1,) and public_key.hash_algo == 'sha2':\n raise AsymmetricKeyError(pretty_message(\n '''\n OpenSSL 0.9.8 only supports DSA keys based on SHA1 (2048 bits or\n less) - this key is based on SHA2 and is %s bits\n ''',\n public_key.bit_size\n ))\n elif public_key.hash_algo is None:\n raise IncompleteAsymmetricKeyError(pretty_message(\n '''\n The DSA key does not contain the necessary p, q and g\n parameters and can not be used\n '''\n ))\n\n data = public_key.dump()\n buffer = buffer_from_bytes(data)\n evp_pkey = libcrypto.d2i_PUBKEY(null(), buffer_pointer(buffer), len(data))\n if is_null(evp_pkey):\n handle_openssl_error(0)\n return PublicKey(evp_pkey, public_key)", "response": "Loads a public key into a base - 16 PublicKey object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload a private key into a private key object.", "response": "def _load_key(private_object):\n \"\"\"\n Loads a private key into a PrivateKey object\n\n :param private_object:\n An asn1crypto.keys.PrivateKeyInfo object\n\n :return:\n A PrivateKey object\n \"\"\"\n\n if libcrypto_version_info < (1,) and private_object.algorithm == 'dsa' and private_object.hash_algo == 'sha2':\n raise AsymmetricKeyError(pretty_message(\n '''\n OpenSSL 0.9.8 only supports DSA keys based on SHA1 (2048 bits or\n less) - this key is based on SHA2 and is %s bits\n ''',\n private_object.bit_size\n ))\n\n source = private_object.unwrap().dump()\n\n buffer = buffer_from_bytes(source)\n evp_pkey = libcrypto.d2i_AutoPrivateKey(null(), buffer_pointer(buffer), len(source))\n if is_null(evp_pkey):\n handle_openssl_error(0)\n return PrivateKey(evp_pkey, private_object)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nencrypting plaintext using an RSA public key or a certificate.", "response": "def _encrypt(certificate_or_public_key, data, padding):\n \"\"\"\n Encrypts plaintext using an RSA public key or certificate\n\n :param certificate_or_public_key:\n A PublicKey, Certificate or PrivateKey object\n\n :param data:\n The byte string to encrypt\n\n :param padding:\n The padding mode to use\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the encrypted data\n \"\"\"\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n rsa = None\n\n try:\n buffer_size = libcrypto.EVP_PKEY_size(certificate_or_public_key.evp_pkey)\n buffer = buffer_from_bytes(buffer_size)\n\n rsa = libcrypto.EVP_PKEY_get1_RSA(certificate_or_public_key.evp_pkey)\n res = libcrypto.RSA_public_encrypt(len(data), data, buffer, rsa, padding)\n handle_openssl_error(res)\n\n return bytes_from_buffer(buffer, res)\n\n finally:\n if rsa:\n libcrypto.RSA_free(rsa)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _decrypt(private_key, ciphertext, padding):\n\n if not isinstance(private_key, PrivateKey):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of the PrivateKey class, not %s\n ''',\n type_name(private_key)\n ))\n\n if not isinstance(ciphertext, byte_cls):\n raise TypeError(pretty_message(\n '''\n ciphertext must be a byte string, not %s\n ''',\n type_name(ciphertext)\n ))\n\n rsa = None\n\n try:\n buffer_size = libcrypto.EVP_PKEY_size(private_key.evp_pkey)\n buffer = buffer_from_bytes(buffer_size)\n\n rsa = libcrypto.EVP_PKEY_get1_RSA(private_key.evp_pkey)\n res = libcrypto.RSA_private_decrypt(len(ciphertext), ciphertext, buffer, rsa, padding)\n handle_openssl_error(res)\n\n return bytes_from_buffer(buffer, res)\n\n finally:\n if rsa:\n libcrypto.RSA_free(rsa)", "response": "Decrypts a byte string using a private key."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=False):\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(signature, byte_cls):\n raise TypeError(pretty_message(\n '''\n signature must be a byte string, not %s\n ''',\n type_name(signature)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n valid_hash_algorithms = set(['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'])\n if certificate_or_public_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms |= set(['raw'])\n\n if hash_algorithm not in valid_hash_algorithms:\n valid_hash_algorithms_error = '\"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\", \"sha512\"'\n if certificate_or_public_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms_error += ', \"raw\"'\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of %s, not %s\n ''',\n valid_hash_algorithms_error,\n repr(hash_algorithm)\n ))\n\n if certificate_or_public_key.algorithm != 'rsa' and rsa_pss_padding:\n raise ValueError(pretty_message(\n '''\n PSS padding can only be used with RSA keys - the key provided is a\n %s key\n ''',\n certificate_or_public_key.algorithm.upper()\n ))\n\n if certificate_or_public_key.algorithm == 'rsa' and hash_algorithm == 'raw':\n if len(data) > certificate_or_public_key.byte_size - 11:\n raise ValueError(pretty_message(\n '''\n data must be 11 bytes shorter than the key size when\n hash_algorithm is \"raw\" - key size is %s bytes, but data is\n %s bytes long\n ''',\n certificate_or_public_key.byte_size,\n len(data)\n ))\n\n rsa = None\n\n try:\n rsa = libcrypto.EVP_PKEY_get1_RSA(certificate_or_public_key.evp_pkey)\n if is_null(rsa):\n handle_openssl_error(0)\n\n buffer_size = libcrypto.EVP_PKEY_size(certificate_or_public_key.evp_pkey)\n decrypted_buffer = buffer_from_bytes(buffer_size)\n decrypted_length = libcrypto.RSA_public_decrypt(\n len(signature),\n signature,\n decrypted_buffer,\n rsa,\n LibcryptoConst.RSA_PKCS1_PADDING\n )\n handle_openssl_error(decrypted_length)\n\n decrypted_bytes = bytes_from_buffer(decrypted_buffer, decrypted_length)\n\n if not constant_compare(data, decrypted_bytes):\n raise SignatureError('Signature is invalid')\n return\n\n finally:\n if rsa:\n libcrypto.RSA_free(rsa)\n\n evp_md_ctx = None\n rsa = None\n dsa = None\n dsa_sig = None\n ec_key = None\n ecdsa_sig = None\n\n try:\n if libcrypto_version_info < (1, 1):\n evp_md_ctx = libcrypto.EVP_MD_CTX_create()\n else:\n evp_md_ctx = libcrypto.EVP_MD_CTX_new()\n\n evp_md = {\n 'md5': libcrypto.EVP_md5,\n 'sha1': libcrypto.EVP_sha1,\n 'sha224': libcrypto.EVP_sha224,\n 'sha256': libcrypto.EVP_sha256,\n 'sha384': libcrypto.EVP_sha384,\n 'sha512': libcrypto.EVP_sha512\n }[hash_algorithm]()\n\n if libcrypto_version_info < (1,):\n if certificate_or_public_key.algorithm == 'rsa' and rsa_pss_padding:\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n rsa = libcrypto.EVP_PKEY_get1_RSA(certificate_or_public_key.evp_pkey)\n if is_null(rsa):\n handle_openssl_error(0)\n\n buffer_size = libcrypto.EVP_PKEY_size(certificate_or_public_key.evp_pkey)\n decoded_buffer = buffer_from_bytes(buffer_size)\n decoded_length = libcrypto.RSA_public_decrypt(\n len(signature),\n signature,\n decoded_buffer,\n rsa,\n LibcryptoConst.RSA_NO_PADDING\n )\n handle_openssl_error(decoded_length)\n\n res = libcrypto.RSA_verify_PKCS1_PSS(\n rsa,\n digest,\n evp_md,\n decoded_buffer,\n LibcryptoConst.EVP_MD_CTX_FLAG_PSS_MDLEN\n )\n\n elif certificate_or_public_key.algorithm == 'rsa':\n res = libcrypto.EVP_DigestInit_ex(evp_md_ctx, evp_md, null())\n handle_openssl_error(res)\n\n res = libcrypto.EVP_DigestUpdate(evp_md_ctx, data, len(data))\n handle_openssl_error(res)\n\n res = libcrypto.EVP_VerifyFinal(\n evp_md_ctx,\n signature,\n len(signature),\n certificate_or_public_key.evp_pkey\n )\n\n elif certificate_or_public_key.algorithm == 'dsa':\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n signature_buffer = buffer_from_bytes(signature)\n signature_pointer = buffer_pointer(signature_buffer)\n dsa_sig = libcrypto.d2i_DSA_SIG(null(), signature_pointer, len(signature))\n if is_null(dsa_sig):\n raise SignatureError('Signature is invalid')\n\n dsa = libcrypto.EVP_PKEY_get1_DSA(certificate_or_public_key.evp_pkey)\n if is_null(dsa):\n handle_openssl_error(0)\n\n res = libcrypto.DSA_do_verify(digest, len(digest), dsa_sig, dsa)\n\n elif certificate_or_public_key.algorithm == 'ec':\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n signature_buffer = buffer_from_bytes(signature)\n signature_pointer = buffer_pointer(signature_buffer)\n ecdsa_sig = libcrypto.d2i_ECDSA_SIG(null(), signature_pointer, len(signature))\n if is_null(ecdsa_sig):\n raise SignatureError('Signature is invalid')\n\n ec_key = libcrypto.EVP_PKEY_get1_EC_KEY(certificate_or_public_key.evp_pkey)\n if is_null(ec_key):\n handle_openssl_error(0)\n\n res = libcrypto.ECDSA_do_verify(digest, len(digest), ecdsa_sig, ec_key)\n\n else:\n evp_pkey_ctx_pointer_pointer = new(libcrypto, 'EVP_PKEY_CTX **')\n res = libcrypto.EVP_DigestVerifyInit(\n evp_md_ctx,\n evp_pkey_ctx_pointer_pointer,\n evp_md,\n null(),\n certificate_or_public_key.evp_pkey\n )\n handle_openssl_error(res)\n evp_pkey_ctx_pointer = unwrap(evp_pkey_ctx_pointer_pointer)\n\n if rsa_pss_padding:\n # Enable PSS padding\n res = libcrypto.EVP_PKEY_CTX_ctrl(\n evp_pkey_ctx_pointer,\n LibcryptoConst.EVP_PKEY_RSA,\n -1, # All operations\n LibcryptoConst.EVP_PKEY_CTRL_RSA_PADDING,\n LibcryptoConst.RSA_PKCS1_PSS_PADDING,\n null()\n )\n handle_openssl_error(res)\n\n # Use the hash algorithm output length as the salt length\n res = libcrypto.EVP_PKEY_CTX_ctrl(\n evp_pkey_ctx_pointer,\n LibcryptoConst.EVP_PKEY_RSA,\n LibcryptoConst.EVP_PKEY_OP_SIGN | LibcryptoConst.EVP_PKEY_OP_VERIFY,\n LibcryptoConst.EVP_PKEY_CTRL_RSA_PSS_SALTLEN,\n -1,\n null()\n )\n handle_openssl_error(res)\n\n res = libcrypto.EVP_DigestUpdate(evp_md_ctx, data, len(data))\n handle_openssl_error(res)\n\n res = libcrypto.EVP_DigestVerifyFinal(evp_md_ctx, signature, len(signature))\n\n if res < 1:\n raise SignatureError('Signature is invalid')\n handle_openssl_error(res)\n\n finally:\n if evp_md_ctx:\n if libcrypto_version_info < (1, 1):\n libcrypto.EVP_MD_CTX_destroy(evp_md_ctx)\n else:\n libcrypto.EVP_MD_CTX_free(evp_md_ctx)\n if rsa:\n libcrypto.RSA_free(rsa)\n if dsa:\n libcrypto.DSA_free(dsa)\n if dsa_sig:\n libcrypto.DSA_SIG_free(dsa_sig)\n if ec_key:\n libcrypto.EC_KEY_free(ec_key)\n if ecdsa_sig:\n libcrypto.ECDSA_SIG_free(ecdsa_sig)", "response": "Verifies a signature with the given certificate_or_public_key."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngenerating a RSASSA - PKCS - v1. 5 signature.", "response": "def rsa_pkcs1v15_sign(private_key, data, hash_algorithm):\n \"\"\"\n Generates an RSASSA-PKCS-v1.5 signature.\n\n When the hash_algorithm is \"raw\", the operation is identical to RSA\n private key encryption. That is: the data is not hashed and no ASN.1\n structure with an algorithm identifier of the hash algorithm is placed in\n the encrypted byte string.\n\n :param private_key:\n The PrivateKey to generate the signature with\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\" or \"raw\"\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the signature\n \"\"\"\n\n if private_key.algorithm != 'rsa':\n raise ValueError(pretty_message(\n '''\n The key specified is not an RSA private key, but %s\n ''',\n private_key.algorithm.upper()\n ))\n\n return _sign(private_key, data, hash_algorithm)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _sign(private_key, data, hash_algorithm, rsa_pss_padding=False):\n\n if not isinstance(private_key, PrivateKey):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of PrivateKey, not %s\n ''',\n type_name(private_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n valid_hash_algorithms = set(['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'])\n if private_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms |= set(['raw'])\n\n if hash_algorithm not in valid_hash_algorithms:\n valid_hash_algorithms_error = '\"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\", \"sha512\"'\n if private_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms_error += ', \"raw\"'\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of %s, not %s\n ''',\n valid_hash_algorithms_error,\n repr(hash_algorithm)\n ))\n\n if private_key.algorithm != 'rsa' and rsa_pss_padding:\n raise ValueError(pretty_message(\n '''\n PSS padding can only be used with RSA keys - the key provided is a\n %s key\n ''',\n private_key.algorithm.upper()\n ))\n\n if private_key.algorithm == 'rsa' and hash_algorithm == 'raw':\n if len(data) > private_key.byte_size - 11:\n raise ValueError(pretty_message(\n '''\n data must be 11 bytes shorter than the key size when\n hash_algorithm is \"raw\" - key size is %s bytes, but data is\n %s bytes long\n ''',\n private_key.byte_size,\n len(data)\n ))\n\n rsa = None\n\n try:\n rsa = libcrypto.EVP_PKEY_get1_RSA(private_key.evp_pkey)\n if is_null(rsa):\n handle_openssl_error(0)\n\n buffer_size = libcrypto.EVP_PKEY_size(private_key.evp_pkey)\n\n signature_buffer = buffer_from_bytes(buffer_size)\n signature_length = libcrypto.RSA_private_encrypt(\n len(data),\n data,\n signature_buffer,\n rsa,\n LibcryptoConst.RSA_PKCS1_PADDING\n )\n handle_openssl_error(signature_length)\n\n return bytes_from_buffer(signature_buffer, signature_length)\n\n finally:\n if rsa:\n libcrypto.RSA_free(rsa)\n\n evp_md_ctx = None\n rsa = None\n dsa = None\n dsa_sig = None\n ec_key = None\n ecdsa_sig = None\n\n try:\n if libcrypto_version_info < (1, 1):\n evp_md_ctx = libcrypto.EVP_MD_CTX_create()\n else:\n evp_md_ctx = libcrypto.EVP_MD_CTX_new()\n\n evp_md = {\n 'md5': libcrypto.EVP_md5,\n 'sha1': libcrypto.EVP_sha1,\n 'sha224': libcrypto.EVP_sha224,\n 'sha256': libcrypto.EVP_sha256,\n 'sha384': libcrypto.EVP_sha384,\n 'sha512': libcrypto.EVP_sha512\n }[hash_algorithm]()\n\n if libcrypto_version_info < (1,):\n if private_key.algorithm == 'rsa' and rsa_pss_padding:\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n rsa = libcrypto.EVP_PKEY_get1_RSA(private_key.evp_pkey)\n if is_null(rsa):\n handle_openssl_error(0)\n\n buffer_size = libcrypto.EVP_PKEY_size(private_key.evp_pkey)\n em_buffer = buffer_from_bytes(buffer_size)\n res = libcrypto.RSA_padding_add_PKCS1_PSS(\n rsa,\n em_buffer,\n digest,\n evp_md,\n LibcryptoConst.EVP_MD_CTX_FLAG_PSS_MDLEN\n )\n handle_openssl_error(res)\n\n signature_buffer = buffer_from_bytes(buffer_size)\n signature_length = libcrypto.RSA_private_encrypt(\n buffer_size,\n em_buffer,\n signature_buffer,\n rsa,\n LibcryptoConst.RSA_NO_PADDING\n )\n handle_openssl_error(signature_length)\n\n elif private_key.algorithm == 'rsa':\n buffer_size = libcrypto.EVP_PKEY_size(private_key.evp_pkey)\n signature_buffer = buffer_from_bytes(buffer_size)\n signature_length = new(libcrypto, 'unsigned int *')\n\n res = libcrypto.EVP_DigestInit_ex(evp_md_ctx, evp_md, null())\n handle_openssl_error(res)\n\n res = libcrypto.EVP_DigestUpdate(evp_md_ctx, data, len(data))\n handle_openssl_error(res)\n\n res = libcrypto.EVP_SignFinal(\n evp_md_ctx,\n signature_buffer,\n signature_length,\n private_key.evp_pkey\n )\n handle_openssl_error(res)\n\n signature_length = deref(signature_length)\n\n elif private_key.algorithm == 'dsa':\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n dsa = libcrypto.EVP_PKEY_get1_DSA(private_key.evp_pkey)\n if is_null(dsa):\n handle_openssl_error(0)\n\n dsa_sig = libcrypto.DSA_do_sign(digest, len(digest), dsa)\n if is_null(dsa_sig):\n handle_openssl_error(0)\n\n buffer_size = libcrypto.i2d_DSA_SIG(dsa_sig, null())\n signature_buffer = buffer_from_bytes(buffer_size)\n signature_pointer = buffer_pointer(signature_buffer)\n signature_length = libcrypto.i2d_DSA_SIG(dsa_sig, signature_pointer)\n handle_openssl_error(signature_length)\n\n elif private_key.algorithm == 'ec':\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n ec_key = libcrypto.EVP_PKEY_get1_EC_KEY(private_key.evp_pkey)\n if is_null(ec_key):\n handle_openssl_error(0)\n\n ecdsa_sig = libcrypto.ECDSA_do_sign(digest, len(digest), ec_key)\n if is_null(ecdsa_sig):\n handle_openssl_error(0)\n\n buffer_size = libcrypto.i2d_ECDSA_SIG(ecdsa_sig, null())\n signature_buffer = buffer_from_bytes(buffer_size)\n signature_pointer = buffer_pointer(signature_buffer)\n signature_length = libcrypto.i2d_ECDSA_SIG(ecdsa_sig, signature_pointer)\n handle_openssl_error(signature_length)\n\n else:\n buffer_size = libcrypto.EVP_PKEY_size(private_key.evp_pkey)\n signature_buffer = buffer_from_bytes(buffer_size)\n signature_length = new(libcrypto, 'size_t *', buffer_size)\n\n evp_pkey_ctx_pointer_pointer = new(libcrypto, 'EVP_PKEY_CTX **')\n res = libcrypto.EVP_DigestSignInit(\n evp_md_ctx,\n evp_pkey_ctx_pointer_pointer,\n evp_md,\n null(),\n private_key.evp_pkey\n )\n handle_openssl_error(res)\n evp_pkey_ctx_pointer = unwrap(evp_pkey_ctx_pointer_pointer)\n\n if rsa_pss_padding:\n # Enable PSS padding\n res = libcrypto.EVP_PKEY_CTX_ctrl(\n evp_pkey_ctx_pointer,\n LibcryptoConst.EVP_PKEY_RSA,\n -1, # All operations\n LibcryptoConst.EVP_PKEY_CTRL_RSA_PADDING,\n LibcryptoConst.RSA_PKCS1_PSS_PADDING,\n null()\n )\n handle_openssl_error(res)\n\n # Use the hash algorithm output length as the salt length\n res = libcrypto.EVP_PKEY_CTX_ctrl(\n evp_pkey_ctx_pointer,\n LibcryptoConst.EVP_PKEY_RSA,\n LibcryptoConst.EVP_PKEY_OP_SIGN | LibcryptoConst.EVP_PKEY_OP_VERIFY,\n LibcryptoConst.EVP_PKEY_CTRL_RSA_PSS_SALTLEN,\n -1,\n null()\n )\n handle_openssl_error(res)\n\n res = libcrypto.EVP_DigestUpdate(evp_md_ctx, data, len(data))\n handle_openssl_error(res)\n\n res = libcrypto.EVP_DigestSignFinal(evp_md_ctx, signature_buffer, signature_length)\n handle_openssl_error(res)\n\n signature_length = deref(signature_length)\n\n return bytes_from_buffer(signature_buffer, signature_length)\n\n finally:\n if evp_md_ctx:\n if libcrypto_version_info < (1, 1):\n libcrypto.EVP_MD_CTX_destroy(evp_md_ctx)\n else:\n libcrypto.EVP_MD_CTX_free(evp_md_ctx)\n if rsa:\n libcrypto.RSA_free(rsa)\n if dsa:\n libcrypto.DSA_free(dsa)\n if dsa_sig:\n libcrypto.DSA_SIG_free(dsa_sig)\n if ec_key:\n libcrypto.EC_KEY_free(ec_key)\n if ecdsa_sig:\n libcrypto.ECDSA_SIG_free(ecdsa_sig)", "response": "Generates a signature for the data and returns the byte string"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the PublicKey object for the public key this certificate contains.", "response": "def public_key(self):\n \"\"\"\n :return:\n The PublicKey object for the public key this certificate contains\n \"\"\"\n\n if not self._public_key and self.x509:\n evp_pkey = libcrypto.X509_get_pubkey(self.x509)\n self._public_key = PublicKey(evp_pkey, self.asn1['tbs_certificate']['subject_public_key_info'])\n\n return self._public_key"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates a public and private key pair for the specified algorithm and bit size.", "response": "def generate_pair(algorithm, bit_size=None, curve=None):\n \"\"\"\n Generates a public/private key pair\n\n :param algorithm:\n The key algorithm - \"rsa\", \"dsa\" or \"ec\"\n\n :param bit_size:\n An integer - used for \"rsa\" and \"dsa\". For \"rsa\" the value maye be 1024,\n 2048, 3072 or 4096. For \"dsa\" the value may be 1024, plus 2048 or 3072\n if on Windows 8 or newer.\n\n :param curve:\n A unicode string - used for \"ec\" keys. Valid values include \"secp256r1\",\n \"secp384r1\" and \"secp521r1\".\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A 2-element tuple of (PublicKey, PrivateKey). The contents of each key\n may be saved by calling .asn1.dump().\n \"\"\"\n\n if algorithm not in set(['rsa', 'dsa', 'ec']):\n raise ValueError(pretty_message(\n '''\n algorithm must be one of \"rsa\", \"dsa\", \"ec\", not %s\n ''',\n repr(algorithm)\n ))\n\n if algorithm == 'rsa':\n if bit_size not in set([1024, 2048, 3072, 4096]):\n raise ValueError(pretty_message(\n '''\n bit_size must be one of 1024, 2048, 3072, 4096, not %s\n ''',\n repr(bit_size)\n ))\n\n elif algorithm == 'dsa':\n # Windows Vista and 7 only support SHA1-based DSA keys\n if _win_version_info < (6, 2) or _backend == 'winlegacy':\n if bit_size != 1024:\n raise ValueError(pretty_message(\n '''\n bit_size must be 1024, not %s\n ''',\n repr(bit_size)\n ))\n else:\n if bit_size not in set([1024, 2048, 3072]):\n raise ValueError(pretty_message(\n '''\n bit_size must be one of 1024, 2048, 3072, not %s\n ''',\n repr(bit_size)\n ))\n\n elif algorithm == 'ec':\n if curve not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise ValueError(pretty_message(\n '''\n curve must be one of \"secp256r1\", \"secp384r1\", \"secp521r1\", not %s\n ''',\n repr(curve)\n ))\n\n if _backend == 'winlegacy':\n if algorithm == 'ec':\n pub_info, priv_info = _pure_python_ec_generate_pair(curve)\n return (PublicKey(None, pub_info), PrivateKey(None, priv_info))\n return _advapi32_generate_pair(algorithm, bit_size)\n else:\n return _bcrypt_generate_pair(algorithm, bit_size, curve)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _advapi32_generate_pair(algorithm, bit_size=None):\n\n if algorithm == 'rsa':\n provider = Advapi32Const.MS_ENH_RSA_AES_PROV\n algorithm_id = Advapi32Const.CALG_RSA_SIGN\n struct_type = 'RSABLOBHEADER'\n else:\n provider = Advapi32Const.MS_ENH_DSS_DH_PROV\n algorithm_id = Advapi32Const.CALG_DSS_SIGN\n struct_type = 'DSSBLOBHEADER'\n\n context_handle = None\n key_handle = None\n\n try:\n context_handle = open_context_handle(provider, verify_only=False)\n\n key_handle_pointer = new(advapi32, 'HCRYPTKEY *')\n flags = (bit_size << 16) | Advapi32Const.CRYPT_EXPORTABLE\n res = advapi32.CryptGenKey(context_handle, algorithm_id, flags, key_handle_pointer)\n handle_error(res)\n\n key_handle = unwrap(key_handle_pointer)\n\n out_len = new(advapi32, 'DWORD *')\n res = advapi32.CryptExportKey(\n key_handle,\n null(),\n Advapi32Const.PRIVATEKEYBLOB,\n 0,\n null(),\n out_len\n )\n handle_error(res)\n\n buffer_length = deref(out_len)\n buffer_ = buffer_from_bytes(buffer_length)\n res = advapi32.CryptExportKey(\n key_handle,\n null(),\n Advapi32Const.PRIVATEKEYBLOB,\n 0,\n buffer_,\n out_len\n )\n handle_error(res)\n\n blob_struct_pointer = struct_from_buffer(advapi32, struct_type, buffer_)\n blob_struct = unwrap(blob_struct_pointer)\n struct_size = sizeof(advapi32, blob_struct)\n\n private_blob = bytes_from_buffer(buffer_, buffer_length)[struct_size:]\n\n if algorithm == 'rsa':\n public_info, private_info = _advapi32_interpret_rsa_key_blob(bit_size, blob_struct, private_blob)\n\n else:\n # The public key for a DSA key is not available in from the private\n # key blob, so we have to separately export the public key\n public_out_len = new(advapi32, 'DWORD *')\n res = advapi32.CryptExportKey(\n key_handle,\n null(),\n Advapi32Const.PUBLICKEYBLOB,\n 0,\n null(),\n public_out_len\n )\n handle_error(res)\n\n public_buffer_length = deref(public_out_len)\n public_buffer = buffer_from_bytes(public_buffer_length)\n res = advapi32.CryptExportKey(\n key_handle,\n null(),\n Advapi32Const.PUBLICKEYBLOB,\n 0,\n public_buffer,\n public_out_len\n )\n handle_error(res)\n\n public_blob = bytes_from_buffer(public_buffer, public_buffer_length)[struct_size:]\n\n public_info, private_info = _advapi32_interpret_dsa_key_blob(bit_size, public_blob, private_blob)\n\n return (load_public_key(public_info), load_private_key(private_info))\n\n finally:\n if context_handle:\n close_context_handle(context_handle)\n if key_handle:\n advapi32.CryptDestroyKey(key_handle)", "response": "Generate a public and private key pair using the specified key algorithm."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _bcrypt_generate_pair(algorithm, bit_size=None, curve=None):\n\n if algorithm == 'rsa':\n alg_constant = BcryptConst.BCRYPT_RSA_ALGORITHM\n struct_type = 'BCRYPT_RSAKEY_BLOB'\n private_blob_type = BcryptConst.BCRYPT_RSAFULLPRIVATE_BLOB\n public_blob_type = BcryptConst.BCRYPT_RSAPUBLIC_BLOB\n\n elif algorithm == 'dsa':\n alg_constant = BcryptConst.BCRYPT_DSA_ALGORITHM\n if bit_size > 1024:\n struct_type = 'BCRYPT_DSA_KEY_BLOB_V2'\n else:\n struct_type = 'BCRYPT_DSA_KEY_BLOB'\n private_blob_type = BcryptConst.BCRYPT_DSA_PRIVATE_BLOB\n public_blob_type = BcryptConst.BCRYPT_DSA_PUBLIC_BLOB\n\n else:\n alg_constant = {\n 'secp256r1': BcryptConst.BCRYPT_ECDSA_P256_ALGORITHM,\n 'secp384r1': BcryptConst.BCRYPT_ECDSA_P384_ALGORITHM,\n 'secp521r1': BcryptConst.BCRYPT_ECDSA_P521_ALGORITHM,\n }[curve]\n bit_size = {\n 'secp256r1': 256,\n 'secp384r1': 384,\n 'secp521r1': 521,\n }[curve]\n struct_type = 'BCRYPT_ECCKEY_BLOB'\n private_blob_type = BcryptConst.BCRYPT_ECCPRIVATE_BLOB\n public_blob_type = BcryptConst.BCRYPT_ECCPUBLIC_BLOB\n\n alg_handle = open_alg_handle(alg_constant)\n key_handle_pointer = new(bcrypt, 'BCRYPT_KEY_HANDLE *')\n res = bcrypt.BCryptGenerateKeyPair(alg_handle, key_handle_pointer, bit_size, 0)\n handle_error(res)\n key_handle = unwrap(key_handle_pointer)\n\n res = bcrypt.BCryptFinalizeKeyPair(key_handle, 0)\n handle_error(res)\n\n private_out_len = new(bcrypt, 'ULONG *')\n res = bcrypt.BCryptExportKey(key_handle, null(), private_blob_type, null(), 0, private_out_len, 0)\n handle_error(res)\n\n private_buffer_length = deref(private_out_len)\n private_buffer = buffer_from_bytes(private_buffer_length)\n res = bcrypt.BCryptExportKey(\n key_handle,\n null(),\n private_blob_type,\n private_buffer,\n private_buffer_length,\n private_out_len,\n 0\n )\n handle_error(res)\n private_blob_struct_pointer = struct_from_buffer(bcrypt, struct_type, private_buffer)\n private_blob_struct = unwrap(private_blob_struct_pointer)\n struct_size = sizeof(bcrypt, private_blob_struct)\n private_blob = bytes_from_buffer(private_buffer, private_buffer_length)[struct_size:]\n\n if algorithm == 'rsa':\n private_key = _bcrypt_interpret_rsa_key_blob('private', private_blob_struct, private_blob)\n elif algorithm == 'dsa':\n if bit_size > 1024:\n private_key = _bcrypt_interpret_dsa_key_blob('private', 2, private_blob_struct, private_blob)\n else:\n private_key = _bcrypt_interpret_dsa_key_blob('private', 1, private_blob_struct, private_blob)\n else:\n private_key = _bcrypt_interpret_ec_key_blob('private', private_blob_struct, private_blob)\n\n public_out_len = new(bcrypt, 'ULONG *')\n res = bcrypt.BCryptExportKey(key_handle, null(), public_blob_type, null(), 0, public_out_len, 0)\n handle_error(res)\n\n public_buffer_length = deref(public_out_len)\n public_buffer = buffer_from_bytes(public_buffer_length)\n res = bcrypt.BCryptExportKey(\n key_handle,\n null(),\n public_blob_type,\n public_buffer,\n public_buffer_length,\n public_out_len,\n 0\n )\n handle_error(res)\n public_blob_struct_pointer = struct_from_buffer(bcrypt, struct_type, public_buffer)\n public_blob_struct = unwrap(public_blob_struct_pointer)\n struct_size = sizeof(bcrypt, public_blob_struct)\n public_blob = bytes_from_buffer(public_buffer, public_buffer_length)[struct_size:]\n\n if algorithm == 'rsa':\n public_key = _bcrypt_interpret_rsa_key_blob('public', public_blob_struct, public_blob)\n elif algorithm == 'dsa':\n if bit_size > 1024:\n public_key = _bcrypt_interpret_dsa_key_blob('public', 2, public_blob_struct, public_blob)\n else:\n public_key = _bcrypt_interpret_dsa_key_blob('public', 1, public_blob_struct, public_blob)\n else:\n public_key = _bcrypt_interpret_ec_key_blob('public', public_blob_struct, public_blob)\n\n return (load_public_key(public_key), load_private_key(private_key))", "response": "Generates a public and private key pair using the specified algorithm."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef generate_dh_parameters(bit_size):\n\n if not isinstance(bit_size, int_types):\n raise TypeError(pretty_message(\n '''\n bit_size must be an integer, not %s\n ''',\n type_name(bit_size)\n ))\n\n if bit_size < 512:\n raise ValueError('bit_size must be greater than or equal to 512')\n\n if bit_size > 4096:\n raise ValueError('bit_size must be less than or equal to 4096')\n\n if bit_size % 64 != 0:\n raise ValueError('bit_size must be a multiple of 64')\n\n alg_handle = None\n\n # The following algorithm has elements taken from OpenSSL. In short, it\n # generates random numbers and then ensures that they are valid for the\n # hardcoded generator of 2, and then ensures the number is a \"safe\" prime\n # by ensuring p//2 is prime also.\n\n # OpenSSL allows use of generator 2 or 5, but we hardcode 2 since it is\n # the default, and what is used by Security.framework on OS X also.\n g = 2\n\n try:\n byte_size = bit_size // 8\n if _backend == 'win':\n alg_handle = open_alg_handle(BcryptConst.BCRYPT_RNG_ALGORITHM)\n buffer = buffer_from_bytes(byte_size)\n\n while True:\n if _backend == 'winlegacy':\n rb = os.urandom(byte_size)\n else:\n res = bcrypt.BCryptGenRandom(alg_handle, buffer, byte_size, 0)\n handle_error(res)\n rb = bytes_from_buffer(buffer)\n\n p = int_from_bytes(rb)\n\n # If a number is even, it can't be prime\n if p % 2 == 0:\n continue\n\n # Perform the generator checks outlined in OpenSSL's\n # dh_builtin_genparams() located in dh_gen.c\n if g == 2:\n if p % 24 != 11:\n continue\n elif g == 5:\n rem = p % 10\n if rem != 3 and rem != 7:\n continue\n\n divisible = False\n for prime in _SMALL_PRIMES:\n if p % prime == 0:\n divisible = True\n break\n\n # If the number is not divisible by any of the small primes, then\n # move on to the full Miller-Rabin test.\n if not divisible and _is_prime(bit_size, p):\n q = p // 2\n if _is_prime(bit_size, q):\n return algos.DHParameters({'p': p, 'g': g})\n\n finally:\n if alg_handle:\n close_alg_handle(alg_handle)", "response": "Generates DH parameters for use with Diffie - Hellman key exchange."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _is_prime(bit_size, n):\n\n r = 0\n s = n - 1\n while s % 2 == 0:\n r += 1\n s //= 2\n\n if bit_size >= 1300:\n k = 2\n elif bit_size >= 850:\n k = 3\n elif bit_size >= 650:\n k = 4\n elif bit_size >= 550:\n k = 5\n elif bit_size >= 450:\n k = 6\n\n for _ in range(k):\n a = random.randrange(2, n - 1)\n x = pow(a, s, n)\n if x == 1 or x == n - 1:\n continue\n for _ in range(r - 1):\n x = pow(x, 2, n)\n if x == n - 1:\n break\n else:\n return False\n\n return True", "response": "A function that checks if a number is prime."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _advapi32_interpret_rsa_key_blob(bit_size, blob_struct, blob):\n\n len1 = bit_size // 8\n len2 = bit_size // 16\n\n prime1_offset = len1\n prime2_offset = prime1_offset + len2\n exponent1_offset = prime2_offset + len2\n exponent2_offset = exponent1_offset + len2\n coefficient_offset = exponent2_offset + len2\n private_exponent_offset = coefficient_offset + len2\n\n public_exponent = blob_struct.rsapubkey.pubexp\n modulus = int_from_bytes(blob[0:prime1_offset][::-1])\n prime1 = int_from_bytes(blob[prime1_offset:prime2_offset][::-1])\n prime2 = int_from_bytes(blob[prime2_offset:exponent1_offset][::-1])\n exponent1 = int_from_bytes(blob[exponent1_offset:exponent2_offset][::-1])\n exponent2 = int_from_bytes(blob[exponent2_offset:coefficient_offset][::-1])\n coefficient = int_from_bytes(blob[coefficient_offset:private_exponent_offset][::-1])\n private_exponent = int_from_bytes(blob[private_exponent_offset:private_exponent_offset + len1][::-1])\n\n public_key_info = keys.PublicKeyInfo({\n 'algorithm': keys.PublicKeyAlgorithm({\n 'algorithm': 'rsa',\n }),\n 'public_key': keys.RSAPublicKey({\n 'modulus': modulus,\n 'public_exponent': public_exponent,\n }),\n })\n\n rsa_private_key = keys.RSAPrivateKey({\n 'version': 'two-prime',\n 'modulus': modulus,\n 'public_exponent': public_exponent,\n 'private_exponent': private_exponent,\n 'prime1': prime1,\n 'prime2': prime2,\n 'exponent1': exponent1,\n 'exponent2': exponent2,\n 'coefficient': coefficient,\n })\n\n private_key_info = keys.PrivateKeyInfo({\n 'version': 0,\n 'private_key_algorithm': keys.PrivateKeyAlgorithm({\n 'algorithm': 'rsa',\n }),\n 'private_key': rsa_private_key,\n })\n\n return (public_key_info, private_key_info)", "response": "Interprets a CryptoAPI RSA private key blob and returns a 2 - element tuple of the ASN. 1 structure for the public and private keys."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntaking a CryptoAPI DSS public key blob and converts it into the ASN. 1 structure for the public and private keys.", "response": "def _advapi32_interpret_dsa_key_blob(bit_size, public_blob, private_blob):\n \"\"\"\n Takes a CryptoAPI DSS private key blob and converts it into the ASN.1\n structures for the public and private keys\n\n :param bit_size:\n The integer bit size of the key\n\n :param public_blob:\n A byte string of the binary data after the public key header\n\n :param private_blob:\n A byte string of the binary data after the private key header\n\n :return:\n A 2-element tuple of (asn1crypto.keys.PublicKeyInfo,\n asn1crypto.keys.PrivateKeyInfo)\n \"\"\"\n\n len1 = 20\n len2 = bit_size // 8\n\n q_offset = len2\n g_offset = q_offset + len1\n x_offset = g_offset + len2\n y_offset = x_offset\n\n p = int_from_bytes(private_blob[0:q_offset][::-1])\n q = int_from_bytes(private_blob[q_offset:g_offset][::-1])\n g = int_from_bytes(private_blob[g_offset:x_offset][::-1])\n x = int_from_bytes(private_blob[x_offset:x_offset + len1][::-1])\n y = int_from_bytes(public_blob[y_offset:y_offset + len2][::-1])\n\n public_key_info = keys.PublicKeyInfo({\n 'algorithm': keys.PublicKeyAlgorithm({\n 'algorithm': 'dsa',\n 'parameters': keys.DSAParams({\n 'p': p,\n 'q': q,\n 'g': g,\n })\n }),\n 'public_key': core.Integer(y),\n })\n\n private_key_info = keys.PrivateKeyInfo({\n 'version': 0,\n 'private_key_algorithm': keys.PrivateKeyAlgorithm({\n 'algorithm': 'dsa',\n 'parameters': keys.DSAParams({\n 'p': p,\n 'q': q,\n 'g': g,\n })\n }),\n 'private_key': core.Integer(x),\n })\n\n return (public_key_info, private_key_info)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _bcrypt_interpret_rsa_key_blob(key_type, blob_struct, blob):\n\n public_exponent_byte_length = native(int, blob_struct.cbPublicExp)\n modulus_byte_length = native(int, blob_struct.cbModulus)\n\n modulus_offset = public_exponent_byte_length\n\n public_exponent = int_from_bytes(blob[0:modulus_offset])\n modulus = int_from_bytes(blob[modulus_offset:modulus_offset + modulus_byte_length])\n\n if key_type == 'public':\n return keys.PublicKeyInfo({\n 'algorithm': keys.PublicKeyAlgorithm({\n 'algorithm': 'rsa',\n }),\n 'public_key': keys.RSAPublicKey({\n 'modulus': modulus,\n 'public_exponent': public_exponent,\n }),\n })\n\n elif key_type == 'private':\n prime1_byte_length = native(int, blob_struct.cbPrime1)\n prime2_byte_length = native(int, blob_struct.cbPrime2)\n\n prime1_offset = modulus_offset + modulus_byte_length\n prime2_offset = prime1_offset + prime1_byte_length\n exponent1_offset = prime2_offset + prime2_byte_length\n exponent2_offset = exponent1_offset + prime2_byte_length\n coefficient_offset = exponent2_offset + prime2_byte_length\n private_exponent_offset = coefficient_offset + prime1_byte_length\n\n prime1 = int_from_bytes(blob[prime1_offset:prime2_offset])\n prime2 = int_from_bytes(blob[prime2_offset:exponent1_offset])\n exponent1 = int_from_bytes(blob[exponent1_offset:exponent2_offset])\n exponent2 = int_from_bytes(blob[exponent2_offset:coefficient_offset])\n coefficient = int_from_bytes(blob[coefficient_offset:private_exponent_offset])\n private_exponent = int_from_bytes(blob[private_exponent_offset:private_exponent_offset + modulus_byte_length])\n\n rsa_private_key = keys.RSAPrivateKey({\n 'version': 'two-prime',\n 'modulus': modulus,\n 'public_exponent': public_exponent,\n 'private_exponent': private_exponent,\n 'prime1': prime1,\n 'prime2': prime2,\n 'exponent1': exponent1,\n 'exponent2': exponent2,\n 'coefficient': coefficient,\n })\n\n return keys.PrivateKeyInfo({\n 'version': 0,\n 'private_key_algorithm': keys.PrivateKeyAlgorithm({\n 'algorithm': 'rsa',\n }),\n 'private_key': rsa_private_key,\n })\n\n else:\n raise ValueError(pretty_message(\n '''\n key_type must be one of \"public\", \"private\", not %s\n ''',\n repr(key_type)\n ))", "response": "Interprets a BCRYPT_RSAFULLPRIVATE_BLOB and returns a ASN. 1 structure of the correct type."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _bcrypt_interpret_dsa_key_blob(key_type, version, blob_struct, blob):\n\n key_byte_length = native(int, blob_struct.cbKey)\n\n if version == 1:\n q = int_from_bytes(native(byte_cls, blob_struct.q))\n\n g_offset = key_byte_length\n public_offset = g_offset + key_byte_length\n private_offset = public_offset + key_byte_length\n\n p = int_from_bytes(blob[0:g_offset])\n g = int_from_bytes(blob[g_offset:public_offset])\n\n elif version == 2:\n seed_byte_length = native(int, blob_struct.cbSeedLength)\n group_byte_length = native(int, blob_struct.cbGroupSize)\n\n q_offset = seed_byte_length\n p_offset = q_offset + group_byte_length\n g_offset = p_offset + key_byte_length\n public_offset = g_offset + key_byte_length\n private_offset = public_offset + key_byte_length\n\n # The seed is skipped since it is not part of the ASN.1 structure\n q = int_from_bytes(blob[q_offset:p_offset])\n p = int_from_bytes(blob[p_offset:g_offset])\n g = int_from_bytes(blob[g_offset:public_offset])\n\n else:\n raise ValueError('version must be 1 or 2, not %s' % repr(version))\n\n if key_type == 'public':\n public = int_from_bytes(blob[public_offset:private_offset])\n return keys.PublicKeyInfo({\n 'algorithm': keys.PublicKeyAlgorithm({\n 'algorithm': 'dsa',\n 'parameters': keys.DSAParams({\n 'p': p,\n 'q': q,\n 'g': g,\n })\n }),\n 'public_key': core.Integer(public),\n })\n\n elif key_type == 'private':\n private = int_from_bytes(blob[private_offset:private_offset + key_byte_length])\n return keys.PrivateKeyInfo({\n 'version': 0,\n 'private_key_algorithm': keys.PrivateKeyAlgorithm({\n 'algorithm': 'dsa',\n 'parameters': keys.DSAParams({\n 'p': p,\n 'q': q,\n 'g': g,\n })\n }),\n 'private_key': core.Integer(private),\n })\n\n else:\n raise ValueError(pretty_message(\n '''\n key_type must be one of \"public\", \"private\", not %s\n ''',\n repr(key_type)\n ))", "response": "Interprets a DSA key blob into a set of ASN. 1 structure."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ninterpreting a CNG BCRYPT_ECCKEY_BLOB and returns a asn1crypto. keys. PrivateKeyInfo or asn1crypto. keys. PublicKeyInfo object based on the key_type param.", "response": "def _bcrypt_interpret_ec_key_blob(key_type, blob_struct, blob):\n \"\"\"\n Take a CNG BCRYPT_ECCKEY_BLOB and converts it into an ASN.1 structure\n\n :param key_type:\n A unicode string of \"private\" or \"public\"\n\n :param blob_struct:\n An instance of BCRYPT_ECCKEY_BLOB\n\n :param blob:\n A byte string of the binary data contained after the struct\n\n :return:\n An asn1crypto.keys.PrivateKeyInfo or asn1crypto.keys.PublicKeyInfo\n object, based on the key_type param\n \"\"\"\n\n magic = native(int, blob_struct.dwMagic)\n key_byte_length = native(int, blob_struct.cbKey)\n\n curve = {\n BcryptConst.BCRYPT_ECDSA_PRIVATE_P256_MAGIC: 'secp256r1',\n BcryptConst.BCRYPT_ECDSA_PRIVATE_P384_MAGIC: 'secp384r1',\n BcryptConst.BCRYPT_ECDSA_PRIVATE_P521_MAGIC: 'secp521r1',\n BcryptConst.BCRYPT_ECDSA_PUBLIC_P256_MAGIC: 'secp256r1',\n BcryptConst.BCRYPT_ECDSA_PUBLIC_P384_MAGIC: 'secp384r1',\n BcryptConst.BCRYPT_ECDSA_PUBLIC_P521_MAGIC: 'secp521r1',\n }[magic]\n\n public = b'\\x04' + blob[0:key_byte_length * 2]\n\n if key_type == 'public':\n return keys.PublicKeyInfo({\n 'algorithm': keys.PublicKeyAlgorithm({\n 'algorithm': 'ec',\n 'parameters': keys.ECDomainParameters(\n name='named',\n value=curve\n )\n }),\n 'public_key': public,\n })\n\n elif key_type == 'private':\n private = int_from_bytes(blob[key_byte_length * 2:key_byte_length * 3])\n return keys.PrivateKeyInfo({\n 'version': 0,\n 'private_key_algorithm': keys.PrivateKeyAlgorithm({\n 'algorithm': 'ec',\n 'parameters': keys.ECDomainParameters(\n name='named',\n value=curve\n )\n }),\n 'private_key': keys.ECPrivateKey({\n 'version': 'ecPrivkeyVer1',\n 'private_key': private,\n 'public_key': public,\n }),\n })\n\n else:\n raise ValueError(pretty_message(\n '''\n key_type must be one of \"public\", \"private\", not %s\n ''',\n repr(key_type)\n ))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nload a certificate public key or private key into a PrivateKey PublicKey or Certificate object based on the container.", "response": "def _load_key(key_object, container):\n \"\"\"\n Loads a certificate, public key or private key into a Certificate,\n PublicKey or PrivateKey object\n\n :param key_object:\n An asn1crypto.x509.Certificate, asn1crypto.keys.PublicKeyInfo or\n asn1crypto.keys.PrivateKeyInfo object\n\n :param container:\n The class of the object to hold the key_handle\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n oscrypto.errors.AsymmetricKeyError - when the key is incompatible with the OS crypto library\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A PrivateKey, PublicKey or Certificate object, based on container\n \"\"\"\n\n key_info = key_object\n if isinstance(key_object, x509.Certificate):\n key_info = key_object['tbs_certificate']['subject_public_key_info']\n\n algo = key_info.algorithm\n curve_name = None\n\n if algo == 'ec':\n curve_type, curve_name = key_info.curve\n if curve_type != 'named':\n raise AsymmetricKeyError(pretty_message(\n '''\n Windows only supports EC keys using named curves\n '''\n ))\n if curve_name not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise AsymmetricKeyError(pretty_message(\n '''\n Windows only supports EC keys using the named curves\n secp256r1, secp384r1 and secp521r1\n '''\n ))\n\n elif algo == 'dsa':\n if key_info.hash_algo is None:\n raise IncompleteAsymmetricKeyError(pretty_message(\n '''\n The DSA key does not contain the necessary p, q and g\n parameters and can not be used\n '''\n ))\n elif key_info.bit_size > 1024 and (_win_version_info < (6, 2) or _backend == 'winlegacy'):\n raise AsymmetricKeyError(pretty_message(\n '''\n Windows XP, 2003, Vista, 7 and Server 2008 only support DSA\n keys based on SHA1 (1024 bits or less) - this key is based\n on %s and is %s bits\n ''',\n key_info.hash_algo.upper(),\n key_info.bit_size\n ))\n elif key_info.bit_size == 2048 and key_info.hash_algo == 'sha1':\n raise AsymmetricKeyError(pretty_message(\n '''\n Windows only supports 2048 bit DSA keys based on SHA2 - this\n key is 2048 bits and based on SHA1, a non-standard\n combination that is usually generated by old versions\n of OpenSSL\n '''\n ))\n\n if _backend == 'winlegacy':\n if algo == 'ec':\n return container(None, key_object)\n return _advapi32_load_key(key_object, key_info, container)\n return _bcrypt_load_key(key_object, key_info, container, curve_name)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nload a certificate public key or private key into a PrivateKey PublicKey or Certificate object based on the container.", "response": "def _advapi32_load_key(key_object, key_info, container):\n \"\"\"\n Loads a certificate, public key or private key into a Certificate,\n PublicKey or PrivateKey object via CryptoAPI\n\n :param key_object:\n An asn1crypto.x509.Certificate, asn1crypto.keys.PublicKeyInfo or\n asn1crypto.keys.PrivateKeyInfo object\n\n :param key_info:\n An asn1crypto.keys.PublicKeyInfo or asn1crypto.keys.PrivateKeyInfo\n object\n\n :param container:\n The class of the object to hold the key_handle\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n oscrypto.errors.AsymmetricKeyError - when the key is incompatible with the OS crypto library\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A PrivateKey, PublicKey or Certificate object, based on container\n \"\"\"\n\n key_type = 'public' if isinstance(key_info, keys.PublicKeyInfo) else 'private'\n algo = key_info.algorithm\n\n if algo == 'rsa':\n provider = Advapi32Const.MS_ENH_RSA_AES_PROV\n else:\n provider = Advapi32Const.MS_ENH_DSS_DH_PROV\n\n context_handle = None\n key_handle = None\n\n try:\n context_handle = open_context_handle(provider, verify_only=key_type == 'public')\n\n blob = _advapi32_create_blob(key_info, key_type, algo)\n buffer_ = buffer_from_bytes(blob)\n\n key_handle_pointer = new(advapi32, 'HCRYPTKEY *')\n res = advapi32.CryptImportKey(\n context_handle,\n buffer_,\n len(blob),\n null(),\n 0,\n key_handle_pointer\n )\n handle_error(res)\n\n key_handle = unwrap(key_handle_pointer)\n output = container(key_handle, key_object)\n output.context_handle = context_handle\n\n if algo == 'rsa':\n ex_blob = _advapi32_create_blob(key_info, key_type, algo, signing=False)\n ex_buffer = buffer_from_bytes(ex_blob)\n\n ex_key_handle_pointer = new(advapi32, 'HCRYPTKEY *')\n res = advapi32.CryptImportKey(\n context_handle,\n ex_buffer,\n len(ex_blob),\n null(),\n 0,\n ex_key_handle_pointer\n )\n handle_error(res)\n\n output.ex_key_handle = unwrap(ex_key_handle_pointer)\n\n return output\n\n except (Exception):\n if key_handle:\n advapi32.CryptDestroyKey(key_handle)\n if context_handle:\n close_context_handle(context_handle)\n raise"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating a blob for importing a key to CryptoAPI.", "response": "def _advapi32_create_blob(key_info, key_type, algo, signing=True):\n \"\"\"\n Generates a blob for importing a key to CryptoAPI\n\n :param key_info:\n An asn1crypto.keys.PublicKeyInfo or asn1crypto.keys.PrivateKeyInfo\n object\n\n :param key_type:\n A unicode string of \"public\" or \"private\"\n\n :param algo:\n A unicode string of \"rsa\" or \"dsa\"\n\n :param signing:\n If the key handle is for signing - may only be False for rsa keys\n\n :return:\n A byte string of a blob to pass to advapi32.CryptImportKey()\n \"\"\"\n\n if key_type == 'public':\n blob_type = Advapi32Const.PUBLICKEYBLOB\n else:\n blob_type = Advapi32Const.PRIVATEKEYBLOB\n\n if algo == 'rsa':\n struct_type = 'RSABLOBHEADER'\n if signing:\n algorithm_id = Advapi32Const.CALG_RSA_SIGN\n else:\n algorithm_id = Advapi32Const.CALG_RSA_KEYX\n else:\n struct_type = 'DSSBLOBHEADER'\n algorithm_id = Advapi32Const.CALG_DSS_SIGN\n\n blob_header_pointer = struct(advapi32, 'BLOBHEADER')\n blob_header = unwrap(blob_header_pointer)\n blob_header.bType = blob_type\n blob_header.bVersion = Advapi32Const.CUR_BLOB_VERSION\n blob_header.reserved = 0\n blob_header.aiKeyAlg = algorithm_id\n\n blob_struct_pointer = struct(advapi32, struct_type)\n blob_struct = unwrap(blob_struct_pointer)\n blob_struct.publickeystruc = blob_header\n\n bit_size = key_info.bit_size\n len1 = bit_size // 8\n len2 = bit_size // 16\n\n if algo == 'rsa':\n pubkey_pointer = struct(advapi32, 'RSAPUBKEY')\n pubkey = unwrap(pubkey_pointer)\n pubkey.bitlen = bit_size\n if key_type == 'public':\n parsed_key_info = key_info['public_key'].parsed\n pubkey.magic = Advapi32Const.RSA1\n pubkey.pubexp = parsed_key_info['public_exponent'].native\n blob_data = int_to_bytes(parsed_key_info['modulus'].native, signed=False, width=len1)[::-1]\n else:\n parsed_key_info = key_info['private_key'].parsed\n pubkey.magic = Advapi32Const.RSA2\n pubkey.pubexp = parsed_key_info['public_exponent'].native\n blob_data = int_to_bytes(parsed_key_info['modulus'].native, signed=False, width=len1)[::-1]\n blob_data += int_to_bytes(parsed_key_info['prime1'].native, signed=False, width=len2)[::-1]\n blob_data += int_to_bytes(parsed_key_info['prime2'].native, signed=False, width=len2)[::-1]\n blob_data += int_to_bytes(parsed_key_info['exponent1'].native, signed=False, width=len2)[::-1]\n blob_data += int_to_bytes(parsed_key_info['exponent2'].native, signed=False, width=len2)[::-1]\n blob_data += int_to_bytes(parsed_key_info['coefficient'].native, signed=False, width=len2)[::-1]\n blob_data += int_to_bytes(parsed_key_info['private_exponent'].native, signed=False, width=len1)[::-1]\n blob_struct.rsapubkey = pubkey\n\n else:\n pubkey_pointer = struct(advapi32, 'DSSPUBKEY')\n pubkey = unwrap(pubkey_pointer)\n pubkey.bitlen = bit_size\n\n if key_type == 'public':\n pubkey.magic = Advapi32Const.DSS1\n params = key_info['algorithm']['parameters'].native\n key_data = int_to_bytes(key_info['public_key'].parsed.native, signed=False, width=len1)[::-1]\n else:\n pubkey.magic = Advapi32Const.DSS2\n params = key_info['private_key_algorithm']['parameters'].native\n key_data = int_to_bytes(key_info['private_key'].parsed.native, signed=False, width=20)[::-1]\n blob_struct.dsspubkey = pubkey\n\n blob_data = int_to_bytes(params['p'], signed=False, width=len1)[::-1]\n blob_data += int_to_bytes(params['q'], signed=False, width=20)[::-1]\n blob_data += int_to_bytes(params['g'], signed=False, width=len1)[::-1]\n blob_data += key_data\n\n dssseed_pointer = struct(advapi32, 'DSSSEED')\n dssseed = unwrap(dssseed_pointer)\n # This indicates no counter or seed info is available\n dssseed.counter = 0xffffffff\n\n blob_data += struct_bytes(dssseed_pointer)\n\n return struct_bytes(blob_struct_pointer) + blob_data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload a certificate public key or private key into a PrivateKey PublicKey or Certificate object based on the container.", "response": "def _bcrypt_load_key(key_object, key_info, container, curve_name):\n \"\"\"\n Loads a certificate, public key or private key into a Certificate,\n PublicKey or PrivateKey object via CNG\n\n :param key_object:\n An asn1crypto.x509.Certificate, asn1crypto.keys.PublicKeyInfo or\n asn1crypto.keys.PrivateKeyInfo object\n\n :param key_info:\n An asn1crypto.keys.PublicKeyInfo or asn1crypto.keys.PrivateKeyInfo\n object\n\n :param container:\n The class of the object to hold the key_handle\n\n :param curve_name:\n None or a unicode string of the curve name for an EC key\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n oscrypto.errors.AsymmetricKeyError - when the key is incompatible with the OS crypto library\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A PrivateKey, PublicKey or Certificate object, based on container\n \"\"\"\n\n alg_handle = None\n key_handle = None\n\n key_type = 'public' if isinstance(key_info, keys.PublicKeyInfo) else 'private'\n algo = key_info.algorithm\n\n try:\n alg_selector = key_info.curve[1] if algo == 'ec' else algo\n alg_constant = {\n 'rsa': BcryptConst.BCRYPT_RSA_ALGORITHM,\n 'dsa': BcryptConst.BCRYPT_DSA_ALGORITHM,\n 'secp256r1': BcryptConst.BCRYPT_ECDSA_P256_ALGORITHM,\n 'secp384r1': BcryptConst.BCRYPT_ECDSA_P384_ALGORITHM,\n 'secp521r1': BcryptConst.BCRYPT_ECDSA_P521_ALGORITHM,\n }[alg_selector]\n alg_handle = open_alg_handle(alg_constant)\n\n if algo == 'rsa':\n if key_type == 'public':\n blob_type = BcryptConst.BCRYPT_RSAPUBLIC_BLOB\n magic = BcryptConst.BCRYPT_RSAPUBLIC_MAGIC\n parsed_key = key_info['public_key'].parsed\n prime1_size = 0\n prime2_size = 0\n else:\n blob_type = BcryptConst.BCRYPT_RSAFULLPRIVATE_BLOB\n magic = BcryptConst.BCRYPT_RSAFULLPRIVATE_MAGIC\n parsed_key = key_info['private_key'].parsed\n prime1 = int_to_bytes(parsed_key['prime1'].native)\n prime2 = int_to_bytes(parsed_key['prime2'].native)\n exponent1 = int_to_bytes(parsed_key['exponent1'].native)\n exponent2 = int_to_bytes(parsed_key['exponent2'].native)\n coefficient = int_to_bytes(parsed_key['coefficient'].native)\n private_exponent = int_to_bytes(parsed_key['private_exponent'].native)\n prime1_size = len(prime1)\n prime2_size = len(prime2)\n\n public_exponent = int_to_bytes(parsed_key['public_exponent'].native)\n modulus = int_to_bytes(parsed_key['modulus'].native)\n\n blob_struct_pointer = struct(bcrypt, 'BCRYPT_RSAKEY_BLOB')\n blob_struct = unwrap(blob_struct_pointer)\n blob_struct.Magic = magic\n blob_struct.BitLength = key_info.bit_size\n blob_struct.cbPublicExp = len(public_exponent)\n blob_struct.cbModulus = len(modulus)\n blob_struct.cbPrime1 = prime1_size\n blob_struct.cbPrime2 = prime2_size\n\n blob = struct_bytes(blob_struct_pointer) + public_exponent + modulus\n if key_type == 'private':\n blob += prime1 + prime2\n blob += fill_width(exponent1, prime1_size)\n blob += fill_width(exponent2, prime2_size)\n blob += fill_width(coefficient, prime1_size)\n blob += fill_width(private_exponent, len(modulus))\n\n elif algo == 'dsa':\n if key_type == 'public':\n blob_type = BcryptConst.BCRYPT_DSA_PUBLIC_BLOB\n public_key = key_info['public_key'].parsed.native\n params = key_info['algorithm']['parameters']\n else:\n blob_type = BcryptConst.BCRYPT_DSA_PRIVATE_BLOB\n public_key = key_info.public_key.native\n private_bytes = int_to_bytes(key_info['private_key'].parsed.native)\n params = key_info['private_key_algorithm']['parameters']\n\n public_bytes = int_to_bytes(public_key)\n p = int_to_bytes(params['p'].native)\n g = int_to_bytes(params['g'].native)\n q = int_to_bytes(params['q'].native)\n\n if key_info.bit_size > 1024:\n q_len = len(q)\n else:\n q_len = 20\n\n key_width = max(len(public_bytes), len(g), len(p))\n\n public_bytes = fill_width(public_bytes, key_width)\n p = fill_width(p, key_width)\n g = fill_width(g, key_width)\n q = fill_width(q, q_len)\n # We don't know the count or seed, so we set them to the max value\n # since setting them to 0 results in a parameter error\n count = b'\\xff' * 4\n seed = b'\\xff' * q_len\n\n if key_info.bit_size > 1024:\n if key_type == 'public':\n magic = BcryptConst.BCRYPT_DSA_PUBLIC_MAGIC_V2\n else:\n magic = BcryptConst.BCRYPT_DSA_PRIVATE_MAGIC_V2\n\n blob_struct_pointer = struct(bcrypt, 'BCRYPT_DSA_KEY_BLOB_V2')\n blob_struct = unwrap(blob_struct_pointer)\n blob_struct.dwMagic = magic\n blob_struct.cbKey = key_width\n # We don't know if SHA256 was used here, but the output is long\n # enough for the generation of q for the supported 2048/224,\n # 2048/256 and 3072/256 FIPS approved pairs\n blob_struct.hashAlgorithm = BcryptConst.DSA_HASH_ALGORITHM_SHA256\n blob_struct.standardVersion = BcryptConst.DSA_FIPS186_3\n blob_struct.cbSeedLength = q_len\n blob_struct.cbGroupSize = q_len\n blob_struct.Count = byte_array(count)\n\n blob = struct_bytes(blob_struct_pointer)\n blob += seed + q + p + g + public_bytes\n if key_type == 'private':\n blob += fill_width(private_bytes, q_len)\n\n else:\n if key_type == 'public':\n magic = BcryptConst.BCRYPT_DSA_PUBLIC_MAGIC\n else:\n magic = BcryptConst.BCRYPT_DSA_PRIVATE_MAGIC\n\n blob_struct_pointer = struct(bcrypt, 'BCRYPT_DSA_KEY_BLOB')\n blob_struct = unwrap(blob_struct_pointer)\n blob_struct.dwMagic = magic\n blob_struct.cbKey = key_width\n blob_struct.Count = byte_array(count)\n blob_struct.Seed = byte_array(seed)\n blob_struct.q = byte_array(q)\n\n blob = struct_bytes(blob_struct_pointer) + p + g + public_bytes\n if key_type == 'private':\n blob += fill_width(private_bytes, q_len)\n\n elif algo == 'ec':\n if key_type == 'public':\n blob_type = BcryptConst.BCRYPT_ECCPUBLIC_BLOB\n public_key = key_info['public_key']\n else:\n blob_type = BcryptConst.BCRYPT_ECCPRIVATE_BLOB\n public_key = key_info.public_key\n private_bytes = int_to_bytes(key_info['private_key'].parsed['private_key'].native)\n\n blob_struct_pointer = struct(bcrypt, 'BCRYPT_ECCKEY_BLOB')\n blob_struct = unwrap(blob_struct_pointer)\n\n magic = {\n ('public', 'secp256r1'): BcryptConst.BCRYPT_ECDSA_PUBLIC_P256_MAGIC,\n ('public', 'secp384r1'): BcryptConst.BCRYPT_ECDSA_PUBLIC_P384_MAGIC,\n ('public', 'secp521r1'): BcryptConst.BCRYPT_ECDSA_PUBLIC_P521_MAGIC,\n ('private', 'secp256r1'): BcryptConst.BCRYPT_ECDSA_PRIVATE_P256_MAGIC,\n ('private', 'secp384r1'): BcryptConst.BCRYPT_ECDSA_PRIVATE_P384_MAGIC,\n ('private', 'secp521r1'): BcryptConst.BCRYPT_ECDSA_PRIVATE_P521_MAGIC,\n }[(key_type, curve_name)]\n\n key_width = {\n 'secp256r1': 32,\n 'secp384r1': 48,\n 'secp521r1': 66\n }[curve_name]\n\n x, y = public_key.to_coords()\n\n x_bytes = int_to_bytes(x)\n y_bytes = int_to_bytes(y)\n\n x_bytes = fill_width(x_bytes, key_width)\n y_bytes = fill_width(y_bytes, key_width)\n\n blob_struct.dwMagic = magic\n blob_struct.cbKey = key_width\n\n blob = struct_bytes(blob_struct_pointer) + x_bytes + y_bytes\n if key_type == 'private':\n blob += fill_width(private_bytes, key_width)\n\n key_handle_pointer = new(bcrypt, 'BCRYPT_KEY_HANDLE *')\n res = bcrypt.BCryptImportKeyPair(\n alg_handle,\n null(),\n blob_type,\n key_handle_pointer,\n blob,\n len(blob),\n BcryptConst.BCRYPT_NO_KEY_VALIDATION\n )\n handle_error(res)\n\n key_handle = unwrap(key_handle_pointer)\n return container(key_handle, key_object)\n\n finally:\n if alg_handle:\n close_alg_handle(alg_handle)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads a public key into a list of PublicKey objects.", "response": "def load_public_key(source):\n \"\"\"\n Loads a public key into a PublicKey object\n\n :param source:\n A byte string of file contents, a unicode string filename or an\n asn1crypto.keys.PublicKeyInfo object\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n oscrypto.errors.AsymmetricKeyError - when the public key is incompatible with the OS crypto library\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A PublicKey object\n \"\"\"\n\n if isinstance(source, keys.PublicKeyInfo):\n public_key = source\n\n elif isinstance(source, byte_cls):\n public_key = parse_public(source)\n\n elif isinstance(source, str_cls):\n with open(source, 'rb') as f:\n public_key = parse_public(f.read())\n\n else:\n raise TypeError(pretty_message(\n '''\n source must be a byte string, unicode string or\n asn1crypto.keys.PublicKeyInfo object, not %s\n ''',\n type_name(public_key)\n ))\n\n return _load_key(public_key, PublicKey)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nload a. p12 or. pfx file into a PrivateKey object and one or more Certificates objects.", "response": "def load_pkcs12(source, password=None):\n \"\"\"\n Loads a .p12 or .pfx file into a PrivateKey object and one or more\n Certificates objects\n\n :param source:\n A byte string of file contents or a unicode string filename\n\n :param password:\n A byte or unicode string to decrypt the PKCS12 file. Unicode strings\n will be encoded using UTF-8.\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n oscrypto.errors.AsymmetricKeyError - when a contained key is incompatible with the OS crypto library\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A three-element tuple containing (PrivateKey, Certificate, [Certificate, ...])\n \"\"\"\n\n if password is not None:\n if isinstance(password, str_cls):\n password = password.encode('utf-8')\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n\n if isinstance(source, str_cls):\n with open(source, 'rb') as f:\n source = f.read()\n\n elif not isinstance(source, byte_cls):\n raise TypeError(pretty_message(\n '''\n source must be a byte string or a unicode string, not %s\n ''',\n type_name(source)\n ))\n\n key_info, cert_info, extra_certs_info = parse_pkcs12(source, password)\n\n key = None\n cert = None\n\n if key_info:\n key = _load_key(key_info, PrivateKey)\n\n if cert_info:\n cert = _load_key(cert_info.public_key, Certificate)\n\n extra_certs = [_load_key(info.public_key, Certificate) for info in extra_certs_info]\n\n return (key, cert, extra_certs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rsa_pkcs1v15_verify(certificate_or_public_key, signature, data, hash_algorithm):\n\n if certificate_or_public_key.algorithm != 'rsa':\n raise ValueError('The key specified is not an RSA public key')\n\n return _verify(certificate_or_public_key, signature, data, hash_algorithm)", "response": "Verifies a RSA - PKCS - v1. 5 signature."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nverifying an RSASSA - PSS signature.", "response": "def rsa_pss_verify(certificate_or_public_key, signature, data, hash_algorithm):\n \"\"\"\n Verifies an RSASSA-PSS signature. For the PSS padding the mask gen algorithm\n will be mgf1 using the same hash algorithm as the signature. The salt length\n with be the length of the hash algorithm, and the trailer field with be the\n standard 0xBC byte.\n\n :param certificate_or_public_key:\n A Certificate or PublicKey instance to verify the signature with\n\n :param signature:\n A byte string of the signature to verify\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha256\", \"sha384\" or \"sha512\"\n\n :raises:\n oscrypto.errors.SignatureError - when the signature is determined to be invalid\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n \"\"\"\n\n if certificate_or_public_key.algorithm != 'rsa':\n raise ValueError('The key specified is not an RSA public key')\n\n return _verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nverify a signature with the given certificate_or_public_key.", "response": "def _verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=False):\n \"\"\"\n Verifies an RSA, DSA or ECDSA signature\n\n :param certificate_or_public_key:\n A Certificate or PublicKey instance to verify the signature with\n\n :param signature:\n A byte string of the signature to verify\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha256\", \"sha384\", \"sha512\" or \"raw\"\n\n :param rsa_pss_padding:\n If PSS padding should be used for RSA keys\n\n :raises:\n oscrypto.errors.SignatureError - when the signature is determined to be invalid\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n \"\"\"\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(signature, byte_cls):\n raise TypeError(pretty_message(\n '''\n signature must be a byte string, not %s\n ''',\n type_name(signature)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n valid_hash_algorithms = set(['md5', 'sha1', 'sha256', 'sha384', 'sha512'])\n if certificate_or_public_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms |= set(['raw'])\n\n if hash_algorithm not in valid_hash_algorithms:\n valid_hash_algorithms_error = '\"md5\", \"sha1\", \"sha256\", \"sha384\", \"sha512\"'\n if certificate_or_public_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms_error += ', \"raw\"'\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of %s, not %s\n ''',\n valid_hash_algorithms_error,\n repr(hash_algorithm)\n ))\n\n if certificate_or_public_key.algorithm != 'rsa' and rsa_pss_padding is not False:\n raise ValueError(pretty_message(\n '''\n PSS padding may only be used with RSA keys - signing via a %s key\n was requested\n ''',\n certificate_or_public_key.algorithm.upper()\n ))\n\n if hash_algorithm == 'raw':\n if len(data) > certificate_or_public_key.byte_size - 11:\n raise ValueError(pretty_message(\n '''\n data must be 11 bytes shorter than the key size when\n hash_algorithm is \"raw\" - key size is %s bytes, but\n data is %s bytes long\n ''',\n certificate_or_public_key.byte_size,\n len(data)\n ))\n\n if _backend == 'winlegacy':\n if certificate_or_public_key.algorithm == 'ec':\n return _pure_python_ecdsa_verify(certificate_or_public_key, signature, data, hash_algorithm)\n return _advapi32_verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding)\n return _bcrypt_verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _advapi32_verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=False):\n\n algo = certificate_or_public_key.algorithm\n\n if algo == 'rsa' and rsa_pss_padding:\n hash_length = {\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }.get(hash_algorithm, 0)\n decrypted_signature = raw_rsa_public_crypt(certificate_or_public_key, signature)\n key_size = certificate_or_public_key.bit_size\n if not verify_pss_padding(hash_algorithm, hash_length, key_size, data, decrypted_signature):\n raise SignatureError('Signature is invalid')\n return\n\n if algo == 'rsa' and hash_algorithm == 'raw':\n padded_plaintext = raw_rsa_public_crypt(certificate_or_public_key, signature)\n try:\n plaintext = remove_pkcs1v15_signature_padding(certificate_or_public_key.byte_size, padded_plaintext)\n if not constant_compare(plaintext, data):\n raise ValueError()\n except (ValueError):\n raise SignatureError('Signature is invalid')\n return\n\n hash_handle = None\n\n try:\n alg_id = {\n 'md5': Advapi32Const.CALG_MD5,\n 'sha1': Advapi32Const.CALG_SHA1,\n 'sha256': Advapi32Const.CALG_SHA_256,\n 'sha384': Advapi32Const.CALG_SHA_384,\n 'sha512': Advapi32Const.CALG_SHA_512,\n }[hash_algorithm]\n\n hash_handle_pointer = new(advapi32, 'HCRYPTHASH *')\n res = advapi32.CryptCreateHash(\n certificate_or_public_key.context_handle,\n alg_id,\n null(),\n 0,\n hash_handle_pointer\n )\n handle_error(res)\n\n hash_handle = unwrap(hash_handle_pointer)\n\n res = advapi32.CryptHashData(hash_handle, data, len(data), 0)\n handle_error(res)\n\n if algo == 'dsa':\n # Windows doesn't use the ASN.1 Sequence for DSA signatures,\n # so we have to convert it here for the verification to work\n try:\n signature = algos.DSASignature.load(signature).to_p1363()\n # Switch the two integers so that the reversal later will\n # result in the correct order\n half_len = len(signature) // 2\n signature = signature[half_len:] + signature[:half_len]\n except (ValueError, OverflowError, TypeError):\n raise SignatureError('Signature is invalid')\n\n # The CryptoAPI expects signatures to be in little endian byte order,\n # which is the opposite of other systems, so we must reverse it\n reversed_signature = signature[::-1]\n\n res = advapi32.CryptVerifySignatureW(\n hash_handle,\n reversed_signature,\n len(signature),\n certificate_or_public_key.key_handle,\n null(),\n 0\n )\n handle_error(res)\n\n finally:\n if hash_handle:\n advapi32.CryptDestroyHash(hash_handle)", "response": "Verifies an advapi32 signature via CryptoAPI."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _bcrypt_verify(certificate_or_public_key, signature, data, hash_algorithm, rsa_pss_padding=False):\n\n if hash_algorithm == 'raw':\n digest = data\n else:\n hash_constant = {\n 'md5': BcryptConst.BCRYPT_MD5_ALGORITHM,\n 'sha1': BcryptConst.BCRYPT_SHA1_ALGORITHM,\n 'sha256': BcryptConst.BCRYPT_SHA256_ALGORITHM,\n 'sha384': BcryptConst.BCRYPT_SHA384_ALGORITHM,\n 'sha512': BcryptConst.BCRYPT_SHA512_ALGORITHM\n }[hash_algorithm]\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n padding_info = null()\n flags = 0\n\n if certificate_or_public_key.algorithm == 'rsa':\n if rsa_pss_padding:\n flags = BcryptConst.BCRYPT_PAD_PSS\n padding_info_struct_pointer = struct(bcrypt, 'BCRYPT_PSS_PADDING_INFO')\n padding_info_struct = unwrap(padding_info_struct_pointer)\n # This has to be assigned to a variable to prevent cffi from gc'ing it\n hash_buffer = buffer_from_unicode(hash_constant)\n padding_info_struct.pszAlgId = cast(bcrypt, 'wchar_t *', hash_buffer)\n padding_info_struct.cbSalt = len(digest)\n else:\n flags = BcryptConst.BCRYPT_PAD_PKCS1\n padding_info_struct_pointer = struct(bcrypt, 'BCRYPT_PKCS1_PADDING_INFO')\n padding_info_struct = unwrap(padding_info_struct_pointer)\n # This has to be assigned to a variable to prevent cffi from gc'ing it\n if hash_algorithm == 'raw':\n padding_info_struct.pszAlgId = null()\n else:\n hash_buffer = buffer_from_unicode(hash_constant)\n padding_info_struct.pszAlgId = cast(bcrypt, 'wchar_t *', hash_buffer)\n padding_info = cast(bcrypt, 'void *', padding_info_struct_pointer)\n else:\n # Windows doesn't use the ASN.1 Sequence for DSA/ECDSA signatures,\n # so we have to convert it here for the verification to work\n try:\n signature = algos.DSASignature.load(signature).to_p1363()\n except (ValueError, OverflowError, TypeError):\n raise SignatureError('Signature is invalid')\n\n res = bcrypt.BCryptVerifySignature(\n certificate_or_public_key.key_handle,\n padding_info,\n digest,\n len(digest),\n signature,\n len(signature),\n flags\n )\n failure = res == BcryptConst.STATUS_INVALID_SIGNATURE\n failure = failure or res == BcryptConst.STATUS_INVALID_PARAMETER\n if failure:\n raise SignatureError('Signature is invalid')\n\n handle_error(res)", "response": "Verifies a signature via BCRYPT using the specified algorithm."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rsa_pkcs1v15_sign(private_key, data, hash_algorithm):\n\n if private_key.algorithm != 'rsa':\n raise ValueError('The key specified is not an RSA private key')\n\n return _sign(private_key, data, hash_algorithm)", "response": "Generates a RSA - PKCS - v1. 5 signature for the data."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a RSASSA - PSS signature for the data.", "response": "def rsa_pss_sign(private_key, data, hash_algorithm):\n \"\"\"\n Generates an RSASSA-PSS signature. For the PSS padding the mask gen\n algorithm will be mgf1 using the same hash algorithm as the signature. The\n salt length with be the length of the hash algorithm, and the trailer field\n with be the standard 0xBC byte.\n\n :param private_key:\n The PrivateKey to generate the signature with\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha256\", \"sha384\" or \"sha512\"\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the signature\n \"\"\"\n\n if private_key.algorithm != 'rsa':\n raise ValueError('The key specified is not an RSA private key')\n\n return _sign(private_key, data, hash_algorithm, rsa_pss_padding=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a DSA signature for the data.", "response": "def dsa_sign(private_key, data, hash_algorithm):\n \"\"\"\n Generates a DSA signature\n\n :param private_key:\n The PrivateKey to generate the signature with\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha256\", \"sha384\" or \"sha512\"\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the signature\n \"\"\"\n\n if private_key.algorithm != 'dsa':\n raise ValueError('The key specified is not a DSA private key')\n\n return _sign(private_key, data, hash_algorithm)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ecdsa_sign(private_key, data, hash_algorithm):\n\n if private_key.algorithm != 'ec':\n raise ValueError('The key specified is not an EC private key')\n\n return _sign(private_key, data, hash_algorithm)", "response": "Generates an ECDSA signature for the given data and hash_algorithm"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef _sign(private_key, data, hash_algorithm, rsa_pss_padding=False):\n\n if not isinstance(private_key, PrivateKey):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of PrivateKey, not %s\n ''',\n type_name(private_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n valid_hash_algorithms = set(['md5', 'sha1', 'sha256', 'sha384', 'sha512'])\n if private_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms |= set(['raw'])\n\n if hash_algorithm not in valid_hash_algorithms:\n valid_hash_algorithms_error = '\"md5\", \"sha1\", \"sha256\", \"sha384\", \"sha512\"'\n if private_key.algorithm == 'rsa' and not rsa_pss_padding:\n valid_hash_algorithms_error += ', \"raw\"'\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of %s, not %s\n ''',\n valid_hash_algorithms_error,\n repr(hash_algorithm)\n ))\n\n if private_key.algorithm != 'rsa' and rsa_pss_padding is not False:\n raise ValueError(pretty_message(\n '''\n PSS padding may only be used with RSA keys - signing via a %s key\n was requested\n ''',\n private_key.algorithm.upper()\n ))\n\n if hash_algorithm == 'raw':\n if len(data) > private_key.byte_size - 11:\n raise ValueError(pretty_message(\n '''\n data must be 11 bytes shorter than the key size when\n hash_algorithm is \"raw\" - key size is %s bytes, but data\n is %s bytes long\n ''',\n private_key.byte_size,\n len(data)\n ))\n\n if _backend == 'winlegacy':\n if private_key.algorithm == 'ec':\n return _pure_python_ecdsa_sign(private_key, data, hash_algorithm)\n return _advapi32_sign(private_key, data, hash_algorithm, rsa_pss_padding)\n return _bcrypt_sign(private_key, data, hash_algorithm, rsa_pss_padding)", "response": "Generates a signature for the given data and returns the byte string."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ngenerate an Advapi32 signature using the given private key.", "response": "def _advapi32_sign(private_key, data, hash_algorithm, rsa_pss_padding=False):\n \"\"\"\n Generates an RSA, DSA or ECDSA signature via CryptoAPI\n\n :param private_key:\n The PrivateKey to generate the signature with\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha256\", \"sha384\", \"sha512\" or \"raw\"\n\n :param rsa_pss_padding:\n If PSS padding should be used for RSA keys\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the signature\n \"\"\"\n\n algo = private_key.algorithm\n\n if algo == 'rsa' and hash_algorithm == 'raw':\n padded_data = add_pkcs1v15_signature_padding(private_key.byte_size, data)\n return raw_rsa_private_crypt(private_key, padded_data)\n\n if algo == 'rsa' and rsa_pss_padding:\n hash_length = {\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }.get(hash_algorithm, 0)\n padded_data = add_pss_padding(hash_algorithm, hash_length, private_key.bit_size, data)\n return raw_rsa_private_crypt(private_key, padded_data)\n\n if private_key.algorithm == 'dsa' and hash_algorithm == 'md5':\n raise ValueError(pretty_message(\n '''\n Windows does not support md5 signatures with DSA keys\n '''\n ))\n\n hash_handle = None\n\n try:\n alg_id = {\n 'md5': Advapi32Const.CALG_MD5,\n 'sha1': Advapi32Const.CALG_SHA1,\n 'sha256': Advapi32Const.CALG_SHA_256,\n 'sha384': Advapi32Const.CALG_SHA_384,\n 'sha512': Advapi32Const.CALG_SHA_512,\n }[hash_algorithm]\n\n hash_handle_pointer = new(advapi32, 'HCRYPTHASH *')\n res = advapi32.CryptCreateHash(\n private_key.context_handle,\n alg_id,\n null(),\n 0,\n hash_handle_pointer\n )\n handle_error(res)\n\n hash_handle = unwrap(hash_handle_pointer)\n\n res = advapi32.CryptHashData(hash_handle, data, len(data), 0)\n handle_error(res)\n\n out_len = new(advapi32, 'DWORD *')\n res = advapi32.CryptSignHashW(\n hash_handle,\n Advapi32Const.AT_SIGNATURE,\n null(),\n 0,\n null(),\n out_len\n )\n handle_error(res)\n\n buffer_length = deref(out_len)\n buffer_ = buffer_from_bytes(buffer_length)\n\n res = advapi32.CryptSignHashW(\n hash_handle,\n Advapi32Const.AT_SIGNATURE,\n null(),\n 0,\n buffer_,\n out_len\n )\n handle_error(res)\n\n output = bytes_from_buffer(buffer_, deref(out_len))\n\n # CryptoAPI outputs the signature in little endian byte order, so we\n # must swap it for compatibility with other systems\n output = output[::-1]\n\n if algo == 'dsa':\n # Switch the two integers because the reversal just before switched\n # then\n half_len = len(output) // 2\n output = output[half_len:] + output[:half_len]\n # Windows doesn't use the ASN.1 Sequence for DSA signatures,\n # so we have to convert it here for the verification to work\n output = algos.DSASignature.from_p1363(output).dump()\n\n return output\n\n finally:\n if hash_handle:\n advapi32.CryptDestroyHash(hash_handle)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a signature for a single object of the type of object.", "response": "def _bcrypt_sign(private_key, data, hash_algorithm, rsa_pss_padding=False):\n \"\"\"\n Generates an RSA, DSA or ECDSA signature via CNG\n\n :param private_key:\n The PrivateKey to generate the signature with\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha256\", \"sha384\", \"sha512\" or \"raw\"\n\n :param rsa_pss_padding:\n If PSS padding should be used for RSA keys\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the signature\n \"\"\"\n\n if hash_algorithm == 'raw':\n digest = data\n else:\n hash_constant = {\n 'md5': BcryptConst.BCRYPT_MD5_ALGORITHM,\n 'sha1': BcryptConst.BCRYPT_SHA1_ALGORITHM,\n 'sha256': BcryptConst.BCRYPT_SHA256_ALGORITHM,\n 'sha384': BcryptConst.BCRYPT_SHA384_ALGORITHM,\n 'sha512': BcryptConst.BCRYPT_SHA512_ALGORITHM\n }[hash_algorithm]\n\n digest = getattr(hashlib, hash_algorithm)(data).digest()\n\n padding_info = null()\n flags = 0\n\n if private_key.algorithm == 'rsa':\n if rsa_pss_padding:\n hash_length = {\n 'md5': 16,\n 'sha1': 20,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }[hash_algorithm]\n\n flags = BcryptConst.BCRYPT_PAD_PSS\n padding_info_struct_pointer = struct(bcrypt, 'BCRYPT_PSS_PADDING_INFO')\n padding_info_struct = unwrap(padding_info_struct_pointer)\n # This has to be assigned to a variable to prevent cffi from gc'ing it\n hash_buffer = buffer_from_unicode(hash_constant)\n padding_info_struct.pszAlgId = cast(bcrypt, 'wchar_t *', hash_buffer)\n padding_info_struct.cbSalt = hash_length\n else:\n flags = BcryptConst.BCRYPT_PAD_PKCS1\n padding_info_struct_pointer = struct(bcrypt, 'BCRYPT_PKCS1_PADDING_INFO')\n padding_info_struct = unwrap(padding_info_struct_pointer)\n # This has to be assigned to a variable to prevent cffi from gc'ing it\n if hash_algorithm == 'raw':\n padding_info_struct.pszAlgId = null()\n else:\n hash_buffer = buffer_from_unicode(hash_constant)\n padding_info_struct.pszAlgId = cast(bcrypt, 'wchar_t *', hash_buffer)\n padding_info = cast(bcrypt, 'void *', padding_info_struct_pointer)\n\n if private_key.algorithm == 'dsa' and private_key.bit_size > 1024 and hash_algorithm in set(['md5', 'sha1']):\n raise ValueError(pretty_message(\n '''\n Windows does not support sha1 signatures with DSA keys based on\n sha224, sha256 or sha512\n '''\n ))\n\n out_len = new(bcrypt, 'DWORD *')\n res = bcrypt.BCryptSignHash(\n private_key.key_handle,\n padding_info,\n digest,\n len(digest),\n null(),\n 0,\n out_len,\n flags\n )\n handle_error(res)\n\n buffer_len = deref(out_len)\n buffer = buffer_from_bytes(buffer_len)\n\n if private_key.algorithm == 'rsa':\n padding_info = cast(bcrypt, 'void *', padding_info_struct_pointer)\n\n res = bcrypt.BCryptSignHash(\n private_key.key_handle,\n padding_info,\n digest,\n len(digest),\n buffer,\n buffer_len,\n out_len,\n flags\n )\n handle_error(res)\n signature = bytes_from_buffer(buffer, deref(out_len))\n\n if private_key.algorithm != 'rsa':\n # Windows doesn't use the ASN.1 Sequence for DSA/ECDSA signatures,\n # so we have to convert it here for the verification to work\n signature = algos.DSASignature.from_p1363(signature).dump()\n\n return signature"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _encrypt(certificate_or_public_key, data, rsa_oaep_padding=False):\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if not isinstance(rsa_oaep_padding, bool):\n raise TypeError(pretty_message(\n '''\n rsa_oaep_padding must be a bool, not %s\n ''',\n type_name(rsa_oaep_padding)\n ))\n\n if _backend == 'winlegacy':\n return _advapi32_encrypt(certificate_or_public_key, data, rsa_oaep_padding)\n return _bcrypt_encrypt(certificate_or_public_key, data, rsa_oaep_padding)", "response": "Encrypts a value using an RSA public key."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _advapi32_encrypt(certificate_or_public_key, data, rsa_oaep_padding=False):\n\n flags = 0\n if rsa_oaep_padding:\n flags = Advapi32Const.CRYPT_OAEP\n\n out_len = new(advapi32, 'DWORD *', len(data))\n res = advapi32.CryptEncrypt(\n certificate_or_public_key.ex_key_handle,\n null(),\n True,\n flags,\n null(),\n out_len,\n 0\n )\n handle_error(res)\n\n buffer_len = deref(out_len)\n buffer = buffer_from_bytes(buffer_len)\n write_to_buffer(buffer, data)\n\n pointer_set(out_len, len(data))\n res = advapi32.CryptEncrypt(\n certificate_or_public_key.ex_key_handle,\n null(),\n True,\n flags,\n buffer,\n out_len,\n buffer_len\n )\n handle_error(res)\n\n return bytes_from_buffer(buffer, deref(out_len))[::-1]", "response": "Encrypts a value using an RSA public key via CryptoAPI."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nencrypt a value using an RSA public key via CNG", "response": "def _bcrypt_encrypt(certificate_or_public_key, data, rsa_oaep_padding=False):\n \"\"\"\n Encrypts a value using an RSA public key via CNG\n\n :param certificate_or_public_key:\n A Certificate or PublicKey instance to encrypt with\n\n :param data:\n A byte string of the data to encrypt\n\n :param rsa_oaep_padding:\n If OAEP padding should be used instead of PKCS#1 v1.5\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the ciphertext\n \"\"\"\n\n flags = BcryptConst.BCRYPT_PAD_PKCS1\n if rsa_oaep_padding is True:\n flags = BcryptConst.BCRYPT_PAD_OAEP\n\n padding_info_struct_pointer = struct(bcrypt, 'BCRYPT_OAEP_PADDING_INFO')\n padding_info_struct = unwrap(padding_info_struct_pointer)\n # This has to be assigned to a variable to prevent cffi from gc'ing it\n hash_buffer = buffer_from_unicode(BcryptConst.BCRYPT_SHA1_ALGORITHM)\n padding_info_struct.pszAlgId = cast(bcrypt, 'wchar_t *', hash_buffer)\n padding_info_struct.pbLabel = null()\n padding_info_struct.cbLabel = 0\n padding_info = cast(bcrypt, 'void *', padding_info_struct_pointer)\n else:\n padding_info = null()\n\n out_len = new(bcrypt, 'ULONG *')\n res = bcrypt.BCryptEncrypt(\n certificate_or_public_key.key_handle,\n data,\n len(data),\n padding_info,\n null(),\n 0,\n null(),\n 0,\n out_len,\n flags\n )\n handle_error(res)\n\n buffer_len = deref(out_len)\n buffer = buffer_from_bytes(buffer_len)\n\n res = bcrypt.BCryptEncrypt(\n certificate_or_public_key.key_handle,\n data,\n len(data),\n padding_info,\n null(),\n 0,\n buffer,\n buffer_len,\n out_len,\n flags\n )\n handle_error(res)\n\n return bytes_from_buffer(buffer, deref(out_len))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _decrypt(private_key, ciphertext, rsa_oaep_padding=False):\n\n if not isinstance(private_key, PrivateKey):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of the PrivateKey class, not %s\n ''',\n type_name(private_key)\n ))\n\n if not isinstance(ciphertext, byte_cls):\n raise TypeError(pretty_message(\n '''\n ciphertext must be a byte string, not %s\n ''',\n type_name(ciphertext)\n ))\n\n if not isinstance(rsa_oaep_padding, bool):\n raise TypeError(pretty_message(\n '''\n rsa_oaep_padding must be a bool, not %s\n ''',\n type_name(rsa_oaep_padding)\n ))\n\n if _backend == 'winlegacy':\n return _advapi32_decrypt(private_key, ciphertext, rsa_oaep_padding)\n return _bcrypt_decrypt(private_key, ciphertext, rsa_oaep_padding)", "response": "Encrypts a value using an RSA private key and returns the plaintext value."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nencrypting a value using an RSA private key via CryptoAPI", "response": "def _advapi32_decrypt(private_key, ciphertext, rsa_oaep_padding=False):\n \"\"\"\n Encrypts a value using an RSA private key via CryptoAPI\n\n :param private_key:\n A PrivateKey instance to decrypt with\n\n :param ciphertext:\n A byte string of the data to decrypt\n\n :param rsa_oaep_padding:\n If OAEP padding should be used instead of PKCS#1 v1.5\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the plaintext\n \"\"\"\n\n flags = 0\n if rsa_oaep_padding:\n flags = Advapi32Const.CRYPT_OAEP\n\n ciphertext = ciphertext[::-1]\n\n buffer = buffer_from_bytes(ciphertext)\n out_len = new(advapi32, 'DWORD *', len(ciphertext))\n res = advapi32.CryptDecrypt(\n private_key.ex_key_handle,\n null(),\n True,\n flags,\n buffer,\n out_len\n )\n handle_error(res)\n\n return bytes_from_buffer(buffer, deref(out_len))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget a set of credentials for use with SChannel.", "response": "def _obtain_credentials(self):\n \"\"\"\n Obtains a credentials handle from secur32.dll for use with SChannel\n \"\"\"\n\n protocol_values = {\n 'SSLv3': Secur32Const.SP_PROT_SSL3_CLIENT,\n 'TLSv1': Secur32Const.SP_PROT_TLS1_CLIENT,\n 'TLSv1.1': Secur32Const.SP_PROT_TLS1_1_CLIENT,\n 'TLSv1.2': Secur32Const.SP_PROT_TLS1_2_CLIENT,\n }\n protocol_bit_mask = 0\n for key, value in protocol_values.items():\n if key in self._protocols:\n protocol_bit_mask |= value\n\n algs = [\n Secur32Const.CALG_AES_128,\n Secur32Const.CALG_AES_256,\n Secur32Const.CALG_3DES,\n Secur32Const.CALG_SHA1,\n Secur32Const.CALG_ECDHE,\n Secur32Const.CALG_DH_EPHEM,\n Secur32Const.CALG_RSA_KEYX,\n Secur32Const.CALG_RSA_SIGN,\n Secur32Const.CALG_ECDSA,\n Secur32Const.CALG_DSS_SIGN,\n ]\n if 'TLSv1.2' in self._protocols:\n algs.extend([\n Secur32Const.CALG_SHA512,\n Secur32Const.CALG_SHA384,\n Secur32Const.CALG_SHA256,\n ])\n\n alg_array = new(secur32, 'ALG_ID[%s]' % len(algs))\n for index, alg in enumerate(algs):\n alg_array[index] = alg\n\n flags = Secur32Const.SCH_USE_STRONG_CRYPTO | Secur32Const.SCH_CRED_NO_DEFAULT_CREDS\n if not self._manual_validation and not self._extra_trust_roots:\n flags |= Secur32Const.SCH_CRED_AUTO_CRED_VALIDATION\n else:\n flags |= Secur32Const.SCH_CRED_MANUAL_CRED_VALIDATION\n\n schannel_cred_pointer = struct(secur32, 'SCHANNEL_CRED')\n schannel_cred = unwrap(schannel_cred_pointer)\n\n schannel_cred.dwVersion = Secur32Const.SCHANNEL_CRED_VERSION\n schannel_cred.cCreds = 0\n schannel_cred.paCred = null()\n schannel_cred.hRootStore = null()\n schannel_cred.cMappers = 0\n schannel_cred.aphMappers = null()\n schannel_cred.cSupportedAlgs = len(alg_array)\n schannel_cred.palgSupportedAlgs = alg_array\n schannel_cred.grbitEnabledProtocols = protocol_bit_mask\n schannel_cred.dwMinimumCipherStrength = 0\n schannel_cred.dwMaximumCipherStrength = 0\n # Default session lifetime is 10 hours\n schannel_cred.dwSessionLifespan = 0\n schannel_cred.dwFlags = flags\n schannel_cred.dwCredFormat = 0\n\n cred_handle_pointer = new(secur32, 'CredHandle *')\n\n result = secur32.AcquireCredentialsHandleW(\n null(),\n Secur32Const.UNISP_NAME,\n Secur32Const.SECPKG_CRED_OUTBOUND,\n null(),\n schannel_cred_pointer,\n null(),\n null(),\n cred_handle_pointer,\n null()\n )\n handle_error(result)\n\n self._credentials_handle = cred_handle_pointer"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef wrap(cls, socket, hostname, session=None):\n\n if not isinstance(socket, socket_.socket):\n raise TypeError(pretty_message(\n '''\n socket must be an instance of socket.socket, not %s\n ''',\n type_name(socket)\n ))\n\n if not isinstance(hostname, str_cls):\n raise TypeError(pretty_message(\n '''\n hostname must be a unicode string, not %s\n ''',\n type_name(hostname)\n ))\n\n if session is not None and not isinstance(session, TLSSession):\n raise TypeError(pretty_message(\n '''\n session must be an instance of oscrypto.tls.TLSSession, not %s\n ''',\n type_name(session)\n ))\n\n new_socket = cls(None, None, session=session)\n new_socket._socket = socket\n new_socket._hostname = hostname\n\n # Since we don't create the socket connection here, we can't try to\n # reconnect with a lower version of the TLS protocol, so we just\n # move the data to public exception type TLSVerificationError()\n try:\n new_socket._handshake()\n except (_TLSDowngradeError) as e:\n new_e = TLSVerificationError(e.message, e.certificate)\n raise new_e\n except (_TLSRetryError) as e:\n new_e = TLSError(e.message)\n raise new_e\n\n return new_socket", "response": "Creates a new socket and adds TLS\n to it"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncreate a list of SecBufferDesc struct and contained SecBuffer objects.", "response": "def _create_buffers(self, number):\n \"\"\"\n Creates a SecBufferDesc struct and contained SecBuffer structs\n\n :param number:\n The number of contains SecBuffer objects to create\n\n :return:\n A tuple of (SecBufferDesc pointer, SecBuffer array)\n \"\"\"\n\n buffers = new(secur32, 'SecBuffer[%d]' % number)\n\n for index in range(0, number):\n buffers[index].cbBuffer = 0\n buffers[index].BufferType = Secur32Const.SECBUFFER_EMPTY\n buffers[index].pvBuffer = null()\n\n sec_buffer_desc_pointer = struct(secur32, 'SecBufferDesc')\n sec_buffer_desc = unwrap(sec_buffer_desc_pointer)\n\n sec_buffer_desc.ulVersion = Secur32Const.SECBUFFER_VERSION\n sec_buffer_desc.cBuffers = number\n sec_buffer_desc.pBuffers = buffers\n\n return (sec_buffer_desc_pointer, buffers)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nperforming a TLS handshake on the current neccesary domain.", "response": "def _handshake(self, renegotiate=False):\n \"\"\"\n Perform an initial TLS handshake, or a renegotiation\n\n :param renegotiate:\n If the handshake is for a renegotiation\n \"\"\"\n\n in_buffers = None\n out_buffers = None\n new_context_handle_pointer = None\n\n try:\n if renegotiate:\n temp_context_handle_pointer = self._context_handle_pointer\n else:\n new_context_handle_pointer = new(secur32, 'CtxtHandle *')\n temp_context_handle_pointer = new_context_handle_pointer\n\n requested_flags = {\n Secur32Const.ISC_REQ_REPLAY_DETECT: 'replay detection',\n Secur32Const.ISC_REQ_SEQUENCE_DETECT: 'sequence detection',\n Secur32Const.ISC_REQ_CONFIDENTIALITY: 'confidentiality',\n Secur32Const.ISC_REQ_ALLOCATE_MEMORY: 'memory allocation',\n Secur32Const.ISC_REQ_INTEGRITY: 'integrity',\n Secur32Const.ISC_REQ_STREAM: 'stream orientation',\n Secur32Const.ISC_REQ_USE_SUPPLIED_CREDS: 'disable automatic client auth',\n }\n\n self._context_flags = 0\n for flag in requested_flags:\n self._context_flags |= flag\n\n in_sec_buffer_desc_pointer, in_buffers = self._create_buffers(2)\n in_buffers[0].BufferType = Secur32Const.SECBUFFER_TOKEN\n\n out_sec_buffer_desc_pointer, out_buffers = self._create_buffers(2)\n out_buffers[0].BufferType = Secur32Const.SECBUFFER_TOKEN\n out_buffers[1].BufferType = Secur32Const.SECBUFFER_ALERT\n\n output_context_flags_pointer = new(secur32, 'ULONG *')\n\n if renegotiate:\n first_handle = temp_context_handle_pointer\n second_handle = null()\n else:\n first_handle = null()\n second_handle = temp_context_handle_pointer\n\n result = secur32.InitializeSecurityContextW(\n self._session._credentials_handle,\n first_handle,\n self._hostname,\n self._context_flags,\n 0,\n 0,\n null(),\n 0,\n second_handle,\n out_sec_buffer_desc_pointer,\n output_context_flags_pointer,\n null()\n )\n if result not in set([Secur32Const.SEC_E_OK, Secur32Const.SEC_I_CONTINUE_NEEDED]):\n handle_error(result, TLSError)\n\n if not renegotiate:\n temp_context_handle_pointer = second_handle\n else:\n temp_context_handle_pointer = first_handle\n\n handshake_server_bytes = b''\n handshake_client_bytes = b''\n\n if out_buffers[0].cbBuffer > 0:\n token = bytes_from_buffer(out_buffers[0].pvBuffer, out_buffers[0].cbBuffer)\n handshake_client_bytes += token\n self._socket.send(token)\n out_buffers[0].cbBuffer = 0\n secur32.FreeContextBuffer(out_buffers[0].pvBuffer)\n out_buffers[0].pvBuffer = null()\n\n in_data_buffer = buffer_from_bytes(32768)\n in_buffers[0].pvBuffer = cast(secur32, 'BYTE *', in_data_buffer)\n\n bytes_read = b''\n while result != Secur32Const.SEC_E_OK:\n try:\n fail_late = False\n bytes_read = self._socket.recv(8192)\n if bytes_read == b'':\n raise_disconnection()\n except (socket_error_cls):\n fail_late = True\n handshake_server_bytes += bytes_read\n self._received_bytes += bytes_read\n\n in_buffers[0].cbBuffer = len(self._received_bytes)\n write_to_buffer(in_data_buffer, self._received_bytes)\n\n result = secur32.InitializeSecurityContextW(\n self._session._credentials_handle,\n temp_context_handle_pointer,\n self._hostname,\n self._context_flags,\n 0,\n 0,\n in_sec_buffer_desc_pointer,\n 0,\n null(),\n out_sec_buffer_desc_pointer,\n output_context_flags_pointer,\n null()\n )\n\n if result == Secur32Const.SEC_E_INCOMPLETE_MESSAGE:\n in_buffers[0].BufferType = Secur32Const.SECBUFFER_TOKEN\n # Windows 10 seems to fill the second input buffer with\n # a BufferType of SECBUFFER_MISSING (4), which if not\n # cleared causes the handshake to fail.\n if in_buffers[1].BufferType != Secur32Const.SECBUFFER_EMPTY:\n in_buffers[1].BufferType = Secur32Const.SECBUFFER_EMPTY\n in_buffers[1].cbBuffer = 0\n if not is_null(in_buffers[1].pvBuffer):\n secur32.FreeContextBuffer(in_buffers[1].pvBuffer)\n in_buffers[1].pvBuffer = null()\n\n if fail_late:\n raise_disconnection()\n\n continue\n\n if result == Secur32Const.SEC_E_ILLEGAL_MESSAGE:\n if detect_client_auth_request(handshake_server_bytes):\n raise_client_auth()\n alert_info = parse_alert(handshake_server_bytes)\n if alert_info and alert_info == (2, 70):\n raise_protocol_version()\n raise_handshake()\n\n if result == Secur32Const.SEC_E_WRONG_PRINCIPAL:\n chain = extract_chain(handshake_server_bytes)\n raise_hostname(chain[0], self._hostname)\n\n if result == Secur32Const.SEC_E_CERT_EXPIRED:\n chain = extract_chain(handshake_server_bytes)\n raise_expired_not_yet_valid(chain[0])\n\n if result == Secur32Const.SEC_E_UNTRUSTED_ROOT:\n chain = extract_chain(handshake_server_bytes)\n cert = chain[0]\n oscrypto_cert = load_certificate(cert)\n if not oscrypto_cert.self_signed:\n raise_no_issuer(cert)\n raise_self_signed(cert)\n\n if result == Secur32Const.SEC_E_INTERNAL_ERROR:\n if get_dh_params_length(handshake_server_bytes) < 1024:\n raise_dh_params()\n\n if result == Secur32Const.SEC_I_INCOMPLETE_CREDENTIALS:\n raise_client_auth()\n\n if result == Crypt32Const.TRUST_E_CERT_SIGNATURE:\n raise_weak_signature(cert)\n\n if result == Secur32Const.SEC_E_INVALID_TOKEN:\n # If an alert it present, there may have been a handshake\n # error due to the server using a certificate path with a\n # trust root using MD2 or MD5 combined with TLS 1.2. To\n # work around this, if the user allows anything other than\n # TLS 1.2, we just remove it from the acceptable protocols\n # and try again.\n if out_buffers[1].cbBuffer > 0:\n alert_bytes = bytes_from_buffer(out_buffers[1].pvBuffer, out_buffers[1].cbBuffer)\n handshake_client_bytes += alert_bytes\n alert_number = alert_bytes[6:7]\n if alert_number == b'\\x28' or alert_number == b'\\x2b':\n if 'TLSv1.2' in self._session._protocols and len(self._session._protocols) > 1:\n chain = extract_chain(handshake_server_bytes)\n raise _TLSDowngradeError(\n 'Server certificate verification failed - weak certificate signature algorithm',\n chain[0]\n )\n if detect_client_auth_request(handshake_server_bytes):\n raise_client_auth()\n if detect_other_protocol(handshake_server_bytes):\n raise_protocol_error(handshake_server_bytes)\n raise_handshake()\n\n # These are semi-common errors with TLSv1.2 on Windows 7 an 8\n # that appears to be due to poor handling of the\n # ServerKeyExchange for DHE_RSA cipher suites. The solution\n # is to retry the handshake.\n if result == Secur32Const.SEC_E_BUFFER_TOO_SMALL or result == Secur32Const.SEC_E_MESSAGE_ALTERED:\n if 'TLSv1.2' in self._session._protocols:\n raise _TLSRetryError('TLS handshake failed')\n\n if fail_late:\n raise_disconnection()\n\n if result == Secur32Const.SEC_E_INVALID_PARAMETER:\n if get_dh_params_length(handshake_server_bytes) < 1024:\n raise_dh_params()\n\n if result not in set([Secur32Const.SEC_E_OK, Secur32Const.SEC_I_CONTINUE_NEEDED]):\n handle_error(result, TLSError)\n\n if out_buffers[0].cbBuffer > 0:\n token = bytes_from_buffer(out_buffers[0].pvBuffer, out_buffers[0].cbBuffer)\n handshake_client_bytes += token\n self._socket.send(token)\n out_buffers[0].cbBuffer = 0\n secur32.FreeContextBuffer(out_buffers[0].pvBuffer)\n out_buffers[0].pvBuffer = null()\n\n if in_buffers[1].BufferType == Secur32Const.SECBUFFER_EXTRA:\n extra_amount = in_buffers[1].cbBuffer\n self._received_bytes = self._received_bytes[-extra_amount:]\n in_buffers[1].BufferType = Secur32Const.SECBUFFER_EMPTY\n in_buffers[1].cbBuffer = 0\n secur32.FreeContextBuffer(in_buffers[1].pvBuffer)\n in_buffers[1].pvBuffer = null()\n\n # The handshake is complete, so discard any extra bytes\n if result == Secur32Const.SEC_E_OK:\n handshake_server_bytes = handshake_server_bytes[-extra_amount:]\n\n else:\n self._received_bytes = b''\n\n connection_info_pointer = struct(secur32, 'SecPkgContext_ConnectionInfo')\n result = secur32.QueryContextAttributesW(\n temp_context_handle_pointer,\n Secur32Const.SECPKG_ATTR_CONNECTION_INFO,\n connection_info_pointer\n )\n handle_error(result, TLSError)\n\n connection_info = unwrap(connection_info_pointer)\n\n self._protocol = {\n Secur32Const.SP_PROT_SSL2_CLIENT: 'SSLv2',\n Secur32Const.SP_PROT_SSL3_CLIENT: 'SSLv3',\n Secur32Const.SP_PROT_TLS1_CLIENT: 'TLSv1',\n Secur32Const.SP_PROT_TLS1_1_CLIENT: 'TLSv1.1',\n Secur32Const.SP_PROT_TLS1_2_CLIENT: 'TLSv1.2',\n }.get(native(int, connection_info.dwProtocol), str_cls(connection_info.dwProtocol))\n\n if self._protocol in set(['SSLv3', 'TLSv1', 'TLSv1.1', 'TLSv1.2']):\n session_info = parse_session_info(handshake_server_bytes, handshake_client_bytes)\n self._cipher_suite = session_info['cipher_suite']\n self._compression = session_info['compression']\n self._session_id = session_info['session_id']\n self._session_ticket = session_info['session_ticket']\n\n output_context_flags = deref(output_context_flags_pointer)\n\n for flag in requested_flags:\n if (flag | output_context_flags) == 0:\n raise OSError(pretty_message(\n '''\n Unable to obtain a credential context with the property %s\n ''',\n requested_flags[flag]\n ))\n\n if not renegotiate:\n self._context_handle_pointer = temp_context_handle_pointer\n new_context_handle_pointer = None\n\n stream_sizes_pointer = struct(secur32, 'SecPkgContext_StreamSizes')\n result = secur32.QueryContextAttributesW(\n self._context_handle_pointer,\n Secur32Const.SECPKG_ATTR_STREAM_SIZES,\n stream_sizes_pointer\n )\n handle_error(result)\n\n stream_sizes = unwrap(stream_sizes_pointer)\n self._header_size = native(int, stream_sizes.cbHeader)\n self._message_size = native(int, stream_sizes.cbMaximumMessage)\n self._trailer_size = native(int, stream_sizes.cbTrailer)\n self._buffer_size = self._header_size + self._message_size + self._trailer_size\n\n if self._session._extra_trust_roots:\n self._extra_trust_root_validation()\n\n except (OSError, socket_.error):\n self.close()\n\n raise\n\n finally:\n if out_buffers:\n if not is_null(out_buffers[0].pvBuffer):\n secur32.FreeContextBuffer(out_buffers[0].pvBuffer)\n if not is_null(out_buffers[1].pvBuffer):\n secur32.FreeContextBuffer(out_buffers[1].pvBuffer)\n if new_context_handle_pointer:\n secur32.DeleteSecurityContext(new_context_handle_pointer)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nread the data from the TLS - wrapped socket and returns it as a byte string.", "response": "def read(self, max_length):\n \"\"\"\n Reads data from the TLS-wrapped socket\n\n :param max_length:\n The number of bytes to read\n\n :raises:\n socket.socket - when a non-TLS socket error occurs\n oscrypto.errors.TLSError - when a TLS-related error occurs\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the data read\n \"\"\"\n\n if not isinstance(max_length, int_types):\n raise TypeError(pretty_message(\n '''\n max_length must be an integer, not %s\n ''',\n type_name(max_length)\n ))\n\n if self._context_handle_pointer is None:\n\n # Allow the user to read any remaining decrypted data\n if self._decrypted_bytes != b'':\n output = self._decrypted_bytes[0:max_length]\n self._decrypted_bytes = self._decrypted_bytes[max_length:]\n return output\n\n self._raise_closed()\n\n # The first time read is called, set up a single contiguous buffer that\n # it used by DecryptMessage() to populate the three output buffers.\n # Since we are creating the buffer, we do not need to free it other\n # than allowing Python to GC it once this object is GCed.\n if not self._decrypt_data_buffer:\n self._decrypt_data_buffer = buffer_from_bytes(self._buffer_size)\n self._decrypt_desc, self._decrypt_buffers = self._create_buffers(4)\n self._decrypt_buffers[0].BufferType = Secur32Const.SECBUFFER_DATA\n self._decrypt_buffers[0].pvBuffer = cast(secur32, 'BYTE *', self._decrypt_data_buffer)\n\n to_recv = max(max_length, self._buffer_size)\n\n # These variables are set to reduce dict access and function calls\n # in the read loop. Also makes the code easier to read.\n null_value = null()\n buf0 = self._decrypt_buffers[0]\n buf1 = self._decrypt_buffers[1]\n buf2 = self._decrypt_buffers[2]\n buf3 = self._decrypt_buffers[3]\n\n def _reset_buffers():\n buf0.BufferType = Secur32Const.SECBUFFER_DATA\n buf0.pvBuffer = cast(secur32, 'BYTE *', self._decrypt_data_buffer)\n buf0.cbBuffer = 0\n\n buf1.BufferType = Secur32Const.SECBUFFER_EMPTY\n buf1.pvBuffer = null_value\n buf1.cbBuffer = 0\n\n buf2.BufferType = Secur32Const.SECBUFFER_EMPTY\n buf2.pvBuffer = null_value\n buf2.cbBuffer = 0\n\n buf3.BufferType = Secur32Const.SECBUFFER_EMPTY\n buf3.pvBuffer = null_value\n buf3.cbBuffer = 0\n\n output = self._decrypted_bytes\n output_len = len(output)\n\n self._decrypted_bytes = b''\n\n # Don't block if we have buffered data available\n if output_len > 0 and not self.select_read(0):\n self._decrypted_bytes = b''\n return output\n\n # This read loop will only be run if there wasn't enough\n # buffered data to fulfill the requested max_length\n do_read = len(self._received_bytes) == 0\n\n while output_len < max_length:\n if do_read:\n self._received_bytes += self._socket.recv(to_recv)\n if len(self._received_bytes) == 0:\n raise_disconnection()\n\n data_len = min(len(self._received_bytes), self._buffer_size)\n if data_len == 0:\n break\n self._decrypt_buffers[0].cbBuffer = data_len\n write_to_buffer(self._decrypt_data_buffer, self._received_bytes[0:data_len])\n\n result = secur32.DecryptMessage(\n self._context_handle_pointer,\n self._decrypt_desc,\n 0,\n null()\n )\n\n do_read = False\n\n if result == Secur32Const.SEC_E_INCOMPLETE_MESSAGE:\n _reset_buffers()\n do_read = True\n continue\n\n elif result == Secur32Const.SEC_I_CONTEXT_EXPIRED:\n self._remote_closed = True\n self.shutdown()\n break\n\n elif result == Secur32Const.SEC_I_RENEGOTIATE:\n self._handshake(renegotiate=True)\n return self.read(max_length)\n\n elif result != Secur32Const.SEC_E_OK:\n handle_error(result, TLSError)\n\n valid_buffer_types = set([\n Secur32Const.SECBUFFER_EMPTY,\n Secur32Const.SECBUFFER_STREAM_HEADER,\n Secur32Const.SECBUFFER_STREAM_TRAILER\n ])\n extra_amount = None\n for buf in (buf0, buf1, buf2, buf3):\n buffer_type = buf.BufferType\n if buffer_type == Secur32Const.SECBUFFER_DATA:\n output += bytes_from_buffer(buf.pvBuffer, buf.cbBuffer)\n output_len = len(output)\n elif buffer_type == Secur32Const.SECBUFFER_EXTRA:\n extra_amount = native(int, buf.cbBuffer)\n elif buffer_type not in valid_buffer_types:\n raise OSError(pretty_message(\n '''\n Unexpected decrypt output buffer of type %s\n ''',\n buffer_type\n ))\n\n if extra_amount:\n self._received_bytes = self._received_bytes[data_len - extra_amount:]\n else:\n self._received_bytes = self._received_bytes[data_len:]\n\n # Here we reset the structs for the next call to DecryptMessage()\n _reset_buffers()\n\n # If we have read something, but there is nothing left to read, we\n # break so that we don't block for longer than necessary\n if self.select_read(0):\n do_read = True\n\n if not do_read and len(self._received_bytes) == 0:\n break\n\n # If the output is more than we requested (because data is decrypted in\n # blocks), we save the extra in a buffer\n if len(output) > max_length:\n self._decrypted_bytes = output[max_length:]\n output = output[0:max_length]\n\n return output"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nblock until the socket is ready to be read.", "response": "def select_read(self, timeout=None):\n \"\"\"\n Blocks until the socket is ready to be read from, or the timeout is hit\n\n :param timeout:\n A float - the period of time to wait for data to be read. None for\n no time limit.\n\n :return:\n A boolean - if data is ready to be read. Will only be False if\n timeout is not None.\n \"\"\"\n\n # If we have buffered data, we consider a read possible\n if len(self._decrypted_bytes) > 0:\n return True\n\n read_ready, _, _ = select.select([self._socket], [], [], timeout)\n return len(read_ready) > 0"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef read_exactly(self, num_bytes):\n\n output = b''\n remaining = num_bytes\n while remaining > 0:\n output += self.read(remaining)\n remaining = num_bytes - len(output)\n\n return output", "response": "Reads exactly the specified number of bytes from the socket and returns a byte string."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef write(self, data):\n\n if self._context_handle_pointer is None:\n self._raise_closed()\n\n if not self._encrypt_data_buffer:\n self._encrypt_data_buffer = buffer_from_bytes(self._header_size + self._message_size + self._trailer_size)\n self._encrypt_desc, self._encrypt_buffers = self._create_buffers(4)\n\n self._encrypt_buffers[0].BufferType = Secur32Const.SECBUFFER_STREAM_HEADER\n self._encrypt_buffers[0].cbBuffer = self._header_size\n self._encrypt_buffers[0].pvBuffer = cast(secur32, 'BYTE *', self._encrypt_data_buffer)\n\n self._encrypt_buffers[1].BufferType = Secur32Const.SECBUFFER_DATA\n self._encrypt_buffers[1].pvBuffer = ref(self._encrypt_data_buffer, self._header_size)\n\n self._encrypt_buffers[2].BufferType = Secur32Const.SECBUFFER_STREAM_TRAILER\n self._encrypt_buffers[2].cbBuffer = self._trailer_size\n self._encrypt_buffers[2].pvBuffer = ref(self._encrypt_data_buffer, self._header_size + self._message_size)\n\n while len(data) > 0:\n to_write = min(len(data), self._message_size)\n write_to_buffer(self._encrypt_data_buffer, data[0:to_write], self._header_size)\n\n self._encrypt_buffers[1].cbBuffer = to_write\n self._encrypt_buffers[2].pvBuffer = ref(self._encrypt_data_buffer, self._header_size + to_write)\n\n result = secur32.EncryptMessage(\n self._context_handle_pointer,\n 0,\n self._encrypt_desc,\n 0\n )\n\n if result != Secur32Const.SEC_E_OK:\n handle_error(result, TLSError)\n\n to_send = native(int, self._encrypt_buffers[0].cbBuffer)\n to_send += native(int, self._encrypt_buffers[1].cbBuffer)\n to_send += native(int, self._encrypt_buffers[2].cbBuffer)\n try:\n self._socket.send(bytes_from_buffer(self._encrypt_data_buffer, to_send))\n except (socket_.error) as e:\n if e.errno == 10053:\n raise_disconnection()\n raise\n\n data = data[to_send:]", "response": "Writes data to the TLS - wrapped socket."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nblocking until the socket is ready to be written to.", "response": "def select_write(self, timeout=None):\n \"\"\"\n Blocks until the socket is ready to be written to, or the timeout is hit\n\n :param timeout:\n A float - the period of time to wait for the socket to be ready to\n written to. None for no time limit.\n\n :return:\n A boolean - if the socket is ready for writing. Will only be False\n if timeout is not None.\n \"\"\"\n\n _, write_ready, _ = select.select([], [self._socket], [], timeout)\n return len(write_ready) > 0"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nshutting down the underlying socket and then shuts down the underlying socket.", "response": "def shutdown(self):\n \"\"\"\n Shuts down the TLS session and then shuts down the underlying socket\n\n :raises:\n OSError - when an error is returned by the OS crypto library\n \"\"\"\n\n if self._context_handle_pointer is None:\n return\n\n out_buffers = None\n try:\n # ApplyControlToken fails with SEC_E_UNSUPPORTED_FUNCTION\n # when called on Windows 7\n if _win_version_info >= (6, 2):\n buffers = new(secur32, 'SecBuffer[1]')\n\n # This is a SCHANNEL_SHUTDOWN token (DWORD of 1)\n buffers[0].cbBuffer = 4\n buffers[0].BufferType = Secur32Const.SECBUFFER_TOKEN\n buffers[0].pvBuffer = cast(secur32, 'BYTE *', buffer_from_bytes(b'\\x01\\x00\\x00\\x00'))\n\n sec_buffer_desc_pointer = struct(secur32, 'SecBufferDesc')\n sec_buffer_desc = unwrap(sec_buffer_desc_pointer)\n\n sec_buffer_desc.ulVersion = Secur32Const.SECBUFFER_VERSION\n sec_buffer_desc.cBuffers = 1\n sec_buffer_desc.pBuffers = buffers\n\n result = secur32.ApplyControlToken(self._context_handle_pointer, sec_buffer_desc_pointer)\n handle_error(result, TLSError)\n\n out_sec_buffer_desc_pointer, out_buffers = self._create_buffers(2)\n out_buffers[0].BufferType = Secur32Const.SECBUFFER_TOKEN\n out_buffers[1].BufferType = Secur32Const.SECBUFFER_ALERT\n\n output_context_flags_pointer = new(secur32, 'ULONG *')\n\n result = secur32.InitializeSecurityContextW(\n self._session._credentials_handle,\n self._context_handle_pointer,\n self._hostname,\n self._context_flags,\n 0,\n 0,\n null(),\n 0,\n null(),\n out_sec_buffer_desc_pointer,\n output_context_flags_pointer,\n null()\n )\n acceptable_results = set([\n Secur32Const.SEC_E_OK,\n Secur32Const.SEC_E_CONTEXT_EXPIRED,\n Secur32Const.SEC_I_CONTINUE_NEEDED\n ])\n if result not in acceptable_results:\n handle_error(result, TLSError)\n\n token = bytes_from_buffer(out_buffers[0].pvBuffer, out_buffers[0].cbBuffer)\n try:\n # If there is an error sending the shutdown, ignore it since the\n # connection is likely gone at this point\n self._socket.send(token)\n except (socket_.error):\n pass\n\n finally:\n if out_buffers:\n if not is_null(out_buffers[0].pvBuffer):\n secur32.FreeContextBuffer(out_buffers[0].pvBuffer)\n if not is_null(out_buffers[1].pvBuffer):\n secur32.FreeContextBuffer(out_buffers[1].pvBuffer)\n\n secur32.DeleteSecurityContext(self._context_handle_pointer)\n self._context_handle_pointer = None\n\n try:\n self._socket.shutdown(socket_.SHUT_RDWR)\n except (socket_.error):\n pass"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nclosing the TLS session and socket.", "response": "def close(self):\n \"\"\"\n Shuts down the TLS session and socket and forcibly closes it\n \"\"\"\n\n try:\n self.shutdown()\n\n finally:\n if self._socket:\n try:\n self._socket.close()\n except (socket_.error):\n pass\n self._socket = None"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _read_certificates(self):\n\n cert_context_pointer_pointer = new(crypt32, 'CERT_CONTEXT **')\n result = secur32.QueryContextAttributesW(\n self._context_handle_pointer,\n Secur32Const.SECPKG_ATTR_REMOTE_CERT_CONTEXT,\n cert_context_pointer_pointer\n )\n handle_error(result, TLSError)\n\n cert_context_pointer = unwrap(cert_context_pointer_pointer)\n cert_context_pointer = cast(crypt32, 'CERT_CONTEXT *', cert_context_pointer)\n cert_context = unwrap(cert_context_pointer)\n\n cert_data = bytes_from_buffer(cert_context.pbCertEncoded, native(int, cert_context.cbCertEncoded))\n self._certificate = x509.Certificate.load(cert_data)\n\n self._intermediates = []\n\n store_handle = None\n try:\n store_handle = cert_context.hCertStore\n context_pointer = crypt32.CertEnumCertificatesInStore(store_handle, null())\n while not is_null(context_pointer):\n context = unwrap(context_pointer)\n data = bytes_from_buffer(context.pbCertEncoded, native(int, context.cbCertEncoded))\n # The cert store seems to include the end-entity certificate as\n # the last entry, but we already have that from the struct.\n if data != cert_data:\n self._intermediates.append(x509.Certificate.load(data))\n context_pointer = crypt32.CertEnumCertificatesInStore(store_handle, context_pointer)\n\n finally:\n if store_handle:\n crypt32.CertCloseStore(store_handle, 0)", "response": "Reads end - entity and intermediate certificates from the secur32."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the asn1crypto. x509. Certificate object of the end - entity certificate that is presented by the server.", "response": "def certificate(self):\n \"\"\"\n An asn1crypto.x509.Certificate object of the end-entity certificate\n presented by the server\n \"\"\"\n\n if self._context_handle_pointer is None:\n self._raise_closed()\n\n if self._certificate is None:\n self._read_certificates()\n\n return self._certificate"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef intermediates(self):\n\n if self._context_handle_pointer is None:\n self._raise_closed()\n\n if self._certificate is None:\n self._read_certificates()\n\n return self._intermediates", "response": "A list of asn1crypto. x509. Certificate objects that were presented as an intermediates by the server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef handle_cf_error(error_pointer):\n\n if is_null(error_pointer):\n return\n\n error = unwrap(error_pointer)\n if is_null(error):\n return\n\n cf_string_domain = CoreFoundation.CFErrorGetDomain(error)\n domain = CFHelpers.cf_string_to_unicode(cf_string_domain)\n CoreFoundation.CFRelease(cf_string_domain)\n num = CoreFoundation.CFErrorGetCode(error)\n\n cf_string_ref = CoreFoundation.CFErrorCopyDescription(error)\n output = CFHelpers.cf_string_to_unicode(cf_string_ref)\n CoreFoundation.CFRelease(cf_string_ref)\n\n if output is None:\n if domain == 'NSOSStatusErrorDomain':\n code_map = {\n -2147416010: 'ACL add failed',\n -2147416025: 'ACL base certs not supported',\n -2147416019: 'ACL challenge callback failed',\n -2147416015: 'ACL change failed',\n -2147416012: 'ACL delete failed',\n -2147416017: 'ACL entry tag not found',\n -2147416011: 'ACL replace failed',\n -2147416021: 'ACL subject type not supported',\n -2147415789: 'Algid mismatch',\n -2147415726: 'Already logged in',\n -2147415040: 'Apple add application ACL subject',\n -2147415036: 'Apple invalid key end date',\n -2147415037: 'Apple invalid key start date',\n -2147415039: 'Apple public key incomplete',\n -2147415038: 'Apple signature mismatch',\n -2147415034: 'Apple SSLv2 rollback',\n -2147415802: 'Attach handle busy',\n -2147415731: 'Block size mismatch',\n -2147415722: 'Crypto data callback failed',\n -2147415804: 'Device error',\n -2147415835: 'Device failed',\n -2147415803: 'Device memory error',\n -2147415836: 'Device reset',\n -2147415728: 'Device verify failed',\n -2147416054: 'Function failed',\n -2147416057: 'Function not implemented',\n -2147415807: 'Input length error',\n -2147415837: 'Insufficient client identification',\n -2147416063: 'Internal error',\n -2147416027: 'Invalid access credentials',\n -2147416026: 'Invalid ACL base certs',\n -2147416020: 'Invalid ACL challenge callback',\n -2147416016: 'Invalid ACL edit mode',\n -2147416018: 'Invalid ACL entry tag',\n -2147416022: 'Invalid ACL subject value',\n -2147415759: 'Invalid algorithm',\n -2147415678: 'Invalid attr access credentials',\n -2147415704: 'Invalid attr alg params',\n -2147415686: 'Invalid attr base',\n -2147415738: 'Invalid attr block size',\n -2147415680: 'Invalid attr dl db handle',\n -2147415696: 'Invalid attr effective bits',\n -2147415692: 'Invalid attr end date',\n -2147415752: 'Invalid attr init vector',\n -2147415682: 'Invalid attr iteration count',\n -2147415754: 'Invalid attr key',\n -2147415740: 'Invalid attr key length',\n -2147415700: 'Invalid attr key type',\n -2147415702: 'Invalid attr label',\n -2147415698: 'Invalid attr mode',\n -2147415708: 'Invalid attr output size',\n -2147415748: 'Invalid attr padding',\n -2147415742: 'Invalid attr passphrase',\n -2147415688: 'Invalid attr prime',\n -2147415674: 'Invalid attr private key format',\n -2147415676: 'Invalid attr public key format',\n -2147415746: 'Invalid attr random',\n -2147415706: 'Invalid attr rounds',\n -2147415750: 'Invalid attr salt',\n -2147415744: 'Invalid attr seed',\n -2147415694: 'Invalid attr start date',\n -2147415684: 'Invalid attr subprime',\n -2147415672: 'Invalid attr symmetric key format',\n -2147415690: 'Invalid attr version',\n -2147415670: 'Invalid attr wrapped key format',\n -2147415760: 'Invalid context',\n -2147416000: 'Invalid context handle',\n -2147415976: 'Invalid crypto data',\n -2147415994: 'Invalid data',\n -2147415768: 'Invalid data count',\n -2147415723: 'Invalid digest algorithm',\n -2147416059: 'Invalid input pointer',\n -2147415766: 'Invalid input vector',\n -2147415792: 'Invalid key',\n -2147415780: 'Invalid keyattr mask',\n -2147415782: 'Invalid keyusage mask',\n -2147415790: 'Invalid key class',\n -2147415776: 'Invalid key format',\n -2147415778: 'Invalid key label',\n -2147415783: 'Invalid key pointer',\n -2147415791: 'Invalid key reference',\n -2147415727: 'Invalid login name',\n -2147416014: 'Invalid new ACL entry',\n -2147416013: 'Invalid new ACL owner',\n -2147416058: 'Invalid output pointer',\n -2147415765: 'Invalid output vector',\n -2147415978: 'Invalid passthrough id',\n -2147416060: 'Invalid pointer',\n -2147416024: 'Invalid sample value',\n -2147415733: 'Invalid signature',\n -2147415787: 'Key blob type incorrect',\n -2147415786: 'Key header inconsistent',\n -2147415724: 'Key label already exists',\n -2147415788: 'Key usage incorrect',\n -2147416061: 'Mds error',\n -2147416062: 'Memory error',\n -2147415677: 'Missing attr access credentials',\n -2147415703: 'Missing attr alg params',\n -2147415685: 'Missing attr base',\n -2147415737: 'Missing attr block size',\n -2147415679: 'Missing attr dl db handle',\n -2147415695: 'Missing attr effective bits',\n -2147415691: 'Missing attr end date',\n -2147415751: 'Missing attr init vector',\n -2147415681: 'Missing attr iteration count',\n -2147415753: 'Missing attr key',\n -2147415739: 'Missing attr key length',\n -2147415699: 'Missing attr key type',\n -2147415701: 'Missing attr label',\n -2147415697: 'Missing attr mode',\n -2147415707: 'Missing attr output size',\n -2147415747: 'Missing attr padding',\n -2147415741: 'Missing attr passphrase',\n -2147415687: 'Missing attr prime',\n -2147415673: 'Missing attr private key format',\n -2147415675: 'Missing attr public key format',\n -2147415745: 'Missing attr random',\n -2147415705: 'Missing attr rounds',\n -2147415749: 'Missing attr salt',\n -2147415743: 'Missing attr seed',\n -2147415693: 'Missing attr start date',\n -2147415683: 'Missing attr subprime',\n -2147415671: 'Missing attr symmetric key format',\n -2147415689: 'Missing attr version',\n -2147415669: 'Missing attr wrapped key format',\n -2147415801: 'Not logged in',\n -2147415840: 'No user interaction',\n -2147416029: 'Object ACL not supported',\n -2147416028: 'Object ACL required',\n -2147416030: 'Object manip auth denied',\n -2147416031: 'Object use auth denied',\n -2147416032: 'Operation auth denied',\n -2147416055: 'OS access denied',\n -2147415806: 'Output length error',\n -2147415725: 'Private key already exists',\n -2147415730: 'Private key not found',\n -2147415989: 'Privilege not granted',\n -2147415805: 'Privilege not supported',\n -2147415729: 'Public key inconsistent',\n -2147415732: 'Query size unknown',\n -2147416023: 'Sample value not supported',\n -2147416056: 'Self check failed',\n -2147415838: 'Service not available',\n -2147415736: 'Staged operation in progress',\n -2147415735: 'Staged operation not started',\n -2147415779: 'Unsupported keyattr mask',\n -2147415781: 'Unsupported keyusage mask',\n -2147415785: 'Unsupported key format',\n -2147415777: 'Unsupported key label',\n -2147415784: 'Unsupported key size',\n -2147415839: 'User canceled',\n -2147415767: 'Vector of bufs unsupported',\n -2147415734: 'Verify failed',\n }\n if num in code_map:\n output = code_map[num]\n\n if not output:\n output = '%s %s' % (domain, num)\n\n raise OSError(output)", "response": "Checks a CFErrorRef and throws an exception if there is an error."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new socket with the specified parameters and adds TLS to it.", "response": "def wrap(cls, socket, hostname, session=None):\n \"\"\"\n Takes an existing socket and adds TLS\n\n :param socket:\n A socket.socket object to wrap with TLS\n\n :param hostname:\n A unicode string of the hostname or IP the socket is connected to\n\n :param session:\n An existing TLSSession object to allow for session reuse, specific\n protocol or manual certificate validation\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n \"\"\"\n\n if not isinstance(socket, socket_.socket):\n raise TypeError(pretty_message(\n '''\n socket must be an instance of socket.socket, not %s\n ''',\n type_name(socket)\n ))\n\n if not isinstance(hostname, str_cls):\n raise TypeError(pretty_message(\n '''\n hostname must be a unicode string, not %s\n ''',\n type_name(hostname)\n ))\n\n if session is not None and not isinstance(session, TLSSession):\n raise TypeError(pretty_message(\n '''\n session must be an instance of oscrypto.tls.TLSSession, not %s\n ''',\n type_name(session)\n ))\n\n new_socket = cls(None, None, session=session)\n new_socket._socket = socket\n new_socket._hostname = hostname\n new_socket._handshake()\n\n return new_socket"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nperform an initial TLS handshake.", "response": "def _handshake(self):\n \"\"\"\n Perform an initial TLS handshake\n \"\"\"\n\n self._ssl = None\n self._rbio = None\n self._wbio = None\n\n try:\n self._ssl = libssl.SSL_new(self._session._ssl_ctx)\n if is_null(self._ssl):\n self._ssl = None\n handle_openssl_error(0)\n\n mem_bio = libssl.BIO_s_mem()\n\n self._rbio = libssl.BIO_new(mem_bio)\n if is_null(self._rbio):\n handle_openssl_error(0)\n\n self._wbio = libssl.BIO_new(mem_bio)\n if is_null(self._wbio):\n handle_openssl_error(0)\n\n libssl.SSL_set_bio(self._ssl, self._rbio, self._wbio)\n\n utf8_domain = self._hostname.encode('utf-8')\n libssl.SSL_ctrl(\n self._ssl,\n LibsslConst.SSL_CTRL_SET_TLSEXT_HOSTNAME,\n LibsslConst.TLSEXT_NAMETYPE_host_name,\n utf8_domain\n )\n\n libssl.SSL_set_connect_state(self._ssl)\n\n if self._session._ssl_session:\n libssl.SSL_set_session(self._ssl, self._session._ssl_session)\n\n self._bio_write_buffer = buffer_from_bytes(self._buffer_size)\n self._read_buffer = buffer_from_bytes(self._buffer_size)\n\n handshake_server_bytes = b''\n handshake_client_bytes = b''\n\n while True:\n result = libssl.SSL_do_handshake(self._ssl)\n handshake_client_bytes += self._raw_write()\n\n if result == 1:\n break\n\n error = libssl.SSL_get_error(self._ssl, result)\n if error == LibsslConst.SSL_ERROR_WANT_READ:\n chunk = self._raw_read()\n if chunk == b'':\n if handshake_server_bytes == b'':\n raise_disconnection()\n if detect_client_auth_request(handshake_server_bytes):\n raise_client_auth()\n raise_protocol_error(handshake_server_bytes)\n handshake_server_bytes += chunk\n\n elif error == LibsslConst.SSL_ERROR_WANT_WRITE:\n handshake_client_bytes += self._raw_write()\n\n elif error == LibsslConst.SSL_ERROR_ZERO_RETURN:\n self._gracefully_closed = True\n self._shutdown(False)\n self._raise_closed()\n\n else:\n info = peek_openssl_error()\n\n if libcrypto_version_info < (1, 1):\n dh_key_info = (\n 20,\n LibsslConst.SSL_F_SSL3_CHECK_CERT_AND_ALGORITHM,\n LibsslConst.SSL_R_DH_KEY_TOO_SMALL\n )\n else:\n dh_key_info = (\n 20,\n LibsslConst.SSL_F_TLS_PROCESS_SKE_DHE,\n LibsslConst.SSL_R_DH_KEY_TOO_SMALL\n )\n if info == dh_key_info:\n raise_dh_params()\n\n if libcrypto_version_info < (1, 1):\n unknown_protocol_info = (\n 20,\n LibsslConst.SSL_F_SSL23_GET_SERVER_HELLO,\n LibsslConst.SSL_R_UNKNOWN_PROTOCOL\n )\n else:\n unknown_protocol_info = (\n 20,\n LibsslConst.SSL_F_SSL3_GET_RECORD,\n LibsslConst.SSL_R_WRONG_VERSION_NUMBER\n )\n if info == unknown_protocol_info:\n raise_protocol_error(handshake_server_bytes)\n\n tls_version_info_error = (\n 20,\n LibsslConst.SSL_F_SSL23_GET_SERVER_HELLO,\n LibsslConst.SSL_R_TLSV1_ALERT_PROTOCOL_VERSION\n )\n if info == tls_version_info_error:\n raise_protocol_version()\n\n handshake_error_info = (\n 20,\n LibsslConst.SSL_F_SSL23_GET_SERVER_HELLO,\n LibsslConst.SSL_R_SSLV3_ALERT_HANDSHAKE_FAILURE\n )\n if info == handshake_error_info:\n raise_handshake()\n\n handshake_failure_info = (\n 20,\n LibsslConst.SSL_F_SSL3_READ_BYTES,\n LibsslConst.SSL_R_SSLV3_ALERT_HANDSHAKE_FAILURE\n )\n if info == handshake_failure_info:\n raise_client_auth()\n\n if libcrypto_version_info < (1, 1):\n cert_verify_failed_info = (\n 20,\n LibsslConst.SSL_F_SSL3_GET_SERVER_CERTIFICATE,\n LibsslConst.SSL_R_CERTIFICATE_VERIFY_FAILED\n )\n else:\n cert_verify_failed_info = (\n 20,\n LibsslConst.SSL_F_TLS_PROCESS_SERVER_CERTIFICATE,\n LibsslConst.SSL_R_CERTIFICATE_VERIFY_FAILED\n )\n\n if info == cert_verify_failed_info:\n verify_result = libssl.SSL_get_verify_result(self._ssl)\n chain = extract_chain(handshake_server_bytes)\n\n self_signed = False\n time_invalid = False\n no_issuer = False\n cert = None\n oscrypto_cert = None\n\n if chain:\n cert = chain[0]\n oscrypto_cert = load_certificate(cert)\n self_signed = oscrypto_cert.self_signed\n\n issuer_error_codes = set([\n LibsslConst.X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT,\n LibsslConst.X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN,\n LibsslConst.X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY\n ])\n if verify_result in issuer_error_codes:\n no_issuer = not self_signed\n\n time_error_codes = set([\n LibsslConst.X509_V_ERR_CERT_HAS_EXPIRED,\n LibsslConst.X509_V_ERR_CERT_NOT_YET_VALID\n ])\n time_invalid = verify_result in time_error_codes\n\n if time_invalid:\n raise_expired_not_yet_valid(cert)\n if no_issuer:\n raise_no_issuer(cert)\n if self_signed:\n raise_self_signed(cert)\n if oscrypto_cert and oscrypto_cert.asn1.hash_algo in set(['md5', 'md2']):\n raise_weak_signature(oscrypto_cert)\n raise_verification(cert)\n\n handle_openssl_error(0, TLSError)\n\n session_info = parse_session_info(\n handshake_server_bytes,\n handshake_client_bytes\n )\n self._protocol = session_info['protocol']\n self._cipher_suite = session_info['cipher_suite']\n self._compression = session_info['compression']\n self._session_id = session_info['session_id']\n self._session_ticket = session_info['session_ticket']\n\n if self._cipher_suite.find('_DHE_') != -1:\n dh_params_length = get_dh_params_length(handshake_server_bytes)\n if dh_params_length < 1024:\n self.close()\n raise_dh_params()\n\n # When saving the session for future requests, we use\n # SSL_get1_session() variant to increase the reference count. This\n # prevents the session from being freed when one connection closes\n # before another is opened. However, since we increase the ref\n # count, we also have to explicitly free any previous session.\n if self._session_id == 'new' or self._session_ticket == 'new':\n if self._session._ssl_session:\n libssl.SSL_SESSION_free(self._session._ssl_session)\n self._session._ssl_session = libssl.SSL_get1_session(self._ssl)\n\n if not self._session._manual_validation:\n if self.certificate.hash_algo in set(['md5', 'md2']):\n raise_weak_signature(self.certificate)\n\n # OpenSSL does not do hostname or IP address checking in the end\n # entity certificate, so we must perform that check\n if not self.certificate.is_valid_domain_ip(self._hostname):\n raise_hostname(self.certificate, self._hostname)\n\n except (OSError, socket_.error):\n if self._ssl:\n libssl.SSL_free(self._ssl)\n self._ssl = None\n self._rbio = None\n self._wbio = None\n # The BIOs are freed by SSL_free(), so we only need to free\n # them if for some reason SSL_free() was not called\n else:\n if self._rbio:\n libssl.BIO_free(self._rbio)\n self._rbio = None\n if self._wbio:\n libssl.BIO_free(self._wbio)\n self._wbio = None\n self.close()\n\n raise"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _raw_read(self):\n\n data = self._raw_bytes\n try:\n data += self._socket.recv(8192)\n except (socket_.error):\n pass\n output = data\n written = libssl.BIO_write(self._rbio, data, len(data))\n self._raw_bytes = data[written:]\n return output", "response": "Reads data from the socket and writes it to the memory."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nwrites the ciphertext from the memory bio to the socket.", "response": "def _raw_write(self):\n \"\"\"\n Takes ciphertext from the memory bio and writes it to the\n socket.\n\n :return:\n A byte string of ciphertext going to the socket. Used\n for debugging the handshake only.\n \"\"\"\n\n data_available = libssl.BIO_ctrl_pending(self._wbio)\n if data_available == 0:\n return b''\n to_read = min(self._buffer_size, data_available)\n read = libssl.BIO_read(self._wbio, self._bio_write_buffer, to_read)\n to_write = bytes_from_buffer(self._bio_write_buffer, read)\n output = to_write\n while len(to_write):\n raise_disconnect = False\n try:\n sent = self._socket.send(to_write)\n except (socket_.error) as e:\n # Handle ECONNRESET and EPIPE\n if e.errno == 104 or e.errno == 32:\n raise_disconnect = True\n else:\n raise\n\n if raise_disconnect:\n raise_disconnection()\n to_write = to_write[sent:]\n if len(to_write):\n self.select_write()\n return output"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read(self, max_length):\n\n if not isinstance(max_length, int_types):\n raise TypeError(pretty_message(\n '''\n max_length must be an integer, not %s\n ''',\n type_name(max_length)\n ))\n\n buffered_length = len(self._decrypted_bytes)\n\n # If we already have enough buffered data, just use that\n if buffered_length >= max_length:\n output = self._decrypted_bytes[0:max_length]\n self._decrypted_bytes = self._decrypted_bytes[max_length:]\n return output\n\n if self._ssl is None:\n self._raise_closed()\n\n # Don't block if we have buffered data available, since it is ok to\n # return less than the max_length\n if buffered_length > 0 and not self.select_read(0):\n output = self._decrypted_bytes\n self._decrypted_bytes = b''\n return output\n\n # Only read enough to get the requested amount when\n # combined with buffered data\n to_read = min(self._buffer_size, max_length - buffered_length)\n\n output = self._decrypted_bytes\n\n # The SSL_read() loop handles renegotiations, so we need to handle\n # requests for both reads and writes\n again = True\n while again:\n again = False\n result = libssl.SSL_read(self._ssl, self._read_buffer, to_read)\n self._raw_write()\n if result <= 0:\n\n error = libssl.SSL_get_error(self._ssl, result)\n if error == LibsslConst.SSL_ERROR_WANT_READ:\n if self._raw_read() != b'':\n again = True\n continue\n raise_disconnection()\n\n elif error == LibsslConst.SSL_ERROR_WANT_WRITE:\n self._raw_write()\n again = True\n continue\n\n elif error == LibsslConst.SSL_ERROR_ZERO_RETURN:\n self._gracefully_closed = True\n self._shutdown(False)\n break\n\n else:\n handle_openssl_error(0, TLSError)\n\n output += bytes_from_buffer(self._read_buffer, result)\n\n if self._gracefully_closed and len(output) == 0:\n self._raise_closed()\n\n self._decrypted_bytes = output[max_length:]\n return output[0:max_length]", "response": "Reads data from the TLS - wrapped socket and returns it as a byte string."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nwrites data to the TLS - wrapped socket.", "response": "def write(self, data):\n \"\"\"\n Writes data to the TLS-wrapped socket\n\n :param data:\n A byte string to write to the socket\n\n :raises:\n socket.socket - when a non-TLS socket error occurs\n oscrypto.errors.TLSError - when a TLS-related error occurs\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n \"\"\"\n\n data_len = len(data)\n while data_len:\n if self._ssl is None:\n self._raise_closed()\n result = libssl.SSL_write(self._ssl, data, data_len)\n self._raw_write()\n if result <= 0:\n\n error = libssl.SSL_get_error(self._ssl, result)\n if error == LibsslConst.SSL_ERROR_WANT_READ:\n if self._raw_read() != b'':\n continue\n raise_disconnection()\n\n elif error == LibsslConst.SSL_ERROR_WANT_WRITE:\n self._raw_write()\n continue\n\n elif error == LibsslConst.SSL_ERROR_ZERO_RETURN:\n self._gracefully_closed = True\n self._shutdown(False)\n self._raise_closed()\n\n else:\n handle_openssl_error(0, TLSError)\n\n data = data[result:]\n data_len = len(data)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _shutdown(self, manual):\n\n if self._ssl is None:\n return\n\n while True:\n result = libssl.SSL_shutdown(self._ssl)\n\n # Don't be noisy if the socket is already closed\n try:\n self._raw_write()\n except (TLSDisconnectError):\n pass\n\n if result >= 0:\n break\n if result < 0:\n error = libssl.SSL_get_error(self._ssl, result)\n if error == LibsslConst.SSL_ERROR_WANT_READ:\n if self._raw_read() != b'':\n continue\n else:\n break\n\n elif error == LibsslConst.SSL_ERROR_WANT_WRITE:\n self._raw_write()\n continue\n\n else:\n handle_openssl_error(0, TLSError)\n\n if manual:\n self._local_closed = True\n\n libssl.SSL_free(self._ssl)\n self._ssl = None\n # BIOs are freed by SSL_free()\n self._rbio = None\n self._wbio = None\n\n try:\n self._socket.shutdown(socket_.SHUT_RDWR)\n except (socket_.error):\n pass", "response": "Shuts down the underlying SSL socket and then shuts down the underlying SSL socket."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _read_certificates(self):\n\n stack_pointer = libssl.SSL_get_peer_cert_chain(self._ssl)\n if is_null(stack_pointer):\n handle_openssl_error(0, TLSError)\n\n if libcrypto_version_info < (1, 1):\n number_certs = libssl.sk_num(stack_pointer)\n else:\n number_certs = libssl.OPENSSL_sk_num(stack_pointer)\n\n self._intermediates = []\n\n for index in range(0, number_certs):\n if libcrypto_version_info < (1, 1):\n x509_ = libssl.sk_value(stack_pointer, index)\n else:\n x509_ = libssl.OPENSSL_sk_value(stack_pointer, index)\n buffer_size = libcrypto.i2d_X509(x509_, null())\n cert_buffer = buffer_from_bytes(buffer_size)\n cert_pointer = buffer_pointer(cert_buffer)\n cert_length = libcrypto.i2d_X509(x509_, cert_pointer)\n handle_openssl_error(cert_length)\n cert_data = bytes_from_buffer(cert_buffer, cert_length)\n\n cert = x509.Certificate.load(cert_data)\n\n if index == 0:\n self._certificate = cert\n else:\n self._intermediates.append(cert)", "response": "Reads end - entity and intermediate certificate information from the SSL session."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef certificate(self):\n\n if self._ssl is None:\n self._raise_closed()\n\n if self._certificate is None:\n self._read_certificates()\n\n return self._certificate", "response": "Returns the asn1crypto. x509. Certificate object of the end - entity certificate presented by the server."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef aes_cbc_no_padding_encrypt(key, data, iv):\n\n if len(key) not in [16, 24, 32]:\n raise ValueError(pretty_message(\n '''\n key must be either 16, 24 or 32 bytes (128, 192 or 256 bits)\n long - is %s\n ''',\n len(key)\n ))\n\n if not iv:\n iv = rand_bytes(16)\n elif len(iv) != 16:\n raise ValueError(pretty_message(\n '''\n iv must be 16 bytes long - is %s\n ''',\n len(iv)\n ))\n\n if len(data) % 16 != 0:\n raise ValueError(pretty_message(\n '''\n data must be a multiple of 16 bytes long - is %s\n ''',\n len(data)\n ))\n\n return (iv, _encrypt('aes', key, data, iv, False))", "response": "Encrypts plaintext using AES with a 128 192 or 256 bit key and no padding."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef tripledes_cbc_pkcs5_encrypt(key, data, iv):\n\n if len(key) != 16 and len(key) != 24:\n raise ValueError(pretty_message(\n '''\n key must be 16 bytes (2 key) or 24 bytes (3 key) long - is %s\n ''',\n len(key)\n ))\n\n if not iv:\n iv = rand_bytes(8)\n elif len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n cipher = 'tripledes_3key'\n if len(key) == 16:\n cipher = 'tripledes_2key'\n\n return (iv, _encrypt(cipher, key, data, iv, True))", "response": "Encrypts the data using 3DES in either 2 or 3 key mode and returns the ciphertext of the plaintext."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncreates a handle for the given cipher key and initialization vector.", "response": "def _advapi32_create_handles(cipher, key, iv):\n \"\"\"\n Creates an HCRYPTPROV and HCRYPTKEY for symmetric encryption/decryption. The\n HCRYPTPROV must be released by close_context_handle() and the\n HCRYPTKEY must be released by advapi32.CryptDestroyKey() when done.\n\n :param cipher:\n A unicode string of \"aes\", \"des\", \"tripledes_2key\", \"tripledes_3key\",\n \"rc2\", \"rc4\"\n\n :param key:\n A byte string of the symmetric key\n\n :param iv:\n The initialization vector - a byte string - unused for RC4\n\n :return:\n A tuple of (HCRYPTPROV, HCRYPTKEY)\n \"\"\"\n\n context_handle = None\n\n if cipher == 'aes':\n algorithm_id = {\n 16: Advapi32Const.CALG_AES_128,\n 24: Advapi32Const.CALG_AES_192,\n 32: Advapi32Const.CALG_AES_256,\n }[len(key)]\n else:\n algorithm_id = {\n 'des': Advapi32Const.CALG_DES,\n 'tripledes_2key': Advapi32Const.CALG_3DES_112,\n 'tripledes_3key': Advapi32Const.CALG_3DES,\n 'rc2': Advapi32Const.CALG_RC2,\n 'rc4': Advapi32Const.CALG_RC4,\n }[cipher]\n\n provider = Advapi32Const.MS_ENH_RSA_AES_PROV\n context_handle = open_context_handle(provider, verify_only=False)\n\n blob_header_pointer = struct(advapi32, 'BLOBHEADER')\n blob_header = unwrap(blob_header_pointer)\n blob_header.bType = Advapi32Const.PLAINTEXTKEYBLOB\n blob_header.bVersion = Advapi32Const.CUR_BLOB_VERSION\n blob_header.reserved = 0\n blob_header.aiKeyAlg = algorithm_id\n\n blob_struct_pointer = struct(advapi32, 'PLAINTEXTKEYBLOB')\n blob_struct = unwrap(blob_struct_pointer)\n blob_struct.hdr = blob_header\n blob_struct.dwKeySize = len(key)\n\n blob = struct_bytes(blob_struct_pointer) + key\n\n flags = 0\n if cipher in set(['rc2', 'rc4']) and len(key) == 5:\n flags = Advapi32Const.CRYPT_NO_SALT\n\n key_handle_pointer = new(advapi32, 'HCRYPTKEY *')\n res = advapi32.CryptImportKey(\n context_handle,\n blob,\n len(blob),\n null(),\n flags,\n key_handle_pointer\n )\n handle_error(res)\n\n key_handle = unwrap(key_handle_pointer)\n\n if cipher == 'rc2':\n buf = new(advapi32, 'DWORD *', len(key) * 8)\n res = advapi32.CryptSetKeyParam(\n key_handle,\n Advapi32Const.KP_EFFECTIVE_KEYLEN,\n buf,\n 0\n )\n handle_error(res)\n\n if cipher != 'rc4':\n res = advapi32.CryptSetKeyParam(\n key_handle,\n Advapi32Const.KP_IV,\n iv,\n 0\n )\n handle_error(res)\n\n buf = new(advapi32, 'DWORD *', Advapi32Const.CRYPT_MODE_CBC)\n res = advapi32.CryptSetKeyParam(\n key_handle,\n Advapi32Const.KP_MODE,\n buf,\n 0\n )\n handle_error(res)\n\n buf = new(advapi32, 'DWORD *', Advapi32Const.PKCS5_PADDING)\n res = advapi32.CryptSetKeyParam(\n key_handle,\n Advapi32Const.KP_PADDING,\n buf,\n 0\n )\n handle_error(res)\n\n return (context_handle, key_handle)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncreating a BCRYPT_KEY_HANDLE for symmetric encryption and decryption.", "response": "def _bcrypt_create_key_handle(cipher, key):\n \"\"\"\n Creates a BCRYPT_KEY_HANDLE for symmetric encryption/decryption. The\n handle must be released by bcrypt.BCryptDestroyKey() when done.\n\n :param cipher:\n A unicode string of \"aes\", \"des\", \"tripledes_2key\", \"tripledes_3key\",\n \"rc2\", \"rc4\"\n\n :param key:\n A byte string of the symmetric key\n\n :return:\n A BCRYPT_KEY_HANDLE\n \"\"\"\n\n alg_handle = None\n\n alg_constant = {\n 'aes': BcryptConst.BCRYPT_AES_ALGORITHM,\n 'des': BcryptConst.BCRYPT_DES_ALGORITHM,\n 'tripledes_2key': BcryptConst.BCRYPT_3DES_112_ALGORITHM,\n 'tripledes_3key': BcryptConst.BCRYPT_3DES_ALGORITHM,\n 'rc2': BcryptConst.BCRYPT_RC2_ALGORITHM,\n 'rc4': BcryptConst.BCRYPT_RC4_ALGORITHM,\n }[cipher]\n\n try:\n alg_handle = open_alg_handle(alg_constant)\n blob_type = BcryptConst.BCRYPT_KEY_DATA_BLOB\n\n blob_struct_pointer = struct(bcrypt, 'BCRYPT_KEY_DATA_BLOB_HEADER')\n blob_struct = unwrap(blob_struct_pointer)\n blob_struct.dwMagic = BcryptConst.BCRYPT_KEY_DATA_BLOB_MAGIC\n blob_struct.dwVersion = BcryptConst.BCRYPT_KEY_DATA_BLOB_VERSION1\n blob_struct.cbKeyData = len(key)\n\n blob = struct_bytes(blob_struct_pointer) + key\n\n if cipher == 'rc2':\n buf = new(bcrypt, 'DWORD *', len(key) * 8)\n res = bcrypt.BCryptSetProperty(\n alg_handle,\n BcryptConst.BCRYPT_EFFECTIVE_KEY_LENGTH,\n buf,\n 4,\n 0\n )\n handle_error(res)\n\n key_handle_pointer = new(bcrypt, 'BCRYPT_KEY_HANDLE *')\n res = bcrypt.BCryptImportKey(\n alg_handle,\n null(),\n blob_type,\n key_handle_pointer,\n null(),\n 0,\n blob,\n len(blob),\n 0\n )\n handle_error(res)\n\n return unwrap(key_handle_pointer)\n\n finally:\n if alg_handle:\n close_alg_handle(alg_handle)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _encrypt(cipher, key, data, iv, padding):\n\n if not isinstance(key, byte_cls):\n raise TypeError(pretty_message(\n '''\n key must be a byte string, not %s\n ''',\n type_name(key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if cipher != 'rc4' and not isinstance(iv, byte_cls):\n raise TypeError(pretty_message(\n '''\n iv must be a byte string, not %s\n ''',\n type_name(iv)\n ))\n\n if cipher != 'rc4' and not padding:\n raise ValueError('padding must be specified')\n\n if _backend == 'winlegacy':\n return _advapi32_encrypt(cipher, key, data, iv, padding)\n return _bcrypt_encrypt(cipher, key, data, iv, padding)", "response": "Encrypts the plaintext data with the specified cipher key and initialization vector."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nencrypts plaintext via CryptoAPI.", "response": "def _advapi32_encrypt(cipher, key, data, iv, padding):\n \"\"\"\n Encrypts plaintext via CryptoAPI\n\n :param cipher:\n A unicode string of \"aes\", \"des\", \"tripledes_2key\", \"tripledes_3key\",\n \"rc2\", \"rc4\"\n\n :param key:\n The encryption key - a byte string 5-16 bytes long\n\n :param data:\n The plaintext - a byte string\n\n :param iv:\n The initialization vector - a byte string - unused for RC4\n\n :param padding:\n Boolean, if padding should be used - unused for RC4\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the ciphertext\n \"\"\"\n\n context_handle = None\n key_handle = None\n\n try:\n context_handle, key_handle = _advapi32_create_handles(cipher, key, iv)\n\n out_len = new(advapi32, 'DWORD *', len(data))\n res = advapi32.CryptEncrypt(\n key_handle,\n null(),\n True,\n 0,\n null(),\n out_len,\n 0\n )\n handle_error(res)\n\n buffer_len = deref(out_len)\n buffer = buffer_from_bytes(buffer_len)\n write_to_buffer(buffer, data)\n\n pointer_set(out_len, len(data))\n res = advapi32.CryptEncrypt(\n key_handle,\n null(),\n True,\n 0,\n buffer,\n out_len,\n buffer_len\n )\n handle_error(res)\n\n output = bytes_from_buffer(buffer, deref(out_len))\n\n # Remove padding when not required. CryptoAPI doesn't support this, so\n # we just manually remove it.\n if cipher == 'aes' and not padding:\n if output[-16:] != (b'\\x10' * 16):\n raise ValueError('Invalid padding generated by OS crypto library')\n output = output[:-16]\n\n return output\n\n finally:\n if key_handle:\n advapi32.CryptDestroyKey(key_handle)\n if context_handle:\n close_context_handle(context_handle)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nencrypts plaintext via BCrypt", "response": "def _bcrypt_encrypt(cipher, key, data, iv, padding):\n \"\"\"\n Encrypts plaintext via CNG\n\n :param cipher:\n A unicode string of \"aes\", \"des\", \"tripledes_2key\", \"tripledes_3key\",\n \"rc2\", \"rc4\"\n\n :param key:\n The encryption key - a byte string 5-16 bytes long\n\n :param data:\n The plaintext - a byte string\n\n :param iv:\n The initialization vector - a byte string - unused for RC4\n\n :param padding:\n Boolean, if padding should be used - unused for RC4\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the ciphertext\n \"\"\"\n\n key_handle = None\n\n try:\n key_handle = _bcrypt_create_key_handle(cipher, key)\n\n if iv is None:\n iv_len = 0\n else:\n iv_len = len(iv)\n\n flags = 0\n if padding is True:\n flags = BcryptConst.BCRYPT_BLOCK_PADDING\n\n out_len = new(bcrypt, 'ULONG *')\n res = bcrypt.BCryptEncrypt(\n key_handle,\n data,\n len(data),\n null(),\n null(),\n 0,\n null(),\n 0,\n out_len,\n flags\n )\n handle_error(res)\n\n buffer_len = deref(out_len)\n buffer = buffer_from_bytes(buffer_len)\n iv_buffer = buffer_from_bytes(iv) if iv else null()\n\n res = bcrypt.BCryptEncrypt(\n key_handle,\n data,\n len(data),\n null(),\n iv_buffer,\n iv_len,\n buffer,\n buffer_len,\n out_len,\n flags\n )\n handle_error(res)\n\n return bytes_from_buffer(buffer, deref(out_len))\n\n finally:\n if key_handle:\n bcrypt.BCryptDestroyKey(key_handle)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _decrypt(cipher, key, data, iv, padding):\n\n if not isinstance(key, byte_cls):\n raise TypeError(pretty_message(\n '''\n key must be a byte string, not %s\n ''',\n type_name(key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if cipher != 'rc4' and not isinstance(iv, byte_cls):\n raise TypeError(pretty_message(\n '''\n iv must be a byte string, not %s\n ''',\n type_name(iv)\n ))\n\n if cipher != 'rc4' and padding is None:\n raise ValueError('padding must be specified')\n\n if _backend == 'winlegacy':\n return _advapi32_decrypt(cipher, key, data, iv, padding)\n return _bcrypt_decrypt(cipher, key, data, iv, padding)", "response": "Decrypts the ciphertext using the specified cipher and key and data."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _advapi32_decrypt(cipher, key, data, iv, padding):\n\n context_handle = None\n key_handle = None\n\n try:\n context_handle, key_handle = _advapi32_create_handles(cipher, key, iv)\n\n # Add removed padding when not required. CryptoAPI doesn't support no\n # padding, so we just add it back in\n if cipher == 'aes' and not padding:\n data += (b'\\x10' * 16)\n\n buffer = buffer_from_bytes(data)\n out_len = new(advapi32, 'DWORD *', len(data))\n res = advapi32.CryptDecrypt(\n key_handle,\n null(),\n True,\n 0,\n buffer,\n out_len\n )\n handle_error(res)\n\n return bytes_from_buffer(buffer, deref(out_len))\n\n finally:\n if key_handle:\n advapi32.CryptDestroyKey(key_handle)\n if context_handle:\n close_context_handle(context_handle)", "response": "Decrypts the ciphertext using the AES - CBC using the specified key and data."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nhandles Windows errors and returns a unicode string", "response": "def handle_error(error_num):\n \"\"\"\n Extracts the last Windows error message into a python unicode string\n\n :param error_num:\n The number to get the error string for\n\n :return:\n A unicode string error message\n \"\"\"\n\n if error_num == 0:\n return\n\n messages = {\n BcryptConst.STATUS_NOT_FOUND: 'The object was not found',\n BcryptConst.STATUS_INVALID_PARAMETER: 'An invalid parameter was passed to a service or function',\n BcryptConst.STATUS_NO_MEMORY: (\n 'Not enough virtual memory or paging file quota is available to complete the specified operation'\n ),\n BcryptConst.STATUS_INVALID_HANDLE: 'An invalid HANDLE was specified',\n BcryptConst.STATUS_INVALID_SIGNATURE: 'The cryptographic signature is invalid',\n BcryptConst.STATUS_NOT_SUPPORTED: 'The request is not supported',\n BcryptConst.STATUS_BUFFER_TOO_SMALL: 'The buffer is too small to contain the entry',\n BcryptConst.STATUS_INVALID_BUFFER_SIZE: 'The size of the buffer is invalid for the specified operation',\n }\n\n output = 'NTSTATUS error 0x%0.2X' % error_num\n\n if error_num is not None and error_num in messages:\n output += ': ' + messages[error_num]\n\n raise OSError(output)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef handle_openssl_error(result, exception_class=None):\n\n if result > 0:\n return\n\n if exception_class is None:\n exception_class = OSError\n\n error_num = libcrypto.ERR_get_error()\n buffer = buffer_from_bytes(120)\n libcrypto.ERR_error_string(error_num, buffer)\n\n # Since we are dealing with a string, it is NULL terminated\n error_string = byte_string_from_buffer(buffer)\n\n raise exception_class(_try_decode(error_string))", "response": "Checks if an error occured and raises an exception."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef peek_openssl_error():\n\n error = libcrypto.ERR_peek_error()\n lib = int((error >> 24) & 0xff)\n func = int((error >> 12) & 0xfff)\n reason = int(error & 0xfff)\n\n return (lib, func, reason)", "response": "Peeks into the error stack and pulls out the lib func and reason"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_pss_padding(hash_algorithm, salt_length, key_length, message):\n\n if _backend != 'winlegacy' and sys.platform != 'darwin':\n raise SystemError(pretty_message(\n '''\n Pure-python RSA PSS signature padding addition code is only for\n Windows XP/2003 and OS X\n '''\n ))\n\n if not isinstance(message, byte_cls):\n raise TypeError(pretty_message(\n '''\n message must be a byte string, not %s\n ''',\n type_name(message)\n ))\n\n if not isinstance(salt_length, int_types):\n raise TypeError(pretty_message(\n '''\n salt_length must be an integer, not %s\n ''',\n type_name(salt_length)\n ))\n\n if salt_length < 0:\n raise ValueError(pretty_message(\n '''\n salt_length must be 0 or more - is %s\n ''',\n repr(salt_length)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 512:\n raise ValueError(pretty_message(\n '''\n key_length must be 512 or more - is %s\n ''',\n repr(key_length)\n ))\n\n if hash_algorithm not in set(['sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n hash_func = getattr(hashlib, hash_algorithm)\n\n # The maximal bit size of a non-negative integer is one less than the bit\n # size of the key since the first bit is used to store sign\n em_bits = key_length - 1\n em_len = int(math.ceil(em_bits / 8))\n\n message_digest = hash_func(message).digest()\n hash_length = len(message_digest)\n\n if em_len < hash_length + salt_length + 2:\n raise ValueError(pretty_message(\n '''\n Key is not long enough to use with specified hash_algorithm and\n salt_length\n '''\n ))\n\n if salt_length > 0:\n salt = os.urandom(salt_length)\n else:\n salt = b''\n\n m_prime = (b'\\x00' * 8) + message_digest + salt\n\n m_prime_digest = hash_func(m_prime).digest()\n\n padding = b'\\x00' * (em_len - salt_length - hash_length - 2)\n\n db = padding + b'\\x01' + salt\n\n db_mask = _mgf1(hash_algorithm, m_prime_digest, em_len - hash_length - 1)\n\n masked_db = int_to_bytes(int_from_bytes(db) ^ int_from_bytes(db_mask))\n masked_db = fill_width(masked_db, len(db_mask))\n\n zero_bits = (8 * em_len) - em_bits\n left_bit_mask = ('0' * zero_bits) + ('1' * (8 - zero_bits))\n left_int_mask = int(left_bit_mask, 2)\n\n if left_int_mask != 255:\n masked_db = chr_cls(left_int_mask & ord(masked_db[0:1])) + masked_db[1:]\n\n return masked_db + m_prime_digest + b'\\xBC'", "response": "Adds padding to a byte string using the EMSA - PSS - Encode operation described in PKCS#1\n v2. 2."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nverifying the PSS padding on an encoded message.", "response": "def verify_pss_padding(hash_algorithm, salt_length, key_length, message, signature):\n \"\"\"\n Verifies the PSS padding on an encoded message\n\n :param hash_algorithm:\n The string name of the hash algorithm to use: \"sha1\", \"sha224\",\n \"sha256\", \"sha384\", \"sha512\"\n\n :param salt_length:\n The length of the salt as an integer - typically the same as the length\n of the output from the hash_algorithm\n\n :param key_length:\n The length of the RSA key, in bits\n\n :param message:\n A byte string of the message to pad\n\n :param signature:\n The signature to verify\n\n :return:\n A boolean indicating if the signature is invalid\n \"\"\"\n\n if _backend != 'winlegacy' and sys.platform != 'darwin':\n raise SystemError(pretty_message(\n '''\n Pure-python RSA PSS signature padding verification code is only for\n Windows XP/2003 and OS X\n '''\n ))\n\n if not isinstance(message, byte_cls):\n raise TypeError(pretty_message(\n '''\n message must be a byte string, not %s\n ''',\n type_name(message)\n ))\n\n if not isinstance(signature, byte_cls):\n raise TypeError(pretty_message(\n '''\n signature must be a byte string, not %s\n ''',\n type_name(signature)\n ))\n\n if not isinstance(salt_length, int_types):\n raise TypeError(pretty_message(\n '''\n salt_length must be an integer, not %s\n ''',\n type_name(salt_length)\n ))\n\n if salt_length < 0:\n raise ValueError(pretty_message(\n '''\n salt_length must be 0 or more - is %s\n ''',\n repr(salt_length)\n ))\n\n if hash_algorithm not in set(['sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n hash_func = getattr(hashlib, hash_algorithm)\n\n em_bits = key_length - 1\n em_len = int(math.ceil(em_bits / 8))\n\n message_digest = hash_func(message).digest()\n hash_length = len(message_digest)\n\n if em_len < hash_length + salt_length + 2:\n return False\n\n if signature[-1:] != b'\\xBC':\n return False\n\n zero_bits = (8 * em_len) - em_bits\n\n masked_db_length = em_len - hash_length - 1\n masked_db = signature[0:masked_db_length]\n\n first_byte = ord(masked_db[0:1])\n bits_that_should_be_zero = first_byte >> (8 - zero_bits)\n if bits_that_should_be_zero != 0:\n return False\n\n m_prime_digest = signature[masked_db_length:masked_db_length + hash_length]\n\n db_mask = _mgf1(hash_algorithm, m_prime_digest, em_len - hash_length - 1)\n\n left_bit_mask = ('0' * zero_bits) + ('1' * (8 - zero_bits))\n left_int_mask = int(left_bit_mask, 2)\n\n if left_int_mask != 255:\n db_mask = chr_cls(left_int_mask & ord(db_mask[0:1])) + db_mask[1:]\n\n db = int_to_bytes(int_from_bytes(masked_db) ^ int_from_bytes(db_mask))\n if len(db) < len(masked_db):\n db = (b'\\x00' * (len(masked_db) - len(db))) + db\n\n zero_length = em_len - hash_length - salt_length - 2\n zero_string = b'\\x00' * zero_length\n if not constant_compare(db[0:zero_length], zero_string):\n return False\n\n if db[zero_length:zero_length + 1] != b'\\x01':\n return False\n\n salt = db[0 - salt_length:]\n\n m_prime = (b'\\x00' * 8) + message_digest + salt\n\n h_prime = hash_func(m_prime).digest()\n\n return constant_compare(m_prime_digest, h_prime)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _mgf1(hash_algorithm, seed, mask_length):\n\n if not isinstance(seed, byte_cls):\n raise TypeError(pretty_message(\n '''\n seed must be a byte string, not %s\n ''',\n type_name(seed)\n ))\n\n if not isinstance(mask_length, int_types):\n raise TypeError(pretty_message(\n '''\n mask_length must be an integer, not %s\n ''',\n type_name(mask_length)\n ))\n\n if mask_length < 1:\n raise ValueError(pretty_message(\n '''\n mask_length must be greater than 0 - is %s\n ''',\n repr(mask_length)\n ))\n\n if hash_algorithm not in set(['sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n output = b''\n\n hash_length = {\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }[hash_algorithm]\n\n iterations = int(math.ceil(mask_length / hash_length))\n\n pack = struct.Struct(b'>I').pack\n hash_func = getattr(hashlib, hash_algorithm)\n\n for counter in range(0, iterations):\n b = pack(counter)\n output += hash_func(seed + b).digest()\n\n return output[0:mask_length]", "response": "Generate a PKCS#1 MGF1 mask from the seed and the desired length."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding padding to a message containing PKCS#1 v1. 5 padding.", "response": "def _add_pkcs1v15_padding(key_length, data, operation):\n \"\"\"\n Adds PKCS#1 v1.5 padding to a message\n\n :param key_length:\n An integer of the number of bytes in the key\n\n :param data:\n A byte string to unpad\n\n :param operation:\n A unicode string of \"encrypting\" or \"signing\"\n\n :return:\n The padded data as a byte string\n \"\"\"\n\n if operation == 'encrypting':\n second_byte = b'\\x02'\n else:\n second_byte = b'\\x01'\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 64:\n raise ValueError(pretty_message(\n '''\n key_length must be 64 or more - is %s\n ''',\n repr(key_length)\n ))\n\n if len(data) > key_length - 11:\n raise ValueError(pretty_message(\n '''\n data must be between 1 and %s bytes long - is %s\n ''',\n key_length - 11,\n len(data)\n ))\n\n required_bytes = key_length - 3 - len(data)\n padding = b''\n while required_bytes > 0:\n temp_padding = rand_bytes(required_bytes)\n # Remove null bytes since they are markers in PKCS#1 v1.5\n temp_padding = b''.join(temp_padding.split(b'\\x00'))\n padding += temp_padding\n required_bytes -= len(temp_padding)\n\n return b'\\x00' + second_byte + padding + b'\\x00' + data"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove PKCS#1 v1.5 padding from a message using constant time operations :param key_length: An integer of the number of bytes in the key :param data: A byte string to unpad :param operation: A unicode string of \"decrypting\" or \"verifying\" :return: The unpadded data as a byte string", "response": "def _remove_pkcs1v15_padding(key_length, data, operation):\n \"\"\"\n Removes PKCS#1 v1.5 padding from a message using constant time operations\n\n :param key_length:\n An integer of the number of bytes in the key\n\n :param data:\n A byte string to unpad\n\n :param operation:\n A unicode string of \"decrypting\" or \"verifying\"\n\n :return:\n The unpadded data as a byte string\n \"\"\"\n\n if operation == 'decrypting':\n second_byte = 2\n else:\n second_byte = 1\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 64:\n raise ValueError(pretty_message(\n '''\n key_length must be 64 or more - is %s\n ''',\n repr(key_length)\n ))\n\n if len(data) != key_length:\n raise ValueError('Error %s' % operation)\n\n error = 0\n trash = 0\n padding_end = 0\n\n # Uses bitwise operations on an error variable and another trash variable\n # to perform constant time error checking/token scanning on the data\n for i in range(0, len(data)):\n byte = data[i:i + 1]\n byte_num = ord(byte)\n\n # First byte should be \\x00\n if i == 0:\n error |= byte_num\n\n # Second byte should be \\x02 for decryption, \\x01 for verification\n elif i == 1:\n error |= int((byte_num | second_byte) != second_byte)\n\n # Bytes 3-10 should not be \\x00\n elif i < 10:\n error |= int((byte_num ^ 0) == 0)\n\n # Byte 11 or after that is zero is end of padding\n else:\n non_zero = byte_num | 0\n if padding_end == 0:\n if non_zero:\n trash |= i\n else:\n padding_end |= i\n else:\n if non_zero:\n trash |= i\n else:\n trash |= i\n\n if error != 0:\n raise ValueError('Error %s' % operation)\n\n return data[padding_end + 1:]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef raw_rsa_private_crypt(private_key, data):\n\n if _backend != 'winlegacy':\n raise SystemError('Pure-python RSA crypt is only for Windows XP/2003')\n\n if not hasattr(private_key, 'asn1') or not isinstance(private_key.asn1, PrivateKeyInfo):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of the\n oscrypto.asymmetric.PrivateKey class, not %s\n ''',\n type_name(private_key)\n ))\n\n algo = private_key.asn1['private_key_algorithm']['algorithm'].native\n if algo != 'rsa':\n raise ValueError(pretty_message(\n '''\n private_key must be an RSA key, not %s\n ''',\n algo.upper()\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n rsa_private_key = private_key.asn1['private_key'].parsed\n transformed_int = pow(\n int_from_bytes(data),\n rsa_private_key['private_exponent'].native,\n rsa_private_key['modulus'].native\n )\n return int_to_bytes(transformed_int, width=private_key.asn1.byte_size)", "response": "This function performs a raw RSA crypt of a byte string using a private key."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef raw_rsa_public_crypt(certificate_or_public_key, data):\n\n if _backend != 'winlegacy':\n raise SystemError('Pure-python RSA crypt is only for Windows XP/2003')\n\n has_asn1 = hasattr(certificate_or_public_key, 'asn1')\n valid_types = (PublicKeyInfo, Certificate)\n if not has_asn1 or not isinstance(certificate_or_public_key.asn1, valid_types):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the\n oscrypto.asymmetric.PublicKey or oscrypto.asymmetric.Certificate\n classes, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n algo = certificate_or_public_key.asn1['algorithm']['algorithm'].native\n if algo != 'rsa':\n raise ValueError(pretty_message(\n '''\n certificate_or_public_key must be an RSA key, not %s\n ''',\n algo.upper()\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n rsa_public_key = certificate_or_public_key.asn1['public_key'].parsed\n transformed_int = pow(\n int_from_bytes(data),\n rsa_public_key['public_exponent'].native,\n rsa_public_key['modulus'].native\n )\n return int_to_bytes(\n transformed_int,\n width=certificate_or_public_key.asn1.byte_size\n )", "response": "This function performs a raw RSA crypt in a byte string using a certificate or public key."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pkcs12_kdf(hash_algorithm, password, salt, iterations, key_length, id_):\n\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n\n if not isinstance(salt, byte_cls):\n raise TypeError(pretty_message(\n '''\n salt must be a byte string, not %s\n ''',\n type_name(salt)\n ))\n\n if not isinstance(iterations, int_types):\n raise TypeError(pretty_message(\n '''\n iterations must be an integer, not %s\n ''',\n type_name(iterations)\n ))\n\n if iterations < 1:\n raise ValueError(pretty_message(\n '''\n iterations must be greater than 0 - is %s\n ''',\n repr(iterations)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 1:\n raise ValueError(pretty_message(\n '''\n key_length must be greater than 0 - is %s\n ''',\n repr(key_length)\n ))\n\n if hash_algorithm not in set(['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"md5\", \"sha1\", \"sha224\", \"sha256\",\n \"sha384\", \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n if id_ not in set([1, 2, 3]):\n raise ValueError(pretty_message(\n '''\n id_ must be one of 1, 2, 3, not %s\n ''',\n repr(id_)\n ))\n\n utf16_password = password.decode('utf-8').encode('utf-16be') + b'\\x00\\x00'\n\n digest_type = {\n 'md5': libcrypto.EVP_md5,\n 'sha1': libcrypto.EVP_sha1,\n 'sha224': libcrypto.EVP_sha224,\n 'sha256': libcrypto.EVP_sha256,\n 'sha384': libcrypto.EVP_sha384,\n 'sha512': libcrypto.EVP_sha512,\n }[hash_algorithm]()\n\n output_buffer = buffer_from_bytes(key_length)\n result = libcrypto.PKCS12_key_gen_uni(\n utf16_password,\n len(utf16_password),\n salt,\n len(salt),\n id_,\n iterations,\n key_length,\n output_buffer,\n digest_type\n )\n handle_openssl_error(result)\n\n return bytes_from_buffer(output_buffer)", "response": "Derives a key from the key in a single pkcs12 format"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nextracting trusted CA certificates from the OS X trusted root keychain.", "response": "def extract_from_system(cert_callback=None, callback_only_on_failure=False):\n \"\"\"\n Extracts trusted CA certificates from the OS X trusted root keychain.\n\n :param cert_callback:\n A callback that is called once for each certificate in the trust store.\n It should accept two parameters: an asn1crypto.x509.Certificate object,\n and a reason. The reason will be None if the certificate is being\n exported, otherwise it will be a unicode string of the reason it won't.\n\n :param callback_only_on_failure:\n A boolean - if the callback should only be called when a certificate is\n not exported.\n\n :raises:\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A list of 3-element tuples:\n - 0: a byte string of a DER-encoded certificate\n - 1: a set of unicode strings that are OIDs of purposes to trust the\n certificate for\n - 2: a set of unicode strings that are OIDs of purposes to reject the\n certificate for\n \"\"\"\n\n certs_pointer_pointer = new(CoreFoundation, 'CFArrayRef *')\n res = Security.SecTrustCopyAnchorCertificates(certs_pointer_pointer)\n handle_sec_error(res)\n\n certs_pointer = unwrap(certs_pointer_pointer)\n\n certificates = {}\n trust_info = {}\n\n all_purposes = '2.5.29.37.0'\n default_trust = (set(), set())\n\n length = CoreFoundation.CFArrayGetCount(certs_pointer)\n for index in range(0, length):\n cert_pointer = CoreFoundation.CFArrayGetValueAtIndex(certs_pointer, index)\n der_cert, cert_hash = _cert_details(cert_pointer)\n certificates[cert_hash] = der_cert\n\n CoreFoundation.CFRelease(certs_pointer)\n\n for domain in [SecurityConst.kSecTrustSettingsDomainUser, SecurityConst.kSecTrustSettingsDomainAdmin]:\n cert_trust_settings_pointer_pointer = new(CoreFoundation, 'CFArrayRef *')\n res = Security.SecTrustSettingsCopyCertificates(domain, cert_trust_settings_pointer_pointer)\n if res == SecurityConst.errSecNoTrustSettings:\n continue\n handle_sec_error(res)\n\n cert_trust_settings_pointer = unwrap(cert_trust_settings_pointer_pointer)\n\n length = CoreFoundation.CFArrayGetCount(cert_trust_settings_pointer)\n for index in range(0, length):\n cert_pointer = CoreFoundation.CFArrayGetValueAtIndex(cert_trust_settings_pointer, index)\n\n trust_settings_pointer_pointer = new(CoreFoundation, 'CFArrayRef *')\n res = Security.SecTrustSettingsCopyTrustSettings(cert_pointer, domain, trust_settings_pointer_pointer)\n\n # In OS X 10.11, this value started being seen. From the comments in\n # the Security Framework Reference, the lack of any settings should\n # indicate \"always trust this certificate\"\n if res == SecurityConst.errSecItemNotFound:\n continue\n\n # If the trust settings for a certificate are invalid, we need to\n # assume the certificate should not be trusted\n if res == SecurityConst.errSecInvalidTrustSettings:\n der_cert, cert_hash = _cert_details(cert_pointer)\n if cert_hash in certificates:\n _cert_callback(\n cert_callback,\n certificates[cert_hash],\n 'invalid trust settings'\n )\n del certificates[cert_hash]\n continue\n\n handle_sec_error(res)\n\n trust_settings_pointer = unwrap(trust_settings_pointer_pointer)\n\n trust_oids = set()\n reject_oids = set()\n settings_length = CoreFoundation.CFArrayGetCount(trust_settings_pointer)\n for settings_index in range(0, settings_length):\n settings_dict_entry = CoreFoundation.CFArrayGetValueAtIndex(trust_settings_pointer, settings_index)\n settings_dict = CFHelpers.cf_dictionary_to_dict(settings_dict_entry)\n\n # No policy OID means the trust result is for all purposes\n policy_oid = settings_dict.get('kSecTrustSettingsPolicy', {}).get('SecPolicyOid', all_purposes)\n\n # 0 = kSecTrustSettingsResultInvalid\n # 1 = kSecTrustSettingsResultTrustRoot\n # 2 = kSecTrustSettingsResultTrustAsRoot\n # 3 = kSecTrustSettingsResultDeny\n # 4 = kSecTrustSettingsResultUnspecified\n trust_result = settings_dict.get('kSecTrustSettingsResult', 1)\n should_trust = trust_result != 0 and trust_result != 3\n\n if should_trust:\n trust_oids.add(policy_oid)\n else:\n reject_oids.add(policy_oid)\n\n der_cert, cert_hash = _cert_details(cert_pointer)\n\n # If rejected for all purposes, we don't export the certificate\n if all_purposes in reject_oids:\n if cert_hash in certificates:\n _cert_callback(\n cert_callback,\n certificates[cert_hash],\n 'explicitly distrusted'\n )\n del certificates[cert_hash]\n else:\n if all_purposes in trust_oids:\n trust_oids = set([all_purposes])\n trust_info[cert_hash] = (trust_oids, reject_oids)\n\n CoreFoundation.CFRelease(trust_settings_pointer)\n\n CoreFoundation.CFRelease(cert_trust_settings_pointer)\n\n output = []\n for cert_hash in certificates:\n if not callback_only_on_failure:\n _cert_callback(cert_callback, certificates[cert_hash], None)\n cert_trust_info = trust_info.get(cert_hash, default_trust)\n output.append((certificates[cert_hash], cert_trust_info[0], cert_trust_info[1]))\n return output"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _cert_callback(callback, der_cert, reason):\n\n if not callback:\n return\n callback(x509.Certificate.load(der_cert), reason)", "response": "This function is used to call the callback function on the ASN. 1. 1 certificate object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the certificate and a hash of the certificate", "response": "def _cert_details(cert_pointer):\n \"\"\"\n Return the certificate and a hash of it\n\n :param cert_pointer:\n A SecCertificateRef\n\n :return:\n A 2-element tuple:\n - [0]: A byte string of the SHA1 hash of the cert\n - [1]: A byte string of the DER-encoded contents of the cert\n \"\"\"\n\n data_pointer = None\n\n try:\n data_pointer = Security.SecCertificateCopyData(cert_pointer)\n der_cert = CFHelpers.cf_data_to_bytes(data_pointer)\n cert_hash = hashlib.sha1(der_cert).digest()\n\n return (der_cert, cert_hash)\n\n finally:\n if data_pointer is not None:\n CoreFoundation.CFRelease(data_pointer)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _extract_error():\n\n error_num = errno()\n\n try:\n error_string = os.strerror(error_num)\n except (ValueError):\n return str_cls(error_num)\n\n if isinstance(error_string, str_cls):\n return error_string\n\n return _try_decode(error_string)", "response": "Extracts the last OS error message into a python unicode string"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pbkdf2(hash_algorithm, password, salt, iterations, key_length):\n\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n\n if not isinstance(salt, byte_cls):\n raise TypeError(pretty_message(\n '''\n salt must be a byte string, not %s\n ''',\n type_name(salt)\n ))\n\n if not isinstance(iterations, int_types):\n raise TypeError(pretty_message(\n '''\n iterations must be an integer, not %s\n ''',\n type_name(iterations)\n ))\n\n if iterations < 1:\n raise ValueError('iterations must be greater than 0')\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 1:\n raise ValueError('key_length must be greater than 0')\n\n if hash_algorithm not in set(['sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n algo = {\n 'sha1': CommonCryptoConst.kCCPRFHmacAlgSHA1,\n 'sha224': CommonCryptoConst.kCCPRFHmacAlgSHA224,\n 'sha256': CommonCryptoConst.kCCPRFHmacAlgSHA256,\n 'sha384': CommonCryptoConst.kCCPRFHmacAlgSHA384,\n 'sha512': CommonCryptoConst.kCCPRFHmacAlgSHA512\n }[hash_algorithm]\n\n output_buffer = buffer_from_bytes(key_length)\n result = CommonCrypto.CCKeyDerivationPBKDF(\n CommonCryptoConst.kCCPBKDF2,\n password,\n len(password),\n salt,\n len(salt),\n algo,\n iterations,\n output_buffer,\n key_length\n )\n if result != 0:\n raise OSError(_extract_error())\n\n return bytes_from_buffer(output_buffer)", "response": "Derives a key from a PKCS#5 key."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rand_bytes(length):\n\n if not isinstance(length, int_types):\n raise TypeError(pretty_message(\n '''\n length must be an integer, not %s\n ''',\n type_name(length)\n ))\n\n if length < 1:\n raise ValueError('length must be greater than 0')\n\n if length > 1024:\n raise ValueError('length must not be greater than 1024')\n\n buffer = buffer_from_bytes(length)\n result = Security.SecRandomCopyBytes(Security.kSecRandomDefault, length, buffer)\n if result != 0:\n raise OSError(_extract_error())\n\n return bytes_from_buffer(buffer)", "response": "Returns a number of random bytes suitable for cryptographic purposes\n "} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cf_number_to_number(value):\n\n type_ = CoreFoundation.CFNumberGetType(_cast_pointer_p(value))\n c_type = {\n 1: c_byte, # kCFNumberSInt8Type\n 2: ctypes.c_short, # kCFNumberSInt16Type\n 3: ctypes.c_int32, # kCFNumberSInt32Type\n 4: ctypes.c_int64, # kCFNumberSInt64Type\n 5: ctypes.c_float, # kCFNumberFloat32Type\n 6: ctypes.c_double, # kCFNumberFloat64Type\n 7: c_byte, # kCFNumberCharType\n 8: ctypes.c_short, # kCFNumberShortType\n 9: ctypes.c_int, # kCFNumberIntType\n 10: c_long, # kCFNumberLongType\n 11: ctypes.c_longlong, # kCFNumberLongLongType\n 12: ctypes.c_float, # kCFNumberFloatType\n 13: ctypes.c_double, # kCFNumberDoubleType\n 14: c_long, # kCFNumberCFIndexType\n 15: ctypes.c_int, # kCFNumberNSIntegerType\n 16: ctypes.c_double, # kCFNumberCGFloatType\n }[type_]\n output = c_type(0)\n CoreFoundation.CFNumberGetValue(_cast_pointer_p(value), type_, byref(output))\n return output.value", "response": "Convert a CFNumber object to a python number."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cf_dictionary_to_dict(dictionary):\n\n dict_length = CoreFoundation.CFDictionaryGetCount(dictionary)\n\n keys = (CFTypeRef * dict_length)()\n values = (CFTypeRef * dict_length)()\n CoreFoundation.CFDictionaryGetKeysAndValues(\n dictionary,\n _cast_pointer_p(keys),\n _cast_pointer_p(values)\n )\n\n output = {}\n for index in range(0, dict_length):\n output[CFHelpers.native(keys[index])] = CFHelpers.native(values[index])\n\n return output", "response": "Converts a CFDictionary object into a python dictionary."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef native(cls, value):\n\n type_id = CoreFoundation.CFGetTypeID(value)\n if type_id in cls._native_map:\n return cls._native_map[type_id](value)\n else:\n return value", "response": "Converts a CF * object into its native equivalent."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a CFString object to a python unicode string.", "response": "def cf_string_to_unicode(value):\n \"\"\"\n Creates a python unicode string from a CFString object\n\n :param value:\n The CFString to convert\n\n :return:\n A python unicode string\n \"\"\"\n\n string = CoreFoundation.CFStringGetCStringPtr(\n _cast_pointer_p(value),\n kCFStringEncodingUTF8\n )\n if string is None:\n buffer = buffer_from_bytes(1024)\n result = CoreFoundation.CFStringGetCString(\n _cast_pointer_p(value),\n buffer,\n 1024,\n kCFStringEncodingUTF8\n )\n if not result:\n raise OSError('Error copying C string from CFStringRef')\n string = byte_string_from_buffer(buffer)\n if string is not None:\n string = string.decode('utf-8')\n return string"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nextracting a bytestring from a CFData object.", "response": "def cf_data_to_bytes(value):\n \"\"\"\n Extracts a bytestring from a CFData object\n\n :param value:\n A CFData object\n\n :return:\n A byte string\n \"\"\"\n\n start = CoreFoundation.CFDataGetBytePtr(value)\n num_bytes = CoreFoundation.CFDataGetLength(value)\n return string_at(start, num_bytes)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a CFArrayRef object from a list of CF * types.", "response": "def cf_array_from_list(values):\n \"\"\"\n Creates a CFArrayRef object from a list of CF* type objects.\n\n :param values:\n A list of CF* type object\n\n :return:\n A CFArrayRef\n \"\"\"\n\n length = len(values)\n values = (CFTypeRef * length)(*values)\n return CoreFoundation.CFArrayCreate(\n CoreFoundation.kCFAllocatorDefault,\n _cast_pointer_p(byref(values)),\n length,\n kCFTypeArrayCallBacks\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef cf_number_from_integer(integer):\n\n integer_as_long = c_long(integer)\n return CoreFoundation.CFNumberCreate(\n CoreFoundation.kCFAllocatorDefault,\n kCFNumberCFIndexType,\n byref(integer_as_long)\n )", "response": "Creates a CFNumber object from an integer."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a CFNumber object to a python number.", "response": "def cf_number_to_number(value):\n \"\"\"\n Converts a CFNumber object to a python float or integer\n\n :param value:\n The CFNumber object\n\n :return:\n A python number (float or integer)\n \"\"\"\n\n type_ = CoreFoundation.CFNumberGetType(value)\n type_name_ = {\n 1: 'int8_t', # kCFNumberSInt8Type\n 2: 'in16_t', # kCFNumberSInt16Type\n 3: 'int32_t', # kCFNumberSInt32Type\n 4: 'int64_t', # kCFNumberSInt64Type\n 5: 'float', # kCFNumberFloat32Type\n 6: 'double', # kCFNumberFloat64Type\n 7: 'char', # kCFNumberCharType\n 8: 'short', # kCFNumberShortType\n 9: 'int', # kCFNumberIntType\n 10: 'long', # kCFNumberLongType\n 11: 'long long', # kCFNumberLongLongType\n 12: 'float', # kCFNumberFloatType\n 13: 'double', # kCFNumberDoubleType\n 14: 'long', # kCFNumberCFIndexType\n 15: 'int', # kCFNumberNSIntegerType\n 16: 'double', # kCFNumberCGFloatType\n }[type_]\n output = new(CoreFoundation, type_name_ + ' *')\n CoreFoundation.CFNumberGetValue(value, type_, output)\n return deref(output)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nconvert a CFString object to a python unicode string.", "response": "def cf_string_to_unicode(value):\n \"\"\"\n Creates a python unicode string from a CFString object\n\n :param value:\n The CFString to convert\n\n :return:\n A python unicode string\n \"\"\"\n\n string_ptr = CoreFoundation.CFStringGetCStringPtr(\n value,\n kCFStringEncodingUTF8\n )\n string = None if is_null(string_ptr) else ffi.string(string_ptr)\n if string is None:\n buffer = buffer_from_bytes(1024)\n result = CoreFoundation.CFStringGetCString(\n value,\n buffer,\n 1024,\n kCFStringEncodingUTF8\n )\n if not result:\n raise OSError('Error copying C string from CFStringRef')\n string = byte_string_from_buffer(buffer)\n if string is not None:\n string = string.decode('utf-8')\n return string"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nextracts a bytestring from a CFData object.", "response": "def cf_data_to_bytes(value):\n \"\"\"\n Extracts a bytestring from a CFData object\n\n :param value:\n A CFData object\n\n :return:\n A byte string\n \"\"\"\n\n start = CoreFoundation.CFDataGetBytePtr(value)\n num_bytes = CoreFoundation.CFDataGetLength(value)\n return ffi.buffer(start, num_bytes)[:]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a CFArrayRef object from a list of CF* types.", "response": "def cf_array_from_list(values):\n \"\"\"\n Creates a CFArrayRef object from a list of CF* type objects.\n\n :param values:\n A list of CF* type object\n\n :return:\n A CFArrayRef\n \"\"\"\n\n length = len(values)\n return CoreFoundation.CFArrayCreate(\n CoreFoundation.kCFAllocatorDefault,\n values,\n length,\n ffi.addressof(CoreFoundation.kCFTypeArrayCallBacks)\n )"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef cf_number_from_integer(integer):\n\n integer_as_long = ffi.new('long *', integer)\n return CoreFoundation.CFNumberCreate(\n CoreFoundation.kCFAllocatorDefault,\n kCFNumberCFIndexType,\n integer_as_long\n )", "response": "Creates a CFNumber object from an integer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dump_dh_parameters(dh_parameters, encoding='pem'):\n\n if encoding not in set(['pem', 'der']):\n raise ValueError(pretty_message(\n '''\n encoding must be one of \"pem\", \"der\", not %s\n ''',\n repr(encoding)\n ))\n\n if not isinstance(dh_parameters, algos.DHParameters):\n raise TypeError(pretty_message(\n '''\n dh_parameters must be an instance of asn1crypto.algos.DHParameters,\n not %s\n ''',\n type_name(dh_parameters)\n ))\n\n output = dh_parameters.dump()\n if encoding == 'pem':\n output = pem.armor('DH PARAMETERS', output)\n return output", "response": "Serializes an asn1crypto. algos. DHParameters object into a byte string of the encoded DH parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dump_public_key(public_key, encoding='pem'):\n\n if encoding not in set(['pem', 'der']):\n raise ValueError(pretty_message(\n '''\n encoding must be one of \"pem\", \"der\", not %s\n ''',\n repr(encoding)\n ))\n\n is_oscrypto = isinstance(public_key, PublicKey)\n if not isinstance(public_key, keys.PublicKeyInfo) and not is_oscrypto:\n raise TypeError(pretty_message(\n '''\n public_key must be an instance of oscrypto.asymmetric.PublicKey or\n asn1crypto.keys.PublicKeyInfo, not %s\n ''',\n type_name(public_key)\n ))\n\n if is_oscrypto:\n public_key = public_key.asn1\n\n output = public_key.dump()\n if encoding == 'pem':\n output = pem.armor('PUBLIC KEY', output)\n return output", "response": "Serializes a public key into a byte string."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nserialize a certificate object into a byte string of the encoded version of the ASN. 1 certificate object.", "response": "def dump_certificate(certificate, encoding='pem'):\n \"\"\"\n Serializes a certificate object into a byte string\n\n :param certificate:\n An oscrypto.asymmetric.Certificate or asn1crypto.x509.Certificate object\n\n :param encoding:\n A unicode string of \"pem\" or \"der\"\n\n :return:\n A byte string of the encoded certificate\n \"\"\"\n\n if encoding not in set(['pem', 'der']):\n raise ValueError(pretty_message(\n '''\n encoding must be one of \"pem\", \"der\", not %s\n ''',\n repr(encoding)\n ))\n\n is_oscrypto = isinstance(certificate, Certificate)\n if not isinstance(certificate, x509.Certificate) and not is_oscrypto:\n raise TypeError(pretty_message(\n '''\n certificate must be an instance of oscrypto.asymmetric.Certificate\n or asn1crypto.x509.Certificate, not %s\n ''',\n type_name(certificate)\n ))\n\n if is_oscrypto:\n certificate = certificate.asn1\n\n output = certificate.dump()\n if encoding == 'pem':\n output = pem.armor('CERTIFICATE', output)\n return output"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nserializes a private key into a byte string of the PKCS#8 format.", "response": "def dump_private_key(private_key, passphrase, encoding='pem', target_ms=200):\n \"\"\"\n Serializes a private key object into a byte string of the PKCS#8 format\n\n :param private_key:\n An oscrypto.asymmetric.PrivateKey or asn1crypto.keys.PrivateKeyInfo\n object\n\n :param passphrase:\n A unicode string of the passphrase to encrypt the private key with.\n A passphrase of None will result in no encryption. A blank string will\n result in a ValueError to help ensure that the lack of passphrase is\n intentional.\n\n :param encoding:\n A unicode string of \"pem\" or \"der\"\n\n :param target_ms:\n Use PBKDF2 with the number of iterations that takes about this many\n milliseconds on the current machine.\n\n :raises:\n ValueError - when a blank string is provided for the passphrase\n\n :return:\n A byte string of the encoded and encrypted public key\n \"\"\"\n\n if encoding not in set(['pem', 'der']):\n raise ValueError(pretty_message(\n '''\n encoding must be one of \"pem\", \"der\", not %s\n ''',\n repr(encoding)\n ))\n\n if passphrase is not None:\n if not isinstance(passphrase, str_cls):\n raise TypeError(pretty_message(\n '''\n passphrase must be a unicode string, not %s\n ''',\n type_name(passphrase)\n ))\n if passphrase == '':\n raise ValueError(pretty_message(\n '''\n passphrase may not be a blank string - pass None to disable\n encryption\n '''\n ))\n\n is_oscrypto = isinstance(private_key, PrivateKey)\n if not isinstance(private_key, keys.PrivateKeyInfo) and not is_oscrypto:\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of oscrypto.asymmetric.PrivateKey\n or asn1crypto.keys.PrivateKeyInfo, not %s\n ''',\n type_name(private_key)\n ))\n\n if is_oscrypto:\n private_key = private_key.asn1\n\n output = private_key.dump()\n\n if passphrase is not None:\n cipher = 'aes256_cbc'\n key_length = 32\n kdf_hmac = 'sha256'\n kdf_salt = rand_bytes(key_length)\n iterations = pbkdf2_iteration_calculator(kdf_hmac, key_length, target_ms=target_ms, quiet=True)\n # Need a bare minimum of 10,000 iterations for PBKDF2 as of 2015\n if iterations < 10000:\n iterations = 10000\n\n passphrase_bytes = passphrase.encode('utf-8')\n key = pbkdf2(kdf_hmac, passphrase_bytes, kdf_salt, iterations, key_length)\n iv, ciphertext = aes_cbc_pkcs7_encrypt(key, output, None)\n\n output = keys.EncryptedPrivateKeyInfo({\n 'encryption_algorithm': {\n 'algorithm': 'pbes2',\n 'parameters': {\n 'key_derivation_func': {\n 'algorithm': 'pbkdf2',\n 'parameters': {\n 'salt': algos.Pbkdf2Salt(\n name='specified',\n value=kdf_salt\n ),\n 'iteration_count': iterations,\n 'prf': {\n 'algorithm': kdf_hmac,\n 'parameters': core.Null()\n }\n }\n },\n 'encryption_scheme': {\n 'algorithm': cipher,\n 'parameters': iv\n }\n }\n },\n 'encrypted_data': ciphertext\n }).dump()\n\n if encoding == 'pem':\n if passphrase is None:\n object_type = 'PRIVATE KEY'\n else:\n object_type = 'ENCRYPTED PRIVATE KEY'\n output = pem.armor(object_type, output)\n\n return output"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dump_openssl_private_key(private_key, passphrase):\n\n if passphrase is not None:\n if not isinstance(passphrase, str_cls):\n raise TypeError(pretty_message(\n '''\n passphrase must be a unicode string, not %s\n ''',\n type_name(passphrase)\n ))\n if passphrase == '':\n raise ValueError(pretty_message(\n '''\n passphrase may not be a blank string - pass None to disable\n encryption\n '''\n ))\n\n is_oscrypto = isinstance(private_key, PrivateKey)\n if not isinstance(private_key, keys.PrivateKeyInfo) and not is_oscrypto:\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of oscrypto.asymmetric.PrivateKey or\n asn1crypto.keys.PrivateKeyInfo, not %s\n ''',\n type_name(private_key)\n ))\n\n if is_oscrypto:\n private_key = private_key.asn1\n\n output = private_key.unwrap().dump()\n\n headers = None\n if passphrase is not None:\n iv = rand_bytes(16)\n\n headers = OrderedDict()\n headers['Proc-Type'] = '4,ENCRYPTED'\n headers['DEK-Info'] = 'AES-128-CBC,%s' % binascii.hexlify(iv).decode('ascii')\n\n key_length = 16\n passphrase_bytes = passphrase.encode('utf-8')\n\n key = hashlib.md5(passphrase_bytes + iv[0:8]).digest()\n while key_length > len(key):\n key += hashlib.md5(key + passphrase_bytes + iv[0:8]).digest()\n key = key[0:key_length]\n\n iv, output = aes_cbc_pkcs7_encrypt(key, output, iv)\n\n if private_key.algorithm == 'ec':\n object_type = 'EC PRIVATE KEY'\n elif private_key.algorithm == 'rsa':\n object_type = 'RSA PRIVATE KEY'\n elif private_key.algorithm == 'dsa':\n object_type = 'DSA PRIVATE KEY'\n\n return pem.armor(object_type, output, headers=headers)", "response": "Serializes a private key into a byte string of PEM formats used by OpenSSL."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck a Security OSStatus error code and raises an exception if there is an error", "response": "def handle_sec_error(error, exception_class=None):\n \"\"\"\n Checks a Security OSStatus error code and throws an exception if there is an\n error to report\n\n :param error:\n An OSStatus\n\n :param exception_class:\n The exception class to use for the exception if an error occurred\n\n :raises:\n OSError - when the OSStatus contains an error\n \"\"\"\n\n if error == 0:\n return\n\n if error in set([SecurityConst.errSSLClosedNoNotify, SecurityConst.errSSLClosedAbort]):\n raise TLSDisconnectError('The remote end closed the connection')\n if error == SecurityConst.errSSLClosedGraceful:\n raise TLSGracefulDisconnectError('The remote end closed the connection')\n\n cf_error_string = Security.SecCopyErrorMessageString(error, null())\n output = CFHelpers.cf_string_to_unicode(cf_error_string)\n CoreFoundation.CFRelease(cf_error_string)\n\n if output is None or output == '':\n output = 'OSStatus %s' % error\n\n if exception_class is None:\n exception_class = OSError\n\n raise exception_class(output)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _get_func_info(docstring, def_lineno, code_lines, prefix):\n\n def_index = def_lineno - 1\n definition = code_lines[def_index]\n definition = definition.rstrip()\n while not definition.endswith(':'):\n def_index += 1\n definition += '\\n' + code_lines[def_index].rstrip()\n\n definition = textwrap.dedent(definition).rstrip(':')\n definition = definition.replace('\\n', '\\n' + prefix)\n\n description = ''\n found_colon = False\n\n params = ''\n\n for line in docstring.splitlines():\n if line and line[0] == ':':\n found_colon = True\n if not found_colon:\n if description:\n description += '\\n'\n description += line\n else:\n if params:\n params += '\\n'\n params += line\n\n description = description.strip()\n description_md = ''\n if description:\n description_md = \"%s%s\" % (prefix, description.replace('\\n', '\\n' + prefix))\n description_md = re.sub('\\n>(\\\\s+)\\n', '\\n>\\n', description_md)\n\n params = params.strip()\n if params:\n definition += (':\\n%s \"\"\"\\n%s ' % (prefix, prefix))\n definition += params.replace('\\n', '\\n%s ' % prefix)\n definition += ('\\n%s \"\"\"' % prefix)\n definition = re.sub('\\n>(\\\\s+)\\n', '\\n>\\n', definition)\n\n for search, replace in definition_replacements.items():\n definition = definition.replace(search, replace)\n\n return (definition, description_md)", "response": "Extracts the function signature and description of a Python function and returns the 2 - element tuple of the function signature and description and markdown snippet of the function description."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _find_sections(md_ast, sections, last, last_class, total_lines=None):\n\n def child_walker(node):\n for child, entering in node.walker():\n if child == node:\n continue\n yield child, entering\n\n for child, entering in child_walker(md_ast):\n if child.t == 'heading':\n start_line = child.sourcepos[0][0]\n\n if child.level == 2:\n if last:\n sections[(last['type_name'], last['identifier'])] = (last['start_line'], start_line - 1)\n last.clear()\n\n if child.level in set([3, 5]):\n heading_elements = []\n for heading_child, _ in child_walker(child):\n heading_elements.append(heading_child)\n if len(heading_elements) != 2:\n continue\n first = heading_elements[0]\n second = heading_elements[1]\n if first.t != 'code':\n continue\n if second.t != 'text':\n continue\n\n type_name = second.literal.strip()\n identifier = first.literal.strip().replace('()', '').lstrip('.')\n\n if last:\n sections[(last['type_name'], last['identifier'])] = (last['start_line'], start_line - 1)\n last.clear()\n\n if type_name == 'function':\n if child.level != 3:\n continue\n\n if type_name == 'class':\n if child.level != 3:\n continue\n last_class.append(identifier)\n\n if type_name in set(['method', 'attribute']):\n if child.level != 5:\n continue\n identifier = last_class[-1] + '.' + identifier\n\n last.update({\n 'type_name': type_name,\n 'identifier': identifier,\n 'start_line': start_line,\n })\n\n elif child.t == 'block_quote':\n find_sections(child, sections, last, last_class)\n\n if last:\n sections[(last['type_name'], last['identifier'])] = (last['start_line'], total_lines)", "response": "Walks through a CommonMark AST to find the sections that should be updated by this script."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef walk_ast(node, code_lines, sections, md_chunks):\n\n if isinstance(node, _ast.FunctionDef):\n key = ('function', node.name)\n if key not in sections:\n return\n\n docstring = ast.get_docstring(node)\n def_lineno = node.lineno + len(node.decorator_list)\n\n definition, description_md = _get_func_info(docstring, def_lineno, code_lines, '> ')\n\n md_chunk = textwrap.dedent(\"\"\"\n ### `%s()` function\n\n > ```python\n > %s\n > ```\n >\n %s\n \"\"\").strip() % (\n node.name,\n definition,\n description_md\n ) + \"\\n\"\n\n md_chunks[key] = md_chunk.replace('>\\n\\n', '')\n\n elif isinstance(node, _ast.ClassDef):\n if ('class', node.name) not in sections:\n return\n\n for subnode in node.body:\n if isinstance(subnode, _ast.FunctionDef):\n node_id = node.name + '.' + subnode.name\n\n method_key = ('method', node_id)\n is_method = method_key in sections\n\n attribute_key = ('attribute', node_id)\n is_attribute = attribute_key in sections\n\n is_constructor = subnode.name == '__init__'\n\n if not is_constructor and not is_attribute and not is_method:\n continue\n\n docstring = ast.get_docstring(subnode)\n def_lineno = subnode.lineno + len(subnode.decorator_list)\n\n if not docstring:\n continue\n\n if is_method or is_constructor:\n definition, description_md = _get_func_info(docstring, def_lineno, code_lines, '> > ')\n\n if is_constructor:\n key = ('class', node.name)\n\n class_docstring = ast.get_docstring(node) or ''\n class_description = textwrap.dedent(class_docstring).strip()\n if class_description:\n class_description_md = \"> %s\\n>\" % (class_description.replace(\"\\n\", \"\\n> \"))\n else:\n class_description_md = ''\n\n md_chunk = textwrap.dedent(\"\"\"\n ### `%s()` class\n\n %s\n > ##### constructor\n >\n > > ```python\n > > %s\n > > ```\n > >\n %s\n \"\"\").strip() % (\n node.name,\n class_description_md,\n definition,\n description_md\n )\n\n md_chunk = md_chunk.replace('\\n\\n\\n', '\\n\\n')\n\n else:\n key = method_key\n\n md_chunk = textwrap.dedent(\"\"\"\n >\n > ##### `.%s()` method\n >\n > > ```python\n > > %s\n > > ```\n > >\n %s\n \"\"\").strip() % (\n subnode.name,\n definition,\n description_md\n )\n\n if md_chunk[-5:] == '\\n> >\\n':\n md_chunk = md_chunk[0:-5]\n\n else:\n key = attribute_key\n\n description = textwrap.dedent(docstring).strip()\n description_md = \"> > %s\" % (description.replace(\"\\n\", \"\\n> > \"))\n\n md_chunk = textwrap.dedent(\"\"\"\n >\n > ##### `.%s` attribute\n >\n %s\n \"\"\").strip() % (\n subnode.name,\n description_md\n )\n\n md_chunks[key] = re.sub('[ \\\\t]+\\n', '\\n', md_chunk.rstrip())\n\n elif isinstance(node, _ast.If):\n for subast in node.body:\n walk_ast(subast, code_lines, sections, md_chunks)\n for subast in node.orelse:\n walk_ast(subast, code_lines, sections, md_chunks)", "response": "This function walks the Python AST and generates markdown markup for the modules that are not in the specified sections."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nruns the update API docs.", "response": "def run():\n \"\"\"\n Looks through the docs/ dir and parses each markdown document, looking for\n sections to update from Python docstrings. Looks for section headers in\n the format:\n\n - ### `ClassName()` class\n - ##### `.method_name()` method\n - ##### `.attribute_name` attribute\n - ### `function_name()` function\n\n The markdown content following these section headers up until the next\n section header will be replaced by new markdown generated from the Python\n docstrings of the associated source files.\n\n By default maps docs/{name}.md to {modulename}/{name}.py. Allows for\n custom mapping via the md_source_map variable.\n \"\"\"\n\n print('Updating API docs...')\n\n md_files = []\n for root, _, filenames in os.walk(os.path.join(package_root, 'docs')):\n for filename in filenames:\n if not filename.endswith('.md'):\n continue\n md_files.append(os.path.join(root, filename))\n\n parser = CommonMark.Parser()\n\n for md_file in md_files:\n md_file_relative = md_file[len(package_root) + 1:]\n if md_file_relative in md_source_map:\n py_files = md_source_map[md_file_relative]\n py_paths = [os.path.join(package_root, py_file) for py_file in py_files]\n else:\n py_files = [os.path.basename(md_file).replace('.md', '.py')]\n py_paths = [os.path.join(package_root, package_name, py_files[0])]\n\n if not os.path.exists(py_paths[0]):\n continue\n\n with open(md_file, 'rb') as f:\n markdown = f.read().decode('utf-8')\n\n original_markdown = markdown\n md_lines = list(markdown.splitlines())\n md_ast = parser.parse(markdown)\n\n last_class = []\n last = {}\n sections = OrderedDict()\n find_sections(md_ast, sections, last, last_class, markdown.count(\"\\n\") + 1)\n\n md_chunks = {}\n\n for index, py_file in enumerate(py_files):\n py_path = py_paths[index]\n\n with open(os.path.join(py_path), 'rb') as f:\n code = f.read().decode('utf-8')\n module_ast = ast.parse(code, filename=py_file)\n code_lines = list(code.splitlines())\n\n for node in ast.iter_child_nodes(module_ast):\n walk_ast(node, code_lines, sections, md_chunks)\n\n added_lines = 0\n\n def _replace_md(key, sections, md_chunk, md_lines, added_lines):\n start, end = sections[key]\n start -= 1\n start += added_lines\n end += added_lines\n new_lines = md_chunk.split('\\n')\n added_lines += len(new_lines) - (end - start)\n\n # Ensure a newline above each class header\n if start > 0 and md_lines[start][0:4] == '### ' and md_lines[start - 1][0:1] == '>':\n added_lines += 1\n new_lines.insert(0, '')\n\n md_lines[start:end] = new_lines\n return added_lines\n\n for key in sections:\n if key not in md_chunks:\n raise ValueError('No documentation found for %s' % key[1])\n added_lines = _replace_md(key, sections, md_chunks[key], md_lines, added_lines)\n\n markdown = '\\n'.join(md_lines).strip() + '\\n'\n\n if original_markdown != markdown:\n with open(md_file, 'wb') as f:\n f.write(markdown.encode('utf-8'))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating a private key pair for the given curve.", "response": "def ec_generate_pair(curve):\n \"\"\"\n Generates a EC public/private key pair\n\n :param curve:\n A unicode string. Valid values include \"secp256r1\", \"secp384r1\" and\n \"secp521r1\".\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n\n :return:\n A 2-element tuple of (asn1crypto.keys.PublicKeyInfo,\n asn1crypto.keys.PrivateKeyInfo)\n \"\"\"\n\n if curve not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise ValueError(pretty_message(\n '''\n curve must be one of \"secp256r1\", \"secp384r1\", \"secp521r1\", not %s\n ''',\n repr(curve)\n ))\n\n curve_num_bytes = CURVE_BYTES[curve]\n curve_base_point = {\n 'secp256r1': SECP256R1_BASE_POINT,\n 'secp384r1': SECP384R1_BASE_POINT,\n 'secp521r1': SECP521R1_BASE_POINT,\n }[curve]\n\n while True:\n private_key_bytes = rand_bytes(curve_num_bytes)\n private_key_int = int_from_bytes(private_key_bytes, signed=False)\n\n if private_key_int > 0 and private_key_int < curve_base_point.order:\n break\n\n private_key_info = keys.PrivateKeyInfo({\n 'version': 0,\n 'private_key_algorithm': keys.PrivateKeyAlgorithm({\n 'algorithm': 'ec',\n 'parameters': keys.ECDomainParameters(\n name='named',\n value=curve\n )\n }),\n 'private_key': keys.ECPrivateKey({\n 'version': 'ecPrivkeyVer1',\n 'private_key': private_key_int\n }),\n })\n private_key_info['private_key'].parsed['public_key'] = private_key_info.public_key\n public_key_info = private_key_info.public_key_info\n\n return (public_key_info, private_key_info)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef ecdsa_sign(private_key, data, hash_algorithm):\n\n if not hasattr(private_key, 'asn1') or not isinstance(private_key.asn1, keys.PrivateKeyInfo):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of the\n oscrypto.asymmetric.PrivateKey class, not %s\n ''',\n type_name(private_key)\n ))\n\n curve_name = private_key.curve\n if curve_name not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise ValueError(pretty_message(\n '''\n private_key does not use one of the named curves secp256r1,\n secp384r1 or secp521r1\n '''\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if hash_algorithm not in set(['sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n hash_func = getattr(hashlib, hash_algorithm)\n\n ec_private_key = private_key.asn1['private_key'].parsed\n private_key_bytes = ec_private_key['private_key'].contents\n private_key_int = ec_private_key['private_key'].native\n\n curve_num_bytes = CURVE_BYTES[curve_name]\n curve_base_point = {\n 'secp256r1': SECP256R1_BASE_POINT,\n 'secp384r1': SECP384R1_BASE_POINT,\n 'secp521r1': SECP521R1_BASE_POINT,\n }[curve_name]\n\n n = curve_base_point.order\n\n # RFC 6979 section 3.2\n\n # a.\n digest = hash_func(data).digest()\n hash_length = len(digest)\n\n h = int_from_bytes(digest, signed=False) % n\n\n # b.\n V = b'\\x01' * hash_length\n\n # c.\n K = b'\\x00' * hash_length\n\n # d.\n K = hmac.new(K, V + b'\\x00' + private_key_bytes + digest, hash_func).digest()\n\n # e.\n V = hmac.new(K, V, hash_func).digest()\n\n # f.\n K = hmac.new(K, V + b'\\x01' + private_key_bytes + digest, hash_func).digest()\n\n # g.\n V = hmac.new(K, V, hash_func).digest()\n\n # h.\n r = 0\n s = 0\n while True:\n # h. 1\n T = b''\n\n # h. 2\n while len(T) < curve_num_bytes:\n V = hmac.new(K, V, hash_func).digest()\n T += V\n\n # h. 3\n k = int_from_bytes(T[0:curve_num_bytes], signed=False)\n if k == 0 or k >= n:\n continue\n\n # Calculate the signature in the loop in case we need a new k\n r = (curve_base_point * k).x % n\n if r == 0:\n continue\n\n s = (inverse_mod(k, n) * (h + (private_key_int * r) % n)) % n\n if s == 0:\n continue\n\n break\n\n return DSASignature({'r': r, 's': s}).dump()", "response": "Generates an ECDSA signature for the given data and hash_algorithm."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef ecdsa_verify(certificate_or_public_key, signature, data, hash_algorithm):\n\n has_asn1 = hasattr(certificate_or_public_key, 'asn1')\n if not has_asn1 or not isinstance(certificate_or_public_key.asn1, (keys.PublicKeyInfo, Certificate)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the\n oscrypto.asymmetric.PublicKey or oscrypto.asymmetric.Certificate\n classes, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n curve_name = certificate_or_public_key.curve\n if curve_name not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise ValueError(pretty_message(\n '''\n certificate_or_public_key does not use one of the named curves\n secp256r1, secp384r1 or secp521r1\n '''\n ))\n\n if not isinstance(signature, byte_cls):\n raise TypeError(pretty_message(\n '''\n signature must be a byte string, not %s\n ''',\n type_name(signature)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if hash_algorithm not in set(['sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n asn1 = certificate_or_public_key.asn1\n if isinstance(asn1, Certificate):\n asn1 = asn1.public_key\n\n curve_base_point = {\n 'secp256r1': SECP256R1_BASE_POINT,\n 'secp384r1': SECP384R1_BASE_POINT,\n 'secp521r1': SECP521R1_BASE_POINT,\n }[curve_name]\n\n x, y = asn1['public_key'].to_coords()\n n = curve_base_point.order\n\n # Validates that the point is valid\n public_key_point = PrimePoint(curve_base_point.curve, x, y, n)\n\n try:\n signature = DSASignature.load(signature)\n r = signature['r'].native\n s = signature['s'].native\n except (ValueError):\n raise SignatureError('Signature is invalid')\n\n invalid = 0\n\n # Check r is valid\n invalid |= r < 1\n invalid |= r >= n\n\n # Check s is valid\n invalid |= s < 1\n invalid |= s >= n\n\n if invalid:\n raise SignatureError('Signature is invalid')\n\n hash_func = getattr(hashlib, hash_algorithm)\n\n digest = hash_func(data).digest()\n\n z = int_from_bytes(digest, signed=False) % n\n w = inverse_mod(s, n)\n u1 = (z * w) % n\n u2 = (r * w) % n\n hash_point = (curve_base_point * u1) + (public_key_point * u2)\n if r != (hash_point.x % n):\n raise SignatureError('Signature is invalid')", "response": "Verifies an ECDSA signature in pure Python."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef system_path():\n\n ca_path = None\n\n # Common CA cert paths\n paths = [\n '/usr/lib/ssl/certs/ca-certificates.crt',\n '/etc/ssl/certs/ca-certificates.crt',\n '/etc/ssl/certs/ca-bundle.crt',\n '/etc/pki/tls/certs/ca-bundle.crt',\n '/etc/ssl/ca-bundle.pem',\n '/usr/local/share/certs/ca-root-nss.crt',\n '/etc/ssl/cert.pem'\n ]\n\n # First try SSL_CERT_FILE\n if 'SSL_CERT_FILE' in os.environ:\n paths.insert(0, os.environ['SSL_CERT_FILE'])\n\n for path in paths:\n if os.path.exists(path) and os.path.getsize(path) > 0:\n ca_path = path\n break\n\n if not ca_path:\n raise OSError(pretty_message(\n '''\n Unable to find a CA certs bundle in common locations - try\n setting the SSL_CERT_FILE environmental variable\n '''\n ))\n\n return ca_path", "response": "Tries to find a CA certs bundle in common locations and returns the full path to the CA certs bundle file"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef extract_from_system(cert_callback=None, callback_only_on_failure=False):\n\n all_purposes = '2.5.29.37.0'\n ca_path = system_path()\n\n output = []\n with open(ca_path, 'rb') as f:\n for armor_type, _, cert_bytes in unarmor(f.read(), multiple=True):\n # Without more info, a certificate is trusted for all purposes\n if armor_type == 'CERTIFICATE':\n if cert_callback:\n cert_callback(Certificate.load(cert_bytes), None)\n output.append((cert_bytes, set(), set()))\n\n # The OpenSSL TRUSTED CERTIFICATE construct adds OIDs for trusted\n # and rejected purposes, so we extract that info.\n elif armor_type == 'TRUSTED CERTIFICATE':\n cert, aux = TrustedCertificate.load(cert_bytes)\n reject_all = False\n trust_oids = set()\n reject_oids = set()\n for purpose in aux['trust']:\n if purpose.dotted == all_purposes:\n trust_oids = set([purpose.dotted])\n break\n trust_oids.add(purpose.dotted)\n for purpose in aux['reject']:\n if purpose.dotted == all_purposes:\n reject_all = True\n break\n reject_oids.add(purpose.dotted)\n if reject_all:\n if cert_callback:\n cert_callback(cert, 'explicitly distrusted')\n continue\n if cert_callback and not callback_only_on_failure:\n cert_callback(cert, None)\n output.append((cert.dump(), trust_oids, reject_oids))\n\n return output", "response": "Extracts trusted CA certs from the system CA cert bundle."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef extract_from_system(cert_callback=None, callback_only_on_failure=False):\n\n certificates = {}\n processed = {}\n\n now = datetime.datetime.utcnow()\n\n for store in [\"ROOT\", \"CA\"]:\n store_handle = crypt32.CertOpenSystemStoreW(null(), store)\n handle_error(store_handle)\n\n context_pointer = null()\n while True:\n context_pointer = crypt32.CertEnumCertificatesInStore(store_handle, context_pointer)\n if is_null(context_pointer):\n break\n context = unwrap(context_pointer)\n\n trust_all = False\n data = None\n digest = None\n\n if context.dwCertEncodingType != Crypt32Const.X509_ASN_ENCODING:\n continue\n\n data = bytes_from_buffer(context.pbCertEncoded, int(context.cbCertEncoded))\n digest = hashlib.sha1(data).digest()\n if digest in processed:\n continue\n\n processed[digest] = True\n cert_info = unwrap(context.pCertInfo)\n\n not_before_seconds = _convert_filetime_to_timestamp(cert_info.NotBefore)\n try:\n not_before = datetime.datetime.fromtimestamp(not_before_seconds)\n if not_before > now:\n if cert_callback:\n cert_callback(Certificate.load(data), 'not yet valid')\n continue\n except (ValueError, OSError):\n # If there is an error converting the not before timestamp,\n # it is almost certainly because it is from too long ago,\n # which means the cert is definitely valid by now.\n pass\n\n not_after_seconds = _convert_filetime_to_timestamp(cert_info.NotAfter)\n try:\n not_after = datetime.datetime.fromtimestamp(not_after_seconds)\n if not_after < now:\n if cert_callback:\n cert_callback(Certificate.load(data), 'no longer valid')\n continue\n except (ValueError, OSError) as e:\n # The only reason we would get an exception here is if the\n # expiration time is so far in the future that it can't be\n # used as a timestamp, or it is before 0. If it is very far\n # in the future, the cert is still valid, so we only raise\n # an exception if the timestamp is less than zero.\n if not_after_seconds < 0:\n message = e.args[0] + ' - ' + str_cls(not_after_seconds)\n e.args = (message,) + e.args[1:]\n raise e\n\n trust_oids = set()\n reject_oids = set()\n\n # Here we grab the extended key usage properties that Windows\n # layers on top of the extended key usage extension that is\n # part of the certificate itself. For highest security, users\n # should only use certificates for the intersection of the two\n # lists of purposes. However, many seen to treat the OS trust\n # list as an override.\n to_read = new(crypt32, 'DWORD *', 0)\n res = crypt32.CertGetEnhancedKeyUsage(\n context_pointer,\n Crypt32Const.CERT_FIND_PROP_ONLY_ENHKEY_USAGE_FLAG,\n null(),\n to_read\n )\n\n # Per the Microsoft documentation, if CRYPT_E_NOT_FOUND is returned\n # from get_error(), it means the certificate is valid for all purposes\n error_code, _ = get_error()\n if not res and error_code != Crypt32Const.CRYPT_E_NOT_FOUND:\n handle_error(res)\n\n if error_code == Crypt32Const.CRYPT_E_NOT_FOUND:\n trust_all = True\n else:\n usage_buffer = buffer_from_bytes(deref(to_read))\n res = crypt32.CertGetEnhancedKeyUsage(\n context_pointer,\n Crypt32Const.CERT_FIND_PROP_ONLY_ENHKEY_USAGE_FLAG,\n cast(crypt32, 'CERT_ENHKEY_USAGE *', usage_buffer),\n to_read\n )\n handle_error(res)\n\n key_usage_pointer = struct_from_buffer(crypt32, 'CERT_ENHKEY_USAGE', usage_buffer)\n key_usage = unwrap(key_usage_pointer)\n\n # Having no enhanced usage properties means a cert is distrusted\n if key_usage.cUsageIdentifier == 0:\n if cert_callback:\n cert_callback(Certificate.load(data), 'explicitly distrusted')\n continue\n\n oids = array_from_pointer(\n crypt32,\n 'LPCSTR',\n key_usage.rgpszUsageIdentifier,\n key_usage.cUsageIdentifier\n )\n for oid in oids:\n trust_oids.add(oid.decode('ascii'))\n\n cert = None\n\n # If the certificate is not under blanket trust, we have to\n # determine what purposes it is rejected for by diffing the\n # set of OIDs from the certificate with the OIDs that are\n # trusted.\n if not trust_all:\n cert = Certificate.load(data)\n if cert.extended_key_usage_value:\n for cert_oid in cert.extended_key_usage_value:\n oid = cert_oid.dotted\n if oid not in trust_oids:\n reject_oids.add(oid)\n\n if cert_callback and not callback_only_on_failure:\n if cert is None:\n cert = Certificate.load(data)\n cert_callback(cert, None)\n\n certificates[digest] = (data, trust_oids, reject_oids)\n\n result = crypt32.CertCloseStore(store_handle, 0)\n handle_error(result)\n store_handle = None\n\n return certificates.values()", "response": "Extracts trusted CA certificates from the Windows certificate store."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nconverting a FILETIME struct to a timestamp.", "response": "def _convert_filetime_to_timestamp(filetime):\n \"\"\"\n Windows returns times as 64-bit unsigned longs that are the number\n of hundreds of nanoseconds since Jan 1 1601. This converts it to\n a datetime object.\n\n :param filetime:\n A FILETIME struct object\n\n :return:\n An integer unix timestamp\n \"\"\"\n\n hundreds_nano_seconds = struct.unpack(\n b'>Q',\n struct.pack(\n b'>LL',\n filetime.dwHighDateTime,\n filetime.dwLowDateTime\n )\n )[0]\n seconds_since_1601 = hundreds_nano_seconds / 10000000\n return seconds_since_1601 - 11644473600"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nextract the X. 509 certificates from the server handshake bytes for use when debugging .", "response": "def extract_chain(server_handshake_bytes):\n \"\"\"\n Extracts the X.509 certificates from the server handshake bytes for use\n when debugging\n\n :param server_handshake_bytes:\n A byte string of the handshake data received from the server\n\n :return:\n A list of asn1crypto.x509.Certificate objects\n \"\"\"\n\n output = []\n\n chain_bytes = None\n\n for record_type, _, record_data in parse_tls_records(server_handshake_bytes):\n if record_type != b'\\x16':\n continue\n for message_type, message_data in parse_handshake_messages(record_data):\n if message_type == b'\\x0b':\n chain_bytes = message_data\n break\n if chain_bytes:\n break\n\n if chain_bytes:\n # The first 3 bytes are the cert chain length\n pointer = 3\n while pointer < len(chain_bytes):\n cert_length = int_from_bytes(chain_bytes[pointer:pointer + 3])\n cert_start = pointer + 3\n cert_end = cert_start + cert_length\n pointer = cert_end\n cert_bytes = chain_bytes[cert_start:cert_end]\n output.append(Certificate.load(cert_bytes))\n\n return output"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef detect_client_auth_request(server_handshake_bytes):\n\n for record_type, _, record_data in parse_tls_records(server_handshake_bytes):\n if record_type != b'\\x16':\n continue\n for message_type, message_data in parse_handshake_messages(record_data):\n if message_type == b'\\x0d':\n return True\n return False", "response": "Detects if a client certificate request is sent from the server."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_dh_params_length(server_handshake_bytes):\n\n output = None\n\n dh_params_bytes = None\n\n for record_type, _, record_data in parse_tls_records(server_handshake_bytes):\n if record_type != b'\\x16':\n continue\n for message_type, message_data in parse_handshake_messages(record_data):\n if message_type == b'\\x0c':\n dh_params_bytes = message_data\n break\n if dh_params_bytes:\n break\n\n if dh_params_bytes:\n output = int_from_bytes(dh_params_bytes[0:2]) * 8\n\n return output", "response": "Determines the length of the DH params from the server key exchange."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nparses the server handshake bytes and returns the alert class.", "response": "def parse_alert(server_handshake_bytes):\n \"\"\"\n Parses the handshake for protocol alerts\n\n :param server_handshake_bytes:\n A byte string of the handshake data received from the server\n\n :return:\n None or an 2-element tuple of integers:\n 0: 1 (warning) or 2 (fatal)\n 1: The alert description (see https://tools.ietf.org/html/rfc5246#section-7.2)\n \"\"\"\n\n for record_type, _, record_data in parse_tls_records(server_handshake_bytes):\n if record_type != b'\\x15':\n continue\n if len(record_data) != 2:\n return None\n return (int_from_bytes(record_data[0:1]), int_from_bytes(record_data[1:2]))\n return None"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nparse the server handshake and returns the information for the server session and client session.", "response": "def parse_session_info(server_handshake_bytes, client_handshake_bytes):\n \"\"\"\n Parse the TLS handshake from the client to the server to extract information\n including the cipher suite selected, if compression is enabled, the\n session id and if a new or reused session ticket exists.\n\n :param server_handshake_bytes:\n A byte string of the handshake data received from the server\n\n :param client_handshake_bytes:\n A byte string of the handshake data sent to the server\n\n :return:\n A dict with the following keys:\n - \"protocol\": unicode string\n - \"cipher_suite\": unicode string\n - \"compression\": boolean\n - \"session_id\": \"new\", \"reused\" or None\n - \"session_ticket: \"new\", \"reused\" or None\n \"\"\"\n\n protocol = None\n cipher_suite = None\n compression = False\n session_id = None\n session_ticket = None\n\n server_session_id = None\n client_session_id = None\n\n for record_type, _, record_data in parse_tls_records(server_handshake_bytes):\n if record_type != b'\\x16':\n continue\n for message_type, message_data in parse_handshake_messages(record_data):\n # Ensure we are working with a ServerHello message\n if message_type != b'\\x02':\n continue\n protocol = {\n b'\\x03\\x00': \"SSLv3\",\n b'\\x03\\x01': \"TLSv1\",\n b'\\x03\\x02': \"TLSv1.1\",\n b'\\x03\\x03': \"TLSv1.2\",\n b'\\x03\\x04': \"TLSv1.3\",\n }[message_data[0:2]]\n\n session_id_length = int_from_bytes(message_data[34:35])\n if session_id_length > 0:\n server_session_id = message_data[35:35 + session_id_length]\n\n cipher_suite_start = 35 + session_id_length\n cipher_suite_bytes = message_data[cipher_suite_start:cipher_suite_start + 2]\n cipher_suite = CIPHER_SUITE_MAP[cipher_suite_bytes]\n\n compression_start = cipher_suite_start + 2\n compression = message_data[compression_start:compression_start + 1] != b'\\x00'\n\n extensions_length_start = compression_start + 1\n extensions_data = message_data[extensions_length_start:]\n for extension_type, extension_data in _parse_hello_extensions(extensions_data):\n if extension_type == 35:\n session_ticket = \"new\"\n break\n break\n\n for record_type, _, record_data in parse_tls_records(client_handshake_bytes):\n if record_type != b'\\x16':\n continue\n for message_type, message_data in parse_handshake_messages(record_data):\n # Ensure we are working with a ClientHello message\n if message_type != b'\\x01':\n continue\n\n session_id_length = int_from_bytes(message_data[34:35])\n if session_id_length > 0:\n client_session_id = message_data[35:35 + session_id_length]\n\n cipher_suite_start = 35 + session_id_length\n cipher_suite_length = int_from_bytes(message_data[cipher_suite_start:cipher_suite_start + 2])\n\n compression_start = cipher_suite_start + 2 + cipher_suite_length\n compression_length = int_from_bytes(message_data[compression_start:compression_start + 1])\n\n # On subsequent requests, the session ticket will only be seen\n # in the ClientHello message\n if server_session_id is None and session_ticket is None:\n extensions_length_start = compression_start + 1 + compression_length\n extensions_data = message_data[extensions_length_start:]\n for extension_type, extension_data in _parse_hello_extensions(extensions_data):\n if extension_type == 35:\n session_ticket = \"reused\"\n break\n break\n\n if server_session_id is not None:\n if client_session_id is None:\n session_id = \"new\"\n else:\n if client_session_id != server_session_id:\n session_id = \"new\"\n else:\n session_id = \"reused\"\n\n return {\n \"protocol\": protocol,\n \"cipher_suite\": cipher_suite,\n \"compression\": compression,\n \"session_id\": session_id,\n \"session_ticket\": session_ticket,\n }"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse the TLS records in a byte string and returns a generator that yields 3 - element tuples of information about each record in the byte string.", "response": "def parse_tls_records(data):\n \"\"\"\n Creates a generator returning tuples of information about each record\n in a byte string of data from a TLS client or server. Stops as soon as it\n find a ChangeCipherSpec message since all data from then on is encrypted.\n\n :param data:\n A byte string of TLS records\n\n :return:\n A generator that yields 3-element tuples:\n [0] Byte string of record type\n [1] Byte string of protocol version\n [2] Byte string of record data\n \"\"\"\n\n pointer = 0\n data_len = len(data)\n while pointer < data_len:\n # Don't try to parse any more once the ChangeCipherSpec is found\n if data[pointer:pointer + 1] == b'\\x14':\n break\n length = int_from_bytes(data[pointer + 3:pointer + 5])\n yield (\n data[pointer:pointer + 1],\n data[pointer + 1:pointer + 3],\n data[pointer + 5:pointer + 5 + length]\n )\n pointer += 5 + length"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse the TLS handshake record and returns a generator that yields 2 - element tuples of message type and data.", "response": "def parse_handshake_messages(data):\n \"\"\"\n Creates a generator returning tuples of information about each message in\n a byte string of data from a TLS handshake record\n\n :param data:\n A byte string of a TLS handshake record data\n\n :return:\n A generator that yields 2-element tuples:\n [0] Byte string of message type\n [1] Byte string of message data\n \"\"\"\n\n pointer = 0\n data_len = len(data)\n while pointer < data_len:\n length = int_from_bytes(data[pointer + 1:pointer + 4])\n yield (\n data[pointer:pointer + 1],\n data[pointer + 4:pointer + 4 + length]\n )\n pointer += 4 + length"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse the server - hello extensions from a TLS ServerHello or ClientHello message and return a generator that yields 2 - element tuples of extension type and extension data.", "response": "def _parse_hello_extensions(data):\n \"\"\"\n Creates a generator returning tuples of information about each extension\n from a byte string of extension data contained in a ServerHello ores\n ClientHello message\n\n :param data:\n A byte string of a extension data from a TLS ServerHello or ClientHello\n message\n\n :return:\n A generator that yields 2-element tuples:\n [0] Byte string of extension type\n [1] Byte string of extension data\n \"\"\"\n\n if data == b'':\n return\n\n extentions_length = int_from_bytes(data[0:2])\n extensions_start = 2\n extensions_end = 2 + extentions_length\n\n pointer = extensions_start\n while pointer < extensions_end:\n extension_type = int_from_bytes(data[pointer:pointer + 2])\n extension_length = int_from_bytes(data[pointer + 2:pointer + 4])\n yield (\n extension_type,\n data[pointer + 4:pointer + 4 + extension_length]\n )\n pointer += 4 + extension_length"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nraises a TLSVerificationError if a hostname mismatch", "response": "def raise_hostname(certificate, hostname):\n \"\"\"\n Raises a TLSVerificationError due to a hostname mismatch\n\n :param certificate:\n An asn1crypto.x509.Certificate object\n\n :raises:\n TLSVerificationError\n \"\"\"\n\n is_ip = re.match('^\\\\d+\\\\.\\\\d+\\\\.\\\\d+\\\\.\\\\d+$', hostname) or hostname.find(':') != -1\n if is_ip:\n hostname_type = 'IP address %s' % hostname\n else:\n hostname_type = 'domain name %s' % hostname\n message = 'Server certificate verification failed - %s does not match' % hostname_type\n valid_ips = ', '.join(certificate.valid_ips)\n valid_domains = ', '.join(certificate.valid_domains)\n if valid_domains:\n message += ' valid domains: %s' % valid_domains\n if valid_domains and valid_ips:\n message += ' or'\n if valid_ips:\n message += ' valid IP addresses: %s' % valid_ips\n raise TLSVerificationError(message, certificate)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nraises a TLSVerificationError if the certificate is expired or not yet valid.", "response": "def raise_expired_not_yet_valid(certificate):\n \"\"\"\n Raises a TLSVerificationError due to certificate being expired, or not yet\n being valid\n\n :param certificate:\n An asn1crypto.x509.Certificate object\n\n :raises:\n TLSVerificationError\n \"\"\"\n\n validity = certificate['tbs_certificate']['validity']\n not_after = validity['not_after'].native\n not_before = validity['not_before'].native\n\n now = datetime.now(timezone.utc)\n\n if not_before > now:\n formatted_before = not_before.strftime('%Y-%m-%d %H:%M:%SZ')\n message = 'Server certificate verification failed - certificate not valid until %s' % formatted_before\n elif not_after < now:\n formatted_after = not_after.strftime('%Y-%m-%d %H:%M:%SZ')\n message = 'Server certificate verification failed - certificate expired %s' % formatted_after\n\n raise TLSVerificationError(message, certificate)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndetecting the other protocol of the server handshake bytes and returns the name of the protocol that is used to connect to it", "response": "def detect_other_protocol(server_handshake_bytes):\n \"\"\"\n Looks at the server handshake bytes to try and detect a different protocol\n\n :param server_handshake_bytes:\n A byte string of the handshake data received from the server\n\n :return:\n None, or a unicode string of \"ftp\", \"http\", \"imap\", \"pop3\", \"smtp\"\n \"\"\"\n\n if server_handshake_bytes[0:5] == b'HTTP/':\n return 'HTTP'\n\n if server_handshake_bytes[0:4] == b'220 ':\n if re.match(b'^[^\\r\\n]*ftp', server_handshake_bytes, re.I):\n return 'FTP'\n else:\n return 'SMTP'\n\n if server_handshake_bytes[0:4] == b'220-':\n return 'FTP'\n\n if server_handshake_bytes[0:4] == b'+OK ':\n return 'POP3'\n\n if server_handshake_bytes[0:4] == b'* OK' or server_handshake_bytes[0:9] == b'* PREAUTH':\n return 'IMAP'\n\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef constant_compare(a, b):\n\n if not isinstance(a, byte_cls):\n raise TypeError(pretty_message(\n '''\n a must be a byte string, not %s\n ''',\n type_name(a)\n ))\n\n if not isinstance(b, byte_cls):\n raise TypeError(pretty_message(\n '''\n b must be a byte string, not %s\n ''',\n type_name(b)\n ))\n\n if len(a) != len(b):\n return False\n\n if sys.version_info < (3,):\n a = [ord(char) for char in a]\n b = [ord(char) for char in b]\n\n result = 0\n for x, y in zip(a, b):\n result |= x ^ y\n return result == 0", "response": "Compares two byte strings in constant time to see if they are equal."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntry to decode a byte string into a unicode string.", "response": "def _try_decode(byte_string):\n \"\"\"\n Tries decoding a byte string from the OS into a unicode string\n\n :param byte_string:\n A byte string\n\n :return:\n A unicode string\n \"\"\"\n\n try:\n return str_cls(byte_string, _encoding)\n\n # If the \"correct\" encoding did not work, try some defaults, and then just\n # obliterate characters that we can't seen to decode properly\n except (UnicodeDecodeError):\n for encoding in _fallback_encodings:\n try:\n return str_cls(byte_string, encoding, errors='strict')\n except (UnicodeDecodeError):\n pass\n\n return str_cls(byte_string, errors='replace')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _read_callback(connection_id, data_buffer, data_length_pointer):\n\n self = None\n try:\n self = _connection_refs.get(connection_id)\n if not self:\n socket = _socket_refs.get(connection_id)\n else:\n socket = self._socket\n\n if not self and not socket:\n return 0\n\n bytes_requested = deref(data_length_pointer)\n\n timeout = socket.gettimeout()\n error = None\n data = b''\n try:\n while len(data) < bytes_requested:\n # Python 2 on Travis CI seems to have issues with blocking on\n # recv() for longer than the socket timeout value, so we select\n if timeout is not None and timeout > 0.0:\n read_ready, _, _ = select.select([socket], [], [], timeout)\n if len(read_ready) == 0:\n raise socket_.error(errno.EAGAIN, 'timed out')\n chunk = socket.recv(bytes_requested - len(data))\n data += chunk\n if chunk == b'':\n if len(data) == 0:\n if timeout is None:\n return SecurityConst.errSSLClosedNoNotify\n return SecurityConst.errSSLClosedAbort\n break\n except (socket_.error) as e:\n error = e.errno\n\n if error is not None and error != errno.EAGAIN:\n if error == errno.ECONNRESET or error == errno.EPIPE:\n return SecurityConst.errSSLClosedNoNotify\n return SecurityConst.errSSLClosedAbort\n\n if self and not self._done_handshake:\n # SecureTransport doesn't bother to check if the TLS record header\n # is valid before asking to read more data, which can result in\n # connection hangs. Here we do basic checks to get around the issue.\n if len(data) >= 3 and len(self._server_hello) == 0:\n # Check to ensure it is an alert or handshake first\n valid_record_type = data[0:1] in set([b'\\x15', b'\\x16'])\n # Check if the protocol version is SSL 3.0 or TLS 1.0-1.3\n valid_protocol_version = data[1:3] in set([\n b'\\x03\\x00',\n b'\\x03\\x01',\n b'\\x03\\x02',\n b'\\x03\\x03',\n b'\\x03\\x04'\n ])\n if not valid_record_type or not valid_protocol_version:\n self._server_hello += data + _read_remaining(socket)\n return SecurityConst.errSSLProtocol\n self._server_hello += data\n\n write_to_buffer(data_buffer, data)\n pointer_set(data_length_pointer, len(data))\n\n if len(data) != bytes_requested:\n return SecurityConst.errSSLWouldBlock\n\n return 0\n except (KeyboardInterrupt) as e:\n if self:\n self._exception = e\n return SecurityConst.errSSLClosedAbort", "response": "This function is called by Secure Transport to actually read the socket and return the status code of the read."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread all available data from the socket and returns them as a byte string.", "response": "def _read_remaining(socket):\n \"\"\"\n Reads everything available from the socket - used for debugging when there\n is a protocol error\n\n :param socket:\n The socket to read from\n\n :return:\n A byte string of the remaining data\n \"\"\"\n\n output = b''\n old_timeout = socket.gettimeout()\n try:\n socket.settimeout(0.0)\n output += socket.recv(8192)\n except (socket_.error):\n pass\n finally:\n socket.settimeout(old_timeout)\n return output"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nperform a TLS handshake on the connection.", "response": "def _handshake(self):\n \"\"\"\n Perform an initial TLS handshake\n \"\"\"\n\n session_context = None\n ssl_policy_ref = None\n crl_search_ref = None\n crl_policy_ref = None\n ocsp_search_ref = None\n ocsp_policy_ref = None\n policy_array_ref = None\n\n try:\n if osx_version_info < (10, 8):\n session_context_pointer = new(Security, 'SSLContextRef *')\n result = Security.SSLNewContext(False, session_context_pointer)\n handle_sec_error(result)\n session_context = unwrap(session_context_pointer)\n\n else:\n session_context = Security.SSLCreateContext(\n null(),\n SecurityConst.kSSLClientSide,\n SecurityConst.kSSLStreamType\n )\n\n result = Security.SSLSetIOFuncs(\n session_context,\n _read_callback_pointer,\n _write_callback_pointer\n )\n handle_sec_error(result)\n\n self._connection_id = id(self) % 2147483647\n _connection_refs[self._connection_id] = self\n _socket_refs[self._connection_id] = self._socket\n result = Security.SSLSetConnection(session_context, self._connection_id)\n handle_sec_error(result)\n\n utf8_domain = self._hostname.encode('utf-8')\n result = Security.SSLSetPeerDomainName(\n session_context,\n utf8_domain,\n len(utf8_domain)\n )\n handle_sec_error(result)\n\n if osx_version_info >= (10, 10):\n disable_auto_validation = self._session._manual_validation or self._session._extra_trust_roots\n explicit_validation = (not self._session._manual_validation) and self._session._extra_trust_roots\n else:\n disable_auto_validation = True\n explicit_validation = not self._session._manual_validation\n\n # Ensure requested protocol support is set for the session\n if osx_version_info < (10, 8):\n for protocol in ['SSLv2', 'SSLv3', 'TLSv1']:\n protocol_const = _PROTOCOL_STRING_CONST_MAP[protocol]\n enabled = protocol in self._session._protocols\n result = Security.SSLSetProtocolVersionEnabled(\n session_context,\n protocol_const,\n enabled\n )\n handle_sec_error(result)\n\n if disable_auto_validation:\n result = Security.SSLSetEnableCertVerify(session_context, False)\n handle_sec_error(result)\n\n else:\n protocol_consts = [_PROTOCOL_STRING_CONST_MAP[protocol] for protocol in self._session._protocols]\n min_protocol = min(protocol_consts)\n max_protocol = max(protocol_consts)\n result = Security.SSLSetProtocolVersionMin(\n session_context,\n min_protocol\n )\n handle_sec_error(result)\n result = Security.SSLSetProtocolVersionMax(\n session_context,\n max_protocol\n )\n handle_sec_error(result)\n\n if disable_auto_validation:\n result = Security.SSLSetSessionOption(\n session_context,\n SecurityConst.kSSLSessionOptionBreakOnServerAuth,\n True\n )\n handle_sec_error(result)\n\n # Disable all sorts of bad cipher suites\n supported_ciphers_pointer = new(Security, 'size_t *')\n result = Security.SSLGetNumberSupportedCiphers(session_context, supported_ciphers_pointer)\n handle_sec_error(result)\n\n supported_ciphers = deref(supported_ciphers_pointer)\n\n cipher_buffer = buffer_from_bytes(supported_ciphers * 4)\n supported_cipher_suites_pointer = cast(Security, 'uint32_t *', cipher_buffer)\n result = Security.SSLGetSupportedCiphers(\n session_context,\n supported_cipher_suites_pointer,\n supported_ciphers_pointer\n )\n handle_sec_error(result)\n\n supported_ciphers = deref(supported_ciphers_pointer)\n supported_cipher_suites = array_from_pointer(\n Security,\n 'uint32_t',\n supported_cipher_suites_pointer,\n supported_ciphers\n )\n good_ciphers = []\n for supported_cipher_suite in supported_cipher_suites:\n cipher_suite = int_to_bytes(supported_cipher_suite, width=2)\n cipher_suite_name = CIPHER_SUITE_MAP.get(cipher_suite, cipher_suite)\n good_cipher = _cipher_blacklist_regex.search(cipher_suite_name) is None\n if good_cipher:\n good_ciphers.append(supported_cipher_suite)\n\n num_good_ciphers = len(good_ciphers)\n good_ciphers_array = new(Security, 'uint32_t[]', num_good_ciphers)\n array_set(good_ciphers_array, good_ciphers)\n good_ciphers_pointer = cast(Security, 'uint32_t *', good_ciphers_array)\n result = Security.SSLSetEnabledCiphers(\n session_context,\n good_ciphers_pointer,\n num_good_ciphers\n )\n handle_sec_error(result)\n\n # Set a peer id from the session to allow for session reuse, the hostname\n # is appended to prevent a bug on OS X 10.7 where it tries to reuse a\n # connection even if the hostnames are different.\n peer_id = self._session._peer_id + self._hostname.encode('utf-8')\n result = Security.SSLSetPeerID(session_context, peer_id, len(peer_id))\n handle_sec_error(result)\n\n handshake_result = Security.SSLHandshake(session_context)\n if self._exception is not None:\n exception = self._exception\n self._exception = None\n raise exception\n while handshake_result == SecurityConst.errSSLWouldBlock:\n handshake_result = Security.SSLHandshake(session_context)\n if self._exception is not None:\n exception = self._exception\n self._exception = None\n raise exception\n\n if osx_version_info < (10, 8) and osx_version_info >= (10, 7):\n do_validation = explicit_validation and handshake_result == 0\n else:\n do_validation = explicit_validation and handshake_result == SecurityConst.errSSLServerAuthCompleted\n\n if do_validation:\n trust_ref_pointer = new(Security, 'SecTrustRef *')\n result = Security.SSLCopyPeerTrust(\n session_context,\n trust_ref_pointer\n )\n handle_sec_error(result)\n trust_ref = unwrap(trust_ref_pointer)\n\n cf_string_hostname = CFHelpers.cf_string_from_unicode(self._hostname)\n ssl_policy_ref = Security.SecPolicyCreateSSL(True, cf_string_hostname)\n result = CoreFoundation.CFRelease(cf_string_hostname)\n handle_cf_error(result)\n\n # Create a new policy for OCSP checking to disable it\n ocsp_oid_pointer = struct(Security, 'CSSM_OID')\n ocsp_oid = unwrap(ocsp_oid_pointer)\n ocsp_oid.Length = len(SecurityConst.APPLE_TP_REVOCATION_OCSP)\n ocsp_oid_buffer = buffer_from_bytes(SecurityConst.APPLE_TP_REVOCATION_OCSP)\n ocsp_oid.Data = cast(Security, 'char *', ocsp_oid_buffer)\n\n ocsp_search_ref_pointer = new(Security, 'SecPolicySearchRef *')\n result = Security.SecPolicySearchCreate(\n SecurityConst.CSSM_CERT_X_509v3,\n ocsp_oid_pointer,\n null(),\n ocsp_search_ref_pointer\n )\n handle_sec_error(result)\n ocsp_search_ref = unwrap(ocsp_search_ref_pointer)\n\n ocsp_policy_ref_pointer = new(Security, 'SecPolicyRef *')\n result = Security.SecPolicySearchCopyNext(ocsp_search_ref, ocsp_policy_ref_pointer)\n handle_sec_error(result)\n ocsp_policy_ref = unwrap(ocsp_policy_ref_pointer)\n\n ocsp_struct_pointer = struct(Security, 'CSSM_APPLE_TP_OCSP_OPTIONS')\n ocsp_struct = unwrap(ocsp_struct_pointer)\n ocsp_struct.Version = SecurityConst.CSSM_APPLE_TP_OCSP_OPTS_VERSION\n ocsp_struct.Flags = (\n SecurityConst.CSSM_TP_ACTION_OCSP_DISABLE_NET |\n SecurityConst.CSSM_TP_ACTION_OCSP_CACHE_READ_DISABLE\n )\n ocsp_struct_bytes = struct_bytes(ocsp_struct_pointer)\n\n cssm_data_pointer = struct(Security, 'CSSM_DATA')\n cssm_data = unwrap(cssm_data_pointer)\n cssm_data.Length = len(ocsp_struct_bytes)\n ocsp_struct_buffer = buffer_from_bytes(ocsp_struct_bytes)\n cssm_data.Data = cast(Security, 'char *', ocsp_struct_buffer)\n\n result = Security.SecPolicySetValue(ocsp_policy_ref, cssm_data_pointer)\n handle_sec_error(result)\n\n # Create a new policy for CRL checking to disable it\n crl_oid_pointer = struct(Security, 'CSSM_OID')\n crl_oid = unwrap(crl_oid_pointer)\n crl_oid.Length = len(SecurityConst.APPLE_TP_REVOCATION_CRL)\n crl_oid_buffer = buffer_from_bytes(SecurityConst.APPLE_TP_REVOCATION_CRL)\n crl_oid.Data = cast(Security, 'char *', crl_oid_buffer)\n\n crl_search_ref_pointer = new(Security, 'SecPolicySearchRef *')\n result = Security.SecPolicySearchCreate(\n SecurityConst.CSSM_CERT_X_509v3,\n crl_oid_pointer,\n null(),\n crl_search_ref_pointer\n )\n handle_sec_error(result)\n crl_search_ref = unwrap(crl_search_ref_pointer)\n\n crl_policy_ref_pointer = new(Security, 'SecPolicyRef *')\n result = Security.SecPolicySearchCopyNext(crl_search_ref, crl_policy_ref_pointer)\n handle_sec_error(result)\n crl_policy_ref = unwrap(crl_policy_ref_pointer)\n\n crl_struct_pointer = struct(Security, 'CSSM_APPLE_TP_CRL_OPTIONS')\n crl_struct = unwrap(crl_struct_pointer)\n crl_struct.Version = SecurityConst.CSSM_APPLE_TP_CRL_OPTS_VERSION\n crl_struct.CrlFlags = 0\n crl_struct_bytes = struct_bytes(crl_struct_pointer)\n\n cssm_data_pointer = struct(Security, 'CSSM_DATA')\n cssm_data = unwrap(cssm_data_pointer)\n cssm_data.Length = len(crl_struct_bytes)\n crl_struct_buffer = buffer_from_bytes(crl_struct_bytes)\n cssm_data.Data = cast(Security, 'char *', crl_struct_buffer)\n\n result = Security.SecPolicySetValue(crl_policy_ref, cssm_data_pointer)\n handle_sec_error(result)\n\n policy_array_ref = CFHelpers.cf_array_from_list([\n ssl_policy_ref,\n crl_policy_ref,\n ocsp_policy_ref\n ])\n\n result = Security.SecTrustSetPolicies(trust_ref, policy_array_ref)\n handle_sec_error(result)\n\n if self._session._extra_trust_roots:\n ca_cert_refs = []\n ca_certs = []\n for cert in self._session._extra_trust_roots:\n ca_cert = load_certificate(cert)\n ca_certs.append(ca_cert)\n ca_cert_refs.append(ca_cert.sec_certificate_ref)\n\n result = Security.SecTrustSetAnchorCertificatesOnly(trust_ref, False)\n handle_sec_error(result)\n\n array_ref = CFHelpers.cf_array_from_list(ca_cert_refs)\n result = Security.SecTrustSetAnchorCertificates(trust_ref, array_ref)\n handle_sec_error(result)\n\n result_pointer = new(Security, 'SecTrustResultType *')\n result = Security.SecTrustEvaluate(trust_ref, result_pointer)\n handle_sec_error(result)\n\n trust_result_code = deref(result_pointer)\n invalid_chain_error_codes = set([\n SecurityConst.kSecTrustResultProceed,\n SecurityConst.kSecTrustResultUnspecified\n ])\n if trust_result_code not in invalid_chain_error_codes:\n handshake_result = SecurityConst.errSSLXCertChainInvalid\n else:\n handshake_result = Security.SSLHandshake(session_context)\n while handshake_result == SecurityConst.errSSLWouldBlock:\n handshake_result = Security.SSLHandshake(session_context)\n\n self._done_handshake = True\n\n handshake_error_codes = set([\n SecurityConst.errSSLXCertChainInvalid,\n SecurityConst.errSSLCertExpired,\n SecurityConst.errSSLCertNotYetValid,\n SecurityConst.errSSLUnknownRootCert,\n SecurityConst.errSSLNoRootCert,\n SecurityConst.errSSLHostNameMismatch,\n SecurityConst.errSSLInternal,\n ])\n\n # In testing, only errSSLXCertChainInvalid was ever returned for\n # all of these different situations, however we include the others\n # for completeness. To get the real reason we have to use the\n # certificate from the handshake and use the deprecated function\n # SecTrustGetCssmResultCode().\n if handshake_result in handshake_error_codes:\n trust_ref_pointer = new(Security, 'SecTrustRef *')\n result = Security.SSLCopyPeerTrust(\n session_context,\n trust_ref_pointer\n )\n handle_sec_error(result)\n trust_ref = unwrap(trust_ref_pointer)\n\n result_code_pointer = new(Security, 'OSStatus *')\n result = Security.SecTrustGetCssmResultCode(trust_ref, result_code_pointer)\n result_code = deref(result_code_pointer)\n\n chain = extract_chain(self._server_hello)\n\n self_signed = False\n revoked = False\n expired = False\n not_yet_valid = False\n no_issuer = False\n cert = None\n bad_hostname = False\n\n if chain:\n cert = chain[0]\n oscrypto_cert = load_certificate(cert)\n self_signed = oscrypto_cert.self_signed\n revoked = result_code == SecurityConst.CSSMERR_TP_CERT_REVOKED\n no_issuer = not self_signed and result_code == SecurityConst.CSSMERR_TP_NOT_TRUSTED\n expired = result_code == SecurityConst.CSSMERR_TP_CERT_EXPIRED\n not_yet_valid = result_code == SecurityConst.CSSMERR_TP_CERT_NOT_VALID_YET\n bad_hostname = result_code == SecurityConst.CSSMERR_APPLETP_HOSTNAME_MISMATCH\n\n # On macOS 10.12, some expired certificates return errSSLInternal\n if osx_version_info >= (10, 12):\n validity = cert['tbs_certificate']['validity']\n not_before = validity['not_before'].chosen.native\n not_after = validity['not_after'].chosen.native\n utcnow = datetime.datetime.now(timezone.utc)\n expired = not_after < utcnow\n not_yet_valid = not_before > utcnow\n\n if chain and chain[0].hash_algo in set(['md5', 'md2']):\n raise_weak_signature(chain[0])\n\n if revoked:\n raise_revoked(cert)\n\n if bad_hostname:\n raise_hostname(cert, self._hostname)\n\n elif expired or not_yet_valid:\n raise_expired_not_yet_valid(cert)\n\n elif no_issuer:\n raise_no_issuer(cert)\n\n elif self_signed:\n raise_self_signed(cert)\n\n if detect_client_auth_request(self._server_hello):\n raise_client_auth()\n\n raise_verification(cert)\n\n if handshake_result == SecurityConst.errSSLPeerHandshakeFail:\n if detect_client_auth_request(self._server_hello):\n raise_client_auth()\n raise_handshake()\n\n if handshake_result == SecurityConst.errSSLWeakPeerEphemeralDHKey:\n raise_dh_params()\n\n if handshake_result == SecurityConst.errSSLPeerProtocolVersion:\n raise_protocol_version()\n\n if handshake_result in set([SecurityConst.errSSLRecordOverflow, SecurityConst.errSSLProtocol]):\n self._server_hello += _read_remaining(self._socket)\n raise_protocol_error(self._server_hello)\n\n if handshake_result in set([SecurityConst.errSSLClosedNoNotify, SecurityConst.errSSLClosedAbort]):\n if not self._done_handshake:\n self._server_hello += _read_remaining(self._socket)\n if detect_other_protocol(self._server_hello):\n raise_protocol_error(self._server_hello)\n raise_disconnection()\n\n if osx_version_info < (10, 10):\n dh_params_length = get_dh_params_length(self._server_hello)\n if dh_params_length is not None and dh_params_length < 1024:\n raise_dh_params()\n\n would_block = handshake_result == SecurityConst.errSSLWouldBlock\n server_auth_complete = handshake_result == SecurityConst.errSSLServerAuthCompleted\n manual_validation = self._session._manual_validation and server_auth_complete\n if not would_block and not manual_validation:\n handle_sec_error(handshake_result, TLSError)\n\n self._session_context = session_context\n\n protocol_const_pointer = new(Security, 'SSLProtocol *')\n result = Security.SSLGetNegotiatedProtocolVersion(\n session_context,\n protocol_const_pointer\n )\n handle_sec_error(result)\n protocol_const = deref(protocol_const_pointer)\n\n self._protocol = _PROTOCOL_CONST_STRING_MAP[protocol_const]\n\n cipher_int_pointer = new(Security, 'SSLCipherSuite *')\n result = Security.SSLGetNegotiatedCipher(\n session_context,\n cipher_int_pointer\n )\n handle_sec_error(result)\n cipher_int = deref(cipher_int_pointer)\n\n cipher_bytes = int_to_bytes(cipher_int, width=2)\n self._cipher_suite = CIPHER_SUITE_MAP.get(cipher_bytes, cipher_bytes)\n\n session_info = parse_session_info(\n self._server_hello,\n self._client_hello\n )\n self._compression = session_info['compression']\n self._session_id = session_info['session_id']\n self._session_ticket = session_info['session_ticket']\n\n except (OSError, socket_.error):\n if session_context:\n if osx_version_info < (10, 8):\n result = Security.SSLDisposeContext(session_context)\n handle_sec_error(result)\n else:\n result = CoreFoundation.CFRelease(session_context)\n handle_cf_error(result)\n\n self._session_context = None\n self.close()\n\n raise\n\n finally:\n # Trying to release crl_search_ref or ocsp_search_ref results in\n # a segmentation fault, so we do not do that\n\n if ssl_policy_ref:\n result = CoreFoundation.CFRelease(ssl_policy_ref)\n handle_cf_error(result)\n ssl_policy_ref = None\n\n if crl_policy_ref:\n result = CoreFoundation.CFRelease(crl_policy_ref)\n handle_cf_error(result)\n crl_policy_ref = None\n\n if ocsp_policy_ref:\n result = CoreFoundation.CFRelease(ocsp_policy_ref)\n handle_cf_error(result)\n ocsp_policy_ref = None\n\n if policy_array_ref:\n result = CoreFoundation.CFRelease(policy_array_ref)\n handle_cf_error(result)\n policy_array_ref = None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreads data from the TLS - wrapped socket and returns it as a byte string.", "response": "def read(self, max_length):\n \"\"\"\n Reads data from the TLS-wrapped socket\n\n :param max_length:\n The number of bytes to read - output may be less than this\n\n :raises:\n socket.socket - when a non-TLS socket error occurs\n oscrypto.errors.TLSError - when a TLS-related error occurs\n oscrypto.errors.TLSDisconnectError - when the connection disconnects\n oscrypto.errors.TLSGracefulDisconnectError - when the remote end gracefully closed the connection\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the data read\n \"\"\"\n\n if not isinstance(max_length, int_types):\n raise TypeError(pretty_message(\n '''\n max_length must be an integer, not %s\n ''',\n type_name(max_length)\n ))\n\n if self._session_context is None:\n # Even if the session is closed, we can use\n # buffered data to respond to read requests\n if self._decrypted_bytes != b'':\n output = self._decrypted_bytes\n self._decrypted_bytes = b''\n return output\n\n self._raise_closed()\n\n buffered_length = len(self._decrypted_bytes)\n\n # If we already have enough buffered data, just use that\n if buffered_length >= max_length:\n output = self._decrypted_bytes[0:max_length]\n self._decrypted_bytes = self._decrypted_bytes[max_length:]\n return output\n\n # Don't block if we have buffered data available, since it is ok to\n # return less than the max_length\n if buffered_length > 0 and not self.select_read(0):\n output = self._decrypted_bytes\n self._decrypted_bytes = b''\n return output\n\n # Only read enough to get the requested amount when\n # combined with buffered data\n to_read = max_length - len(self._decrypted_bytes)\n\n read_buffer = buffer_from_bytes(to_read)\n processed_pointer = new(Security, 'size_t *')\n result = Security.SSLRead(\n self._session_context,\n read_buffer,\n to_read,\n processed_pointer\n )\n if self._exception is not None:\n exception = self._exception\n self._exception = None\n raise exception\n if result and result not in set([SecurityConst.errSSLWouldBlock, SecurityConst.errSSLClosedGraceful]):\n handle_sec_error(result, TLSError)\n\n if result and result == SecurityConst.errSSLClosedGraceful:\n self._gracefully_closed = True\n self._shutdown(False)\n self._raise_closed()\n\n bytes_read = deref(processed_pointer)\n output = self._decrypted_bytes + bytes_from_buffer(read_buffer, bytes_read)\n\n self._decrypted_bytes = output[max_length:]\n return output[0:max_length]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nread data from the socket until a marker is found.", "response": "def read_until(self, marker):\n \"\"\"\n Reads data from the socket until a marker is found. Data read includes\n the marker.\n\n :param marker:\n A byte string or regex object from re.compile(). Used to determine\n when to stop reading. Regex objects are more inefficient since\n they must scan the entire byte string of read data each time data\n is read off the socket.\n\n :return:\n A byte string of the data read, including the marker\n \"\"\"\n\n if not isinstance(marker, byte_cls) and not isinstance(marker, Pattern):\n raise TypeError(pretty_message(\n '''\n marker must be a byte string or compiled regex object, not %s\n ''',\n type_name(marker)\n ))\n\n output = b''\n\n is_regex = isinstance(marker, Pattern)\n\n while True:\n if len(self._decrypted_bytes) > 0:\n chunk = self._decrypted_bytes\n self._decrypted_bytes = b''\n else:\n to_read = self._os_buffered_size() or 8192\n chunk = self.read(to_read)\n\n offset = len(output)\n output += chunk\n\n if is_regex:\n match = marker.search(output)\n if match is not None:\n end = match.end()\n break\n else:\n # If the marker was not found last time, we have to start\n # at a position where the marker would have its final char\n # in the newly read chunk\n start = max(0, offset - len(marker) - 1)\n match = output.find(marker, start)\n if match != -1:\n end = match + len(marker)\n break\n\n self._decrypted_bytes = output[end:] + self._decrypted_bytes\n return output[0:end]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _os_buffered_size(self):\n\n num_bytes_pointer = new(Security, 'size_t *')\n result = Security.SSLGetBufferedReadSize(\n self._session_context,\n num_bytes_pointer\n )\n handle_sec_error(result)\n\n return deref(num_bytes_pointer)", "response": "Returns the number of bytes of decrypted data stored in the Secure\n Transport read buffer."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nwriting data to the TLS - wrapped socket.", "response": "def write(self, data):\n \"\"\"\n Writes data to the TLS-wrapped socket\n\n :param data:\n A byte string to write to the socket\n\n :raises:\n socket.socket - when a non-TLS socket error occurs\n oscrypto.errors.TLSError - when a TLS-related error occurs\n oscrypto.errors.TLSDisconnectError - when the connection disconnects\n oscrypto.errors.TLSGracefulDisconnectError - when the remote end gracefully closed the connection\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n \"\"\"\n\n if self._session_context is None:\n self._raise_closed()\n\n processed_pointer = new(Security, 'size_t *')\n\n data_len = len(data)\n while data_len:\n write_buffer = buffer_from_bytes(data)\n result = Security.SSLWrite(\n self._session_context,\n write_buffer,\n data_len,\n processed_pointer\n )\n if self._exception is not None:\n exception = self._exception\n self._exception = None\n raise exception\n handle_sec_error(result, TLSError)\n\n bytes_written = deref(processed_pointer)\n data = data[bytes_written:]\n data_len = len(data)\n if data_len > 0:\n self.select_write()"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _shutdown(self, manual):\n\n if self._session_context is None:\n return\n\n # Ignore error during close in case other end closed already\n result = Security.SSLClose(self._session_context)\n\n if osx_version_info < (10, 8):\n result = Security.SSLDisposeContext(self._session_context)\n handle_sec_error(result)\n else:\n result = CoreFoundation.CFRelease(self._session_context)\n handle_cf_error(result)\n\n self._session_context = None\n\n if manual:\n self._local_closed = True\n\n try:\n self._socket.shutdown(socket_.SHUT_RDWR)\n except (socket_.error):\n pass", "response": "Shuts down the TLS session and then shuts down the underlying socket."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef close(self):\n\n try:\n self.shutdown()\n\n finally:\n if self._socket:\n try:\n self._socket.close()\n except (socket_.error):\n pass\n self._socket = None\n\n if self._connection_id in _socket_refs:\n del _socket_refs[self._connection_id]", "response": "Closes the TLS session and socket and forcibly closes it."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreading end - entity and intermediate certificate information from the TLS session and returns the result.", "response": "def _read_certificates(self):\n \"\"\"\n Reads end-entity and intermediate certificate information from the\n TLS session\n \"\"\"\n\n trust_ref = None\n cf_data_ref = None\n result = None\n\n try:\n trust_ref_pointer = new(Security, 'SecTrustRef *')\n result = Security.SSLCopyPeerTrust(\n self._session_context,\n trust_ref_pointer\n )\n handle_sec_error(result)\n\n trust_ref = unwrap(trust_ref_pointer)\n\n number_certs = Security.SecTrustGetCertificateCount(trust_ref)\n\n self._intermediates = []\n\n for index in range(0, number_certs):\n sec_certificate_ref = Security.SecTrustGetCertificateAtIndex(\n trust_ref,\n index\n )\n cf_data_ref = Security.SecCertificateCopyData(sec_certificate_ref)\n\n cert_data = CFHelpers.cf_data_to_bytes(cf_data_ref)\n\n result = CoreFoundation.CFRelease(cf_data_ref)\n handle_cf_error(result)\n cf_data_ref = None\n\n cert = x509.Certificate.load(cert_data)\n\n if index == 0:\n self._certificate = cert\n else:\n self._intermediates.append(cert)\n\n finally:\n if trust_ref:\n result = CoreFoundation.CFRelease(trust_ref)\n handle_cf_error(result)\n if cf_data_ref:\n result = CoreFoundation.CFRelease(cf_data_ref)\n handle_cf_error(result)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the asn1crypto. x509. Certificate object of the end - entity certificate that is presented by the server.", "response": "def certificate(self):\n \"\"\"\n An asn1crypto.x509.Certificate object of the end-entity certificate\n presented by the server\n \"\"\"\n\n if self._session_context is None:\n self._raise_closed()\n\n if self._certificate is None:\n self._read_certificates()\n\n return self._certificate"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the path to a file that contains OpenSSL - compatible CA certs.", "response": "def get_path(temp_dir=None, cache_length=24, cert_callback=None):\n \"\"\"\n Get the filesystem path to a file that contains OpenSSL-compatible CA certs.\n\n On OS X and Windows, there are extracted from the system certificate store\n and cached in a file on the filesystem. This path should not be writable\n by other users, otherwise they could inject CA certs into the trust list.\n\n :param temp_dir:\n The temporary directory to cache the CA certs in on OS X and Windows.\n Needs to have secure permissions so other users can not modify the\n contents.\n\n :param cache_length:\n The number of hours to cache the CA certs on OS X and Windows\n\n :param cert_callback:\n A callback that is called once for each certificate in the trust store.\n It should accept two parameters: an asn1crypto.x509.Certificate object,\n and a reason. The reason will be None if the certificate is being\n exported, otherwise it will be a unicode string of the reason it won't.\n This is only called on Windows and OS X when passed to this function.\n\n :raises:\n oscrypto.errors.CACertsError - when an error occurs exporting/locating certs\n\n :return:\n The full filesystem path to a CA certs file\n \"\"\"\n\n ca_path, temp = _ca_path(temp_dir)\n\n # Windows and OS X\n if temp and _cached_path_needs_update(ca_path, cache_length):\n empty_set = set()\n\n any_purpose = '2.5.29.37.0'\n apple_ssl = '1.2.840.113635.100.1.3'\n win_server_auth = '1.3.6.1.5.5.7.3.1'\n\n with path_lock:\n if _cached_path_needs_update(ca_path, cache_length):\n with open(ca_path, 'wb') as f:\n for cert, trust_oids, reject_oids in extract_from_system(cert_callback, True):\n if sys.platform == 'darwin':\n if trust_oids != empty_set and any_purpose not in trust_oids \\\n and apple_ssl not in trust_oids:\n if cert_callback:\n cert_callback(Certificate.load(cert), 'implicitly distrusted for TLS')\n continue\n if reject_oids != empty_set and (apple_ssl in reject_oids\n or any_purpose in reject_oids):\n if cert_callback:\n cert_callback(Certificate.load(cert), 'explicitly distrusted for TLS')\n continue\n elif sys.platform == 'win32':\n if trust_oids != empty_set and any_purpose not in trust_oids \\\n and win_server_auth not in trust_oids:\n if cert_callback:\n cert_callback(Certificate.load(cert), 'implicitly distrusted for TLS')\n continue\n if reject_oids != empty_set and (win_server_auth in reject_oids\n or any_purpose in reject_oids):\n if cert_callback:\n cert_callback(Certificate.load(cert), 'explicitly distrusted for TLS')\n continue\n if cert_callback:\n cert_callback(Certificate.load(cert), None)\n f.write(armor('CERTIFICATE', cert))\n\n if not ca_path:\n raise CACertsError('No CA certs found')\n\n return ca_path"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a list of the CA certs from the OS.", "response": "def get_list(cache_length=24, map_vendor_oids=True, cert_callback=None):\n \"\"\"\n Retrieves (and caches in memory) the list of CA certs from the OS. Includes\n trust information from the OS - purposes the certificate should be trusted\n or rejected for.\n\n Trust information is encoded via object identifiers (OIDs) that are sourced\n from various RFCs and vendors (Apple and Microsoft). This trust information\n augments what is in the certificate itself. Any OID that is in the set of\n trusted purposes indicates the certificate has been explicitly trusted for\n a purpose beyond the extended key purpose extension. Any OID in the reject\n set is a purpose that the certificate should not be trusted for, even if\n present in the extended key purpose extension.\n\n *A list of common trust OIDs can be found as part of the `KeyPurposeId()`\n class in the `asn1crypto.x509` module of the `asn1crypto` package.*\n\n :param cache_length:\n The number of hours to cache the CA certs in memory before they are\n refreshed\n\n :param map_vendor_oids:\n A bool indicating if the following mapping of OIDs should happen for\n trust information from the OS trust list:\n - 1.2.840.113635.100.1.3 (apple_ssl) -> 1.3.6.1.5.5.7.3.1 (server_auth)\n - 1.2.840.113635.100.1.3 (apple_ssl) -> 1.3.6.1.5.5.7.3.2 (client_auth)\n - 1.2.840.113635.100.1.8 (apple_smime) -> 1.3.6.1.5.5.7.3.4 (email_protection)\n - 1.2.840.113635.100.1.9 (apple_eap) -> 1.3.6.1.5.5.7.3.13 (eap_over_ppp)\n - 1.2.840.113635.100.1.9 (apple_eap) -> 1.3.6.1.5.5.7.3.14 (eap_over_lan)\n - 1.2.840.113635.100.1.11 (apple_ipsec) -> 1.3.6.1.5.5.7.3.5 (ipsec_end_system)\n - 1.2.840.113635.100.1.11 (apple_ipsec) -> 1.3.6.1.5.5.7.3.6 (ipsec_tunnel)\n - 1.2.840.113635.100.1.11 (apple_ipsec) -> 1.3.6.1.5.5.7.3.7 (ipsec_user)\n - 1.2.840.113635.100.1.11 (apple_ipsec) -> 1.3.6.1.5.5.7.3.17 (ipsec_ike)\n - 1.2.840.113635.100.1.16 (apple_code_signing) -> 1.3.6.1.5.5.7.3.3 (code_signing)\n - 1.2.840.113635.100.1.20 (apple_time_stamping) -> 1.3.6.1.5.5.7.3.8 (time_stamping)\n - 1.3.6.1.4.1.311.10.3.2 (microsoft_time_stamp_signing) -> 1.3.6.1.5.5.7.3.8 (time_stamping)\n\n :param cert_callback:\n A callback that is called once for each certificate in the trust store.\n It should accept two parameters: an asn1crypto.x509.Certificate object,\n and a reason. The reason will be None if the certificate is being\n exported, otherwise it will be a unicode string of the reason it won't.\n\n :raises:\n oscrypto.errors.CACertsError - when an error occurs exporting/locating certs\n\n :return:\n A (copied) list of 3-element tuples containing CA certs from the OS\n trust ilst:\n - 0: an asn1crypto.x509.Certificate object\n - 1: a set of unicode strings of OIDs of trusted purposes\n - 2: a set of unicode strings of OIDs of rejected purposes\n \"\"\"\n\n if not _in_memory_up_to_date(cache_length):\n with memory_lock:\n if not _in_memory_up_to_date(cache_length):\n certs = []\n for cert_bytes, trust_oids, reject_oids in extract_from_system(cert_callback):\n if map_vendor_oids:\n trust_oids = _map_oids(trust_oids)\n reject_oids = _map_oids(reject_oids)\n certs.append((Certificate.load(cert_bytes), trust_oids, reject_oids))\n _module_values['certs'] = certs\n _module_values['last_update'] = time.time()\n\n return list(_module_values['certs'])"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nclearing any cached info that was exported from the OS trust store.", "response": "def clear_cache(temp_dir=None):\n \"\"\"\n Clears any cached info that was exported from the OS trust store. This will\n ensure the latest changes are returned from calls to get_list() and\n get_path(), but at the expense of re-exporting and parsing all certificates.\n\n :param temp_dir:\n The temporary directory to cache the CA certs in on OS X and Windows.\n Needs to have secure permissions so other users can not modify the\n contents. Must be the same value passed to get_path().\n \"\"\"\n\n with memory_lock:\n _module_values['last_update'] = None\n _module_values['certs'] = None\n\n ca_path, temp = _ca_path(temp_dir)\n if temp:\n with path_lock:\n if os.path.exists(ca_path):\n os.remove(ca_path)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the file path to the CA certs file in the specified temporary directory.", "response": "def _ca_path(temp_dir=None):\n \"\"\"\n Returns the file path to the CA certs file\n\n :param temp_dir:\n The temporary directory to cache the CA certs in on OS X and Windows.\n Needs to have secure permissions so other users can not modify the\n contents.\n\n :return:\n A 2-element tuple:\n - 0: A unicode string of the file path\n - 1: A bool if the file is a temporary file\n \"\"\"\n\n ca_path = system_path()\n\n # Windows and OS X\n if ca_path is None:\n if temp_dir is None:\n temp_dir = tempfile.gettempdir()\n\n if not os.path.isdir(temp_dir):\n raise CACertsError(pretty_message(\n '''\n The temp dir specified, \"%s\", is not a directory\n ''',\n temp_dir\n ))\n\n ca_path = os.path.join(temp_dir, 'oscrypto-ca-bundle.crt')\n return (ca_path, True)\n\n return (ca_path, False)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nchecks to see if a cache file needs to be refreshed and if so checks if the file exists and if it is up - to - date and returns True if the file exists and False if it is up - to - date", "response": "def _cached_path_needs_update(ca_path, cache_length):\n \"\"\"\n Checks to see if a cache file needs to be refreshed\n\n :param ca_path:\n A unicode string of the path to the cache file\n\n :param cache_length:\n An integer representing the number of hours the cache is valid for\n\n :return:\n A boolean - True if the cache needs to be updated, False if the file\n is up-to-date\n \"\"\"\n\n exists = os.path.exists(ca_path)\n if not exists:\n return True\n\n stats = os.stat(ca_path)\n\n if stats.st_mtime < time.time() - cache_length * 60 * 60:\n return True\n\n if stats.st_size == 0:\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rand_bytes(length):\n\n if not isinstance(length, int_types):\n raise TypeError(pretty_message(\n '''\n length must be an integer, not %s\n ''',\n type_name(length)\n ))\n\n if length < 1:\n raise ValueError('length must be greater than 0')\n\n if length > 1024:\n raise ValueError('length must not be greater than 1024')\n\n return os.urandom(length)", "response": "Returns a random number of bytes suitable for cryptographic purposes\nAttributeNames"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef pbkdf2(hash_algorithm, password, salt, iterations, key_length):\n\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n\n if not isinstance(salt, byte_cls):\n raise TypeError(pretty_message(\n '''\n salt must be a byte string, not %s\n ''',\n type_name(salt)\n ))\n\n if not isinstance(iterations, int_types):\n raise TypeError(pretty_message(\n '''\n iterations must be an integer, not %s\n ''',\n type_name(iterations)\n ))\n\n if iterations < 1:\n raise ValueError(pretty_message(\n '''\n iterations must be greater than 0 - is %s\n ''',\n repr(iterations)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 1:\n raise ValueError(pretty_message(\n '''\n key_length must be greater than 0 - is %s\n ''',\n repr(key_length)\n ))\n\n if hash_algorithm not in set(['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"md5\", \"sha1\", \"sha224\", \"sha256\",\n \"sha384\", \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n algo = getattr(hashlib, hash_algorithm)\n\n hash_length = {\n 'md5': 16,\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }[hash_algorithm]\n\n blocks = int(math.ceil(key_length / hash_length))\n\n original_hmac = hmac.new(password, None, algo)\n\n int_pack = struct.Struct(b'>I').pack\n\n output = b''\n for block in range(1, blocks + 1):\n prf = original_hmac.copy()\n prf.update(salt + int_pack(block))\n last = prf.digest()\n u = int_from_bytes(last)\n for _ in range(2, iterations + 1):\n prf = original_hmac.copy()\n prf.update(last)\n last = prf.digest()\n u ^= int_from_bytes(last)\n t = int_to_bytes(u)\n output += t\n\n return output[0:key_length]", "response": "Implements PBKDF2 from PKCS#5 v2. 2 in pure Python module."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a public key pair for the specified algorithm bit_size and curve.", "response": "def generate_pair(algorithm, bit_size=None, curve=None):\n \"\"\"\n Generates a public/private key pair\n\n :param algorithm:\n The key algorithm - \"rsa\", \"dsa\" or \"ec\"\n\n :param bit_size:\n An integer - used for \"rsa\" and \"dsa\". For \"rsa\" the value maye be 1024,\n 2048, 3072 or 4096. For \"dsa\" the value may be 1024.\n\n :param curve:\n A unicode string - used for \"ec\" keys. Valid values include \"secp256r1\",\n \"secp384r1\" and \"secp521r1\".\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A 2-element tuple of (PublicKey, PrivateKey). The contents of each key\n may be saved by calling .asn1.dump().\n \"\"\"\n\n if algorithm not in set(['rsa', 'dsa', 'ec']):\n raise ValueError(pretty_message(\n '''\n algorithm must be one of \"rsa\", \"dsa\", \"ec\", not %s\n ''',\n repr(algorithm)\n ))\n\n if algorithm == 'rsa':\n if bit_size not in set([1024, 2048, 3072, 4096]):\n raise ValueError(pretty_message(\n '''\n bit_size must be one of 1024, 2048, 3072, 4096, not %s\n ''',\n repr(bit_size)\n ))\n\n elif algorithm == 'dsa':\n if bit_size not in set([1024]):\n raise ValueError(pretty_message(\n '''\n bit_size must be 1024, not %s\n ''',\n repr(bit_size)\n ))\n\n elif algorithm == 'ec':\n if curve not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise ValueError(pretty_message(\n '''\n curve must be one of \"secp256r1\", \"secp384r1\", \"secp521r1\", not %s\n ''',\n repr(curve)\n ))\n\n cf_dict = None\n public_key_ref = None\n private_key_ref = None\n cf_data_public = None\n cf_data_private = None\n cf_string = None\n sec_access_ref = None\n\n try:\n key_type = {\n 'dsa': Security.kSecAttrKeyTypeDSA,\n 'ec': Security.kSecAttrKeyTypeECDSA,\n 'rsa': Security.kSecAttrKeyTypeRSA,\n }[algorithm]\n\n if algorithm == 'ec':\n key_size = {\n 'secp256r1': 256,\n 'secp384r1': 384,\n 'secp521r1': 521,\n }[curve]\n else:\n key_size = bit_size\n\n private_key_pointer = new(Security, 'SecKeyRef *')\n public_key_pointer = new(Security, 'SecKeyRef *')\n\n cf_string = CFHelpers.cf_string_from_unicode(\"Temporary key from oscrypto python library - safe to delete\")\n\n # For some reason Apple decided that DSA keys were not a valid type of\n # key to be generated via SecKeyGeneratePair(), thus we have to use the\n # lower level, deprecated SecKeyCreatePair()\n if algorithm == 'dsa':\n sec_access_ref_pointer = new(Security, 'SecAccessRef *')\n result = Security.SecAccessCreate(cf_string, null(), sec_access_ref_pointer)\n sec_access_ref = unwrap(sec_access_ref_pointer)\n\n result = Security.SecKeyCreatePair(\n null(),\n SecurityConst.CSSM_ALGID_DSA,\n key_size,\n 0,\n SecurityConst.CSSM_KEYUSE_VERIFY,\n SecurityConst.CSSM_KEYATTR_EXTRACTABLE | SecurityConst.CSSM_KEYATTR_PERMANENT,\n SecurityConst.CSSM_KEYUSE_SIGN,\n SecurityConst.CSSM_KEYATTR_EXTRACTABLE | SecurityConst.CSSM_KEYATTR_PERMANENT,\n sec_access_ref,\n public_key_pointer,\n private_key_pointer\n )\n handle_sec_error(result)\n else:\n cf_dict = CFHelpers.cf_dictionary_from_pairs([\n (Security.kSecAttrKeyType, key_type),\n (Security.kSecAttrKeySizeInBits, CFHelpers.cf_number_from_integer(key_size)),\n (Security.kSecAttrLabel, cf_string)\n ])\n result = Security.SecKeyGeneratePair(cf_dict, public_key_pointer, private_key_pointer)\n handle_sec_error(result)\n\n public_key_ref = unwrap(public_key_pointer)\n private_key_ref = unwrap(private_key_pointer)\n\n cf_data_public_pointer = new(CoreFoundation, 'CFDataRef *')\n result = Security.SecItemExport(public_key_ref, 0, 0, null(), cf_data_public_pointer)\n handle_sec_error(result)\n cf_data_public = unwrap(cf_data_public_pointer)\n public_key_bytes = CFHelpers.cf_data_to_bytes(cf_data_public)\n\n cf_data_private_pointer = new(CoreFoundation, 'CFDataRef *')\n result = Security.SecItemExport(private_key_ref, 0, 0, null(), cf_data_private_pointer)\n handle_sec_error(result)\n cf_data_private = unwrap(cf_data_private_pointer)\n private_key_bytes = CFHelpers.cf_data_to_bytes(cf_data_private)\n\n # Clean the new keys out of the keychain\n result = Security.SecKeychainItemDelete(public_key_ref)\n handle_sec_error(result)\n result = Security.SecKeychainItemDelete(private_key_ref)\n handle_sec_error(result)\n\n finally:\n if cf_dict:\n CoreFoundation.CFRelease(cf_dict)\n if public_key_ref:\n CoreFoundation.CFRelease(public_key_ref)\n if private_key_ref:\n CoreFoundation.CFRelease(private_key_ref)\n if cf_data_public:\n CoreFoundation.CFRelease(cf_data_public)\n if cf_data_private:\n CoreFoundation.CFRelease(cf_data_private)\n if cf_string:\n CoreFoundation.CFRelease(cf_string)\n if sec_access_ref:\n CoreFoundation.CFRelease(sec_access_ref)\n\n return (load_public_key(public_key_bytes), load_private_key(private_key_bytes))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngenerates DH parameters for use with Diffie - Hellman key exchange.", "response": "def generate_dh_parameters(bit_size):\n \"\"\"\n Generates DH parameters for use with Diffie-Hellman key exchange. Returns\n a structure in the format of DHParameter defined in PKCS#3, which is also\n used by the OpenSSL dhparam tool.\n\n THIS CAN BE VERY TIME CONSUMING!\n\n :param bit_size:\n The integer bit size of the parameters to generate. Must be between 512\n and 4096, and divisible by 64. Recommended secure value as of early 2016\n is 2048, with an absolute minimum of 1024.\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n An asn1crypto.algos.DHParameters object. Use\n oscrypto.asymmetric.dump_dh_parameters() to save to disk for usage with\n web servers.\n \"\"\"\n\n if not isinstance(bit_size, int_types):\n raise TypeError(pretty_message(\n '''\n bit_size must be an integer, not %s\n ''',\n type_name(bit_size)\n ))\n\n if bit_size < 512:\n raise ValueError('bit_size must be greater than or equal to 512')\n\n if bit_size > 4096:\n raise ValueError('bit_size must be less than or equal to 4096')\n\n if bit_size % 64 != 0:\n raise ValueError('bit_size must be a multiple of 64')\n\n public_key_ref = None\n private_key_ref = None\n cf_data_public = None\n cf_data_private = None\n cf_string = None\n sec_access_ref = None\n\n try:\n public_key_pointer = new(Security, 'SecKeyRef *')\n private_key_pointer = new(Security, 'SecKeyRef *')\n\n cf_string = CFHelpers.cf_string_from_unicode(\"Temporary key from oscrypto python library - safe to delete\")\n\n sec_access_ref_pointer = new(Security, 'SecAccessRef *')\n result = Security.SecAccessCreate(cf_string, null(), sec_access_ref_pointer)\n sec_access_ref = unwrap(sec_access_ref_pointer)\n\n result = Security.SecKeyCreatePair(\n null(),\n SecurityConst.CSSM_ALGID_DH,\n bit_size,\n 0,\n 0,\n SecurityConst.CSSM_KEYATTR_EXTRACTABLE | SecurityConst.CSSM_KEYATTR_PERMANENT,\n 0,\n SecurityConst.CSSM_KEYATTR_EXTRACTABLE | SecurityConst.CSSM_KEYATTR_PERMANENT,\n sec_access_ref,\n public_key_pointer,\n private_key_pointer\n )\n handle_sec_error(result)\n\n public_key_ref = unwrap(public_key_pointer)\n private_key_ref = unwrap(private_key_pointer)\n\n cf_data_private_pointer = new(CoreFoundation, 'CFDataRef *')\n result = Security.SecItemExport(private_key_ref, 0, 0, null(), cf_data_private_pointer)\n handle_sec_error(result)\n cf_data_private = unwrap(cf_data_private_pointer)\n private_key_bytes = CFHelpers.cf_data_to_bytes(cf_data_private)\n\n # Clean the new keys out of the keychain\n result = Security.SecKeychainItemDelete(public_key_ref)\n handle_sec_error(result)\n\n result = Security.SecKeychainItemDelete(private_key_ref)\n handle_sec_error(result)\n\n return algos.KeyExchangeAlgorithm.load(private_key_bytes)['parameters']\n\n finally:\n if public_key_ref:\n CoreFoundation.CFRelease(public_key_ref)\n if private_key_ref:\n CoreFoundation.CFRelease(private_key_ref)\n if cf_data_public:\n CoreFoundation.CFRelease(cf_data_public)\n if cf_data_private:\n CoreFoundation.CFRelease(cf_data_private)\n if cf_string:\n CoreFoundation.CFRelease(cf_string)\n if sec_access_ref:\n CoreFoundation.CFRelease(sec_access_ref)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _load_x509(certificate):\n\n source = certificate.dump()\n\n cf_source = None\n try:\n cf_source = CFHelpers.cf_data_from_bytes(source)\n sec_key_ref = Security.SecCertificateCreateWithData(CoreFoundation.kCFAllocatorDefault, cf_source)\n return Certificate(sec_key_ref, certificate)\n\n finally:\n if cf_source:\n CoreFoundation.CFRelease(cf_source)", "response": "Loads an ASN. 1 object of an x509 certificate into a Certificate object."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nloading public and private keys into a set of PublicKey and PrivateKey objects.", "response": "def _load_key(key_object):\n \"\"\"\n Common code to load public and private keys into PublicKey and PrivateKey\n objects\n\n :param key_object:\n An asn1crypto.keys.PublicKeyInfo or asn1crypto.keys.PrivateKeyInfo\n object\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n oscrypto.errors.AsymmetricKeyError - when the key is incompatible with the OS crypto library\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A PublicKey or PrivateKey object\n \"\"\"\n\n if key_object.algorithm == 'ec':\n curve_type, details = key_object.curve\n if curve_type != 'named':\n raise AsymmetricKeyError('OS X only supports EC keys using named curves')\n if details not in set(['secp256r1', 'secp384r1', 'secp521r1']):\n raise AsymmetricKeyError(pretty_message(\n '''\n OS X only supports EC keys using the named curves secp256r1,\n secp384r1 and secp521r1\n '''\n ))\n\n elif key_object.algorithm == 'dsa' and key_object.hash_algo == 'sha2':\n raise AsymmetricKeyError(pretty_message(\n '''\n OS X only supports DSA keys based on SHA1 (2048 bits or less) - this\n key is based on SHA2 and is %s bits\n ''',\n key_object.bit_size\n ))\n\n elif key_object.algorithm == 'dsa' and key_object.hash_algo is None:\n raise IncompleteAsymmetricKeyError(pretty_message(\n '''\n The DSA key does not contain the necessary p, q and g parameters\n and can not be used\n '''\n ))\n\n if isinstance(key_object, keys.PublicKeyInfo):\n source = key_object.dump()\n key_class = Security.kSecAttrKeyClassPublic\n else:\n source = key_object.unwrap().dump()\n key_class = Security.kSecAttrKeyClassPrivate\n\n cf_source = None\n cf_dict = None\n cf_output = None\n\n try:\n cf_source = CFHelpers.cf_data_from_bytes(source)\n key_type = {\n 'dsa': Security.kSecAttrKeyTypeDSA,\n 'ec': Security.kSecAttrKeyTypeECDSA,\n 'rsa': Security.kSecAttrKeyTypeRSA,\n }[key_object.algorithm]\n cf_dict = CFHelpers.cf_dictionary_from_pairs([\n (Security.kSecAttrKeyType, key_type),\n (Security.kSecAttrKeyClass, key_class),\n (Security.kSecAttrCanSign, CoreFoundation.kCFBooleanTrue),\n (Security.kSecAttrCanVerify, CoreFoundation.kCFBooleanTrue),\n ])\n error_pointer = new(CoreFoundation, 'CFErrorRef *')\n sec_key_ref = Security.SecKeyCreateFromData(cf_dict, cf_source, error_pointer)\n handle_cf_error(error_pointer)\n\n if key_class == Security.kSecAttrKeyClassPublic:\n return PublicKey(sec_key_ref, key_object)\n\n if key_class == Security.kSecAttrKeyClassPrivate:\n return PrivateKey(sec_key_ref, key_object)\n\n finally:\n if cf_source:\n CoreFoundation.CFRelease(cf_source)\n if cf_dict:\n CoreFoundation.CFRelease(cf_dict)\n if cf_output:\n CoreFoundation.CFRelease(cf_output)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nencrypt a byte string using an RSA public key or certificate. Uses PKCS#1. 5 padding.", "response": "def rsa_pkcs1v15_encrypt(certificate_or_public_key, data):\n \"\"\"\n Encrypts a byte string using an RSA public key or certificate. Uses PKCS#1\n v1.5 padding.\n\n :param certificate_or_public_key:\n A PublicKey or Certificate object\n\n :param data:\n A byte string, with a maximum length 11 bytes less than the key length\n (in bytes)\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the encrypted data\n \"\"\"\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n key_length = certificate_or_public_key.byte_size\n buffer = buffer_from_bytes(key_length)\n output_length = new(Security, 'size_t *', key_length)\n result = Security.SecKeyEncrypt(\n certificate_or_public_key.sec_key_ref,\n SecurityConst.kSecPaddingPKCS1,\n data,\n len(data),\n buffer,\n output_length\n )\n handle_sec_error(result)\n\n return bytes_from_buffer(buffer, deref(output_length))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef rsa_pkcs1v15_decrypt(private_key, ciphertext):\n\n if not isinstance(private_key, PrivateKey):\n raise TypeError(pretty_message(\n '''\n private_key must an instance of the PrivateKey class, not %s\n ''',\n type_name(private_key)\n ))\n\n if not isinstance(ciphertext, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(ciphertext)\n ))\n\n key_length = private_key.byte_size\n buffer = buffer_from_bytes(key_length)\n output_length = new(Security, 'size_t *', key_length)\n\n if osx_version_info < (10, 8):\n padding = SecurityConst.kSecPaddingNone\n else:\n padding = SecurityConst.kSecPaddingPKCS1\n\n result = Security.SecKeyDecrypt(\n private_key.sec_key_ref,\n padding,\n ciphertext,\n len(ciphertext),\n buffer,\n output_length\n )\n handle_sec_error(result)\n\n output = bytes_from_buffer(buffer, deref(output_length))\n\n if osx_version_info < (10, 8):\n output = remove_pkcs1v15_encryption_padding(key_length, output)\n\n return output", "response": "Decrypts a byte string using an RSA private key. Uses PKCS1 v1. 5 padding."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _encrypt(certificate_or_public_key, data, padding):\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if not padding:\n raise ValueError('padding must be specified')\n\n cf_data = None\n sec_transform = None\n\n try:\n cf_data = CFHelpers.cf_data_from_bytes(data)\n\n error_pointer = new(CoreFoundation, 'CFErrorRef *')\n sec_transform = Security.SecEncryptTransformCreate(\n certificate_or_public_key.sec_key_ref,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n if padding:\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecPaddingKey,\n padding,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecTransformInputAttributeName,\n cf_data,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n ciphertext = Security.SecTransformExecute(sec_transform, error_pointer)\n handle_cf_error(error_pointer)\n\n return CFHelpers.cf_data_to_bytes(ciphertext)\n\n finally:\n if cf_data:\n CoreFoundation.CFRelease(cf_data)\n if sec_transform:\n CoreFoundation.CFRelease(sec_transform)", "response": "Encrypts the data using a RSA public key or a certificate."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef rsa_pss_verify(certificate_or_public_key, signature, data, hash_algorithm):\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if certificate_or_public_key.algorithm != 'rsa':\n raise ValueError('The key specified is not an RSA public key')\n\n hash_length = {\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }.get(hash_algorithm, 0)\n\n key_length = certificate_or_public_key.byte_size\n buffer = buffer_from_bytes(key_length)\n output_length = new(Security, 'size_t *', key_length)\n result = Security.SecKeyEncrypt(\n certificate_or_public_key.sec_key_ref,\n SecurityConst.kSecPaddingNone,\n signature,\n len(signature),\n buffer,\n output_length\n )\n handle_sec_error(result)\n\n plaintext = bytes_from_buffer(buffer, deref(output_length))\n if not verify_pss_padding(hash_algorithm, hash_length, certificate_or_public_key.bit_size, data, plaintext):\n raise SignatureError('Signature is invalid')", "response": "Verifies an RSA - PSS signature."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nverifying a RSA DSA or ECDSA signature with the specified certificate or public key.", "response": "def _verify(certificate_or_public_key, signature, data, hash_algorithm):\n \"\"\"\n Verifies an RSA, DSA or ECDSA signature\n\n :param certificate_or_public_key:\n A Certificate or PublicKey instance to verify the signature with\n\n :param signature:\n A byte string of the signature to verify\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\" or \"sha512\"\n\n :raises:\n oscrypto.errors.SignatureError - when the signature is determined to be invalid\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n \"\"\"\n\n if not isinstance(certificate_or_public_key, (Certificate, PublicKey)):\n raise TypeError(pretty_message(\n '''\n certificate_or_public_key must be an instance of the Certificate or\n PublicKey class, not %s\n ''',\n type_name(certificate_or_public_key)\n ))\n\n if not isinstance(signature, byte_cls):\n raise TypeError(pretty_message(\n '''\n signature must be a byte string, not %s\n ''',\n type_name(signature)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n valid_hash_algorithms = set(['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'])\n if certificate_or_public_key.algorithm == 'rsa':\n valid_hash_algorithms |= set(['raw'])\n\n if hash_algorithm not in valid_hash_algorithms:\n valid_hash_algorithms_error = '\"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\", \"sha512\"'\n if certificate_or_public_key.algorithm == 'rsa':\n valid_hash_algorithms_error += ', \"raw\"'\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of %s, not %s\n ''',\n valid_hash_algorithms_error,\n repr(hash_algorithm)\n ))\n\n if certificate_or_public_key.algorithm == 'rsa' and hash_algorithm == 'raw':\n if len(data) > certificate_or_public_key.byte_size - 11:\n raise ValueError(pretty_message(\n '''\n data must be 11 bytes shorter than the key size when\n hash_algorithm is \"raw\" - key size is %s bytes, but data\n is %s bytes long\n ''',\n certificate_or_public_key.byte_size,\n len(data)\n ))\n\n result = Security.SecKeyRawVerify(\n certificate_or_public_key.sec_key_ref,\n SecurityConst.kSecPaddingPKCS1,\n data,\n len(data),\n signature,\n len(signature)\n )\n # errSSLCrypto is returned in some situations on macOS 10.12\n if result == SecurityConst.errSecVerifyFailed or result == SecurityConst.errSSLCrypto:\n raise SignatureError('Signature is invalid')\n handle_sec_error(result)\n return\n\n cf_signature = None\n cf_data = None\n cf_hash_length = None\n sec_transform = None\n\n try:\n error_pointer = new(CoreFoundation, 'CFErrorRef *')\n cf_signature = CFHelpers.cf_data_from_bytes(signature)\n sec_transform = Security.SecVerifyTransformCreate(\n certificate_or_public_key.sec_key_ref,\n cf_signature,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n hash_constant = {\n 'md5': Security.kSecDigestMD5,\n 'sha1': Security.kSecDigestSHA1,\n 'sha224': Security.kSecDigestSHA2,\n 'sha256': Security.kSecDigestSHA2,\n 'sha384': Security.kSecDigestSHA2,\n 'sha512': Security.kSecDigestSHA2\n }[hash_algorithm]\n\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecDigestTypeAttribute,\n hash_constant,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n if hash_algorithm in set(['sha224', 'sha256', 'sha384', 'sha512']):\n hash_length = {\n 'sha224': 224,\n 'sha256': 256,\n 'sha384': 384,\n 'sha512': 512\n }[hash_algorithm]\n\n cf_hash_length = CFHelpers.cf_number_from_integer(hash_length)\n\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecDigestLengthAttribute,\n cf_hash_length,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n if certificate_or_public_key.algorithm == 'rsa':\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecPaddingKey,\n Security.kSecPaddingPKCS1Key,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n cf_data = CFHelpers.cf_data_from_bytes(data)\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecTransformInputAttributeName,\n cf_data,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n res = Security.SecTransformExecute(sec_transform, error_pointer)\n if not is_null(error_pointer):\n error = unwrap(error_pointer)\n if not is_null(error):\n raise SignatureError('Signature is invalid')\n\n res = bool(CoreFoundation.CFBooleanGetValue(res))\n\n if not res:\n raise SignatureError('Signature is invalid')\n\n finally:\n if sec_transform:\n CoreFoundation.CFRelease(sec_transform)\n if cf_signature:\n CoreFoundation.CFRelease(cf_signature)\n if cf_data:\n CoreFoundation.CFRelease(cf_data)\n if cf_hash_length:\n CoreFoundation.CFRelease(cf_hash_length)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates an RSASSA - PSS signature for the data.", "response": "def rsa_pss_sign(private_key, data, hash_algorithm):\n \"\"\"\n Generates an RSASSA-PSS signature. For the PSS padding the mask gen\n algorithm will be mgf1 using the same hash algorithm as the signature. The\n salt length with be the length of the hash algorithm, and the trailer field\n with be the standard 0xBC byte.\n\n :param private_key:\n The PrivateKey to generate the signature with\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\" or\n \"sha512\"\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the signature\n \"\"\"\n\n if not isinstance(private_key, PrivateKey):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of the PrivateKey class, not %s\n ''',\n type_name(private_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if private_key.algorithm != 'rsa':\n raise ValueError('The key specified is not an RSA private key')\n\n hash_length = {\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }.get(hash_algorithm, 0)\n\n encoded_data = add_pss_padding(hash_algorithm, hash_length, private_key.bit_size, data)\n\n key_length = private_key.byte_size\n buffer = buffer_from_bytes(key_length)\n output_length = new(Security, 'size_t *', key_length)\n result = Security.SecKeyDecrypt(\n private_key.sec_key_ref,\n SecurityConst.kSecPaddingNone,\n encoded_data,\n len(encoded_data),\n buffer,\n output_length\n )\n handle_sec_error(result)\n\n return bytes_from_buffer(buffer, deref(output_length))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating a RSA DSA or ECDSA signature for the given data and hash_algorithm.", "response": "def _sign(private_key, data, hash_algorithm):\n \"\"\"\n Generates an RSA, DSA or ECDSA signature\n\n :param private_key:\n The PrivateKey to generate the signature with\n\n :param data:\n A byte string of the data the signature is for\n\n :param hash_algorithm:\n A unicode string of \"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\" or\n \"sha512\"\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the signature\n \"\"\"\n\n if not isinstance(private_key, PrivateKey):\n raise TypeError(pretty_message(\n '''\n private_key must be an instance of PrivateKey, not %s\n ''',\n type_name(private_key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n valid_hash_algorithms = set(['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512'])\n if private_key.algorithm == 'rsa':\n valid_hash_algorithms |= set(['raw'])\n\n if hash_algorithm not in valid_hash_algorithms:\n valid_hash_algorithms_error = '\"md5\", \"sha1\", \"sha224\", \"sha256\", \"sha384\", \"sha512\"'\n if private_key.algorithm == 'rsa':\n valid_hash_algorithms_error += ', \"raw\"'\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of %s, not %s\n ''',\n valid_hash_algorithms_error,\n repr(hash_algorithm)\n ))\n\n if private_key.algorithm == 'rsa' and hash_algorithm == 'raw':\n if len(data) > private_key.byte_size - 11:\n raise ValueError(pretty_message(\n '''\n data must be 11 bytes shorter than the key size when\n hash_algorithm is \"raw\" - key size is %s bytes, but\n data is %s bytes long\n ''',\n private_key.byte_size,\n len(data)\n ))\n\n key_length = private_key.byte_size\n buffer = buffer_from_bytes(key_length)\n output_length = new(Security, 'size_t *', key_length)\n result = Security.SecKeyRawSign(\n private_key.sec_key_ref,\n SecurityConst.kSecPaddingPKCS1,\n data,\n len(data),\n buffer,\n output_length\n )\n handle_sec_error(result)\n\n return bytes_from_buffer(buffer, deref(output_length))\n\n cf_signature = None\n cf_data = None\n cf_hash_length = None\n sec_transform = None\n\n try:\n error_pointer = new(CoreFoundation, 'CFErrorRef *')\n sec_transform = Security.SecSignTransformCreate(private_key.sec_key_ref, error_pointer)\n handle_cf_error(error_pointer)\n\n hash_constant = {\n 'md5': Security.kSecDigestMD5,\n 'sha1': Security.kSecDigestSHA1,\n 'sha224': Security.kSecDigestSHA2,\n 'sha256': Security.kSecDigestSHA2,\n 'sha384': Security.kSecDigestSHA2,\n 'sha512': Security.kSecDigestSHA2\n }[hash_algorithm]\n\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecDigestTypeAttribute,\n hash_constant,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n if hash_algorithm in set(['sha224', 'sha256', 'sha384', 'sha512']):\n hash_length = {\n 'sha224': 224,\n 'sha256': 256,\n 'sha384': 384,\n 'sha512': 512\n }[hash_algorithm]\n\n cf_hash_length = CFHelpers.cf_number_from_integer(hash_length)\n\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecDigestLengthAttribute,\n cf_hash_length,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n if private_key.algorithm == 'rsa':\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecPaddingKey,\n Security.kSecPaddingPKCS1Key,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n cf_data = CFHelpers.cf_data_from_bytes(data)\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecTransformInputAttributeName,\n cf_data,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n cf_signature = Security.SecTransformExecute(sec_transform, error_pointer)\n handle_cf_error(error_pointer)\n\n return CFHelpers.cf_data_to_bytes(cf_signature)\n\n finally:\n if sec_transform:\n CoreFoundation.CFRelease(sec_transform)\n if cf_signature:\n CoreFoundation.CFRelease(cf_signature)\n if cf_data:\n CoreFoundation.CFRelease(cf_data)\n if cf_hash_length:\n CoreFoundation.CFRelease(cf_hash_length)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef public_key(self):\n\n if not self._public_key and self.sec_certificate_ref:\n sec_public_key_ref_pointer = new(Security, 'SecKeyRef *')\n res = Security.SecCertificateCopyPublicKey(self.sec_certificate_ref, sec_public_key_ref_pointer)\n handle_sec_error(res)\n sec_public_key_ref = unwrap(sec_public_key_ref_pointer)\n self._public_key = PublicKey(sec_public_key_ref, self.asn1['tbs_certificate']['subject_public_key_info'])\n\n return self._public_key", "response": "Returns the PublicKey object for the public key this certificate contains."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef backend():\n\n if _module_values['backend'] is not None:\n return _module_values['backend']\n\n with _backend_lock:\n if _module_values['backend'] is not None:\n return _module_values['backend']\n\n if sys.platform == 'win32':\n # Windows XP was major version 5, Vista was 6\n if sys.getwindowsversion()[0] < 6:\n _module_values['backend'] = 'winlegacy'\n else:\n _module_values['backend'] = 'win'\n elif sys.platform == 'darwin':\n _module_values['backend'] = 'osx'\n else:\n _module_values['backend'] = 'openssl'\n\n return _module_values['backend']", "response": "A unicode string of the backend being used."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nconfiguring the oscrypto to use OpenSSL dynamic libraries.", "response": "def use_openssl(libcrypto_path, libssl_path, trust_list_path=None):\n \"\"\"\n Forces using OpenSSL dynamic libraries on OS X (.dylib) or Windows (.dll),\n or using a specific dynamic library on Linux/BSD (.so).\n\n This can also be used to configure oscrypto to use LibreSSL dynamic\n libraries.\n\n This method must be called before any oscrypto submodules are imported.\n\n :param libcrypto_path:\n A unicode string of the file path to the OpenSSL/LibreSSL libcrypto\n dynamic library.\n\n :param libssl_path:\n A unicode string of the file path to the OpenSSL/LibreSSL libssl\n dynamic library.\n\n :param trust_list_path:\n An optional unicode string of the path to a file containing\n OpenSSL-compatible CA certificates in PEM format. If this is not\n provided and the platform is OS X or Windows, the system trust roots\n will be exported from the OS and used for all TLS connections.\n\n :raises:\n ValueError - when one of the paths is not a unicode string\n OSError - when the trust_list_path does not exist on the filesystem\n oscrypto.errors.LibraryNotFoundError - when one of the path does not exist on the filesystem\n RuntimeError - when this function is called after another part of oscrypto has been imported\n \"\"\"\n\n if not isinstance(libcrypto_path, str_cls):\n raise ValueError('libcrypto_path must be a unicode string, not %s' % type_name(libcrypto_path))\n\n if not isinstance(libssl_path, str_cls):\n raise ValueError('libssl_path must be a unicode string, not %s' % type_name(libssl_path))\n\n if not os.path.exists(libcrypto_path):\n raise LibraryNotFoundError('libcrypto does not exist at %s' % libcrypto_path)\n\n if not os.path.exists(libssl_path):\n raise LibraryNotFoundError('libssl does not exist at %s' % libssl_path)\n\n if trust_list_path is not None:\n if not isinstance(trust_list_path, str_cls):\n raise ValueError('trust_list_path must be a unicode string, not %s' % type_name(trust_list_path))\n if not os.path.exists(trust_list_path):\n raise OSError('trust_list_path does not exist at %s' % trust_list_path)\n\n with _backend_lock:\n if _module_values['backend'] is not None:\n raise RuntimeError('Another part of oscrypto has already been imported, unable to force use of OpenSSL')\n\n _module_values['backend'] = 'openssl'\n _module_values['backend_config'] = {\n 'libcrypto_path': libcrypto_path,\n 'libssl_path': libssl_path,\n 'trust_list_path': trust_list_path,\n }"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef use_winlegacy():\n\n if sys.platform != 'win32':\n plat = platform.system() or sys.platform\n if plat == 'Darwin':\n plat = 'OS X'\n raise EnvironmentError('The winlegacy backend can only be used on Windows, not %s' % plat)\n\n with _backend_lock:\n if _module_values['backend'] is not None:\n raise RuntimeError(\n 'Another part of oscrypto has already been imported, unable to force use of Windows legacy CryptoAPI'\n )\n _module_values['backend'] = 'winlegacy'", "response": "Force use of the Windows legacy CryptoAPI."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef pkcs12_kdf(hash_algorithm, password, salt, iterations, key_length, id_):\n\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n type_name(password)\n ))\n\n if not isinstance(salt, byte_cls):\n raise TypeError(pretty_message(\n '''\n salt must be a byte string, not %s\n ''',\n type_name(salt)\n ))\n\n if not isinstance(iterations, int_types):\n raise TypeError(pretty_message(\n '''\n iterations must be an integer, not %s\n ''',\n type_name(iterations)\n ))\n\n if iterations < 1:\n raise ValueError(pretty_message(\n '''\n iterations must be greater than 0 - is %s\n ''',\n repr(iterations)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 1:\n raise ValueError(pretty_message(\n '''\n key_length must be greater than 0 - is %s\n ''',\n repr(key_length)\n ))\n\n if hash_algorithm not in set(['md5', 'sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"md5\", \"sha1\", \"sha224\", \"sha256\",\n \"sha384\", \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n if id_ not in set([1, 2, 3]):\n raise ValueError(pretty_message(\n '''\n id_ must be one of 1, 2, 3, not %s\n ''',\n repr(id_)\n ))\n\n utf16_password = password.decode('utf-8').encode('utf-16be') + b'\\x00\\x00'\n\n algo = getattr(hashlib, hash_algorithm)\n\n # u and v values are bytes (not bits as in the RFC)\n u = {\n 'md5': 16,\n 'sha1': 20,\n 'sha224': 28,\n 'sha256': 32,\n 'sha384': 48,\n 'sha512': 64\n }[hash_algorithm]\n\n if hash_algorithm in ['sha384', 'sha512']:\n v = 128\n else:\n v = 64\n\n # Step 1\n d = chr_cls(id_) * v\n\n # Step 2\n s = b''\n if salt != b'':\n s_len = v * int(math.ceil(float(len(salt)) / v))\n while len(s) < s_len:\n s += salt\n s = s[0:s_len]\n\n # Step 3\n p = b''\n if utf16_password != b'':\n p_len = v * int(math.ceil(float(len(utf16_password)) / v))\n while len(p) < p_len:\n p += utf16_password\n p = p[0:p_len]\n\n # Step 4\n i = s + p\n\n # Step 5\n c = int(math.ceil(float(key_length) / u))\n\n a = b'\\x00' * (c * u)\n\n for num in range(1, c + 1):\n # Step 6A\n a2 = algo(d + i).digest()\n for _ in range(2, iterations + 1):\n a2 = algo(a2).digest()\n\n if num < c:\n # Step 6B\n b = b''\n while len(b) < v:\n b += a2\n\n b = int_from_bytes(b[0:v]) + 1\n\n # Step 6C\n for num2 in range(0, len(i) // v):\n start = num2 * v\n end = (num2 + 1) * v\n i_num2 = i[start:end]\n\n i_num2 = int_to_bytes(int_from_bytes(i_num2) + b)\n\n # Ensure the new slice is the right size\n i_num2_l = len(i_num2)\n if i_num2_l > v:\n i_num2 = i_num2[i_num2_l - v:]\n\n i = i[0:start] + i_num2 + i[end:]\n\n # Step 7 (one peice at a time)\n begin = (num - 1) * u\n to_copy = min(key_length, u)\n a = a[0:begin] + a2[0:to_copy] + a[begin + to_copy:]\n\n return a[0:key_length]", "response": "Derives a key from the key in a single pkcs12 file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pbkdf2_iteration_calculator(hash_algorithm, key_length, target_ms=100, quiet=False):\n\n if hash_algorithm not in set(['sha1', 'sha224', 'sha256', 'sha384', 'sha512']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"sha1\", \"sha224\", \"sha256\", \"sha384\",\n \"sha512\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n type_name(key_length)\n ))\n\n if key_length < 1:\n raise ValueError(pretty_message(\n '''\n key_length must be greater than 0 - is %s\n ''',\n repr(key_length)\n ))\n\n if not isinstance(target_ms, int_types):\n raise TypeError(pretty_message(\n '''\n target_ms must be an integer, not %s\n ''',\n type_name(target_ms)\n ))\n\n if target_ms < 1:\n raise ValueError(pretty_message(\n '''\n target_ms must be greater than 0 - is %s\n ''',\n repr(target_ms)\n ))\n\n if pbkdf2.pure_python:\n raise OSError(pretty_message(\n '''\n Only a very slow, pure-python version of PBKDF2 is available,\n making this function useless\n '''\n ))\n\n iterations = 10000\n password = 'this is a test'.encode('utf-8')\n salt = rand_bytes(key_length)\n\n def _measure():\n start = _get_start()\n pbkdf2(hash_algorithm, password, salt, iterations, key_length)\n observed_ms = _get_elapsed(start)\n if not quiet:\n print('%s iterations in %sms' % (iterations, observed_ms))\n return 1.0 / target_ms * observed_ms\n\n # Measure the initial guess, then estimate how many iterations it would\n # take to reach 1/2 of the target ms and try it to get a good final number\n fraction = _measure()\n iterations = int(iterations / fraction / 2.0)\n\n fraction = _measure()\n iterations = iterations / fraction\n\n # < 20,000 round to 1000\n # 20,000-100,000 round to 5,000\n # > 100,000 round to 10,000\n round_factor = -3 if iterations < 100000 else -4\n result = int(round(iterations, round_factor))\n if result > 20000:\n result = (result // 5000) * 5000\n return result", "response": "This function calculates the approximate number of iterations to use for a given hash algorithm and length."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pbkdf1(hash_algorithm, password, salt, iterations, key_length):\n\n if not isinstance(password, byte_cls):\n raise TypeError(pretty_message(\n '''\n password must be a byte string, not %s\n ''',\n (type_name(password))\n ))\n\n if not isinstance(salt, byte_cls):\n raise TypeError(pretty_message(\n '''\n salt must be a byte string, not %s\n ''',\n (type_name(salt))\n ))\n\n if not isinstance(iterations, int_types):\n raise TypeError(pretty_message(\n '''\n iterations must be an integer, not %s\n ''',\n (type_name(iterations))\n ))\n\n if iterations < 1:\n raise ValueError(pretty_message(\n '''\n iterations must be greater than 0 - is %s\n ''',\n repr(iterations)\n ))\n\n if not isinstance(key_length, int_types):\n raise TypeError(pretty_message(\n '''\n key_length must be an integer, not %s\n ''',\n (type_name(key_length))\n ))\n\n if key_length < 1:\n raise ValueError(pretty_message(\n '''\n key_length must be greater than 0 - is %s\n ''',\n repr(key_length)\n ))\n\n if hash_algorithm not in set(['md2', 'md5', 'sha1']):\n raise ValueError(pretty_message(\n '''\n hash_algorithm must be one of \"md2\", \"md5\", \"sha1\", not %s\n ''',\n repr(hash_algorithm)\n ))\n\n if key_length > 16 and hash_algorithm in set(['md2', 'md5']):\n raise ValueError(pretty_message(\n '''\n key_length can not be longer than 16 for %s - is %s\n ''',\n (hash_algorithm, repr(key_length))\n ))\n\n if key_length > 20 and hash_algorithm == 'sha1':\n raise ValueError(pretty_message(\n '''\n key_length can not be longer than 20 for sha1 - is %s\n ''',\n repr(key_length)\n ))\n\n algo = getattr(hashlib, hash_algorithm)\n output = algo(password + salt).digest()\n for _ in range(2, iterations + 1):\n output = algo(output).digest()\n\n return output[:key_length]", "response": "Derives a key from a set of keys."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nhandle the Windows error message returned by get_error and returns the error message.", "response": "def handle_error(result):\n \"\"\"\n Extracts the last Windows error message into a python unicode string\n\n :param result:\n A function result, 0 or None indicates failure\n\n :return:\n A unicode string error message\n \"\"\"\n\n if result:\n return\n\n code, error_string = get_error()\n\n if code == Advapi32Const.NTE_BAD_SIGNATURE:\n raise SignatureError('Signature is invalid')\n\n if not isinstance(error_string, str_cls):\n error_string = _try_decode(error_string)\n\n raise OSError(error_string)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef aes_cbc_no_padding_encrypt(key, data, iv):\n\n if len(key) not in [16, 24, 32]:\n raise ValueError(pretty_message(\n '''\n key must be either 16, 24 or 32 bytes (128, 192 or 256 bits) long - is %s\n ''',\n len(key)\n ))\n\n if not iv:\n iv = rand_bytes(16)\n elif len(iv) != 16:\n raise ValueError(pretty_message(\n '''\n iv must be 16 bytes long - is %s\n ''',\n len(iv)\n ))\n\n if len(data) % 16 != 0:\n raise ValueError(pretty_message(\n '''\n data must be a multiple of 16 bytes long - is %s\n ''',\n len(data)\n ))\n\n return (iv, _encrypt(Security.kSecAttrKeyTypeAES, key, data, iv, Security.kSecPaddingNoneKey))", "response": "Encrypts plaintext using AES with a 128 192 or 256 bit key and no padding."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef aes_cbc_no_padding_decrypt(key, data, iv):\n\n if len(key) not in [16, 24, 32]:\n raise ValueError(pretty_message(\n '''\n key must be either 16, 24 or 32 bytes (128, 192 or 256 bits) long - is %s\n ''',\n len(key)\n ))\n\n if len(iv) != 16:\n raise ValueError(pretty_message(\n '''\n iv must be 16 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return _decrypt(Security.kSecAttrKeyTypeAES, key, data, iv, Security.kSecPaddingNoneKey)", "response": "Decrypts a ciphertext in CBC mode using a 128 192 or 256 bit key and no padding."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef aes_cbc_pkcs7_encrypt(key, data, iv):\n\n if len(key) not in [16, 24, 32]:\n raise ValueError(pretty_message(\n '''\n key must be either 16, 24 or 32 bytes (128, 192 or 256 bits) long - is %s\n ''',\n len(key)\n ))\n\n if not iv:\n iv = rand_bytes(16)\n elif len(iv) != 16:\n raise ValueError(pretty_message(\n '''\n iv must be 16 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return (iv, _encrypt(Security.kSecAttrKeyTypeAES, key, data, iv, Security.kSecPaddingPKCS7Key))", "response": "Encrypts the plaintext using AES with a 128 192 or 256 bit key and PKCS#7 padding."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nencrypting plaintext using RC4 with a 40 - 128 bit key", "response": "def rc4_encrypt(key, data):\n \"\"\"\n Encrypts plaintext using RC4 with a 40-128 bit key\n\n :param key:\n The encryption key - a byte string 5-16 bytes long\n\n :param data:\n The plaintext - a byte string\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the ciphertext\n \"\"\"\n\n if len(key) < 5 or len(key) > 16:\n raise ValueError(pretty_message(\n '''\n key must be 5 to 16 bytes (40 to 128 bits) long - is %s\n ''',\n len(key)\n ))\n\n return _encrypt(Security.kSecAttrKeyTypeRC4, key, data, None, None)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef rc2_cbc_pkcs5_encrypt(key, data, iv):\n\n if len(key) < 5 or len(key) > 16:\n raise ValueError(pretty_message(\n '''\n key must be 5 to 16 bytes (40 to 128 bits) long - is %s\n ''',\n len(key)\n ))\n\n if not iv:\n iv = rand_bytes(8)\n elif len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return (iv, _encrypt(Security.kSecAttrKeyTypeRC2, key, data, iv, Security.kSecPaddingPKCS5Key))", "response": "Encrypts plaintext using RC2 with a 64 bit key and returns the ciphertext."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef rc2_cbc_pkcs5_decrypt(key, data, iv):\n\n if len(key) < 5 or len(key) > 16:\n raise ValueError(pretty_message(\n '''\n key must be 5 to 16 bytes (40 to 128 bits) long - is %s\n ''',\n len(key)\n ))\n\n if len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return _decrypt(Security.kSecAttrKeyTypeRC2, key, data, iv, Security.kSecPaddingPKCS5Key)", "response": "Decrypts RC2 ciphertext using a 64 bit key."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nencrypting plaintext using 3DES using the specified key and data and iv.", "response": "def tripledes_cbc_pkcs5_encrypt(key, data, iv):\n \"\"\"\n Encrypts plaintext using 3DES in either 2 or 3 key mode\n\n :param key:\n The encryption key - a byte string 16 or 24 bytes long (2 or 3 key mode)\n\n :param data:\n The plaintext - a byte string\n\n :param iv:\n The 8-byte initialization vector to use - a byte string - set as None\n to generate an appropriate one\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A tuple of two byte strings (iv, ciphertext)\n \"\"\"\n\n if len(key) != 16 and len(key) != 24:\n raise ValueError(pretty_message(\n '''\n key must be 16 bytes (2 key) or 24 bytes (3 key) long - %s\n ''',\n len(key)\n ))\n\n if not iv:\n iv = rand_bytes(8)\n elif len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - %s\n ''',\n len(iv)\n ))\n\n # Expand 2-key to actual 24 byte byte string used by cipher\n if len(key) == 16:\n key = key + key[0:8]\n\n return (iv, _encrypt(Security.kSecAttrKeyType3DES, key, data, iv, Security.kSecPaddingPKCS5Key))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tripledes_cbc_pkcs5_decrypt(key, data, iv):\n\n if len(key) != 16 and len(key) != 24:\n raise ValueError(pretty_message(\n '''\n key must be 16 bytes (2 key) or 24 bytes (3 key) long - is %s\n ''',\n len(key)\n ))\n\n if len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n # Expand 2-key to actual 24 byte byte string used by cipher\n if len(key) == 16:\n key = key + key[0:8]\n\n return _decrypt(Security.kSecAttrKeyType3DES, key, data, iv, Security.kSecPaddingPKCS5Key)", "response": "Decrypts 3DES ciphertext in either 2 or 3 key mode and returns the ciphertext in either 2 or 3 key mode"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nencrypts a plaintext using DES with a 56 bit key and returns the ciphertext.", "response": "def des_cbc_pkcs5_encrypt(key, data, iv):\n \"\"\"\n Encrypts plaintext using DES with a 56 bit key\n\n :param key:\n The encryption key - a byte string 8 bytes long (includes error correction bits)\n\n :param data:\n The plaintext - a byte string\n\n :param iv:\n The 8-byte initialization vector to use - a byte string - set as None\n to generate an appropriate one\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A tuple of two byte strings (iv, ciphertext)\n \"\"\"\n\n if len(key) != 8:\n raise ValueError(pretty_message(\n '''\n key must be 8 bytes (56 bits + 8 parity bits) long - is %s\n ''',\n len(key)\n ))\n\n if not iv:\n iv = rand_bytes(8)\n elif len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return (iv, _encrypt(Security.kSecAttrKeyTypeDES, key, data, iv, Security.kSecPaddingPKCS5Key))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef des_cbc_pkcs5_decrypt(key, data, iv):\n\n if len(key) != 8:\n raise ValueError(pretty_message(\n '''\n key must be 8 bytes (56 bits + 8 parity bits) long - is %s\n ''',\n len(key)\n ))\n\n if len(iv) != 8:\n raise ValueError(pretty_message(\n '''\n iv must be 8 bytes long - is %s\n ''',\n len(iv)\n ))\n\n return _decrypt(Security.kSecAttrKeyTypeDES, key, data, iv, Security.kSecPaddingPKCS5Key)", "response": "Decrypts DES ciphertext using a 56 bit key"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nencrypting a plaintext object with the specified cipher key and data and initialization vector.", "response": "def _encrypt(cipher, key, data, iv, padding):\n \"\"\"\n Encrypts plaintext\n\n :param cipher:\n A kSecAttrKeyType* value that specifies the cipher to use\n\n :param key:\n The encryption key - a byte string 5-16 bytes long\n\n :param data:\n The plaintext - a byte string\n\n :param iv:\n The initialization vector - a byte string - unused for RC4\n\n :param padding:\n The padding mode to use, specified as a kSecPadding*Key value - unused for RC4\n\n :raises:\n ValueError - when any of the parameters contain an invalid value\n TypeError - when any of the parameters are of the wrong type\n OSError - when an error is returned by the OS crypto library\n\n :return:\n A byte string of the ciphertext\n \"\"\"\n\n if not isinstance(key, byte_cls):\n raise TypeError(pretty_message(\n '''\n key must be a byte string, not %s\n ''',\n type_name(key)\n ))\n\n if not isinstance(data, byte_cls):\n raise TypeError(pretty_message(\n '''\n data must be a byte string, not %s\n ''',\n type_name(data)\n ))\n\n if cipher != Security.kSecAttrKeyTypeRC4 and not isinstance(iv, byte_cls):\n raise TypeError(pretty_message(\n '''\n iv must be a byte string, not %s\n ''',\n type_name(iv)\n ))\n\n if cipher != Security.kSecAttrKeyTypeRC4 and not padding:\n raise ValueError('padding must be specified')\n\n cf_dict = None\n cf_key = None\n cf_data = None\n cf_iv = None\n sec_key = None\n sec_transform = None\n\n try:\n cf_dict = CFHelpers.cf_dictionary_from_pairs([(Security.kSecAttrKeyType, cipher)])\n cf_key = CFHelpers.cf_data_from_bytes(key)\n cf_data = CFHelpers.cf_data_from_bytes(data)\n\n error_pointer = new(CoreFoundation, 'CFErrorRef *')\n sec_key = Security.SecKeyCreateFromData(cf_dict, cf_key, error_pointer)\n handle_cf_error(error_pointer)\n\n sec_transform = Security.SecEncryptTransformCreate(sec_key, error_pointer)\n handle_cf_error(error_pointer)\n\n if cipher != Security.kSecAttrKeyTypeRC4:\n Security.SecTransformSetAttribute(sec_transform, Security.kSecModeCBCKey, null(), error_pointer)\n handle_cf_error(error_pointer)\n\n Security.SecTransformSetAttribute(sec_transform, Security.kSecPaddingKey, padding, error_pointer)\n handle_cf_error(error_pointer)\n\n cf_iv = CFHelpers.cf_data_from_bytes(iv)\n Security.SecTransformSetAttribute(sec_transform, Security.kSecIVKey, cf_iv, error_pointer)\n handle_cf_error(error_pointer)\n\n Security.SecTransformSetAttribute(\n sec_transform,\n Security.kSecTransformInputAttributeName,\n cf_data,\n error_pointer\n )\n handle_cf_error(error_pointer)\n\n ciphertext = Security.SecTransformExecute(sec_transform, error_pointer)\n handle_cf_error(error_pointer)\n\n return CFHelpers.cf_data_to_bytes(ciphertext)\n\n finally:\n if cf_dict:\n CoreFoundation.CFRelease(cf_dict)\n if cf_key:\n CoreFoundation.CFRelease(cf_key)\n if cf_data:\n CoreFoundation.CFRelease(cf_data)\n if cf_iv:\n CoreFoundation.CFRelease(cf_iv)\n if sec_key:\n CoreFoundation.CFRelease(sec_key)\n if sec_transform:\n CoreFoundation.CFRelease(sec_transform)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef version(self):\n ver = Version()\n ver.conn = self.conn\n\n ver.attrs = {\n # Parent params\n 'service_id': self.attrs['id'],\n }\n\n ver.save()\n\n return ver", "response": "Create a new version under this service."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef vcl(self, name, content):\n vcl = VCL()\n vcl.conn = self.conn\n\n vcl.attrs = {\n # Parent params\n 'service_id': self.attrs['service_id'],\n 'version': self.attrs['number'],\n\n # New instance params\n 'name': name,\n 'content': content,\n }\n\n vcl.save()\n\n return vcl", "response": "Create a new VCL under this version."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting the column to a dictionary representation accepted by the Citrination server.", "response": "def to_dict(self):\n \"\"\"\n Converts the column to a dictionary representation accepted\n by the Citrination server.\n\n :return: Dictionary with basic options, plus any column type specific\n options held under the \"options\" key\n :rtype: dict\n \"\"\"\n return {\n \"type\": self.type,\n \"name\": self.name,\n \"group_by_key\": self.group_by_key,\n \"role\": self.role,\n \"units\": self.units,\n \"options\": self.build_options()\n }"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_descriptor(self, descriptor, role='ignore', group_by_key=False):\n\n descriptor.validate()\n\n if descriptor.key in self.configuration[\"roles\"]:\n raise ValueError(\"Cannot add a descriptor with the same name twice\")\n\n self.configuration['descriptors'].append(descriptor.as_dict())\n self.configuration[\"roles\"][descriptor.key] = role\n\n if group_by_key:\n self.configuration[\"group_by\"].append(descriptor.key)", "response": "Adds a descriptor column."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get(self, route, headers=None, failure_message=None):\n headers = self._get_headers(headers)\n response_lambda = (\n lambda: requests.get(self._get_qualified_route(route), headers=headers, verify=False, proxies=self.proxies)\n )\n response = check_for_rate_limiting(response_lambda(), response_lambda)\n return self._handle_response(response, failure_message)", "response": "Execute a post request and return the result\n "} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes a post request and return the result.", "response": "def _post(self, route, data, headers=None, failure_message=None):\n \"\"\"\n Execute a post request and return the result\n :param data:\n :param headers:\n :return:\n \"\"\"\n headers = self._get_headers(headers)\n response_lambda = (\n lambda: requests.post(\n self._get_qualified_route(route), headers=headers, data=data, verify=False, proxies=self.proxies\n )\n )\n response = check_for_rate_limiting(response_lambda(), response_lambda)\n return self._handle_response(response, failure_message)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting a put request and return the result.", "response": "def _put(self, route, data, headers=None, failure_message=None):\n \"\"\"\n Execute a put request and return the result\n :param data:\n :param headers:\n :return:\n \"\"\"\n headers = self._get_headers(headers)\n response_lambda = (\n lambda: requests.put(\n self._get_qualified_route(route), headers=headers, data=data, verify=False, proxies=self.proxies\n )\n )\n response = check_for_rate_limiting(response_lambda(), response_lambda)\n return self._handle_response(response, failure_message)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nexecutes a patch request and return the result", "response": "def _patch(self, route, data, headers=None, failure_message=None):\n \"\"\"\n Execute a patch request and return the result\n \"\"\"\n headers = self._get_headers(headers)\n response_lambda = (\n lambda: requests.patch(\n self._get_qualified_route(route), headers=headers, data=data, verify=False, proxies=self.proxies\n )\n )\n response = check_for_rate_limiting(response_lambda(), response_lambda)\n return self._handle_response(response, failure_message)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _delete(self, route, headers=None, failure_message=None):\n headers = self._get_headers(headers)\n response_lambda = (lambda: requests.delete(\n self._get_qualified_route(route), headers=headers, verify=False, proxies=self.proxies)\n )\n response = check_for_rate_limiting(response_lambda(), response_lambda)\n return self._handle_response(response, failure_message)", "response": "Execute a delete request and return the result\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _validate_search_query(self, returning_query):\n\n start_index = returning_query.from_index or 0\n size = returning_query.size or 0\n\n if start_index < 0:\n raise CitrinationClientError(\n \"start_index cannot be negative. Please enter a value greater than or equal to zero\")\n if size < 0:\n raise CitrinationClientError(\"Size cannot be negative. Please enter a value greater than or equal to zero\")\n if start_index + size > MAX_QUERY_DEPTH:\n raise CitrinationClientError(\n \"Citrination does not support pagination past the {0}th result. Please reduce either the from_index and/or size such that their sum is below {0}\".format(\n MAX_QUERY_DEPTH))", "response": "Checks to see if the query will not exceed the maximum query depth."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef pif_search(self, pif_system_returning_query):\n\n self._validate_search_query(pif_system_returning_query)\n return self._execute_search_query(\n pif_system_returning_query,\n PifSearchResult\n )", "response": "Execute a PIF system query against the Citrination."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dataset_search(self, dataset_returning_query):\n\n self._validate_search_query(dataset_returning_query)\n return self._execute_search_query(\n dataset_returning_query,\n DatasetSearchResult\n )", "response": "Run a dataset query against Citrination."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexecutes a PIF query against the Citrination.", "response": "def _execute_search_query(self, returning_query, result_class):\n \"\"\"\n Run a PIF query against Citrination.\n\n :param returning_query: :class:`BaseReturningQuery` to execute.\n :param result_class: The class of the result to return.\n :return: ``result_class`` object with the results of the query.\n \"\"\"\n if returning_query.from_index:\n from_index = returning_query.from_index\n else:\n from_index = 0\n\n if returning_query.size != None:\n size = min(returning_query.size, client_config.max_query_size)\n else:\n size = client_config.max_query_size\n\n if (size == client_config.max_query_size and\n size != returning_query.size):\n self._warn(\"Query size greater than max system size - only {} results will be returned\".format(size))\n\n time = 0.0;\n hits = [];\n while True:\n sub_query = deepcopy(returning_query)\n sub_query.from_index = from_index + len(hits)\n partial_results = self._search_internal(sub_query, result_class)\n total = partial_results.total_num_hits\n time += partial_results.took\n if partial_results.hits is not None:\n hits.extend(partial_results.hits)\n if len(hits) >= size or len(hits) >= total or sub_query.from_index >= total:\n break\n\n return result_class(hits=hits, total_num_hits=total, took=time)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef pif_multi_search(self, multi_query):\n failure_message = \"Error while making PIF multi search request\"\n response_dict = self._get_success_json(\n self._post(routes.pif_multi_search, data=json.dumps(multi_query, cls=QueryEncoder),\n failure_message=failure_message))\n\n return PifMultiSearchResult(**keys_to_snake_case(response_dict['results']))", "response": "Run each in a list of PIF queries against Citrination."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef check_for_rate_limiting(response, response_lambda, timeout=1, attempts=0):\n if attempts >= 3:\n raise RateLimitingException()\n if response.status_code == 429:\n sleep(timeout)\n new_timeout = timeout + 1\n new_attempts = attempts + 1\n return check_for_rate_limiting(response_lambda(timeout, attempts), response_lambda, timeout=new_timeout, attempts=new_attempts)\n return response", "response": "Checks if the response is rate limited and retries it if it is not."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef create(self, configuration, name, description):\n\n data = {\n \"configuration\":\n configuration,\n \"name\":\n name,\n \"description\":\n description\n }\n\n failure_message = \"Dataview creation failed\"\n\n result = self._get_success_json(self._post_json(\n 'v1/data_views', data, failure_message=failure_message))\n data_view_id = result['data']['id']\n\n return data_view_id", "response": "Creates a data view from the search template and ml template given the name and description."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef update(self, id, configuration, name, description):\n\n data = {\n \"configuration\":\n configuration,\n \"name\":\n name,\n \"description\":\n description\n }\n\n failure_message = \"Dataview creation failed\"\n\n self._patch_json(\n 'v1/data_views/' + id, data, failure_message=failure_message)", "response": "Updates an existing data view from the search template and ml template given the parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get(self, data_view_id):\n\n failure_message = \"Dataview get failed\"\n return self._get_success_json(self._get(\n 'v1/data_views/' + data_view_id, None, failure_message=failure_message))['data']['data_view']", "response": "Gets basic information about a data view"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nretrieves the status of all of the services associated with a data view.", "response": "def get_data_view_service_status(self, data_view_id):\n \"\"\"\n Retrieves the status for all of the services associated with a data view:\n - predict\n - experimental_design\n - data_reports\n - model_reports\n\n :param data_view_id: The ID number of the data view to which the\n run belongs, as a string\n :type data_view_id: str\n :return: A :class:`DataViewStatus`\n :rtype: DataViewStatus\n \"\"\"\n\n url = \"data_views/{}/status\".format(data_view_id)\n\n response = self._get(url).json()\n result = response[\"data\"][\"status\"]\n\n return DataViewStatus(\n predict=ServiceStatus.from_response_dict(result[\"predict\"]),\n experimental_design=ServiceStatus.from_response_dict(result[\"experimental_design\"]),\n data_reports=ServiceStatus.from_response_dict(result[\"data_reports\"]),\n model_reports=ServiceStatus.from_response_dict(result[\"model_reports\"])\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_ml_configuration_from_datasets(self, dataset_ids):\n available_columns = self.search_template_client.get_available_columns(dataset_ids)\n\n # Create a search template from dataset ids\n search_template = self.search_template_client.create(dataset_ids, available_columns)\n return self.create_ml_configuration(search_template, available_columns, dataset_ids)", "response": "Creates an ml configuration from dataset ids and extract_as_keys\n "} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __snake_case(self, descriptor):\n newdict = {}\n for i, (k, v) in enumerate(descriptor.items()):\n newkey = \"\"\n for j, c in enumerate(k):\n if c.isupper():\n if len(newkey) != 0:\n newkey += '_'\n newkey += c.lower()\n else:\n newkey += c\n newdict[newkey] = v\n\n return newdict", "response": "Utility method to convert camelcase to snakecase"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __get_ml_configuration_status(self, job_id):\n\n failure_message = \"Get status on ml configuration failed\"\n response = self._get_success_json(self._get(\n 'v1/descriptors/builders/simple/default/' + job_id + '/status', None, failure_message=failure_message))[\n 'data']\n return response", "response": "Get the status of the ml configuration job."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the t - SNE analysis from the data view.", "response": "def tsne(self, data_view_id):\n \"\"\"\n Get the t-SNE projection, including responses and tags.\n\n :param data_view_id: The ID of the data view to retrieve TSNE from\n :type data_view_id: int\n :return: The TSNE analysis\n :rtype: :class:`Tsne`\n \"\"\"\n analysis = self._data_analysis(data_view_id)\n projections = analysis['projections']\n tsne = Tsne()\n for k, v in projections.items():\n projection = Projection(\n xs=v['x'],\n ys=v['y'],\n responses=v['label'],\n tags=v['inputs'],\n uids=v['uid']\n )\n tsne.add_projection(k, projection)\n\n return tsne"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\npredicts endpoint. This simply wraps the async methods (submit and poll for status/results). :param data_view_id: The ID of the data view to use for prediction :type data_view_id: str :param candidates: A list of candidates to make predictions on :type candidates: list of dicts :param method: Method for propagating predictions through model graphs. \"scalar\" uses linearized uncertainty propagation, whereas \"scalar_from_distribution\" still returns scalar predictions but uses sampling to propagate uncertainty without a linear approximation. :type method: str (\"scalar\" or \"scalar_from_distribution\") :param use_prior: Whether to apply prior values implied by the property descriptors :type use_prior: bool :return: The results of the prediction :rtype: list of :class:`PredictionResult`", "response": "def predict(self, data_view_id, candidates, method=\"scalar\", use_prior=True):\n \"\"\"\n Predict endpoint. This simply wraps the async methods (submit and poll for status/results).\n\n :param data_view_id: The ID of the data view to use for prediction\n :type data_view_id: str\n :param candidates: A list of candidates to make predictions on\n :type candidates: list of dicts\n :param method: Method for propagating predictions through model graphs. \"scalar\" uses linearized uncertainty\n propagation, whereas \"scalar_from_distribution\" still returns scalar predictions but uses sampling to\n propagate uncertainty without a linear approximation.\n :type method: str (\"scalar\" or \"scalar_from_distribution\")\n :param use_prior: Whether to apply prior values implied by the property descriptors\n :type use_prior: bool\n :return: The results of the prediction\n :rtype: list of :class:`PredictionResult`\n \"\"\"\n\n uid = self.submit_predict_request(data_view_id, candidates, method, use_prior)\n\n while self.check_predict_status(data_view_id, uid)['status'] not in [\"Finished\", \"Failed\", \"Killed\"]:\n time.sleep(1)\n\n result = self.check_predict_status(data_view_id, uid)\n if result[\"status\"] == \"Finished\":\n\n paired = zip(result[\"results\"][\"candidates\"], result[\"results\"][\"loss\"])\n prediction_result_format = [{k: (p[0][k], p[1][k]) for k in p[0].keys()} for p in paired]\n\n return list(map(\n lambda c: _get_prediction_result_from_candidate(c), prediction_result_format\n ))\n else:\n raise RuntimeError(\n \"Prediction failed: UID={}, result={}\".format(uid, result[\"status\"])\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nretrain the dataview with the current dataview_id", "response": "def retrain(self, dataview_id):\n \"\"\"\n Start a model retraining\n :param dataview_id: The ID of the views\n :return:\n \"\"\"\n url = 'data_views/{}/retrain'.format(dataview_id)\n response = self._post_json(url, data={})\n if response.status_code != requests.codes.ok:\n raise RuntimeError('Retrain requested ' + str(response.status_code) + ' response: ' + str(response.message))\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets the data analysis for a specific data view", "response": "def _data_analysis(self, data_view_id):\n \"\"\"\n Data analysis endpoint.\n\n :param data_view_id: The model identifier (id number for data views)\n :type data_view_id: str\n :return: dictionary containing information about the data, e.g. dCorr and tsne\n \"\"\"\n failure_message = \"Error while retrieving data analysis for data view {}\".format(data_view_id)\n return self._get_success_json(self._get(routes.data_analysis(data_view_id), failure_message=failure_message))"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef submit_predict_request(self, data_view_id, candidates, prediction_source='scalar', use_prior=True):\n\n data = {\n \"prediction_source\":\n prediction_source,\n \"use_prior\":\n use_prior,\n \"candidates\":\n candidates\n }\n\n failure_message = \"Configuration creation failed\"\n post_url = 'v1/data_views/' + str(data_view_id) + '/predict/submit'\n return self._get_success_json(\n self._post_json(post_url, data, failure_message=failure_message)\n )['data']['uid']", "response": "Submit a prediction request to the data view."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nchecks the status of the prediction job.", "response": "def check_predict_status(self, view_id, predict_request_id):\n \"\"\"\n Returns a string indicating the status of the prediction job\n\n :param view_id: The data view id returned from data view create\n :param predict_request_id: The id returned from predict\n :return: Status data, also includes results if state is finished\n \"\"\"\n\n failure_message = \"Get status on predict failed\"\n\n bare_response = self._get_success_json(self._get(\n 'v1/data_views/' + str(view_id) + '/predict/' + str(predict_request_id) + '/status',\n None, failure_message=failure_message))\n\n result = bare_response[\"data\"]\n # result.update({\"message\": bare_response[\"message\"]})\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsubmitting a new experimental design run.", "response": "def submit_design_run(self, data_view_id, num_candidates, effort, target=None, constraints=[], sampler=\"Default\"):\n \"\"\"\n Submits a new experimental design run.\n\n :param data_view_id: The ID number of the data view to which the\n run belongs, as a string\n :type data_view_id: str\n :param num_candidates: The number of candidates to return\n :type num_candidates: int\n :param target: An :class:``Target`` instance representing\n the design run optimization target\n :type target: :class:``Target``\n :param constraints: An array of design constraints (instances of\n objects which extend :class:``BaseConstraint``)\n :type constraints: list of :class:``BaseConstraint``\n :param sampler: The name of the sampler to use during the design run:\n either \"Default\" or \"This view\"\n :type sampler: str\n :return: A :class:`DesignRun` instance containing the UID of the\n new run\n \"\"\"\n if effort > 30:\n raise CitrinationClientError(\"Parameter effort must be less than 30 to trigger a design run\")\n\n if target is not None:\n target = target.to_dict()\n\n constraint_dicts = [c.to_dict() for c in constraints]\n\n body = {\n \"num_candidates\": num_candidates,\n \"target\": target,\n \"effort\": effort,\n \"constraints\": constraint_dicts,\n \"sampler\": sampler\n }\n\n url = routes.submit_data_view_design(data_view_id)\n\n response = self._post_json(url, body).json()\n\n return DesignRun(response[\"data\"][\"design_run\"][\"uid\"])"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_design_run_status(self, data_view_id, run_uuid):\n\n url = routes.get_data_view_design_status(data_view_id, run_uuid)\n\n response = self._get(url).json()\n\n status = response[\"data\"]\n\n return ProcessStatus(\n result=status.get(\"result\"),\n progress=status.get(\"progress\"),\n status=status.get(\"status\"),\n messages=status.get(\"messages\")\n )", "response": "Retrieves the status of an in progress or completed design run"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_design_run_results(self, data_view_id, run_uuid):\n\n url = routes.get_data_view_design_results(data_view_id, run_uuid)\n\n response = self._get(url).json()\n\n result = response[\"data\"]\n\n return DesignResults(\n best_materials=result.get(\"best_material_results\"),\n next_experiments=result.get(\"next_experiment_results\")\n )", "response": "Retrieves the results of an existing design run."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nretrieving a summary of information for a given data view.", "response": "def get_data_view(self, data_view_id):\n \"\"\"\n Retrieves a summary of information for a given data view\n - view id\n - name\n - description\n - columns\n\n :param data_view_id: The ID number of the data view to which the\n run belongs, as a string\n :type data_view_id: str\n \"\"\"\n\n url = routes.get_data_view(data_view_id)\n\n response = self._get(url).json()\n\n result = response[\"data\"][\"data_view\"]\n\n datasets_list = []\n for dataset in result[\"datasets\"]:\n datasets_list.append(Dataset(\n name=dataset[\"name\"],\n id=dataset[\"id\"],\n description=dataset[\"description\"]\n ))\n\n columns_list = []\n for column in result[\"columns\"]:\n columns_list.append(ColumnFactory.from_dict(column))\n\n return DataView(\n view_id=data_view_id,\n name=result[\"name\"],\n description=result[\"description\"],\n datasets=datasets_list,\n columns=columns_list,\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nkill an in progress experimental design run", "response": "def kill_design_run(self, data_view_id, run_uuid):\n \"\"\"\n Kills an in progress experimental design run\n\n :param data_view_id: The ID number of the data view to which the\n run belongs, as a string\n :type data_view_id: str\n :param run_uuid: The UUID of the design run to kill\n :type run_uuid: str\n :return: The UUID of the design run\n \"\"\"\n\n url = routes.kill_data_view_design_run(data_view_id, run_uuid)\n\n response = self._delete(url).json()\n return response[\"data\"][\"uid\"]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads the file at the given path as a dictionary from YAML", "response": "def load_file_as_yaml(path):\n \"\"\"\n Given a filepath, loads the file as a dictionary from YAML\n\n :param path: The path to a YAML file\n \"\"\"\n with open(path, \"r\") as f:\n raw_yaml = f.read()\n parsed_dict = yaml.load(raw_yaml)\n return parsed_dict"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nextracting credentials from the yaml formatted credential filepath", "response": "def get_credentials_from_file(filepath):\n \"\"\"\n Extracts credentials from the yaml formatted credential filepath\n passed in. Uses the default profile if the CITRINATION_PROFILE env var\n is not set, otherwise looks for a profile with that name in the credentials file.\n\n :param filepath: The path of the credentials file\n \"\"\"\n try:\n creds = load_file_as_yaml(filepath)\n except Exception:\n creds = {}\n\n profile_name = os.environ.get(citr_env_vars.CITRINATION_PROFILE)\n if profile_name is None or len(profile_name) == 0:\n profile_name = DEFAULT_CITRINATION_PROFILE\n api_key = None\n site = None\n try:\n profile = creds[profile_name]\n api_key = profile[CREDENTIALS_API_KEY_KEY]\n site = profile[CREDENTIALS_SITE_KEY]\n except KeyError:\n pass\n\n return (api_key, site)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngives an API key, a site url and a credentials file path, runs through a prioritized list of credential sources to find credentials. Specifically, this method ranks credential priority as follows: 1. Those passed in as the first two parameters to this method 2. Those found in the environment as variables 3. Those found in the credentials file at the profile specified by the profile environment variable 4. Those found in the default stanza in the credentials file :param api_key: A Citrination API Key or None :param site: A Citrination site URL or None :param cred_file: The path to a credentials file", "response": "def get_preferred_credentials(api_key, site, cred_file=DEFAULT_CITRINATION_CREDENTIALS_FILE):\n \"\"\"\n Given an API key, a site url and a credentials file path, runs through a prioritized list of credential sources to find credentials.\n\n Specifically, this method ranks credential priority as follows:\n 1. Those passed in as the first two parameters to this method\n 2. Those found in the environment as variables\n 3. Those found in the credentials file at the profile specified\n by the profile environment variable\n 4. Those found in the default stanza in the credentials file\n\n :param api_key: A Citrination API Key or None\n :param site: A Citrination site URL or None\n :param cred_file: The path to a credentials file\n \"\"\"\n profile_api_key, profile_site = get_credentials_from_file(cred_file)\n if api_key is None:\n api_key = os.environ.get(citr_env_vars.CITRINATION_API_KEY)\n if api_key is None or len(api_key) == 0:\n api_key = profile_api_key\n\n if site is None:\n site = os.environ.get(citr_env_vars.CITRINATION_SITE)\n if site is None or len(site) == 0:\n site = profile_site\n if site is None:\n site = \"https://citrination.com\"\n\n return api_key, site"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nuploading a file to the dataset with the given source and destination paths.", "response": "def upload(self, dataset_id, source_path, dest_path=None):\n \"\"\"\n Upload a file, specifying source and dest paths a file (acts as the scp command).asdfasdf\n\n :param source_path: The path to the file on the source host asdf\n :type source_path: str\n :param dest_path: The path to the file where the contents of the upload will be written (on the dest host)\n :type dest_path: str\n :return: The result of the upload process\n :rtype: :class:`UploadResult`\n \"\"\"\n upload_result = UploadResult()\n source_path = str(source_path)\n if not dest_path:\n dest_path = source_path\n else:\n dest_path = str(dest_path)\n if os.path.isdir(source_path):\n for path, subdirs, files in os.walk(source_path):\n relative_path = os.path.relpath(path, source_path)\n current_dest_prefix = dest_path\n if relative_path is not \".\":\n current_dest_prefix = os.path.join(current_dest_prefix, relative_path)\n for name in files:\n current_dest_path = os.path.join(current_dest_prefix, name)\n current_source_path = os.path.join(path, name)\n try:\n if self.upload(dataset_id, current_source_path, current_dest_path).successful():\n upload_result.add_success(current_source_path)\n else:\n upload_result.add_failure(current_source_path,\"Upload failure\")\n except (CitrinationClientError, ValueError) as e:\n upload_result.add_failure(current_source_path, str(e))\n return upload_result\n elif os.path.isfile(source_path):\n file_data = { \"dest_path\": str(dest_path), \"src_path\": str(source_path)}\n j = self._get_success_json(self._post_json(routes.upload_to_dataset(dataset_id), data=file_data))\n s3url = _get_s3_presigned_url(j)\n with open(source_path, 'rb') as f:\n if os.stat(source_path).st_size == 0:\n # Upload a null character as a placeholder for\n # the empty file since Citrination does not support\n # truly empty files\n data = \"\\0\"\n else:\n data = f\n r = requests.put(s3url, data=data, headers=j[\"required_headers\"])\n if r.status_code == 200:\n data = {'s3object': j['url']['path'], 's3bucket': j['bucket']}\n self._post_json(routes.update_file(j['file_id']), data=data)\n upload_result.add_success(source_path)\n return upload_result\n else:\n raise CitrinationClientError(\"Failure to upload {} to Citrination\".format(source_path))\n else:\n raise ValueError(\"No file at specified path {}\".format(source_path))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef list_files(self, dataset_id, glob=\".\", is_dir=False):\n data = {\n \"list\": {\n \"glob\": glob,\n \"isDir\": is_dir\n }\n }\n return self._get_success_json(self._post_json(routes.list_files(dataset_id), data, failure_message=\"Failed to list files for dataset {}\".format(dataset_id)))['files']", "response": "List files in a dataset on Citrination."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef matched_file_count(self, dataset_id, glob=\".\", is_dir=False):\n list_result = self.list_files(dataset_id, glob, is_dir)\n return len(list_result)", "response": "Returns the number of files matching a pattern in a dataset."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the current status of the dataset ingestion.", "response": "def get_ingest_status(self, dataset_id):\n \"\"\"\n Returns the current status of dataset ingestion. If any file uploaded to a dataset is in an error/failure state\n this endpoint will return error/failure. If any files are still processing, will return processing.\n\n :param dataset_id: Dataset identifier\n :return: Status of dataset ingestion as a string\n \"\"\"\n failure_message = \"Failed to create dataset ingest status for dataset {}\".format(dataset_id)\n response = self._get_success_json(\n self._get('v1/datasets/' + str(dataset_id) + '/ingest-status',\n failure_message=failure_message))['data']\n\n if 'status' in response:\n return response['status']\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nretrieve URLs for the files matched by a glob or a path to a directory containing one or more files in a dataset.", "response": "def get_dataset_files(self, dataset_id, glob=\".\", is_dir=False, version_number=None):\n \"\"\"\n Retrieves URLs for the files matched by a glob or a path to a directory\n in a given dataset.\n\n :param dataset_id: The id of the dataset to retrieve files from\n :type dataset_id: int\n :param glob: A regex used to select one or more files in the dataset\n :type glob: str\n :param is_dir: Whether or not the supplied pattern should be treated as a directory to search in\n :type is_dir: bool\n :param version_number: The version number of the dataset to retrieve files from\n :type version_number: int\n :return: A list of dataset files whose paths match the provided pattern.\n :rtype: list of :class:`DatasetFile`\n \"\"\"\n if version_number is None:\n latest = True\n else:\n latest = False\n\n data = {\n \"download_request\": {\n \"glob\": glob,\n \"isDir\": is_dir,\n \"latest\": latest\n }\n }\n\n failure_message = \"Failed to get matched files in dataset {}\".format(dataset_id)\n\n versions = self._get_success_json(self._post_json(routes.matched_files(dataset_id), data, failure_message=failure_message))['versions']\n\n # if you don't provide a version number, only the latest\n # will be included in the response body\n if version_number is None:\n version = versions[0]\n else:\n try:\n version = list(filter(lambda v: v['number'] == version_number, versions))[0]\n except IndexError:\n raise ResourceNotFoundException()\n\n return list(\n map(\n lambda f: DatasetFile(path=f['filename'], url=f['url']), version['files']\n )\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_dataset_file(self, dataset_id, file_path, version = None):\n return self.get_dataset_files(dataset_id, \"^{}$\".format(file_path), version_number=version)[0]", "response": "Retrieves a dataset file matching a provided file path"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndownloading the files from the dataset_files list to the destination.", "response": "def download_files(self, dataset_files, destination='.'):\n \"\"\"\n Downloads file(s) to a local destination.\n\n :param dataset_files:\n :type dataset_files: list of :class: `DatasetFile`\n :param destination: The path to the desired local download destination\n :type destination: str\n :param chunk: Whether or not to chunk the file. Default True\n :type chunk: bool\n \"\"\"\n if not isinstance(dataset_files, list):\n dataset_files = [dataset_files]\n\n for f in dataset_files:\n filename = f.path.lstrip('/')\n local_path = os.path.join(destination, filename)\n\n if not os.path.isdir(os.path.dirname(local_path)):\n os.makedirs(os.path.dirname(local_path))\n\n r = requests.get(f.url, stream=True)\n\n with open(local_path, 'wb') as output_file:\n shutil.copyfileobj(r.raw, output_file)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_pif(self, dataset_id, uid, dataset_version = None):\n failure_message = \"An error occurred retrieving PIF {}\".format(uid)\n if dataset_version == None:\n response = self._get(routes.pif_dataset_uid(dataset_id, uid), failure_message=failure_message)\n else:\n response = self._get(routes.pif_dataset_version_uid(dataset_id, uid, dataset_version), failure_message=failure_message)\n\n return pif.loads(response.content.decode(\"utf-8\"))", "response": "Retrieves a PIF from a given dataset."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef create_dataset(self, name=None, description=None, public=False):\n data = {\n \"public\": _convert_bool_to_public_value(public)\n }\n if name:\n data[\"name\"] = name\n if description:\n data[\"description\"] = description\n dataset = {\"dataset\": data}\n failure_message = \"Unable to create dataset\"\n result = self._get_success_json(self._post_json(routes.create_dataset(), dataset, failure_message=failure_message))\n\n return _dataset_from_response_dict(result)", "response": "Creates a new data set."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef update_dataset(self, dataset_id, name=None, description=None, public=None):\n data = {\n \"public\": _convert_bool_to_public_value(public)\n }\n\n if name:\n data[\"name\"] = name\n if description:\n data[\"description\"] = description\n\n dataset = {\"dataset\": data}\n failure_message = \"Failed to update dataset {}\".format(dataset_id)\n response = self._get_success_json(self._post_json(routes.update_dataset(dataset_id), data=dataset, failure_message=failure_message))\n\n return _dataset_from_response_dict(response)", "response": "Update the name description and public attributes of a dataset."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncreates a new dataset version.", "response": "def create_dataset_version(self, dataset_id):\n \"\"\"\n Create a new data set version.\n\n :param dataset_id: The ID of the dataset for which the version must be bumped.\n :type dataset_id: int\n :return: The new dataset version.\n :rtype: :class:`DatasetVersion`\n \"\"\"\n failure_message = \"Failed to create dataset version for dataset {}\".format(dataset_id)\n number = self._get_success_json(self._post_json(routes.create_dataset_version(dataset_id), data={}, failure_message=failure_message))['dataset_scoped_id']\n\n return DatasetVersion(number=number)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nretrieves the set of columns from the combination of dataset ids given", "response": "def get_available_columns(self, dataset_ids):\n \"\"\"\n Retrieves the set of columns from the combination of dataset ids given\n\n :param dataset_ids: The id of the dataset to retrieve columns from\n :type dataset_ids: list of int\n :return: A list of column names from the dataset ids given.\n :rtype: list of str\n \"\"\"\n if not isinstance(dataset_ids, list):\n dataset_ids = [dataset_ids]\n\n data = {\n \"dataset_ids\":\n dataset_ids\n }\n\n failure_message = \"Failed to get available columns in dataset(s) {}\".format(dataset_ids)\n\n return self._get_success_json(self._post_json(\n 'v1/datasets/get-available-columns', data, failure_message=failure_message))['data']"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating a search template based on the available columns in the dataset ids given.", "response": "def __generate_search_template(self, dataset_ids):\n \"\"\"\n Generates a default search templates from the available columns in the dataset ids given.\n\n :param dataset_ids: The id of the dataset to retrieve files from\n :type dataset_ids: list of int\n :return: A search template based on the columns in the datasets given\n \"\"\"\n\n data = {\n \"dataset_ids\":\n dataset_ids\n }\n\n failure_message = \"Failed to generate a search template from columns in dataset(s) {}\".format(dataset_ids)\n\n return self._get_success_json(self._post_json(\n 'v1/search_templates/builders/from-dataset-ids', data, failure_message=failure_message))['data']"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef __prune_search_template(self, extract_as_keys, search_template):\n\n data = {\n \"extract_as_keys\":\n extract_as_keys,\n \"search_template\":\n search_template\n }\n\n failure_message = \"Failed to prune a search template\"\n\n return self._get_success_json(self._post_json(\n 'v1/search_templates/prune-to-extract-as', data, failure_message=failure_message))['data']", "response": "Prune a search template with only the given extract as keys."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef default(self, obj):\n if obj is None:\n return []\n elif isinstance(obj, list):\n return [i.as_dictionary() for i in obj]\n elif isinstance(obj, dict):\n return self._keys_to_camel_case(obj)\n else:\n return obj.as_dictionary()", "response": "Convert an object to a list of dictionaries each representing a physical information object ready to be serialized to json."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _keys_to_camel_case(self, obj):\n return dict((to_camel_case(key), value) for (key, value) in obj.items())", "response": "Convert all keys in obj to camel case."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef validate(self, ml_template):\n\n data = {\n \"ml_template\":\n ml_template\n }\n\n failure_message = \"ML template validation invoke failed\"\n\n res = self._get_success_json(self._post_json(\n 'ml_templates/validate', data, failure_message=failure_message))['data']\n if res['valid']:\n return 'OK'\n return res['reason']", "response": "Validates the template against the validation endpoint returns a message indicating status of the template validation"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _validate_course_key(course_key):\n if not validators.course_key_is_valid(course_key):\n exceptions.raise_exception(\n \"CourseKey\",\n course_key,\n exceptions.InvalidCourseKeyException\n )", "response": "Validate the course key."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_organization_course(organization_data, course_key):\n _validate_course_key(course_key)\n _validate_organization_data(organization_data)\n data.create_organization_course(\n organization=organization_data,\n course_key=course_key\n )", "response": "Adds a organization - course link to the system\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nremove the specified course from the specified organization.", "response": "def remove_organization_course(organization, course_key):\n \"\"\"\n Removes the specfied course from the specified organization\n \"\"\"\n _validate_organization_data(organization)\n _validate_course_key(course_key)\n return data.delete_organization_course(course_key=course_key, organization=organization)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nraising an exception if the given entity is not valid.", "response": "def raise_exception(entity_type, entity, exception):\n \"\"\" Exception helper \"\"\"\n raise exception(\n u'The {} you have provided is not valid: {}'.format(entity_type, entity).encode('utf-8')\n )"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef course_key_is_valid(course_key):\n if course_key is None:\n return False\n try:\n CourseKey.from_string(text_type(course_key))\n except (InvalidKeyError, UnicodeDecodeError):\n return False\n return True", "response": "Checks if the given course key is valid."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef organization_data_is_valid(organization_data):\n if organization_data is None:\n return False\n if 'id' in organization_data and not organization_data.get('id'):\n return False\n if 'name' in organization_data and not organization_data.get('name'):\n return False\n return True", "response": "Checks if organization data is valid."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _activate_organization(organization):\n [_activate_organization_course_relationship(record) for record\n in internal.OrganizationCourse.objects.filter(organization_id=organization.id, active=False)]\n\n [_activate_record(record) for record\n in internal.Organization.objects.filter(id=organization.id, active=False)]", "response": "Activates an inactivated organization as well as any inactive relationships\n "} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _inactivate_organization(organization):\n [_inactivate_organization_course_relationship(record) for record\n in internal.OrganizationCourse.objects.filter(organization_id=organization.id, active=True)]\n\n [_inactivate_record(record) for record\n in internal.Organization.objects.filter(id=organization.id, active=True)]", "response": "Inactivates an activated organization as well as any active relationships\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nactivates an inactive organization - course relationship", "response": "def _activate_organization_course_relationship(relationship): # pylint: disable=invalid-name\n \"\"\"\n Activates an inactive organization-course relationship\n \"\"\"\n # If the relationship doesn't exist or the organization isn't active we'll want to raise an error\n relationship = internal.OrganizationCourse.objects.get(\n id=relationship.id,\n active=False,\n organization__active=True\n )\n _activate_record(relationship)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _inactivate_organization_course_relationship(relationship): # pylint: disable=invalid-name\n relationship = internal.OrganizationCourse.objects.get(\n id=relationship.id,\n active=True\n )\n _inactivate_record(relationship)", "response": "Inactivates an active organization - course relationship"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef create_organization(organization):\n # Trust, but verify...\n if not organization.get('name'):\n exceptions.raise_exception(\"organization\", organization, exceptions.InvalidOrganizationException)\n organization_obj = serializers.deserialize_organization(organization)\n try:\n organization = internal.Organization.objects.get(\n name=organization_obj.name,\n )\n # If the organization exists, but was inactivated, we can simply turn it back on\n if not organization.active:\n _activate_organization(organization_obj)\n except internal.Organization.DoesNotExist:\n organization = internal.Organization.objects.create(\n name=organization_obj.name,\n short_name=organization_obj.short_name,\n description=organization_obj.description,\n logo=organization_obj.logo,\n active=True\n )\n return serializers.serialize_organization(organization)", "response": "Creates a new organization into app / local state given the following dictionary"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nupdates an existing organization in app / local state Returns a dictionary representation of the organization", "response": "def update_organization(organization):\n \"\"\"\n Updates an existing organization in app/local state\n Returns a dictionary representation of the object\n \"\"\"\n organization_obj = serializers.deserialize_organization(organization)\n try:\n organization = internal.Organization.objects.get(id=organization_obj.id)\n organization.name = organization_obj.name\n organization.short_name = organization_obj.short_name\n organization.description = organization_obj.description\n organization.logo = organization_obj.logo\n organization.active = organization_obj.active\n except internal.Organization.DoesNotExist:\n exceptions.raise_exception(\"organization\", organization, exceptions.InvalidOrganizationException)\n return serializers.serialize_organization(organization)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a specific organization from app / local state", "response": "def fetch_organization(organization_id):\n \"\"\"\n Retrieves a specific organization from app/local state\n Returns a dictionary representation of the object\n \"\"\"\n organization = {'id': organization_id}\n if not organization_id:\n exceptions.raise_exception(\"organization\", organization, exceptions.InvalidOrganizationException)\n organizations = serializers.serialize_organizations(\n internal.Organization.objects.filter(id=organization_id, active=True)\n )\n if not organizations:\n exceptions.raise_exception(\"organization\", organization, exceptions.InvalidOrganizationException)\n return organizations[0]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fetch_organization_by_short_name(organization_short_name):\n organization = {'short_name': organization_short_name}\n if not organization_short_name:\n exceptions.raise_exception(\"organization\", organization, exceptions.InvalidOrganizationException)\n organizations = serializers.serialize_organizations(internal.Organization.objects.filter(\n active=True, short_name=organization_short_name\n ))\n if not organizations:\n exceptions.raise_exception(\"organization\", organization, exceptions.InvalidOrganizationException)\n return organizations[0]", "response": "Returns a specific organization from app state by short name"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef create_organization_course(organization, course_key):\n organization_obj = serializers.deserialize_organization(organization)\n try:\n relationship = internal.OrganizationCourse.objects.get(\n organization=organization_obj,\n course_id=text_type(course_key)\n )\n # If the relationship exists, but was inactivated, we can simply turn it back on\n if not relationship.active:\n _activate_organization_course_relationship(relationship)\n except internal.OrganizationCourse.DoesNotExist:\n relationship = internal.OrganizationCourse.objects.create(\n organization=organization_obj,\n course_id=text_type(course_key),\n active=True\n )", "response": "Creates a new organization - course relationship into app - local state"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nremoves an existing organization - course relationship from app / local state", "response": "def delete_organization_course(organization, course_key):\n \"\"\"\n Removes an existing organization-course relationship from app/local state\n No response currently defined for this operation\n \"\"\"\n try:\n relationship = internal.OrganizationCourse.objects.get(\n organization=organization['id'],\n course_id=text_type(course_key),\n active=True,\n )\n _inactivate_organization_course_relationship(relationship)\n except internal.OrganizationCourse.DoesNotExist:\n # If we're being asked to delete an organization-course link\n # that does not exist in the database then our work is done\n pass"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef fetch_organization_courses(organization):\n organization_obj = serializers.deserialize_organization(organization)\n queryset = internal.OrganizationCourse.objects.filter(\n organization=organization_obj,\n active=True\n ).select_related('organization')\n return [serializers.serialize_organization_with_course(organization) for organization in queryset]", "response": "Returns the set of courses currently linked to the organization"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nretrieves the organizations linked to the specified course", "response": "def fetch_course_organizations(course_key):\n \"\"\"\n Retrieves the organizations linked to the specified course\n \"\"\"\n queryset = internal.OrganizationCourse.objects.filter(\n course_id=text_type(course_key),\n active=True\n ).select_related('organization')\n return [serializers.serialize_organization_with_course(organization) for organization in queryset]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef delete_course_references(course_key):\n [_inactivate_record(record) for record in internal.OrganizationCourse.objects.filter(\n course_id=text_type(course_key),\n active=True\n )]", "response": "Delete all references to the given course key within this app."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nserialize an organization object to a dictionary.", "response": "def serialize_organization(organization):\n \"\"\"\n Organization object-to-dict serialization\n \"\"\"\n return {\n 'id': organization.id,\n 'name': organization.name,\n 'short_name': organization.short_name,\n 'description': organization.description,\n 'logo': organization.logo\n }"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef serialize_organization_with_course(organization_course):\n return {\n 'id': organization_course.organization.id,\n 'name': organization_course.organization.name,\n 'short_name': organization_course.organization.short_name,\n 'description': organization_course.organization.description,\n 'logo': organization_course.organization.logo,\n 'course_id': organization_course.course_id\n }", "response": "Serialize organization_course object into a composite object."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deserialize_organization(organization_dict):\n return models.Organization(\n id=organization_dict.get('id'),\n name=organization_dict.get('name', ''),\n short_name=organization_dict.get('short_name', ''),\n description=organization_dict.get('description', ''),\n logo=organization_dict.get('logo', '')\n )", "response": "Deserialize an organization dict - to - object serialization"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_large_images(self, node, parent_depth_level, sibling_depth_level):\n good_images = self.get_image_candidates(node)\n\n if good_images:\n scored_images = self.fetch_images(good_images, parent_depth_level)\n if scored_images:\n highscore_image = sorted(list(scored_images.items()),\n key=lambda x: x[1], reverse=True)[0][0]\n main_image = Image()\n main_image._src = highscore_image.src\n main_image._width = highscore_image.width\n main_image._height = highscore_image.height\n main_image._extraction_type = \"bigimage\"\n score_len = len(scored_images)\n main_image._confidence_score = 100 / score_len if score_len > 0 else 0\n return main_image\n\n depth_obj = self.get_depth_level(node, parent_depth_level, sibling_depth_level)\n if depth_obj:\n return self.check_large_images(depth_obj.node, depth_obj.parent_depth,\n depth_obj.sibling_depth)\n\n return None", "response": "This method checks if the node has any images that are too large and returns the Image object that is the same as the parent."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef is_banner_dimensions(width, height):\n if width == height:\n return False\n\n if width > height:\n diff = float(width / height)\n if diff > 5:\n return True\n\n if height > width:\n diff = float(height / width)\n if diff > 5:\n return True\n\n return False", "response": "\\ Returns True if we think this is a bannery dimension."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef is_valid_filename(self, image_node):\n src = self.parser.getAttribute(image_node, attr='src')\n\n if not src:\n return False\n\n if self.badimages_names_re.search(src):\n return False\n\n return True", "response": "\\ This method checks if the image src attribute is a valid filename."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nloop through all the images and find the ones that have the best bytez to even make them a candidate", "response": "def get_images_bytesize_match(self, images):\n \"\"\"\\\n loop through all the images and find the ones\n that have the best bytez to even make them a candidate\n \"\"\"\n cnt = 0\n max_bytes_size = 15728640\n good_images = []\n for image in images:\n if cnt > 30:\n return good_images\n src = self.parser.getAttribute(image, attr='src')\n src = self.build_image_path(src)\n src = self.add_schema_if_none(src)\n local_image = self.get_local_image(src)\n if local_image:\n filesize = local_image.bytes\n if (filesize == 0 or filesize > self.images_min_bytes) and filesize < max_bytes_size:\n good_images.append(image)\n else:\n images.remove(image)\n cnt += 1\n return good_images if len(good_images) > 0 else None"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking if the image is in the known schemas and returns the image object.", "response": "def check_known_schemas(self):\n \"\"\"\\\n checks to see if we were able to find the image via known schemas:\n\n Supported Schemas\n - Open Graph\n - schema.org\n \"\"\"\n if 'image' in self.article.opengraph:\n return self.get_image(self.article.opengraph[\"image\"],\n extraction_type='opengraph')\n elif (self.article.schema and 'image' in self.article.schema and\n \"url\" in self.article.schema[\"image\"]):\n return self.get_image(self.article.schema[\"image\"][\"url\"],\n extraction_type='schema.org')\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_local_image(self, src):\n return ImageUtils.store_image(self.fetcher, self.article.link_hash, src, self.config)", "response": "\\ Returns the bytes of the image on disk"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_video(self, node):\n video = Video()\n video._embed_code = self.get_embed_code(node)\n video._embed_type = self.get_embed_type(node)\n video._width = self.get_width(node)\n video._height = self.get_height(node)\n video._src = self.get_src(node)\n video._provider = self.get_provider(video.src)\n return video", "response": "Create a video object from a video embed node"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_encodings_from_content(content):\n if isinstance(content, bytes):\n find_charset = re.compile(\n br']', flags=re.I\n ).findall\n\n find_xml = re.compile(\n br'^<\\?xml.*?encoding=[\"\\']*([a-z0-9\\-_]+?) *?[\"\\'>]'\n ).findall\n return [encoding.decode('utf-8') for encoding in\n find_charset(content) + find_xml(content)]\n else:\n find_charset = re.compile(\n r']', flags=re.I\n ).findall\n\n find_xml = re.compile(\n r'^<\\?xml.*?encoding=[\"\\']*([a-z0-9\\-_]+?) *?[\"\\'>]'\n ).findall\n return find_charset(content) + find_xml(content)", "response": "Return encodings from given content string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef store_image(cls, http_client, link_hash, src, config):\n # check for a cache hit already on disk\n image = cls.read_localfile(link_hash, src, config)\n if image:\n return image\n\n # no cache found; do something else\n\n # parse base64 image\n if src.startswith('data:image'):\n image = cls.write_localfile_base64(link_hash, src, config)\n return image\n\n # download the image\n data = http_client.fetch(src)\n if data:\n image = cls.write_localfile(data, link_hash, src, config)\n if image:\n return image\n\n return None", "response": "\\ Writes an image src http string to disk as a temporary file\n and returns the LocallyStoredImage object that has the info you need on the image."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the list of known context patterns.", "response": "def known_context_patterns(self, val):\n ''' val must be an ArticleContextPattern, a dictionary, or list of \\\n dictionaries\n e.g., {'attr': 'class', 'value': 'my-article-class'}\n or [{'attr': 'class', 'value': 'my-article-class'},\n {'attr': 'id', 'value': 'my-article-id'}]\n '''\n def create_pat_from_dict(val):\n '''Helper function used to create an ArticleContextPattern from a dictionary\n '''\n if \"tag\" in val:\n pat = ArticleContextPattern(tag=val[\"tag\"])\n if \"attr\" in val:\n pat.attr = val[\"attr\"]\n pat.value = val[\"value\"]\n elif \"attr\" in val:\n pat = ArticleContextPattern(attr=val[\"attr\"], value=val[\"value\"])\n\n if \"domain\" in val:\n pat.domain = val[\"domain\"]\n\n return pat\n\n if isinstance(val, list):\n self._known_context_patterns = [\n x if isinstance(x, ArticleContextPattern) else create_pat_from_dict(x)\n for x in val\n ] + self.known_context_patterns\n elif isinstance(val, ArticleContextPattern):\n self._known_context_patterns.insert(0, val)\n elif isinstance(val, dict):\n self._known_context_patterns.insert(0, create_pat_from_dict(val))\n else:\n raise Exception(\"Unknown type: {}. Use a ArticleContextPattern.\".format(type(val)))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef known_publish_date_tags(self, val):\n ''' val must be a dictionary or list of dictionaries\n e.g., {'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'}\n or [{'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'},\n {'attrribute': 'property', 'value': 'pub_time', 'content': 'content'}]\n '''\n def create_pat_from_dict(val):\n '''Helper function used to create an PublishDatePattern from a dictionary\n '''\n if \"tag\" in val:\n pat = PublishDatePattern(tag=val[\"tag\"])\n if \"attribute\" in val:\n pat.attr = val[\"attribute\"]\n pat.value = val[\"value\"]\n elif \"attribute\" in val:\n pat = PublishDatePattern(attr=val[\"attribute\"], value=val[\"value\"],\n content=val[\"content\"])\n if \"subcontent\" in val:\n pat.subcontent = val[\"subcontent\"]\n\n if \"domain\" in val:\n pat.domain = val[\"domain\"]\n\n return pat\n\n if isinstance(val, list):\n self._known_publish_date_tags = [\n x if isinstance(x, PublishDatePattern) else create_pat_from_dict(x)\n for x in val\n ] + self.known_publish_date_tags\n elif isinstance(val, PublishDatePattern):\n self._known_publish_date_tags.insert(0, val)\n elif isinstance(val, dict):\n self._known_publish_date_tags.insert(0, create_pat_from_dict(val))\n else:\n raise Exception(\"Unknown type: {}. Use a PublishDatePattern.\".format(type(val)))", "response": "Set self. _known_publish_date_tags based on a dictionary or list of dictionaries."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting self. _known_author_patterns based on a dictionary or list of dictionaries.", "response": "def known_author_patterns(self, val):\n ''' val must be a dictionary or list of dictionaries\n e.g., {'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'}\n or [{'attrribute': 'name', 'value': 'my-pubdate', 'content': 'datetime'},\n {'attrribute': 'property', 'value': 'pub_time', 'content': 'content'}]\n '''\n\n def create_pat_from_dict(val):\n '''Helper function used to create an AuthorPatterns from a dictionary\n '''\n if \"tag\" in val:\n pat = AuthorPattern(tag=val[\"tag\"])\n if \"attribute\" in val:\n pat.attr = val[\"attribute\"]\n pat.value = val[\"value\"]\n elif \"attribute\" in val:\n pat = AuthorPattern(attr=val[\"attribute\"], value=val[\"value\"],\n content=val[\"content\"])\n if \"subpattern\" in val:\n pat.subpattern = create_pat_from_dict(val[\"subpattern\"])\n\n return pat\n\n if isinstance(val, list):\n self._known_author_patterns = [\n x if isinstance(x, AuthorPattern) else create_pat_from_dict(x)\n for x in val\n ] + self.known_author_patterns\n elif isinstance(val, AuthorPattern):\n self._known_author_patterns.insert(0, val)\n elif isinstance(val, dict):\n self._known_author_patterns.insert(0, create_pat_from_dict(val))\n else:\n raise Exception(\"Unknown type: {}. Use an AuthorPattern.\".format(type(val)))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_siblings_content(self, current_sibling, baselinescore_siblings_para):\n if current_sibling.tag == 'p' and self.parser.getText(current_sibling):\n tmp = current_sibling\n if tmp.tail:\n tmp = deepcopy(tmp)\n tmp.tail = ''\n return [tmp]\n else:\n potential_paragraphs = self.parser.getElementsByTag(current_sibling, tag='p')\n if potential_paragraphs is None:\n return None\n\n paragraphs = list()\n for first_paragraph in potential_paragraphs:\n text = self.parser.getText(first_paragraph)\n if text: # no len(text) > 0\n word_stats = self.stopwords_class(language=self.get_language()).get_stopword_count(text)\n paragraph_score = word_stats.get_stopword_count()\n sibling_baseline_score = float(.30)\n high_link_density = self.is_highlink_density(first_paragraph)\n score = float(baselinescore_siblings_para * sibling_baseline_score)\n if score < paragraph_score and not high_link_density:\n para = self.parser.createElement(tag='p', text=text, tail=None)\n paragraphs.append(para)\n return paragraphs", "response": "returns a list of paragraphs that are more than baselinescore_siblings_para"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef is_highlink_density(self, element):\n links = self.parser.getElementsByTag(element, tag='a')\n if not links:\n return False\n\n text = self.parser.getText(element)\n words = text.split(' ')\n words_number = float(len(words))\n link_text_parts = []\n for link in links:\n link_text_parts.append(self.parser.getText(link))\n\n link_text = ''.join(link_text_parts)\n link_words = link_text.split(' ')\n number_of_link_words = float(len(link_words))\n number_of_links = float(len(links))\n link_divisor = float(number_of_link_words / words_number)\n score = float(link_divisor * number_of_links)\n if score >= 1.0:\n return True\n return False", "response": "Checks the density of links within a node"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef post_cleanup(self):\n parse_tags = ['p']\n if self.config.parse_lists:\n parse_tags.extend(['ul', 'ol'])\n if self.config.parse_headers:\n parse_tags.extend(['h1', 'h2', 'h3', 'h4', 'h5', 'h6'])\n\n target_node = self.article.top_node\n node = self.add_siblings(target_node)\n for elm in self.parser.getChildren(node):\n e_tag = self.parser.getTag(elm)\n if e_tag not in parse_tags:\n if (self.is_highlink_density(elm) or self.is_table_and_no_para_exist(elm) or\n not self.is_nodescore_threshold_met(node, elm)):\n self.parser.remove(elm)\n return node", "response": "\\ Removes any divs that looks like non - content links or paras with no gusto ArcGIS or no gusto ArcGIS."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncleaning the title with the use of og : site_name and use TITLE_SPLITTERS to reformat it", "response": "def clean_title(self, title):\n \"\"\"Clean title with the use of og:site_name\n in this case try to get rid of site name\n and use TITLE_SPLITTERS to reformat title\n \"\"\"\n # check if we have the site name in opengraph data\n if \"site_name\" in list(self.article.opengraph.keys()):\n site_name = self.article.opengraph['site_name']\n # remove the site name from title\n title = title.replace(site_name, '').strip()\n elif (self.article.schema and \"publisher\" in self.article.schema and\n \"name\" in self.article.schema[\"publisher\"]):\n site_name = self.article.schema[\"publisher\"][\"name\"]\n # remove the site name from title\n title = title.replace(site_name, '').strip()\n\n # try to remove the domain from url\n if self.article.domain:\n pattern = re.compile(self.article.domain, re.IGNORECASE)\n title = pattern.sub(\"\", title).strip()\n\n # split the title in words\n # TechCrunch | my wonderfull article\n # my wonderfull article | TechCrunch\n title_words = title.split()\n\n # check if first letter is in TITLE_SPLITTERS\n # if so remove it\n if title_words and title_words[0] in TITLE_SPLITTERS:\n title_words.pop(0)\n\n # check for a title that is empty or consists of only a\n # title splitter to avoid a IndexError below\n if not title_words:\n return \"\"\n\n # check if last letter is in TITLE_SPLITTERS\n # if so remove it\n if title_words[-1] in TITLE_SPLITTERS:\n title_words.pop(-1)\n\n # rebuild the title\n title = \" \".join(title_words).strip()\n\n return title"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nfetch the article title and analyze it", "response": "def get_title(self):\n \"\"\"\\\n Fetch the article title and analyze it\n \"\"\"\n title = ''\n\n # rely on opengraph in case we have the data\n if \"title\" in list(self.article.opengraph.keys()):\n return self.clean_title(self.article.opengraph['title'])\n elif self.article.schema and \"headline\" in self.article.schema:\n return self.clean_title(self.article.schema['headline'])\n\n # try to fetch the meta headline\n meta_headline = self.parser.getElementsByTag(self.article.doc,\n tag=\"meta\",\n attr=\"name\",\n value=\"headline\")\n if meta_headline is not None and len(meta_headline) > 0:\n title = self.parser.getAttribute(meta_headline[0], 'content')\n return self.clean_title(title)\n\n # otherwise use the title meta\n title_element = self.parser.getElementsByTag(self.article.doc, tag='title')\n if title_element is not None and len(title_element) > 0:\n title = self.parser.getText(title_element[0])\n return self.clean_title(title)\n\n return title"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the canonical link for the article.", "response": "def get_canonical_link(self):\n \"\"\"\n if the article has meta canonical link set in the url\n \"\"\"\n if self.article.final_url:\n kwargs = {'tag': 'link', 'attr': 'rel', 'value': 'canonical'}\n meta = self.parser.getElementsByTag(self.article.doc, **kwargs)\n if meta is not None and len(meta) > 0:\n href = self.parser.getAttribute(meta[0], 'href')\n if href:\n href = href.strip()\n o = urlparse(href)\n if not o.hostname:\n tmp = urlparse(self.article.final_url)\n domain = '%s://%s' % (tmp.scheme, tmp.hostname)\n href = urljoin(domain, href)\n return href\n return self.article.final_url"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nmake any list element read like a list", "response": "def make_list_elms_pretty(self):\n \"\"\" make any list element read like a list\n \"\"\"\n for elm in self.parser.getElementsByTag(self.top_node, tag='li'):\n elm.text = r'\u2022 {}'.format(elm.text)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef close(self):\n ''' Close the network connection and perform any other required cleanup\n\n Note:\n Auto closed when using goose as a context manager or when garbage collected '''\n if self.fetcher is not None:\n self.shutdown_network()\n self.finalizer.atexit = False", "response": "Close the network connection and perform any other required cleanup\n "} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nextract the most likely article content from the html page and return the article content as a string.", "response": "def extract(self, url=None, raw_html=None):\n ''' Extract the most likely article content from the html page\n\n Args:\n url (str): URL to pull and parse\n raw_html (str): String representation of the HTML page\n Returns:\n Article: Representation of the article contents \\\n including other parsed and extracted metadata '''\n crawl_candidate = CrawlCandidate(self.config, url, raw_html)\n return self.__crawl(crawl_candidate)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef __crawl(self, crawl_candidate):\n ''' wrap the crawling functionality '''\n def crawler_wrapper(parser, parsers_lst, crawl_candidate):\n try:\n crawler = Crawler(self.config, self.fetcher)\n article = crawler.crawl(crawl_candidate)\n except (UnicodeDecodeError, ValueError) as ex:\n if parsers_lst:\n parser = parsers_lst.pop(0) # remove it also!\n return crawler_wrapper(parser, parsers_lst, crawl_candidate)\n else:\n raise ex\n return article\n\n # use the wrapper\n parsers = list(self.config.available_parsers)\n parsers.remove(self.config.parser_class)\n return crawler_wrapper(self.config.parser_class, parsers, crawl_candidate)", "response": "wrap the crawling functionality"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef smart_unicode(string, encoding='utf-8', strings_only=False, errors='strict'):\n # if isinstance(s, Promise):\n # # The input is the result of a gettext_lazy() call.\n # return s\n return force_unicode(string, encoding, strings_only, errors)", "response": "Returns a unicode object representing string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nforcing unicode string to Unicode.", "response": "def force_unicode(string, encoding='utf-8', strings_only=False, errors='strict'):\n \"\"\"\n Similar to smart_unicode, except that lazy instances are resolved to\n strings, rather than kept as lazy objects.\n\n If strings_only is True, don't convert (some) non-string-like objects.\n \"\"\"\n # Handle the common case first, saves 30-40% in performance when s\n # is an instance of unicode. This function gets called often in that\n # setting.\n if isinstance(string, str):\n return string\n if strings_only and is_protected_type(string):\n return string\n try:\n if not isinstance(string, str):\n if hasattr(string, '__unicode__'):\n string = string.__unicode__()\n else:\n try:\n string = str(string, encoding, errors)\n except UnicodeEncodeError:\n if not isinstance(string, Exception):\n raise\n # If we get to here, the caller has passed in an Exception\n # subclass populated with non-ASCII data without special\n # handling to display as a string. We need to handle this\n # without raising a further exception. We do an\n # approximation to what the Exception's standard str()\n # output should be.\n string = ' '.join([force_unicode(arg, encoding,\n strings_only,\n errors) for arg in string])\n elif not isinstance(string, str):\n # Note: We use .decode() here, instead of unicode(s, encoding,\n # errors), so that if s is a SafeString, it ends up being a\n # SafeUnicode at the end.\n string = string.decode(encoding, errors)\n except UnicodeDecodeError as ex:\n if not isinstance(string, Exception):\n raise DjangoUnicodeDecodeError(string, *ex.args)\n else:\n # If we get to here, the caller has passed in an Exception\n # subclass populated with non-ASCII bytestring data without a\n # working unicode method. Try to handle this without raising a\n # further exception by individually forcing the exception args\n # to unicode.\n string = ' '.join([force_unicode(arg, encoding, strings_only,\n errors) for arg in string])\n return string"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef smart_str(string, encoding='utf-8', strings_only=False, errors='strict'):\n if strings_only and isinstance(string, (type(None), int)):\n return string\n # if isinstance(s, Promise):\n # return unicode(s).encode(encoding, errors)\n if isinstance(string, str):\n try:\n return string.encode(encoding, errors)\n except UnicodeEncodeError:\n return string.encode('utf-8', errors)\n elif not isinstance(string, bytes):\n try:\n return str(string).encode(encoding, errors)\n except UnicodeEncodeError:\n if isinstance(string, Exception):\n # An Exception subclass containing non-ASCII data that doesn't\n # know how to print itself properly. We shouldn't raise a\n # further exception.\n return ' '.join([smart_str(arg, encoding, strings_only,\n errors) for arg in string])\n return str(string).encode(encoding, errors)\n else:\n return string", "response": "Returns a bytestring version of s encoded as specified in encoding."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_urls(self):\n urls = patterns(\n '',\n url(r'^upload/$', self.admin_site.admin_view(self.handle_upload), name='quill-file-upload'),\n )\n return urls + super(QuillAdmin, self).get_urls()", "response": "Add URLs needed to handle image uploads."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef handle_upload(self, request):\n if request.method != 'POST':\n raise Http404\n\n if request.is_ajax():\n try:\n filename = request.GET['quillUploadFile']\n data = request\n is_raw = True\n except KeyError:\n return HttpResponseBadRequest(\"Invalid file upload.\")\n else:\n if len(request.FILES) != 1:\n return HttpResponseBadRequest(\"Can only upload 1 file at a time.\")\n try:\n data = request.FILES['quillUploadFile']\n filename = data.name\n is_raw = False\n except KeyError:\n return HttpResponseBadRequest('Missing image `quillUploadFile`.')\n\n url = save_file(data, filename, is_raw, default_storage)\n response_data = {}\n response_data['url'] = url\n\n # Response content type needs to be text/html here or else\n # IE will try to download the file.\n return HttpResponse(json.dumps(response_data), content_type=\"text/html; charset=utf-8\")", "response": "Handle file uploads from WYSIWYG."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef render(self, name, value, attrs={}):\n if value is None:\n value = ''\n final_attrs = self.build_attrs(attrs, name=name)\n quill_app = apps.get_app_config('quill')\n quill_config = getattr(quill_app, self.config)\n\n return mark_safe(render_to_string(quill_config['template'], {\n 'final_attrs': flatatt(final_attrs),\n 'value': value,\n 'id': final_attrs['id'],\n 'config': self.config,\n }))", "response": "Render the Quill WYSIWYG."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef formfield(self, **kwargs):\n defaults = {\n 'form_class': RichTextFormField,\n 'config': self.config,\n }\n defaults.update(kwargs)\n return super(RichTextField, self).formfield(**defaults)", "response": "Get the form for field."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrender the toolbar for the given config.", "response": "def render_toolbar(context, config):\n \"\"\"Render the toolbar for the given config.\"\"\"\n quill_config = getattr(quill_app, config)\n t = template.loader.get_template(quill_config['toolbar_template'])\n return t.render(context)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef read(self, filename=None):\n self._init_filename(filename)\n\n def BLANK(i):\n return \"B{0:04d}\".format(i)\n def COMMENT(i):\n return \"C{0:04d}\".format(i)\n\n data = odict()\n iblank = icomment = 0\n with open(self.real_filename) as mdp:\n for line in mdp:\n line = line.strip()\n if len(line) == 0:\n iblank += 1\n data[BLANK(iblank)] = ''\n continue\n m = self.COMMENT.match(line)\n if m:\n icomment += 1\n data[COMMENT(icomment)] = m.group('value')\n continue\n # parameter\n m = self.PARAMETER.match(line)\n if m:\n # check for comments after parameter?? -- currently discarded\n parameter = m.group('parameter')\n value = self._transform(m.group('value'))\n data[parameter] = value\n else:\n errmsg = '{filename!r}: unknown line in mdp file, {line!r}'.format(**vars())\n self.logger.error(errmsg)\n raise ParseError(errmsg)\n\n super(MDP,self).update(data)", "response": "Read and parse mdp file filename."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting the dictionary to the specified mdp file.", "response": "def write(self, filename=None, skipempty=False):\n \"\"\"Write mdp file to *filename*.\n\n :Keywords:\n *filename*\n output mdp file; default is the filename the mdp\n was read from\n *skipempty* : boolean\n ``True`` removes any parameter lines from output that\n contain empty values [``False``]\n\n .. Note:: Overwrites the file that the mdp was read from if no\n *filename* supplied.\n \"\"\"\n\n with open(self.filename(filename, ext='mdp'), 'w') as mdp:\n for k,v in self.items():\n if k[0] == 'B': # blank line\n mdp.write(\"\\n\")\n elif k[0] == 'C': # comment\n mdp.write(\"; {v!s}\\n\".format(**vars()))\n else: # parameter = value\n if skipempty and (v == '' or v is None):\n continue\n if isinstance(v, six.string_types) or not hasattr(v, '__iter__'):\n mdp.write(\"{k!s} = {v!s}\\n\".format(**vars()))\n else:\n mdp.write(\"{} = {}\\n\".format(k,' '.join(map(str, v))))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding the first Gromacs command that can be found on the PATH.", "response": "def find_gromacs_command(commands):\n \"\"\"Return *driver* and *name* of the first command that can be found on :envvar:`PATH`\"\"\"\n\n # We could try executing 'name' or 'driver name' but to keep things lean we\n # just check if the executables can be found and then hope for the best.\n\n commands = utilities.asiterable(commands)\n for command in commands:\n try:\n driver, name = command.split()\n except ValueError:\n driver, name = None, command\n\n executable = driver if driver else name\n if utilities.which(executable):\n break\n else:\n raise OSError(errno.ENOENT, \"No Gromacs executable found in\", \", \".join(commands))\n\n return driver, name"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef check_mdrun_success(logfile):\n if not os.path.exists(logfile):\n return None\n with open(logfile, 'rb') as log:\n log.seek(-1024, 2)\n for line in log:\n line = line.decode('ASCII')\n if line.startswith(\"Finished mdrun on\"):\n return True\n return False", "response": "Check if mdrun finished successfully."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn double precision mdrun or single precision gromacs. mdrun_d if it cannot run.", "response": "def get_double_or_single_prec_mdrun():\n \"\"\"Return double precision ``mdrun`` or fall back to single precision.\n\n This convenience function tries :func:`gromacs.mdrun_d` first and\n if it cannot run it, falls back to :func:`gromacs.mdrun` (without\n further checking).\n\n .. versionadded:: 0.5.1\n \"\"\"\n try:\n gromacs.mdrun_d(h=True, stdout=False, stderr=False)\n logger.debug(\"using double precision gromacs.mdrun_d\")\n return gromacs.mdrun_d\n except (AttributeError, GromacsError, OSError):\n # fall back to mdrun if no double precision binary\n wmsg = \"No 'mdrun_d' binary found so trying 'mdrun' instead.\\n\"\\\n \"(Note that energy minimization runs better with mdrun_d.)\"\n logger.warn(wmsg)\n warnings.warn(wmsg, category=AutoCorrectionWarning)\n return gromacs.mdrun"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a simple command line to invoke the mdrun launcher.", "response": "def commandline(self, **mpiargs):\n \"\"\"Returns simple command line to invoke mdrun.\n\n If :attr:`mpiexec` is set then :meth:`mpicommand` provides the mpi\n launcher command that prefixes the actual ``mdrun`` invocation:\n\n :attr:`mpiexec` [*mpiargs*] :attr:`mdrun` [*mdrun-args*]\n\n The *mdrun-args* are set on initializing the class. Override\n :meth:`mpicommand` to fit your system if the simple default\n OpenMP launcher is not appropriate.\n \"\"\"\n cmd = self.MDRUN.commandline()\n if self.mpiexec:\n cmd = self.mpicommand(**mpiargs) + cmd\n return cmd"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the mpi command portion of the commandline.", "response": "def mpicommand(self, *args, **kwargs):\n \"\"\"Return a list of the mpi command portion of the commandline.\n\n Only allows primitive mpi at the moment:\n *mpiexec* -n *ncores* *mdrun* *mdrun-args*\n\n (This is a primitive example for OpenMP. Override it for more\n complicated cases.)\n \"\"\"\n if self.mpiexec is None:\n raise NotImplementedError(\"Override mpiexec to enable the simple OpenMP launcher\")\n # example implementation\n ncores = kwargs.pop('ncores', 8)\n return [self.mpiexec, '-n', str(ncores)]"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexecutes the MDrun command and return the exit code.", "response": "def run(self, pre=None, post=None, mdrunargs=None, **mpiargs):\n \"\"\"Execute the mdrun command (possibly as a MPI command) and run the simulation.\n\n :Keywords:\n *pre*\n a dictionary containing keyword arguments for the :meth:`prehook`\n *post*\n a dictionary containing keyword arguments for the :meth:`posthook`\n *mdrunargs*\n a dictionary with keyword arguments for :program:`mdrun` which supersede\n **and update** the defaults given to the class constructor\n *mpiargs*\n all other keyword arguments that are processed by :meth:`mpicommand`\n \"\"\"\n\n if pre is None:\n pre = {}\n if post is None:\n post = {}\n if mdrunargs is not None:\n try:\n self.MDRUN.gmxargs.update(mdrunargs)\n except (ValueError, TypeError):\n msg = \"mdrunargs must be a dict of mdrun options, not {0}\".format(mdrunargs)\n logger.error(msg)\n raise\n\n cmd = self.commandline(**mpiargs)\n\n with utilities.in_dir(self.dirname, create=False):\n try:\n self.prehook(**pre)\n logger.info(\" \".join(cmd))\n rc = subprocess.call(cmd)\n except:\n logger.exception(\"Failed MD run for unknown reasons.\")\n raise\n finally:\n self.posthook(**post)\n if rc == 0:\n logger.info(\"MDrun completed ok, returncode = {0:d}\".format(rc))\n else:\n logger.critical(\"Failure in MDrun, returncode = {0:d}\".format(rc))\n return rc"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nrunning the mdrun and check if it completes.", "response": "def run_check(self, **kwargs):\n \"\"\"Run :program:`mdrun` and check if run completed when it finishes.\n\n This works by looking at the mdrun log file for 'Finished\n mdrun on node'. It is useful to implement robust simulation\n techniques.\n\n :Arguments:\n *kwargs* are keyword arguments that are passed on to\n :meth:`run` (typically used for mpi things)\n\n :Returns:\n - ``True`` if run completed successfully\n - ``False`` otherwise\n \"\"\"\n rc = None # set to something in case we ever want to look at it later (and bomb in the try block)\n try:\n rc = self.run(**kwargs)\n except:\n logger.exception(\"run_check: caught exception\")\n status = self.check_success()\n if status:\n logger.info(\"run_check: Hooray! mdrun finished successfully\")\n else:\n logger.error(\"run_check: mdrun failed to complete run\")\n return status"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndefine functions for the top level name space.", "response": "def _define_canned_commands():\n \"\"\"Define functions for the top level name space.\n\n Definitions are collected here so that they can all be wrapped in\n a try-except block that avoids code failing when the Gromacs tools\n are not available --- in some cases they are not necessary to use\n parts of GromacsWrapper.\n\n .. Note:: Any function defined here **must be listed in ``global``**!\n \"\"\"\n global trj_compact, rmsd_backbone, trj_fitted, trj_xyfitted\n\n trj_compact = tools.Trjconv(ur='compact', center=True, boxcenter='tric', pbc='mol',\n input=('protein','system'),\n doc=\"\"\"\nWrites a compact representation of the system centered on the protein\"\"\")\n\n rmsd_backbone = tools.G_rms(what='rmsd', fit='rot+trans',\n input=('Backbone','Backbone'),\n doc=\"\"\"\nComputes RMSD of backbone after fitting to the backbone.\"\"\")\n\n trj_fitted = tools.Trjconv(fit='rot+trans',\n input=('backbone', 'system'),\n doc=\"\"\"\nWrites a trajectory fitted to the protein backbone.\n\nNote that this does *not* center; if center is required, the *input*\nselection should have the group to be centered on in second position,\ne.g. ``input = ('backbone', 'Protein', System')``.\n\"\"\")\n\n\n # Gromacs 4.x\n trj_xyfitted = tools.Trjconv(fit='rotxy+transxy',\n input=('backbone', 'protein','system'),\n doc=\"\"\"\nWrites a trajectory fitted to the protein in the XY-plane only.\n\nThis is useful for membrane proteins. The system *must* be oriented so\nthat the membrane is in the XY plane. The protein backbone is used\nfor the least square fit, centering is done for the whole protein.\n\nNote that centering together with fitting does not always work well\nand that one sometimes need two runs of trjconv: one to center and\none to fit.\n\n.. Note:: Gromacs 4.x only\"\"\")"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef trj_fitandcenter(xy=False, **kwargs):\n if xy:\n fitmode = 'rotxy+transxy'\n kwargs.pop('fit', None)\n else:\n fitmode = kwargs.pop('fit', 'rot+trans') # user can use progressive, too\n\n intrj = kwargs.pop('f', None)\n # get the correct suffix for the intermediate step: only trr will\n # keep velocities/forces!\n suffix = os.path.splitext(intrj)[1]\n if not suffix in ('xtc', 'trr'):\n suffix = '.xtc'\n outtrj = kwargs.pop('o', None)\n\n ndx = kwargs.pop('n', None)\n ndxcompact = kwargs.pop('n1', ndx)\n\n structures = kwargs.pop('s', None)\n if type(structures) in (tuple, list):\n try:\n compact_structure, fit_structure = structures\n except:\n raise ValueError(\"argument s must be a pair of tpr/pdb files or a single structure file\")\n else:\n compact_structure = fit_structure = structures\n\n\n inpfit = kwargs.pop('input', ('backbone', 'protein','system'))\n try:\n _inpcompact = inpfit[1:] # use 2nd and 3rd group for compact\n except TypeError:\n _inpcompact = None\n inpcompact = kwargs.pop('input1', _inpcompact) # ... or the user supplied ones\n\n fd, tmptrj = tempfile.mkstemp(suffix=suffix, prefix='pbc_compact_')\n\n logger.info(\"Input structure for PBC: {compact_structure!r}\".format(**vars()))\n logger.info(\"Input structure for fit: {fit_structure!r}\".format(**vars()))\n logger.info(\"Input trajectory: {intrj!r}\".format(**vars()))\n logger.info(\"Output trajectory: {outtrj!r}\".format(**vars()))\n logger.debug(\"Writing temporary trajectory {tmptrj!r} (will be auto-cleaned).\".format(**vars()))\n sys.stdout.flush()\n try:\n gromacs.trjconv(s=compact_structure, f=intrj, o=tmptrj, n=ndxcompact,\n ur='compact', center=True, boxcenter='tric', pbc='mol',\n input=inpcompact, **kwargs)\n # explicitly set pbc=\"none\" for the fitting stage (anything else will produce rubbish and/or\n # complaints from Gromacs)\n kwargs['pbc'] = \"none\"\n if compact_structure == fit_structure:\n # fit as ususal, including centering\n # (Is center=True really necessary? -- note, if I remove center=True then\n # I MUST fiddle inpfit as below!!)\n gromacs.trjconv(s=fit_structure, f=tmptrj, o=outtrj, n=ndx, fit=fitmode, center=True, input=inpfit, **kwargs)\n else:\n # make sure that we fit EXACTLY as the user wants\n inpfit = [inpfit[0], inpfit[-1]]\n gromacs.trjconv(s=fit_structure, f=tmptrj, o=outtrj, n=ndx, fit=fitmode, input=inpfit, **kwargs)\n finally:\n utilities.unlink_gmx(tmptrj)", "response": "This function takes a list of structure files and fits the system to a compact representation and creates a compact representation of the system and centers it with the system."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef cat(prefix=\"md\", dirname=os.path.curdir, partsdir=\"parts\", fulldir=\"full\",\n resolve_multi=\"pass\"):\n \"\"\"Concatenate all parts of a simulation.\n\n The xtc, trr, and edr files in *dirname* such as prefix.xtc,\n prefix.part0002.xtc, prefix.part0003.xtc, ... are\n\n 1) moved to the *partsdir* (under *dirname*)\n 2) concatenated with the Gromacs tools to yield prefix.xtc, prefix.trr,\n prefix.edr, prefix.gro (or prefix.md) in *dirname*\n 3) Store these trajectories in *fulldir*\n\n .. Note:: Trajectory files are *never* deleted by this function to avoid\n data loss in case of bugs. You will have to clean up yourself\n by deleting *dirname*/*partsdir*.\n\n Symlinks for the trajectories are *not* handled well and\n break the function. Use hard links instead.\n\n .. Warning:: If an exception occurs when running this function then make\n doubly and triply sure where your files are before running\n this function again; otherwise you might **overwrite data**.\n Possibly you will need to manually move the files from *partsdir*\n back into the working directory *dirname*; this should onlu overwrite\n generated files so far but *check carefully*!\n\n :Keywords:\n *prefix*\n deffnm of the trajectories [md]\n *resolve_multi\"\n how to deal with multiple \"final\" gro or pdb files: normally there should\n only be one but in case of restarting from the checkpoint of a finished\n simulation one can end up with multiple identical ones.\n - \"pass\" : do nothing and log a warning\n - \"guess\" : take prefix.pdb or prefix.gro if it exists, otherwise the one of\n prefix.partNNNN.gro|pdb with the highes NNNN\n *dirname*\n change to *dirname* and assume all tarjectories are located there [.]\n *partsdir*\n directory where to store the input files (they are moved out of the way);\n *partsdir* must be manually deleted [parts]\n *fulldir*\n directory where to store the final results [full]\n \"\"\"\n\n gmxcat = {'xtc': gromacs.trjcat,\n 'trr': gromacs.trjcat,\n 'edr': gromacs.eneconv,\n 'log': utilities.cat,\n }\n\n def _cat(prefix, ext, partsdir=partsdir, fulldir=fulldir):\n filenames = glob_parts(prefix, ext)\n if ext.startswith('.'):\n ext = ext[1:]\n outfile = os.path.join(fulldir, prefix + '.' + ext)\n if not filenames:\n return None\n nonempty_files = []\n for f in filenames:\n if os.stat(f).st_size == 0:\n logger.warn(\"File {f!r} is empty, skipping\".format(**vars()))\n continue\n if os.path.islink(f):\n # TODO: re-write the symlink to point to the original file\n errmsg = \"Symbolic links do not work (file %(f)r), sorry. \" \\\n \"CHECK LOCATION OF FILES MANUALLY BEFORE RUNNING gromacs.cbook.cat() AGAIN!\" % vars()\n logger.exception(errmsg)\n raise NotImplementedError(errmsg)\n shutil.move(f, partsdir)\n nonempty_files.append(f)\n filepaths = [os.path.join(partsdir, f) for f in nonempty_files]\n gmxcat[ext](f=filepaths, o=outfile)\n return outfile\n\n _resolve_options = (\"pass\", \"guess\")\n if not resolve_multi in _resolve_options:\n raise ValueError(\"resolve_multi must be one of %(_resolve_options)r, \"\n \"not %(resolve_multi)r\" % vars())\n\n if fulldir == os.path.curdir:\n wmsg = \"Using the current directory as fulldir can potentially lead to data loss if you run this function multiple times.\"\n logger.warning(wmsg)\n warnings.warn(wmsg, category=BadParameterWarning)\n\n with utilities.in_dir(dirname, create=False):\n utilities.mkdir_p(partsdir)\n utilities.mkdir_p(fulldir)\n for ext in ('log', 'edr', 'trr', 'xtc'):\n logger.info(\"[%(dirname)s] concatenating %(ext)s files...\", vars())\n outfile = _cat(prefix, ext, partsdir)\n logger.info(\"[%(dirname)s] created %(outfile)r\", vars())\n for ext in ('gro', 'pdb'): # XXX: ugly, make method out of parts?\n filenames = glob_parts(prefix, ext)\n if len(filenames) == 0:\n continue # goto next ext\n elif len(filenames) == 1:\n pick = filenames[0]\n else:\n if resolve_multi == \"pass\":\n logger.warning(\"[%(dirname)s] too many output structures %(filenames)r, \"\n \"cannot decide which one --- resolve manually!\", vars())\n for f in filenames:\n shutil.move(f, partsdir)\n continue # goto next ext\n elif resolve_multi == \"guess\":\n pick = prefix + '.' + ext\n if not pick in filenames:\n pick = filenames[-1] # filenames are ordered with highest parts at end\n final = os.path.join(fulldir, prefix + '.' + ext)\n shutil.copy(pick, final) # copy2 fails on nfs with Darwin at least\n for f in filenames:\n shutil.move(f, partsdir)\n logger.info(\"[%(dirname)s] collected final structure %(final)r \"\n \"(from %(pick)r)\", vars())\n\n\n partsdirpath = utilities.realpath(dirname, partsdir)\n logger.warn(\"[%(dirname)s] cat() complete in %(fulldir)r but original files \"\n \"in %(partsdirpath)r must be manually removed\", vars())", "response": "Concatenate all parts of a simulation into a single file."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfinds files from a continuation run", "response": "def glob_parts(prefix, ext):\n \"\"\"Find files from a continuation run\"\"\"\n if ext.startswith('.'):\n ext = ext[1:]\n files = glob.glob(prefix+'.'+ext) + glob.glob(prefix+'.part[0-9][0-9][0-9][0-9].'+ext)\n files.sort() # at least some rough sorting...\n return files"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef grompp_qtot(*args, **kwargs):\n qtot_pattern = re.compile('System has non-zero total charge: *(?P[-+]?\\d*\\.\\d+([eE][-+]\\d+)?)')\n # make sure to capture ALL output\n kwargs['stdout'] = False\n kwargs['stderr'] = False\n rc, output, error = grompp_warnonly(*args, **kwargs)\n gmxoutput = \"\\n\".join([x for x in [output, error] if x is not None])\n if rc != 0:\n # error occured and we want to see the whole output for debugging\n msg = \"grompp_qtot() failed. See warning and screen output for clues.\"\n logger.error(msg)\n import sys\n sys.stderr.write(\"=========== grompp (stdout/stderr) ============\\n\")\n sys.stderr.write(gmxoutput)\n sys.stderr.write(\"===============================================\\n\")\n sys.stderr.flush()\n raise GromacsError(rc, msg)\n qtot = 0\n for line in gmxoutput.split('\\n'):\n m = qtot_pattern.search(line)\n if m:\n qtot = float(m.group('qtot'))\n break\n logger.info(\"system total charge qtot = {qtot!r}\".format(**vars()))\n return qtot", "response": "Run gromacs. grompp and return the total charge of the system."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _mdp_include_string(dirs):\n include_paths = [os.path.expanduser(p) for p in dirs]\n return ' -I'.join([''] + include_paths)", "response": "Generate a string that can be added to a mdp include line."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef add_mdp_includes(topology=None, kwargs=None):\n if kwargs is None:\n kwargs = {}\n\n include_dirs = ['.', '..'] # should . & .. always be added?\n if topology is not None:\n # half-hack: find additional itps in the same directory as the topology\n topology_dir = os.path.dirname(topology)\n include_dirs.append(topology_dir)\n\n include_dirs.extend(asiterable(kwargs.pop('includes', []))) # includes can be a list or a string\n\n # 1. setdefault: we do nothing if user defined include\n # 2. modify input in place!\n kwargs.setdefault('include', _mdp_include_string(include_dirs))\n return kwargs", "response": "This function adds the mdp include files to the top - level file that are included in the current directory."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef filter_grompp_options(**kwargs):\n grompp_options = ('f','po','c','r','rb','n','p','pp','o','t','e', # files\n 'h', 'noh', 'version', 'noversion', 'nice', 'v', 'nov',\n 'time', 'rmvsbds', 'normvsbds', 'maxwarn', 'zero', 'nozero',\n 'renum', 'norenum')\n grompp = dict((k,v) for k,v in kwargs.items() if k in grompp_options)\n other = dict((k,v) for k,v in kwargs.items() if k not in grompp_options)\n return grompp, other", "response": "Returns one dictionary only containing valid grompp options and everything else."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a portable topology file.", "response": "def create_portable_topology(topol, struct, **kwargs):\n \"\"\"Create a processed topology.\n\n The processed (or portable) topology file does not contain any\n ``#include`` statements and hence can be easily copied around. It\n also makes it possible to re-grompp without having any special itp\n files available.\n\n :Arguments:\n *topol*\n topology file\n *struct*\n coordinat (structure) file\n\n :Keywords:\n *processed*\n name of the new topology file; if not set then it is named like\n *topol* but with ``pp_`` prepended\n *includes*\n path or list of paths of directories in which itp files are\n searched for\n *grompp_kwargs**\n other options for :program:`grompp` such as ``maxwarn=2`` can\n also be supplied\n\n :Returns: full path to the processed topology\n \"\"\"\n _topoldir, _topol = os.path.split(topol)\n processed = kwargs.pop('processed', os.path.join(_topoldir, 'pp_'+_topol))\n grompp_kwargs, mdp_kwargs = filter_grompp_options(**kwargs)\n mdp_kwargs = add_mdp_includes(topol, mdp_kwargs)\n with tempfile.NamedTemporaryFile(suffix='.mdp') as mdp:\n mdp.write('; empty mdp file\\ninclude = {include!s}\\n'.format(**mdp_kwargs))\n mdp.flush()\n grompp_kwargs['p'] = topol\n grompp_kwargs['pp'] = processed\n grompp_kwargs['f'] = mdp.name\n grompp_kwargs['c'] = struct\n grompp_kwargs['v'] = False\n try:\n gromacs.grompp(**grompp_kwargs)\n finally:\n utilities.unlink_gmx('topol.tpr', 'mdout.mdp')\n return utilities.realpath(processed)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_volume(f):\n fd, temp = tempfile.mkstemp('.gro')\n try:\n rc,out,err = gromacs.editconf(f=f, o=temp, stdout=False)\n finally:\n os.unlink(temp)\n return [float(x.split()[1]) for x in out.splitlines()\n if x.startswith('Volume:')][0]", "response": "Return the volume in nm^3 of structure file f."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nedits values in a Gromacs mdp file.", "response": "def edit_mdp(mdp, new_mdp=None, extend_parameters=None, **substitutions):\n \"\"\"Change values in a Gromacs mdp file.\n\n Parameters and values are supplied as substitutions, eg ``nsteps=1000``.\n\n By default the template mdp file is **overwritten in place**.\n\n If a parameter does not exist in the template then it cannot be substituted\n and the parameter/value pair is returned. The user has to check the\n returned list in order to make sure that everything worked as expected. At\n the moment it is not possible to automatically append the new values to the\n mdp file because of ambiguities when having to replace dashes in parameter\n names with underscores (see the notes below on dashes/underscores).\n\n If a parameter is set to the value ``None`` then it will be ignored.\n\n :Arguments:\n *mdp* : filename\n filename of input (and output filename of ``new_mdp=None``)\n *new_mdp* : filename\n filename of alternative output mdp file [None]\n *extend_parameters* : string or list of strings\n single parameter or list of parameters for which the new values\n should be appended to the existing value in the mdp file. This\n makes mostly sense for a single parameter, namely 'include', which\n is set as the default. Set to ``[]`` to disable. ['include']\n *substitutions*\n parameter=value pairs, where parameter is defined by the Gromacs\n mdp file; dashes in parameter names have to be replaced by\n underscores. If a value is a list-like object then the items are\n written as a sequence, joined with spaces, e.g. ::\n\n ref_t=[310,310,310] ---> ref_t = 310 310 310\n\n :Returns:\n Dict of parameters that have *not* been substituted.\n\n **Example** ::\n\n edit_mdp('md.mdp', new_mdp='long_md.mdp', nsteps=100000, nstxtcout=1000, lincs_iter=2)\n\n .. Note::\n\n * Dashes in Gromacs mdp parameters have to be replaced by an underscore\n when supplied as python keyword arguments (a limitation of python). For example\n the MDP syntax is ``lincs-iter = 4`` but the corresponding keyword would be\n ``lincs_iter = 4``.\n * If the keyword is set as a dict key, eg ``mdp_params['lincs-iter']=4`` then one\n does not have to substitute.\n * Parameters *aa_bb* and *aa-bb* are considered the same (although this should\n not be a problem in practice because there are no mdp parameters that only\n differ by a underscore).\n * This code is more compact in ``Perl`` as one can use ``s///`` operators:\n ``s/^(\\s*${key}\\s*=\\s*).*/$1${val}/``\n\n .. SeeAlso:: One can also load the mdp file with\n :class:`gromacs.formats.MDP`, edit the object (a dict), and save it again.\n \"\"\"\n if new_mdp is None:\n new_mdp = mdp\n if extend_parameters is None:\n extend_parameters = ['include']\n else:\n extend_parameters = list(asiterable(extend_parameters))\n\n # None parameters should be ignored (simple way to keep the template defaults)\n substitutions = {k: v for k,v in substitutions.items() if v is not None}\n\n params = list(substitutions.keys()) # list will be reduced for each match\n\n def demangled(p):\n \"\"\"Return a RE string that matches the parameter.\"\"\"\n return p.replace('_', '[-_]') # must catch either - or _\n\n patterns = {parameter:\n re.compile(\"\"\"\\\n (?P\\s*{0!s}\\s*=\\s*) # parameter == everything before the value\n (?P[^;]*) # value (stop before comment=;)\n (?P\\s*;.*)? # optional comment\n \"\"\".format(demangled(parameter)), re.VERBOSE)\n for parameter in substitutions}\n\n with tempfile.TemporaryFile() as target:\n with open(mdp, 'rb') as src:\n logger.info(\"editing mdp = {0!r}: {1!r}\".format(mdp, substitutions.keys()))\n for line in src:\n line = line.decode('utf-8')\n new_line = line.strip() # \\n must be stripped to ensure that new line is built without break\n for p in params[:]:\n m = patterns[p].match(new_line)\n if m:\n # I am too stupid to replace a specific region in the string so I rebuild it\n # (matching a line and then replacing value requires TWO re calls)\n #print 'line:' + new_line\n #print m.groupdict()\n if m.group('comment') is None:\n comment = ''\n else:\n comment = \" \"+m.group('comment')\n assignment = m.group('assignment')\n if not assignment.endswith(' '):\n assignment += ' '\n # build new line piece-wise:\n new_line = assignment\n if p in extend_parameters:\n # keep original value and add new stuff at end\n new_line += str(m.group('value')) + ' '\n # automatically transform lists into space-separated string values\n value = \" \".join(map(str, asiterable(substitutions[p])))\n new_line += value + comment\n params.remove(p)\n break\n target.write((new_line+'\\n').encode('utf-8'))\n target.seek(0)\n # XXX: Is there a danger of corrupting the original mdp if something went wrong?\n with open(new_mdp, 'wb') as final:\n shutil.copyfileobj(target, final)\n\n # return all parameters that have NOT been substituted\n if len(params) > 0:\n logger.warn(\"Not substituted in {new_mdp!r}: {params!r}\".format(**vars()))\n return {p: substitutions[p] for p in params}"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef edit_txt(filename, substitutions, newname=None):\n if newname is None:\n newname = filename\n\n # No sanity checks (figure out later how to give decent diagnostics).\n # Filter out any rules that have None in replacement.\n _substitutions = [{'lRE': re.compile(str(lRE)),\n 'sRE': re.compile(str(sRE)),\n 'repl': repl}\n for lRE,sRE,repl in substitutions if repl is not None]\n\n with tempfile.TemporaryFile() as target:\n with open(filename, 'rb') as src:\n logger.info(\"editing txt = {0!r} ({1:d} substitutions)\".format(filename, len(substitutions)))\n for line in src:\n line = line.decode(\"utf-8\")\n keep_line = True\n for subst in _substitutions:\n m = subst['lRE'].match(line)\n if m: # apply substition to this line?\n logger.debug('match: '+line.rstrip())\n if subst['repl'] is False: # special rule: delete line\n keep_line = False\n else: # standard replacement\n line = subst['sRE'].sub(str(subst['repl']), line)\n logger.debug('replaced: '+line.rstrip())\n if keep_line:\n target.write(line.encode('utf-8'))\n else:\n logger.debug(\"Deleting line %r\", line)\n\n target.seek(0)\n with open(newname, 'wb') as final:\n shutil.copyfileobj(target, final)\n logger.info(\"edited txt = {newname!r}\".format(**vars()))", "response": "Primitive text file stream editor."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef remove_molecules_from_topology(filename, **kwargs):\n marker = kwargs.pop('marker', None)\n if marker is None:\n marker = \"; Gromacs auto-generated entries follow:\"\n logger.debug(\"Scrubbed [ molecules ]: marker = %(marker)r\", vars())\n\n p_marker = re.compile(\"\\s*{0!s}\".format(marker))\n p_molecule = re.compile(\"\\s*[\\w+_-]+\\s+\\d+\\s*(;.*)?$\")\n with tempfile.TemporaryFile() as target:\n with open(filename, 'rb') as src:\n autogenerated = False\n n_removed = 0\n for line in src:\n line = line.decode('utf-8')\n if p_marker.match(line):\n autogenerated = True\n if autogenerated and p_molecule.match(line):\n n_removed += 1\n continue # remove by skipping\n target.write(line.encode('utf-8'))\n if autogenerated and n_removed > 0:\n target.seek(0)\n with open(filename, 'wb') as final: # overwrite original!\n shutil.copyfileobj(target, final)\n logger.info(\"Removed %(n_removed)d autogenerated [ molecules ] from \"\n \"topol = %(filename)r\" % vars())\n return n_removed", "response": "Remove molecules from a topology file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef make_ndx_captured(**kwargs):\n kwargs['stdout']=False # required for proper output as described in doc\n user_input = kwargs.pop('input',[])\n user_input = [cmd for cmd in user_input if cmd != 'q'] # filter any quit\n kwargs['input'] = user_input + ['', 'q'] # necessary commands\n return gromacs.make_ndx(**kwargs)", "response": "This function will take a list of n - dimensional items and return a new list of n - dimensional items."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn a list of index groups in the index file ndx.", "response": "def get_ndx_groups(ndx, **kwargs):\n \"\"\"Return a list of index groups in the index file *ndx*.\n\n :Arguments:\n - *ndx* is a Gromacs index file.\n - kwargs are passed to :func:`make_ndx_captured`.\n\n :Returns:\n list of groups as supplied by :func:`parse_ndxlist`\n\n Alternatively, load the index file with\n :class:`gromacs.formats.NDX` for full control.\n \"\"\"\n fd, tmp_ndx = tempfile.mkstemp(suffix='.ndx')\n kwargs['o'] = tmp_ndx\n try:\n g = parse_ndxlist(make_ndx_captured(n=ndx, **kwargs)[1])\n finally:\n utilities.unlink_gmx(tmp_ndx)\n return g"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nparse output from make_ndx to build list of index groups", "response": "def parse_ndxlist(output):\n \"\"\"Parse output from make_ndx to build list of index groups::\n\n groups = parse_ndxlist(output)\n\n output should be the standard output from ``make_ndx``, e.g.::\n\n rc,output,junk = gromacs.make_ndx(..., input=('', 'q'), stdout=False, stderr=True)\n\n (or simply use\n\n rc,output,junk = cbook.make_ndx_captured(...)\n\n which presets input, stdout and stderr; of course input can be overriden.)\n\n :Returns:\n The function returns a list of dicts (``groups``) with fields\n\n name\n name of the groups\n nr\n number of the group (starts at 0)\n natoms\n number of atoms in the group\n\n \"\"\"\n\n m = NDXLIST.search(output) # make sure we pick up a proper full list\n grouplist = m.group('LIST')\n return parse_groups(grouplist)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef parse_groups(output):\n groups = []\n for line in output.split('\\n'):\n m = NDXGROUP.match(line)\n if m:\n d = m.groupdict()\n groups.append({'name': d['GROUPNAME'],\n 'nr': int(d['GROUPNUMBER']),\n 'natoms': int(d['NATOMS'])})\n return groups", "response": "Parse make_ndx output and return groups as a list of dicts."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef gmx_resid(self, resid):\n try:\n gmx_resid = int(self.offset[resid])\n except (TypeError, IndexError):\n gmx_resid = resid + self.offset\n except KeyError:\n raise KeyError(\"offset must be a dict that contains the gmx resid for {0:d}\".format(resid))\n return gmx_resid", "response": "Returns the Gromacs index by transforming with offset."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef combine(self, name_all=None, out_ndx=None, operation='|', defaultgroups=False):\n if not operation in ('|', '&', False):\n raise ValueError(\"Illegal operation {0!r}, only '|' (OR) and '&' (AND) or False allowed.\".format(\n operation))\n if name_all is None and operation:\n name_all = self.name_all or operation.join(self.indexfiles)\n if out_ndx is None:\n out_ndx = self.output\n\n if defaultgroups:\n # make a default file (using the original ndx where provided!!)\n fd, default_ndx = tempfile.mkstemp(suffix='.ndx', prefix='default__')\n try:\n self.make_ndx(o=default_ndx, input=['q'])\n except:\n utilities.unlink_gmx(default_ndx)\n raise\n ndxfiles = [default_ndx]\n else:\n ndxfiles = []\n\n ndxfiles.extend(self.indexfiles.values())\n\n if operation:\n # combine multiple selections and name them\n try:\n fd, tmp_ndx = tempfile.mkstemp(suffix='.ndx', prefix='combined__')\n # combine all selections by loading ALL temporary index files\n operation = ' '+operation.strip()+' '\n cmd = [operation.join(['\"{0!s}\"'.format(gname) for gname in self.indexfiles]),\n '', 'q']\n rc,out,err = self.make_ndx(n=ndxfiles, o=tmp_ndx, input=cmd)\n if self._is_empty_group(out):\n warnings.warn(\"No atoms found for {cmd!r}\".format(**vars()),\n category=BadParameterWarning)\n\n # second pass for naming, sigh (or: use NDX ?)\n groups = parse_ndxlist(out)\n last = groups[-1]\n # name this group\n name_cmd = [\"name {0:d} {1!s}\".format(last['nr'], name_all), 'q']\n rc,out,err = self.make_ndx(n=tmp_ndx, o=out_ndx, input=name_cmd)\n # For debugging, look at out and err or set stdout=True, stderr=True\n # TODO: check out if at least 1 atom selected\n ##print \"DEBUG: combine()\"\n ##print out\n finally:\n utilities.unlink_gmx(tmp_ndx)\n if defaultgroups:\n utilities.unlink_gmx(default_ndx)\n else:\n # just write individual groups in one file (name_all --> None)\n rc,out,err = self.make_ndx(n=ndxfiles, o=out_ndx, input=['','q'])\n\n return name_all, out_ndx", "response": "Combine individual groups into a single one and write output."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting individual named groups to out_ndx.", "response": "def write(self, out_ndx=None, defaultgroups=False):\n \"\"\"Write individual (named) groups to *out_ndx*.\"\"\"\n name_all, out_ndx = self.combine(operation=False, out_ndx=out_ndx, defaultgroups=defaultgroups)\n return out_ndx"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef cat(self, out_ndx=None):\n if out_ndx is None:\n out_ndx = self.output\n self.make_ndx(o=out_ndx, input=['q'])\n return out_ndx", "response": "Concatenate input index files into a new index file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef parse_selection(self, selection, name=None):\n\n if type(selection) is tuple:\n # range\n process = self._process_range\n elif selection.startswith('@'):\n # verbatim make_ndx command\n process = self._process_command\n selection = selection[1:]\n else:\n process = self._process_residue\n return process(selection, name)", "response": "Retuns ( groupname filename ) with index group."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _process_command(self, command, name=None):\n\n self._command_counter += 1\n if name is None:\n name = \"CMD{0:03d}\".format(self._command_counter)\n\n # Need to build it with two make_ndx calls because I cannot reliably\n # name the new group without knowing its number.\n try:\n fd, tmp_ndx = tempfile.mkstemp(suffix='.ndx', prefix='tmp_'+name+'__')\n cmd = [command, '', 'q'] # empty command '' necessary to get list\n # This sometimes fails with 'OSError: Broken Pipe' --- hard to debug\n rc,out,err = self.make_ndx(o=tmp_ndx, input=cmd)\n self.check_output(out, \"No atoms found for selection {command!r}.\".format(**vars()), err=err)\n # For debugging, look at out and err or set stdout=True, stderr=True\n # TODO: check ' 0 r_300_&_ALA_&_O : 1 atoms' has at least 1 atom\n ##print \"DEBUG: _process_command()\"\n ##print out\n groups = parse_ndxlist(out)\n last = groups[-1]\n # reduce and name this group\n fd, ndx = tempfile.mkstemp(suffix='.ndx', prefix=name+'__')\n name_cmd = [\"keep {0:d}\".format(last['nr']),\n \"name 0 {0!s}\".format(name), 'q']\n rc,out,err = self.make_ndx(n=tmp_ndx, o=ndx, input=name_cmd)\n finally:\n utilities.unlink_gmx(tmp_ndx)\n\n return name, ndx", "response": "Process a make_ndx command and return name and temp index file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _process_residue(self, selection, name=None):\n\n if name is None:\n name = selection.replace(':', '_')\n\n # XXX: use _translate_residue() ....\n m = self.RESIDUE.match(selection)\n if not m:\n raise ValueError(\"Selection {selection!r} is not valid.\".format(**vars()))\n\n gmx_resid = self.gmx_resid(int(m.group('resid')))\n residue = m.group('aa')\n if len(residue) == 1:\n gmx_resname = utilities.convert_aa_code(residue) # only works for AA\n else:\n gmx_resname = residue # use 3-letter for any resname\n gmx_atomname = m.group('atom')\n if gmx_atomname is None:\n gmx_atomname = 'CA'\n\n #: select residue atom \n _selection = 'r {gmx_resid:d} & r {gmx_resname!s} & a {gmx_atomname!s}'.format(**vars())\n cmd = ['keep 0', 'del 0',\n _selection,\n 'name 0 {name!s}'.format(**vars()),\n 'q']\n fd, ndx = tempfile.mkstemp(suffix='.ndx', prefix=name+'__')\n rc,out,err = self.make_ndx(n=self.ndx, o=ndx, input=cmd)\n self.check_output(out, \"No atoms found for \"\n \"%(selection)r --> %(_selection)r\" % vars())\n # For debugging, look at out and err or set stdout=True, stderr=True\n ##print \"DEBUG: _process_residue()\"\n ##print out\n\n return name, ndx", "response": "Process a residue atom selection and return name and temp index file."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprocesses a range selection.", "response": "def _process_range(self, selection, name=None):\n \"\"\"Process a range selection.\n\n (\"S234\", \"A300\", \"CA\") --> selected all CA in this range\n (\"S234\", \"A300\") --> selected all atoms in this range\n\n .. Note:: Ignores residue type, only cares about the resid (but still required)\n \"\"\"\n\n try:\n first, last, gmx_atomname = selection\n except ValueError:\n try:\n first, last = selection\n gmx_atomname = '*'\n except:\n logger.error(\"%r is not a valid range selection\", selection)\n raise\n if name is None:\n name = \"{first!s}-{last!s}_{gmx_atomname!s}\".format(**vars())\n\n _first = self._translate_residue(first, default_atomname=gmx_atomname)\n _last = self._translate_residue(last, default_atomname=gmx_atomname)\n\n _selection = 'r {0:d} - {1:d} & & a {2!s}'.format(_first['resid'], _last['resid'], gmx_atomname)\n cmd = ['keep 0', 'del 0',\n _selection,\n 'name 0 {name!s}'.format(**vars()),\n 'q']\n fd, ndx = tempfile.mkstemp(suffix='.ndx', prefix=name+'__')\n rc,out,err = self.make_ndx(n=self.ndx, o=ndx, input=cmd)\n self.check_output(out, \"No atoms found for \"\n \"%(selection)r --> %(_selection)r\" % vars())\n # For debugging, look at out and err or set stdout=True, stderr=True\n ##print \"DEBUG: _process_residue()\"\n ##print out\n\n return name, ndx"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntranslate a single residue to make_ndx syntax.", "response": "def _translate_residue(self, selection, default_atomname='CA'):\n \"\"\"Translate selection for a single res to make_ndx syntax.\"\"\"\n m = self.RESIDUE.match(selection)\n if not m:\n errmsg = \"Selection {selection!r} is not valid.\".format(**vars())\n logger.error(errmsg)\n raise ValueError(errmsg)\n\n gmx_resid = self.gmx_resid(int(m.group('resid'))) # magic offset correction\n residue = m.group('aa')\n if len(residue) == 1:\n gmx_resname = utilities.convert_aa_code(residue) # only works for AA\n else:\n gmx_resname = residue # use 3-letter for any resname\n\n gmx_atomname = m.group('atom')\n if gmx_atomname is None:\n gmx_atomname = default_atomname\n\n return {'resname':gmx_resname, 'resid':gmx_resid, 'atomname':gmx_atomname}"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rp(self, *args):\n try:\n p = os.path.join(*args)\n if os.path.isabs(p):\n return p\n except TypeError:\n pass\n return utilities.realpath(self.dirname, *args)", "response": "Return canonical path to file under *dirname* with components *args*"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncenter the tpr and xtc of the current tpr reference structure.", "response": "def center_fit(self, **kwargs):\n \"\"\"Write compact xtc that is fitted to the tpr reference structure.\n\n See :func:`gromacs.cbook.trj_fitandcenter` for details and\n description of *kwargs* (including *input*, *input1*, *n* and\n *n1* for how to supply custom index groups). The most important ones are listed\n here but in most cases the defaults should work.\n\n :Keywords:\n *s*\n Input structure (typically the default tpr file but can be set to\n some other file with a different conformation for fitting)\n *n*\n Alternative index file.\n *o*\n Name of the output trajectory.\n *xy* : Boolean\n If ``True`` then only fit in xy-plane (useful for a membrane normal\n to z). The default is ``False``.\n *force*\n - ``True``: overwrite existing trajectories\n - ``False``: throw a IOError exception\n - ``None``: skip existing and log a warning [default]\n\n :Returns:\n dictionary with keys *tpr*, *xtc*, which are the names of the\n the new files\n \"\"\"\n kwargs.setdefault('s', self.tpr)\n kwargs.setdefault('n', self.ndx)\n kwargs['f'] = self.xtc\n kwargs.setdefault('o', self.outfile(self.infix_filename(None, self.xtc, '_centfit', 'xtc')))\n force = kwargs.pop('force', self.force)\n\n logger.info(\"Centering and fitting trajectory {f!r}...\".format(**kwargs))\n with utilities.in_dir(self.dirname):\n if not self.check_file_exists(kwargs['o'], resolve=\"indicate\", force=force):\n trj_fitandcenter(**kwargs)\n logger.info(\"Centered and fit trajectory: {o!r}.\".format(**kwargs))\n return {'tpr': self.rp(kwargs['s']), 'xtc': self.rp(kwargs['o'])}"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fit(self, xy=False, **kwargs):\n kwargs.setdefault('s', self.tpr)\n kwargs.setdefault('n', self.ndx)\n kwargs['f'] = self.xtc\n force = kwargs.pop('force', self.force)\n if xy:\n fitmode = 'rotxy+transxy'\n kwargs.pop('fit', None)\n infix_default = '_fitxy'\n else:\n fitmode = kwargs.pop('fit', 'rot+trans') # user can use 'progressive', too\n infix_default = '_fit'\n\n dt = kwargs.get('dt')\n if dt:\n infix_default += '_dt{0:d}ps'.format(int(dt)) # dt in ps\n\n kwargs.setdefault('o', self.outfile(self.infix_filename(None, self.xtc, infix_default, 'xtc')))\n fitgroup = kwargs.pop('fitgroup', 'backbone')\n kwargs.setdefault('input', [fitgroup, \"system\"])\n\n if kwargs.get('center', False):\n logger.warn(\"Transformer.fit(): center=%(center)r used: centering should not be combined with fitting.\", kwargs)\n if len(kwargs['inputs']) != 3:\n logger.error(\"If you insist on centering you must provide three groups in the 'input' kwarg: (center, fit, output)\")\n raise ValuError(\"Insufficient index groups for centering,fitting,output\")\n\n logger.info(\"Fitting trajectory %r to with xy=%r...\", kwargs['f'], xy)\n logger.info(\"Fitting on index group %(fitgroup)r\", vars())\n with utilities.in_dir(self.dirname):\n if self.check_file_exists(kwargs['o'], resolve=\"indicate\", force=force):\n logger.warn(\"File %r exists; force regenerating it with force=True.\", kwargs['o'])\n else:\n gromacs.trjconv(fit=fitmode, **kwargs)\n logger.info(\"Fitted trajectory (fitmode=%s): %r.\", fitmode, kwargs['o'])\n return {'tpr': self.rp(kwargs['s']), 'xtc': self.rp(kwargs['o'])}", "response": "Fit the tpr reference structure to the output trajectory."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef strip_water(self, os=None, o=None, on=None, compact=False,\n resn=\"SOL\", groupname=\"notwater\", **kwargs):\n \"\"\"Write xtc and tpr with water (by resname) removed.\n\n :Keywords:\n *os*\n Name of the output tpr file; by default use the original but\n insert \"nowater\" before suffix.\n *o*\n Name of the output trajectory; by default use the original name but\n insert \"nowater\" before suffix.\n *on*\n Name of a new index file (without water).\n *compact*\n ``True``: write a compact and centered trajectory\n ``False``: use trajectory as it is [``False``]\n *centergroup*\n Index group used for centering [\"Protein\"]\n\n .. Note:: If *input* is provided (see below under *kwargs*)\n then *centergroup* is ignored and the group for\n centering is taken as the first entry in *input*.\n\n *resn*\n Residue name of the water molecules; all these residues are excluded.\n *groupname*\n Name of the group that is generated by subtracting all waters\n from the system.\n *force* : Boolean\n - ``True``: overwrite existing trajectories\n - ``False``: throw a IOError exception\n - ``None``: skip existing and log a warning [default]\n *kwargs*\n are passed on to :func:`gromacs.cbook.trj_compact` (unless the\n values have to be set to certain values such as s, f, n, o\n keywords). The *input* keyword is always mangled: Only the first\n entry (the group to centre the trajectory on) is kept, and as a\n second group (the output group) *groupname* is used.\n\n :Returns:\n dictionary with keys *tpr*, *xtc*, *ndx* which are the names of the\n the new files\n\n .. warning:: The input tpr file should *not* have *any position restraints*;\n otherwise Gromacs will throw a hissy-fit and say\n\n *Software inconsistency error: Position restraint coordinates are\n missing*\n\n (This appears to be a bug in Gromacs 4.x.)\n \"\"\"\n force = kwargs.pop('force', self.force)\n\n newtpr = self.outfile(self.infix_filename(os, self.tpr, '_nowater'))\n newxtc = self.outfile(self.infix_filename(o, self.xtc, '_nowater'))\n newndx = self.outfile(self.infix_filename(on, self.tpr, '_nowater', 'ndx'))\n\n nowater_ndx = self._join_dirname(newtpr, \"nowater.ndx\") # refers to original tpr\n\n if compact:\n TRJCONV = trj_compact\n # input overrides centergroup\n if kwargs.get('centergroup') is not None and 'input' in kwargs:\n logger.warn(\"centergroup = %r will be superceded by input[0] = %r\", kwargs['centergroup'], kwargs['input'][0])\n _input = kwargs.get('input', [kwargs.get('centergroup', 'Protein')])\n kwargs['input'] = [_input[0], groupname] # [center group, write-out selection]\n del _input\n logger.info(\"Creating a compact trajectory centered on group %r\", kwargs['input'][0])\n logger.info(\"Writing %r to the output trajectory\", kwargs['input'][1])\n else:\n TRJCONV = gromacs.trjconv\n kwargs['input'] = [groupname]\n logger.info(\"Writing %r to the output trajectory (no centering)\", kwargs['input'][0])\n # clean kwargs, only legal arguments for Gromacs tool trjconv should remain\n kwargs.pop(\"centergroup\", None)\n\n NOTwater = \"! r {resn!s}\".format(**vars()) # make_ndx selection (\"not water residues\")\n with utilities.in_dir(self.dirname):\n # ugly because I cannot break from the block\n if not self.check_file_exists(newxtc, resolve=\"indicate\", force=force):\n # make no-water index\n B = IndexBuilder(struct=self.tpr, selections=['@'+NOTwater],\n ndx=self.ndx, out_ndx=nowater_ndx)\n B.combine(name_all=groupname, operation=\"|\", defaultgroups=True)\n logger.debug(\"Index file for water removal: %r\", nowater_ndx)\n\n logger.info(\"TPR file without water {newtpr!r}\".format(**vars()))\n gromacs.tpbconv(s=self.tpr, o=newtpr, n=nowater_ndx, input=[groupname])\n\n logger.info(\"NDX of the new system %r\", newndx)\n gromacs.make_ndx(f=newtpr, o=newndx, input=['q'], stderr=False, stdout=False)\n # PROBLEM: If self.ndx contained a custom group required for fitting then we are loosing\n # this group here. We could try to merge only this group but it is possible that\n # atom indices changed. The only way to solve this is to regenerate the group with\n # a selection or only use Gromacs default groups.\n\n logger.info(\"Trajectory without water {newxtc!r}\".format(**vars()))\n kwargs['s'] = self.tpr\n kwargs['f'] = self.xtc\n kwargs['n'] = nowater_ndx\n kwargs['o'] = newxtc\n TRJCONV(**kwargs)\n\n logger.info(\"pdb and gro for visualization\")\n for ext in 'pdb', 'gro':\n try:\n # see warning in doc ... so we don't use the new xtc but the old one\n kwargs['o'] = self.filename(newtpr, ext=ext)\n TRJCONV(dump=0, stdout=False, stderr=False, **kwargs) # silent\n except:\n logger.exception(\"Failed building the water-less %(ext)s. \"\n \"Position restraints in tpr file (see docs)?\" % vars())\n logger.info(\"strip_water() complete\")\n\n self.nowater[self.rp(newxtc)] = Transformer(dirname=self.dirname, s=newtpr,\n f=newxtc, n=newndx, force=force)\n return {'tpr':self.rp(newtpr), 'xtc':self.rp(newxtc), 'ndx':self.rp(newndx)}", "response": "This function is used to remove water from the output tpr file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef strip_fit(self, **kwargs):\n kwargs.setdefault('fit', 'rot+trans')\n kw_fit = {}\n for k in ('xy', 'fit', 'fitgroup', 'input'):\n if k in kwargs:\n kw_fit[k] = kwargs.pop(k)\n\n kwargs['input'] = kwargs.pop('strip_input', ['Protein'])\n kwargs['force'] = kw_fit['force'] = kwargs.pop('force', self.force)\n\n paths = self.strip_water(**kwargs) # updates self.nowater\n transformer_nowater = self.nowater[paths['xtc']] # make sure to get the one we just produced\n return transformer_nowater.fit(**kw_fit)", "response": "Strip water and fit to the remaining system."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _join_dirname(self, *args):\n # extra function because I need to use it in a method that defines\n # the kwarg 'os', which collides with os.path...\n return os.path.join(os.path.dirname(args[0]), *args[1:])", "response": "join_dirname - Joins the directory name of the base directory of the log file."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a top level logger with the given name and logfile.", "response": "def create(logger_name, logfile='gromacs.log'):\n \"\"\"Create a top level logger.\n\n - The file logger logs everything (including DEBUG).\n - The console logger only logs INFO and above.\n\n Logging to a file and the console.\n \n See http://docs.python.org/library/logging.html?#logging-to-multiple-destinations\n \n The top level logger of the library is named 'gromacs'. Note that\n we are configuring this logger with console output. If the root\n logger also does this then we will get two output lines to the\n console. We'll live with this because this is a simple\n convenience library...\n \"\"\"\n\n logger = logging.getLogger(logger_name)\n\n logger.setLevel(logging.DEBUG)\n\n logfile = logging.FileHandler(logfile)\n logfile_formatter = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s')\n logfile.setFormatter(logfile_formatter)\n logger.addHandler(logfile)\n\n # define a Handler which writes INFO messages or higher to the sys.stderr\n console = logging.StreamHandler()\n console.setLevel(logging.INFO)\n # set a format which is simpler for console use\n formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')\n console.setFormatter(formatter)\n\n logger.addHandler(console)\n\n return logger"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating a dictionary of template filenames and their contents.", "response": "def _generate_template_dict(dirname):\n \"\"\"Generate a list of included files *and* extract them to a temp space.\n\n Templates have to be extracted from the egg because they are used\n by external code. All template filenames are stored in\n :data:`config.templates`.\n \"\"\"\n return dict((resource_basename(fn), resource_filename(__name__, dirname +'/'+fn))\n for fn in resource_listdir(__name__, dirname)\n if not fn.endswith('~'))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef resource_basename(resource):\n if resource.endswith('/'):\n resource = resource[:-1]\n parts = resource.split('/')\n return parts[-1]", "response": "Returns the last component of a resource."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nfinding template file t and return its real path.", "response": "def get_template(t):\n \"\"\"Find template file *t* and return its real path.\n\n *t* can be a single string or a list of strings. A string\n should be one of\n\n 1. a relative or absolute path,\n 2. a file in one of the directories listed in :data:`gromacs.config.path`,\n 3. a filename in the package template directory (defined in the template dictionary\n :data:`gromacs.config.templates`) or\n 4. a key into :data:`~gromacs.config.templates`.\n\n The first match (in this order) is returned. If the argument is a\n single string then a single string is returned, otherwise a list\n of strings.\n\n :Arguments: *t* : template file or key (string or list of strings)\n :Returns: os.path.realpath(*t*) (or a list thereof)\n :Raises: :exc:`ValueError` if no file can be located.\n\n \"\"\"\n templates = [_get_template(s) for s in utilities.asiterable(t)]\n if len(templates) == 1:\n return templates[0]\n return templates"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a single template file.", "response": "def _get_template(t):\n \"\"\"Return a single template *t*.\"\"\"\n if os.path.exists(t): # 1) Is it an accessible file?\n pass\n else:\n _t = t\n _t_found = False\n for d in path: # 2) search config.path\n p = os.path.join(d, _t)\n if os.path.exists(p):\n t = p\n _t_found = True\n break\n _t = os.path.basename(t)\n if not _t_found: # 3) try template dirs\n for p in templates.values():\n if _t == os.path.basename(p):\n t = p\n _t_found = True # NOTE: in principle this could match multiple\n break # times if more than one template dir existed.\n if not _t_found: # 4) try it as a key into templates\n try:\n t = templates[t]\n except KeyError:\n pass\n else:\n _t_found = True\n if not _t_found: # 5) nothing else to try...\n raise ValueError(\"Failed to locate the template file {t!r}.\".format(**vars()))\n return os.path.realpath(t)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_configuration(filename=CONFIGNAME):\n global cfg, configuration # very iffy --- most of the whole config mod should a class\n\n #: :data:`cfg` is the instance of :class:`GMXConfigParser` that makes all\n #: global configuration data accessible\n cfg = GMXConfigParser(filename=filename) # update module-level cfg\n globals().update(cfg.configuration) # update configdir, templatesdir ...\n configuration = cfg.configuration # update module-level configuration\n return cfg", "response": "Reads and parses the configuration file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\npreparing a default GromacsWrapper global environment.", "response": "def setup(filename=CONFIGNAME):\n \"\"\"Prepare a default GromacsWrapper global environment.\n\n 1) Create the global config file.\n 2) Create the directories in which the user can store template and config files.\n\n This function can be run repeatedly without harm.\n \"\"\"\n # setup() must be separate and NOT run automatically when config\n # is loaded so that easy_install installations work\n # (otherwise we get a sandbox violation)\n # populate cfg with defaults (or existing data)\n get_configuration()\n if not os.path.exists(filename):\n with open(filename, 'w') as configfile:\n cfg.write(configfile) # write the default file so that user can edit\n msg = \"NOTE: GromacsWrapper created the configuration file \\n\\t%r\\n\" \\\n \" for you. Edit the file to customize the package.\" % filename\n print(msg)\n\n # directories\n for d in config_directories:\n utilities.mkdir_p(d)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_setup():\n\n if \"GROMACSWRAPPER_SUPPRESS_SETUP_CHECK\" in os.environ:\n return True\n\n missing = [d for d in config_directories if not os.path.exists(d)]\n if len(missing) > 0:\n print(\"NOTE: Some configuration directories are not set up yet: \")\n print(\"\\t{0!s}\".format('\\n\\t'.join(missing)))\n print(\"NOTE: You can create the configuration file and directories with:\")\n print(\"\\t>>> import gromacs\")\n print(\"\\t>>> gromacs.config.setup()\")\n return False\n return True", "response": "Check if templates directories are setup and issue a warning and help."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nsets the environment variables loaded by GMXRC into the current Python environment.", "response": "def set_gmxrc_environment(gmxrc):\n \"\"\"Set the environment from ``GMXRC`` provided in *gmxrc*.\n\n Runs ``GMXRC`` in a subprocess and puts environment variables loaded by it\n into this Python environment.\n\n If *gmxrc* evaluates to ``False`` then nothing is done. If errors occur\n then only a warning will be logged. Thus, it should be safe to just call\n this function.\n \"\"\"\n # only v5: 'GMXPREFIX', 'GROMACS_DIR'\n envvars = ['GMXBIN', 'GMXLDLIB', 'GMXMAN', 'GMXDATA',\n 'LD_LIBRARY_PATH', 'MANPATH', 'PKG_CONFIG_PATH',\n 'PATH',\n 'GMXPREFIX', 'GROMACS_DIR']\n # in order to keep empty values, add ___ sentinels around result\n # (will be removed later)\n cmdargs = ['bash', '-c', \". {0} && echo {1}\".format(gmxrc,\n ' '.join(['___${{{0}}}___'.format(v) for v in envvars]))]\n\n if not gmxrc:\n logger.debug(\"set_gmxrc_environment(): no GMXRC, nothing done.\")\n return\n\n try:\n out = subprocess.check_output(cmdargs)\n out = out.strip().split()\n for key, value in zip(envvars, out):\n value = str(value.decode('ascii').replace('___', '')) # remove sentinels\n os.environ[key] = value\n logger.debug(\"set_gmxrc_environment(): %s = %r\", key, value)\n except (subprocess.CalledProcessError, OSError):\n logger.warning(\"Failed to automatically set the Gromacs environment\"\n \"from GMXRC=%r\", gmxrc)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets all tool names from all configured groups.", "response": "def get_tool_names():\n \"\"\" Get tool names from all configured groups.\n\n :return: list of tool names\n \"\"\"\n names = []\n for group in cfg.get('Gromacs', 'groups').split():\n names.extend(cfg.get('Gromacs', group).split())\n return names"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn a dictionary of variables that we can use as globals in the module.", "response": "def configuration(self):\n \"\"\"Dict of variables that we make available as globals in the module.\n\n Can be used as ::\n\n globals().update(GMXConfigParser.configuration) # update configdir, templatesdir ...\n \"\"\"\n configuration = {\n 'configfilename': self.filename,\n 'logfilename': self.getpath('Logging', 'logfilename'),\n 'loglevel_console': self.getLogLevel('Logging', 'loglevel_console'),\n 'loglevel_file': self.getLogLevel('Logging', 'loglevel_file'),\n 'configdir': self.getpath('DEFAULT', 'configdir'),\n 'qscriptdir': self.getpath('DEFAULT', 'qscriptdir'),\n 'templatesdir': self.getpath('DEFAULT', 'templatesdir'),\n }\n configuration['path'] = [os.path.curdir,\n configuration['qscriptdir'],\n configuration['templatesdir']]\n return configuration"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns option as an expanded path.", "response": "def getpath(self, section, option):\n \"\"\"Return option as an expanded path.\"\"\"\n return os.path.expanduser(os.path.expandvars(self.get(section, option)))"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the textual representation of the logging level option or the number.", "response": "def getLogLevel(self, section, option):\n \"\"\"Return the textual representation of logging level 'option' or the number.\n\n Note that option is always interpreted as an UPPERCASE string\n and hence integer log levels will not be recognized.\n\n .. SeeAlso: :mod:`logging` and :func:`logging.getLevelName`\n \"\"\"\n return logging.getLevelName(self.get(section, option).upper())"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef save(self, filename):\n cPickle.dump(self, open(self._canonicalize(filename), 'wb'), \n protocol=cPickle.HIGHEST_PROTOCOL)", "response": "Save the whole collection to filename."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load(self, filename, append=False):\n tmp = cPickle.load(open(self._canonicalize(filename), 'rb'))\n if append:\n self.extend(tmp)\n else:\n self[:] = tmp[:]\n del tmp", "response": "Load a collection from a pickled file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _canonicalize(self, filename):\n path, ext = os.path.splitext(filename)\n if not ext:\n ext = \".collection\"\n return path + ext", "response": "Use .collection as extension unless provided"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef register(self,flag):\n super(Flags,self).__setitem__(flag.name,flag)", "response": "Register a new flag instance with the Flags registry."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef update(self,*flags):\n super(Flags,self).update([(flag.name,flag) for flag in flags])", "response": "Update the flags registry with a list of : class : Flag instances."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nscales dihedrals in a molecule.", "response": "def scale_dihedrals(mol, dihedrals, scale, banned_lines=None):\n \"\"\"Scale dihedral angles\"\"\"\n\n if banned_lines is None:\n banned_lines = []\n new_dihedrals = []\n for dh in mol.dihedrals:\n atypes = dh.atom1.get_atomtype(), dh.atom2.get_atomtype(), dh.atom3.get_atomtype(), dh.atom4.get_atomtype()\n atypes = [a.replace(\"_\", \"\").replace(\"=\",\"\") for a in atypes]\n\n # special-case: this is a [ dihedral ] override in molecule block, continue and don't match\n if dh.gromacs['param'] != []:\n for p in dh.gromacs['param']:\n p['kch'] *= scale\n new_dihedrals.append(dh)\n continue\n\n for iswitch in range(32):\n if (iswitch%2==0 ):\n a1=atypes[0]; a2=atypes[1]; a3=atypes[2]; a4=atypes[3]\n else:\n a1=atypes[3]; a2=atypes[2]; a3=atypes[1]; a4=atypes[0]\n\n if((iswitch//2)%2==1): a1=\"X\";\n if((iswitch//4)%2==1): a2=\"X\";\n if((iswitch//8)%2==1): a3=\"X\";\n if((iswitch//16)%2==1): a4=\"X\";\n key = \"{0}-{1}-{2}-{3}-{4}\".format(a1, a2, a3, a4, dh.gromacs['func'])\n if (key in dihedrals):\n for i, dt in enumerate(dihedrals[key]):\n dhA = copy.deepcopy(dh)\n param = copy.deepcopy(dt.gromacs['param'])\n # Only check the first dihedral in a list\n if not dihedrals[key][0].line in banned_lines:\n for p in param: p['kchi'] *= scale\n dhA.gromacs['param'] = param\n #if key == \"CT3-C-NH1-CT1-9\": print i, dt, key\n if i == 0:\n dhA.comment = \"; banned lines {0} found={1}\\n\".format(\" \".join(\n map(str, banned_lines)), 1 if dt.line in banned_lines else 0)\n dhA.comment += \"; parameters for types {}-{}-{}-{}-9 at LINE({})\\n\".format(\n dhA.atom1.atomtype, dhA.atom2.atomtype, dhA.atom3.atomtype,\n dhA.atom4.atomtype, dt.line).replace(\"_\",\"\")\n name = \"{}-{}-{}-{}-9\".format(dhA.atom1.atomtype, dhA.atom2.atomtype,\n dhA.atom3.atomtype, dhA.atom4.atomtype).replace(\"_\",\"\")\n #if name == \"CL-CTL2-CTL2-HAL2-9\": print dihedrals[key], key\n new_dihedrals.append(dhA)\n break\n\n\n mol.dihedrals = new_dihedrals\n #assert(len(mol.dihedrals) == new_dihedrals)\n return mol"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef partial_tempering(topfile=\"processed.top\", outfile=\"scaled.top\", banned_lines='',\n scale_lipids=1.0, scale_protein=1.0):\n \"\"\"Set up topology for partial tempering (REST2) replica exchange.\n\n\n .. versionchanged:: 0.7.0\n Use keyword arguments instead of an `args` Namespace object.\n \"\"\"\n\n banned_lines = map(int, banned_lines.split())\n top = TOP(topfile)\n groups = [(\"_\", float(scale_protein)), (\"=\", float(scale_lipids))]\n\n #\n # CMAPTYPES\n #\n cmaptypes = []\n for ct in top.cmaptypes:\n cmaptypes.append(ct)\n for gr, scale in groups:\n ctA = copy.deepcopy(ct)\n ctA.atype1 += gr\n ctA.atype2 += gr\n ctA.atype3 += gr\n ctA.atype4 += gr\n ctA.atype8 += gr\n ctA.gromacs['param'] = [ v*scale for v in ct.gromacs['param'] ]\n cmaptypes.append(ctA)\n logger.debug(\"cmaptypes was {0}, is {1}\".format(len(top.cmaptypes), len(cmaptypes)))\n top.cmaptypes = cmaptypes\n\n\n #\n # ATOMTYPES\n #\n atomtypes = []\n for at in top.atomtypes:\n atomtypes.append(at)\n for gr, scale in groups:\n atA = copy.deepcopy(at)\n atA.atnum = atA.atype\n atA.atype += gr\n atA.gromacs['param']['lje'] *= scale\n atomtypes.append(atA)\n top.atomtypes = atomtypes\n\n #\n # PAIRTYPES\n #\n pairtypes = []\n for pt in top.pairtypes:\n pairtypes.append(pt)\n for gr, scale in groups:\n ptA = copy.deepcopy(pt)\n ptA.atype1 += gr\n ptA.atype2 += gr\n ptA.gromacs['param']['lje14'] *= scale\n\n pairtypes.append(ptA)\n top.pairtypes = pairtypes\n\n #\n # BONDTYPES\n #\n bondtypes = []\n for bt in top.bondtypes:\n bondtypes.append(bt)\n for gr, scale in groups:\n btA = copy.deepcopy(bt)\n btA.atype1 += gr\n btA.atype2 += gr\n bondtypes.append(btA)\n top.bondtypes = bondtypes\n\n\n #\n # ANGLETYPES\n #\n angletypes = []\n for at in top.angletypes:\n angletypes.append(at)\n for gr, scale in groups:\n atA = copy.deepcopy(at)\n atA.atype1 += gr\n atA.atype2 += gr\n atA.atype3 += gr\n angletypes.append(atA)\n top.angletypes = angletypes\n\n #\n # Build dihedral dictionary\n #\n dihedraltypes = {}\n for dt in top.dihedraltypes:\n dt.disabled = True\n dt.comment = \"; type={0!s}-{1!s}-{2!s}-{3!s}-9\\n; LINE({4:d}) \".format(\n dt.atype1, dt.atype2, dt.atype3, dt.atype4, dt.line)\n dt.comment = dt.comment.replace(\"_\",\"\")\n\n #if \"X-CTL2-CTL2-X-9\" in dt.comment: print dt\n name = \"{0}-{1}-{2}-{3}-{4}\".format(dt.atype1, dt.atype2, dt.atype3, dt.atype4, dt.gromacs['func'])\n if not name in dihedraltypes:\n dihedraltypes[name] = []\n dihedraltypes[name].append(dt)\n logger.debug(\"Build dihedraltypes dictionary with {0} entries\".format(len(dihedraltypes)))\n\n #\n # Build improper dictionary\n #\n impropertypes = {}\n for it in top.impropertypes:\n it.disabled = True\n it.comment = \"; LINE({0:d}) \".format(it.line)\n name = \"{0}-{1}-{2}-{3}-{4}\".format(\n it.atype1, it.atype2, it.atype3, it.atype4, it.gromacs['func'])\n if not name in impropertypes:\n impropertypes[name] = []\n impropertypes[name].append(it)\n logger.debug(\"Build impropertypes dictionary with {0} entries\".format(len(impropertypes)))\n\n for molname_mol in top.dict_molname_mol:\n if not 'Protein' in molname_mol:\n continue\n mol = top.dict_molname_mol[molname_mol]\n for at in mol.atoms:\n at.charge *= math.sqrt(scale_protein)\n mol = scale_dihedrals(mol, dihedraltypes, scale_protein, banned_lines)\n mol = scale_impropers(mol, impropertypes, 1.0, banned_lines)\n\n top.write(outfile)", "response": "Set up topology for partial tempering."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nconvert obj to unicode if it can be converted.", "response": "def to_unicode(obj):\n \"\"\"Convert obj to unicode (if it can be be converted).\n\n Conversion is only attempted if `obj` is a string type (as\n determined by :data:`six.string_types`).\n\n .. versionchanged:: 0.7.0\n removed `encoding keyword argument\n\n \"\"\"\n if not isinstance(obj, six.string_types):\n return obj\n\n try:\n obj = six.text_type(obj)\n except TypeError:\n pass\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert string x to the most useful type i. e. int float or unicode string.", "response": "def besttype(x):\n \"\"\"Convert string x to the most useful type, i.e. int, float or unicode string.\n\n If x is a quoted string (single or double quotes) then the quotes\n are stripped and the enclosed string returned.\n\n .. Note::\n\n Strings will be returned as Unicode strings (using :func:`to_unicode`).\n\n .. versionchanged:: 0.7.0\n removed `encoding keyword argument\n \"\"\"\n x = to_unicode(x) # make unicode as soon as possible\n try:\n x = x.strip()\n except AttributeError:\n pass\n m = re.match(r\"\"\"['\"](?P.*)[\"']$\"\"\", x)\n if m is None:\n # not a quoted string, try different types\n for converter in int, float, to_unicode: # try them in increasing order of lenience\n try:\n return converter(x)\n except ValueError:\n pass\n else:\n # quoted string\n x = to_unicode(m.group('value'))\n return x"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_int64(a):\n # build new dtype and replace i4 --> i8\n def promote_i4(typestr):\n if typestr[1:] == 'i4':\n typestr = typestr[0]+'i8'\n return typestr\n\n dtype = [(name, promote_i4(typestr)) for name,typestr in a.dtype.descr]\n return a.astype(dtype)", "response": "Return view of the recarray with all int32 cast to int64."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nslowing conversion of a recarray into a list of records with python types.", "response": "def irecarray_to_py(a):\n \"\"\"Slow conversion of a recarray into a list of records with python types.\n\n Get the field names from :attr:`a.dtype.names`.\n\n :Returns: iterator so that one can handle big input arrays\n \"\"\"\n pytypes = [pyify(typestr) for name,typestr in a.dtype.descr]\n def convert_record(r):\n return tuple([converter(value) for converter, value in zip(pytypes,r)])\n return (convert_record(r) for r in a)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _convert_fancy(self, field):\n if self.sep is False:\n x = self._convert_singlet(field)\n else:\n x = tuple([self._convert_singlet(s) for s in field.split(self.sep)])\n if len(x) == 0:\n x = ''\n elif len(x) == 1:\n x = x[0]\n #print \"%r --> %r\" % (field, x)\n return x", "response": "Convert to a list and convert list elements."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nparsing the xpm file and populate the XPM. array attribute.", "response": "def parse(self):\n \"\"\"Parse the xpm file and populate :attr:`XPM.array`.\"\"\"\n with utilities.openany(self.real_filename) as xpm:\n # Read in lines until we find the start of the array\n meta = [xpm.readline()]\n while not meta[-1].startswith(\"static char *gromacs_xpm[]\"):\n meta.append(xpm.readline())\n\n # The next line will contain the dimensions of the array\n dim = xpm.readline()\n # There are four integers surrounded by quotes\n # nx: points along x, ny: points along y, nc: ?, nb: stride x\n nx, ny, nc, nb = [int(i) for i in self.unquote(dim).split()]\n\n # The next dim[2] lines contain the color definitions\n # Each pixel is encoded by dim[3] bytes, and a comment\n # at the end of the line contains the corresponding value\n colors = dict([self.col(xpm.readline()) for i in range(nc)])\n\n\n if self.autoconvert:\n autoconverter = Autoconverter(mode=\"singlet\")\n for symbol, value in colors.items():\n colors[symbol] = autoconverter.convert(value)\n self.logger.debug(\"Autoconverted colours: %r\", colors)\n\n # make an array containing all possible values and let numpy figure out the dtype\n dtype = numpy.array(colors.values()).dtype\n self.logger.debug(\"Guessed array type: %s\", dtype.name)\n\n # pre-allocate array\n data = numpy.zeros((int(nx/nb), ny), dtype=dtype)\n\n self.logger.debug(\"dimensions: NX=%d NY=%d strideX=%d (NC=%d) --> (%d, %d)\",\n nx, ny, nb, nc, nx/nb, ny)\n\n iy = 0\n xval = []\n yval = []\n autoconverter = Autoconverter(mode=\"singlet\")\n for line in xpm:\n if line.startswith(\"/*\"):\n # lines '/* x-axis:' ... and '/* y-axis:' contain the\n # values of x and y coordinates\n s = self.uncomment(line).strip()\n if s.startswith('x-axis:'):\n xval.extend([autoconverter.convert(x) for x in s[7:].split()])\n elif s.startswith('y-axis:'):\n yval.extend([autoconverter.convert(y) for y in s[7:].split()])\n continue\n s = self.unquote(line)\n # Joao M. Damas suggests on gmx-users (24 Oct 2014)\n # that the next line should read:\n #\n # data[:, iy] = [colors[j[k:k+nb]] for k in range(0,nx*nb,nb)]\n #\n # \"if one is using higher -nlevels for the .xpm construction (in g_rms, for example)\"\n # However, without a test case I am not eager to change it right away so in\n # case some faulty behavior is discovered with the XPM reader then this comment\n # might be helpful. --- Oliver 2014-10-25\n data[:, iy] = [colors[s[k:k+nb]] for k in range(0,nx,nb)]\n self.logger.debug(\"read row %d with %d columns: '%s....%s'\",\n iy, data.shape[0], s[:4], s[-4:])\n iy += 1 # for next row\n\n self.xvalues = numpy.array(xval)\n if self.reverse:\n self.logger.debug(\"reversed row order, reverse=%r\", self.reverse)\n self.__array = data[:, ::-1]\n self.yvalues = numpy.array(yval)\n else:\n self.__array = data\n self.yvalues = numpy.array(yval)[::-1]"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nparses a colour specification and return color and value.", "response": "def col(self, c):\n \"\"\"Parse colour specification\"\"\"\n m = self.COLOUR.search(c)\n if not m:\n self.logger.fatal(\"Cannot parse colour specification %r.\", c)\n raise ParseError(\"XPM reader: Cannot parse colour specification {0!r}.\".format(c))\n value = m.group('value')\n color = m.group('symbol')\n self.logger.debug(\"%s: %s %s\\n\", c.strip(), color, value)\n return color, value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrun the command ; args and kwargs are added or replaced the ones given to the constructor.", "response": "def run(self, *args, **kwargs):\n \"\"\"Run the command; args/kwargs are added or replace the ones given to the constructor.\"\"\"\n _args, _kwargs = self._combine_arglist(args, kwargs)\n results, p = self._run_command(*_args, **_kwargs)\n return results"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _combine_arglist(self, args, kwargs):\n _args = self.args + args\n _kwargs = self.kwargs.copy()\n _kwargs.update(kwargs)\n return _args, _kwargs", "response": "Combine the default values and the supplied values."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nexecutes the command and return the output and stderr.", "response": "def _run_command(self, *args, **kwargs):\n \"\"\"Execute the command; see the docs for __call__.\n\n :Returns: a tuple of the *results* tuple ``(rc, stdout, stderr)`` and\n the :class:`Popen` instance.\n \"\"\"\n # hack to run command WITHOUT input (-h...) even though user defined\n # input (should have named it \"ignore_input\" with opposite values...)\n use_input = kwargs.pop('use_input', True)\n\n # logic for capturing output (see docs on I/O and the flags)\n capturefile = None\n if environment.flags['capture_output'] is True:\n # capture into Python vars (see subprocess.Popen.communicate())\n kwargs.setdefault('stderr', PIPE)\n kwargs.setdefault('stdout', PIPE)\n elif environment.flags['capture_output'] == \"file\":\n if 'stdout' in kwargs and 'stderr' in kwargs:\n pass\n else:\n # XXX: not race or thread proof; potentially many commands write to the same file\n fn = environment.flags['capture_output_filename']\n capturefile = file(fn, \"w\") # overwrite (clobber) capture file\n if 'stdout' in kwargs and 'stderr' not in kwargs:\n # special case of stdout used by code but stderr should be captured to file\n kwargs.setdefault('stderr', capturefile)\n else:\n # merge stderr with stdout and write stdout to file\n # (stderr comes *before* stdout in capture file, could split...)\n kwargs.setdefault('stderr', STDOUT)\n kwargs.setdefault('stdout', capturefile)\n\n try:\n p = self.Popen(*args, **kwargs)\n out, err = p.communicate(use_input=use_input) # special Popen knows input!\n except:\n if capturefile is not None:\n logger.error(\"Use captured command output in %r for diagnosis.\", capturefile)\n raise\n finally:\n if capturefile is not None:\n capturefile.close()\n rc = p.returncode\n return (rc, out, err), p"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _commandline(self, *args, **kwargs):\n # transform_args() is a hook (used in GromacsCommand very differently!)\n return [self.command_name] + self.transform_args(*args, **kwargs)", "response": "Returns the command line as a list."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef commandline(self, *args, **kwargs):\n # this mirrors the setup in run()\n _args, _kwargs = self._combine_arglist(args, kwargs)\n return self._commandline(*_args, **_kwargs)", "response": "Returns the commandline that run() uses."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef Popen(self, *args, **kwargs):\n stderr = kwargs.pop('stderr', None) # default: print to stderr (if STDOUT then merge)\n if stderr is False: # False: capture it\n stderr = PIPE\n elif stderr is True:\n stderr = None # use stderr\n\n stdout = kwargs.pop('stdout', None) # either set to PIPE for capturing output\n if stdout is False: # ... or to False\n stdout = PIPE\n elif stdout is True:\n stdout = None # for consistency, make True write to screen\n\n stdin = kwargs.pop('stdin', None)\n input = kwargs.pop('input', None)\n\n use_shell = kwargs.pop('use_shell', False)\n if input:\n stdin = PIPE\n if isinstance(input, six.string_types) and not input.endswith('\\n'):\n # make sure that input is a simple string with \\n line endings\n input = six.text_type(input) + '\\n'\n else:\n try:\n # make sure that input is a simple string with \\n line endings\n input = '\\n'.join(map(six.text_type, input)) + '\\n'\n except TypeError:\n # so maybe we are a file or something ... and hope for the best\n pass\n\n cmd = self._commandline(*args, **kwargs) # lots of magic happening here\n # (cannot move out of method because filtering of stdin etc)\n try:\n p = PopenWithInput(cmd, stdin=stdin, stderr=stderr, stdout=stdout,\n universal_newlines=True, input=input, shell=use_shell)\n except OSError as err:\n logger.error(\" \".join(cmd)) # log command line\n if err.errno == errno.ENOENT:\n errmsg = \"Failed to find Gromacs command {0!r}, maybe its not on PATH or GMXRC must be sourced?\".format(self.command_name)\n logger.fatal(errmsg)\n raise OSError(errmsg)\n else:\n logger.exception(\"Setting up Gromacs command {0!r} raised an exception.\".format(self.command_name))\n raise\n logger.debug(p.command_string)\n return p", "response": "Returns a special Popen instance with input."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntransforms arguments and return them as a list suitable for Popen.", "response": "def transform_args(self, *args, **kwargs):\n \"\"\"Transform arguments and return them as a list suitable for Popen.\"\"\"\n options = []\n for option,value in kwargs.items():\n if not option.startswith('-'):\n # heuristic for turning key=val pairs into options\n # (fails for commands such as 'find' -- then just use args)\n if len(option) == 1:\n option = '-' + option # POSIX style\n else:\n option = '--' + option # GNU option\n if value is True:\n options.append(option)\n continue\n elif value is False:\n raise ValueError('A False value is ambiguous for option {0!r}'.format(option))\n\n if option[:2] == '--':\n options.append(option + '=' + str(value)) # GNU option\n else:\n options.extend((option, str(value))) # POSIX style\n return options + list(args)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprints help ; same as using? in IPython. long = True also gives call signature.", "response": "def help(self, long=False):\n \"\"\"Print help; same as using ``?`` in ``ipython``. long=True also gives call signature.\"\"\"\n print(\"\\ncommand: {0!s}\\n\\n\".format(self.command_name))\n print(self.__doc__)\n if long:\n print(\"\\ncall method: command():\\n\")\n print(self.__call__.__doc__)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncombining the default values and the supplied values.", "response": "def _combine_arglist(self, args, kwargs):\n \"\"\"Combine the default values and the supplied values.\"\"\"\n gmxargs = self.gmxargs.copy()\n gmxargs.update(self._combineargs(*args, **kwargs))\n return (), gmxargs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncombine the arguments of the command line.", "response": "def _combineargs(self, *args, **kwargs):\n \"\"\"Add switches as 'options' with value True to the options dict.\"\"\"\n d = {arg: True for arg in args} # switches are kwargs with value True\n d.update(kwargs)\n return d"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nbuild a list of arguments from the dictionary.", "response": "def _build_arg_list(self, **kwargs):\n \"\"\"Build list of arguments from the dict; keys must be valid gromacs flags.\"\"\"\n arglist = []\n for flag, value in kwargs.items():\n # XXX: check flag against allowed values\n flag = str(flag)\n if flag.startswith('_'):\n flag = flag[1:] # python-illegal keywords are '_'-quoted\n if not flag.startswith('-'):\n flag = '-' + flag # now flag is guaranteed to start with '-'\n if value is True:\n arglist.append(flag) # simple command line flag\n elif value is False:\n if flag.startswith('-no'):\n # negate a negated flag ('noX=False' --> X=True --> -X ... but who uses that?)\n arglist.append('-' + flag[3:])\n else:\n arglist.append('-no' + flag[1:]) # gromacs switches booleans by prefixing 'no'\n elif value is None:\n pass # ignore flag = None\n else:\n try:\n arglist.extend([flag] + value) # option with value list\n except TypeError:\n arglist.extend([flag, value]) # option with single value\n return list(map(str, arglist))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef _run_command(self,*args,**kwargs):\n result, p = super(GromacsCommand, self)._run_command(*args, **kwargs)\n self.check_failure(result, command_string=p.command_string)\n return result, p", "response": "Execute the gromacs command ; see the docs for __call__."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _commandline(self, *args, **kwargs):\n if(self.driver is not None):\n return [self.driver, self.command_name] + self.transform_args(*args, **kwargs)\n return [self.command_name] + self.transform_args(*args, **kwargs)", "response": "Returns the command line as a list. Inserts driver if present"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncombine arguments and turn them into gromacs tool arguments.", "response": "def transform_args(self,*args,**kwargs):\n \"\"\"Combine arguments and turn them into gromacs tool arguments.\"\"\"\n newargs = self._combineargs(*args, **kwargs)\n return self._build_arg_list(**newargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nextracts standard gromacs docstrings from the command line and return them as a string.", "response": "def _get_gmx_docs(self):\n \"\"\"Extract standard gromacs doc\n\n Extract by running the program and chopping the header to keep from\n 'DESCRIPTION' onwards.\n \"\"\"\n if self._doc_cache is not None:\n return self._doc_cache\n\n try:\n logging.disable(logging.CRITICAL)\n rc, header, docs = self.run('h', stdout=PIPE, stderr=PIPE, use_input=False)\n except:\n logging.critical(\"Invoking command {0} failed when determining its doc string. Proceed with caution\".format(self.command_name))\n self._doc_cache = \"(No Gromacs documentation available)\"\n return self._doc_cache\n finally:\n # ALWAYS restore logging...\n logging.disable(logging.NOTSET)\n\n # The header is on STDOUT and is ignored. The docs are read from STDERR in GMX 4.\n m = re.match(self.doc_pattern, docs, re.DOTALL)\n\n if m is None:\n # In GMX 5, the opposite is true (Grrr)\n m = re.match(self.doc_pattern, header, re.DOTALL)\n if m is None:\n self._doc_cache = \"(No Gromacs documentation available)\"\n return self._doc_cache\n\n self._doc_cache = m.group('DOCS')\n return self._doc_cache"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef communicate(self, use_input=True):\n if use_input:\n return super(PopenWithInput, self).communicate(self.input)\n else:\n return super(PopenWithInput, self).communicate()", "response": "Run the command using the input that was set up on __init__."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef autoconvert(s):\n if type(s) is not str:\n return s\n for converter in int, float, str: # try them in increasing order of lenience\n try:\n s = [converter(i) for i in s.split()]\n if len(s) == 1:\n return s[0]\n else:\n return numpy.array(s)\n except (ValueError, AttributeError):\n pass\n raise ValueError(\"Failed to autoconvert {0!r}\".format(s))", "response": "Convert input to a numerical type if possible."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef openany(datasource, mode='rt', reset=True):\n stream = anyopen(datasource, mode=mode, reset=reset)\n try:\n yield stream\n finally:\n stream.close()", "response": "A context manager for opening any file or stream."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef anyopen(datasource, mode='rt', reset=True):\n handlers = {'bz2': bz2_open, 'gz': gzip.open, '': open}\n\n if mode.startswith('r'):\n if isstream(datasource):\n stream = datasource\n try:\n filename = str(stream.name) # maybe that does not always work?\n except AttributeError:\n filename = \"\"\n if reset:\n try:\n stream.reset()\n except (AttributeError, IOError):\n try:\n stream.seek(0)\n except (AttributeError, IOError):\n warnings.warn(\"Stream {0}: not guaranteed to be at the beginning.\"\n \"\".format(filename),\n category=StreamWarning)\n else:\n stream = None\n filename = datasource\n for ext in ('bz2', 'gz', ''): # file == '' should be last\n openfunc = handlers[ext]\n stream = _get_stream(datasource, openfunc, mode=mode)\n if stream is not None:\n break\n if stream is None:\n raise IOError(errno.EIO, \"Cannot open file or stream in mode={mode!r}.\".format(**vars()), repr(filename))\n elif mode.startswith('w') or mode.startswith('a'): # append 'a' not tested...\n if isstream(datasource):\n stream = datasource\n try:\n filename = str(stream.name) # maybe that does not always work?\n except AttributeError:\n filename = \"\"\n else:\n stream = None\n filename = datasource\n name, ext = os.path.splitext(filename)\n if ext.startswith('.'):\n ext = ext[1:]\n if not ext in ('bz2', 'gz'):\n ext = '' # anything else but bz2 or gz is just a normal file\n openfunc = handlers[ext]\n stream = openfunc(datasource, mode=mode)\n if stream is None:\n raise IOError(errno.EIO, \"Cannot open file or stream in mode={mode!r}.\".format(**vars()), repr(filename))\n else:\n raise NotImplementedError(\"Sorry, mode={mode!r} is not implemented for {datasource!r}\".format(**vars()))\n try:\n stream.name = filename\n except (AttributeError, TypeError):\n pass # can't set name (e.g. cStringIO.StringIO)\n return stream", "response": "Open a file or a stream."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _get_stream(filename, openfunction=open, mode='r'):\n try:\n stream = openfunction(filename, mode=mode)\n except (IOError, OSError) as err:\n # An exception might be raised due to two reasons, first the openfunction is unable to open the file, in this\n # case we have to ignore the error and return None. Second is when openfunction can't open the file because\n # either the file isn't there or the permissions don't allow access.\n if errno.errorcode[err.errno] in ['ENOENT', 'EACCES']:\n six.reraise(*sys.exc_info())\n return None\n if mode.startswith('r'):\n # additional check for reading (eg can we uncompress) --- is this needed?\n try:\n stream.readline()\n except IOError:\n stream.close()\n stream = None\n except:\n stream.close()\n raise\n else:\n stream.close()\n stream = openfunction(filename, mode=mode)\n return stream", "response": "Return open stream if filename can be opened with openfunction or else None."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef hasmethod(obj, m):\n return hasattr(obj, m) and callable(getattr(obj, m))", "response": "Return True if object *obj* contains the method m."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ndetects if an object is a stream.", "response": "def isstream(obj):\n \"\"\"Detect if `obj` is a stream.\n\n We consider anything a stream that has the methods\n\n - ``close()``\n\n and either set of the following\n\n - ``read()``, ``readline()``, ``readlines()``\n - ``write()``, ``writeline()``, ``writelines()``\n\n :Arguments:\n *obj*\n stream or str\n\n :Returns:\n *bool*, ``True`` if `obj` is a stream, ``False`` otherwise\n\n .. SeeAlso::\n :mod:`io`\n\n\n .. versionadded:: 0.7.1\n \"\"\"\n signature_methods = (\"close\",)\n alternative_methods = (\n (\"read\", \"readline\", \"readlines\"),\n (\"write\", \"writeline\", \"writelines\"))\n\n # Must have ALL the signature methods\n for m in signature_methods:\n if not hasmethod(obj, m):\n return False\n # Must have at least one complete set of alternative_methods\n alternative_results = [\n numpy.all([hasmethod(obj, m) for m in alternatives])\n for alternatives in alternative_methods]\n return numpy.any(alternative_results)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting between 1 - letter amino acid codes and 3 - letter amino acid codes.", "response": "def convert_aa_code(x):\n \"\"\"Converts between 3-letter and 1-letter amino acid codes.\"\"\"\n if len(x) == 1:\n return amino_acid_codes[x.upper()]\n elif len(x) == 3:\n return inverse_aa_codes[x.upper()]\n else:\n raise ValueError(\"Can only convert 1-letter or 3-letter amino acid codes, \"\n \"not %r\" % x)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef realpath(*args):\n if None in args:\n return None\n return os.path.realpath(\n os.path.expandvars(os.path.expanduser(os.path.join(*args))))", "response": "Join all args and return the real path rooted at /."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfinding first filename with a suffix from suffices.", "response": "def find_first(filename, suffices=None):\n \"\"\"Find first *filename* with a suffix from *suffices*.\n\n :Arguments:\n *filename*\n base filename; this file name is checked first\n *suffices*\n list of suffices that are tried in turn on the root of *filename*; can contain the\n ext separator (:data:`os.path.extsep`) or not\n\n :Returns: The first match or ``None``.\n \"\"\"\n # struct is not reliable as it depends on qscript so now we just try everything...\n\n root,extension = os.path.splitext(filename)\n if suffices is None:\n suffices = []\n else:\n suffices = withextsep(suffices)\n extensions = [extension] + suffices # native name is first\n for ext in extensions:\n fn = root + ext\n if os.path.exists(fn):\n return fn\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef withextsep(extensions):\n def dottify(x):\n if x.startswith(os.path.extsep):\n return x\n return os.path.extsep + x\n return [dottify(x) for x in asiterable(extensions)]", "response": "Return list of files in which each element is guaranteed to start with the given extensions."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef iterable(obj):\n if isinstance(obj, string_types):\n return False # avoid iterating over characters of a string\n if hasattr(obj, 'next'):\n return True # any iterator will do\n try:\n len(obj) # anything else that might work\n except TypeError:\n return False\n return True", "response": "Returns True if obj can be iterated over and is not a string."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef unlink_f(path):\n try:\n os.unlink(path)\n except OSError as err:\n if err.errno != errno.ENOENT:\n raise", "response": "Unlink a file but do not complain if file does not exist."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mkdir_p(path):\n try:\n os.makedirs(path)\n except OSError as err:\n if err.errno != errno.EEXIST:\n raise", "response": "Create a directory path with subdirs but do not complain if it exists."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconcatenating files f and write to o", "response": "def cat(f=None, o=None):\n \"\"\"Concatenate files *f*=[...] and write to *o*\"\"\"\n # need f, o to be compatible with trjcat and eneconv\n if f is None or o is None:\n return\n target = o\n infiles = asiterable(f)\n logger.debug(\"cat {0!s} > {1!s} \".format(\" \".join(infiles), target))\n with open(target, 'w') as out:\n rc = subprocess.call(['cat'] + infiles, stdout=out)\n if rc != 0:\n msg = \"failed with return code {0:d}: cat {1!r} > {2!r} \".format(rc, \" \".join(infiles), target)\n logger.exception(msg)\n raise OSError(errno.EIO, msg, target)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef activate_subplot(numPlot):\n # see http://www.mail-archive.com/matplotlib-users@lists.sourceforge.net/msg07156.html\n from pylab import gcf, axes\n numPlot -= 1 # index is 0-based, plots are 1-based\n return axes(gcf().get_axes()[numPlot])", "response": "Make subplot numPlot active on the canvas."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef remove_legend(ax=None):\n from pylab import gca, draw\n if ax is None:\n ax = gca()\n ax.legend_ = None\n draw()", "response": "Remove legend for axes or gca.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef number_pdbs(*args, **kwargs):\n\n format = kwargs.pop('format', \"%(num)04d\")\n name_format = \"%(prefix)s\" + format +\".%(suffix)s\"\n\n for f in itertools.chain.from_iterable(map(glob.glob, args)):\n m = NUMBERED_PDB.search(f)\n if m is None:\n continue\n num = int(m.group('NUMBER'))\n prefix = m.group('PREFIX')\n suffix = m.group('SUFFIX')\n newname = name_format % vars()\n logger.info(\"Renaming {f!r} --> {newname!r}\".format(**vars()))\n try:\n os.rename(f, newname)\n except OSError:\n logger.exception(\"renaming failed\")", "response": "Rename pdbs x1. pdb... x345. pdb... x0345. pdb... x0345. pdb... x01. pdb... x0345. pdb... x01. pdb... x0345. pdb... x01. pdb... x0345. pdb... x01. pdb... x01. pdb... x01. pdb... x0345. pdb... x01. pdb... x01. pdb... x01. pdb... x01. pdb"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ninitialize the current filename of the object.", "response": "def _init_filename(self, filename=None, ext=None):\n \"\"\"Initialize the current filename :attr:`FileUtils.real_filename` of the object.\n\n Bit of a hack.\n\n - The first invocation must have ``filename != None``; this will set a\n default filename with suffix :attr:`FileUtils.default_extension`\n unless another one was supplied.\n\n - Subsequent invocations either change the filename accordingly or\n ensure that the default filename is set with the proper suffix.\n\n \"\"\"\n\n extension = ext or self.default_extension\n filename = self.filename(filename, ext=extension, use_my_ext=True, set_default=True)\n #: Current full path of the object for reading and writing I/O.\n self.real_filename = os.path.realpath(filename)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nsupply a filename for the class object.", "response": "def filename(self,filename=None,ext=None,set_default=False,use_my_ext=False):\n \"\"\"Supply a file name for the class object.\n\n Typical uses::\n\n fn = filename() ---> \n fn = filename('name.ext') ---> 'name'\n fn = filename(ext='pickle') ---> '.pickle'\n fn = filename('name.inp','pdf') --> 'name.pdf'\n fn = filename('foo.pdf',ext='png',use_my_ext=True) --> 'foo.pdf'\n\n The returned filename is stripped of the extension\n (``use_my_ext=False``) and if provided, another extension is\n appended. Chooses a default if no filename is given.\n\n Raises a ``ValueError`` exception if no default file name is known.\n\n If ``set_default=True`` then the default filename is also set.\n\n ``use_my_ext=True`` lets the suffix of a provided filename take\n priority over a default ``ext`` tension.\n\n .. versionchanged:: 0.3.1\n An empty string as *ext* = \"\" will suppress appending an extension.\n \"\"\"\n if filename is None:\n if not hasattr(self,'_filename'):\n self._filename = None # add attribute to class\n if self._filename:\n filename = self._filename\n else:\n raise ValueError(\"A file name is required because no default file name was defined.\")\n my_ext = None\n else:\n filename, my_ext = os.path.splitext(filename)\n if set_default: # replaces existing default file name\n self._filename = filename\n if my_ext and use_my_ext:\n ext = my_ext\n if ext is not None:\n if ext.startswith(os.extsep):\n ext = ext[1:] # strip a dot to avoid annoying mistakes\n if ext != \"\":\n filename = filename + os.extsep + ext\n return filename"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_file_exists(self, filename, resolve='exception', force=None):\n def _warn(x):\n msg = \"File {0!r} already exists.\".format(x)\n logger.warn(msg)\n warnings.warn(msg)\n return True\n def _raise(x):\n msg = \"File {0!r} already exists.\".format(x)\n logger.error(msg)\n raise IOError(errno.EEXIST, x, msg)\n solutions = {'ignore': lambda x: False, # file exists, but we pretend that it doesn't\n 'indicate': lambda x: True, # yes, file exists\n 'warn': _warn,\n 'warning': _warn,\n 'exception': _raise,\n 'raise': _raise,\n }\n\n if force is True:\n resolve = 'ignore'\n elif force is False:\n resolve = 'exception'\n\n if not os.path.isfile(filename):\n return False\n else:\n return solutions[resolve](filename)", "response": "Check if a file exists."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef infix_filename(self, name, default, infix, ext=None):\n if name is None:\n p, oldext = os.path.splitext(default)\n if ext is None:\n ext = oldext\n if ext.startswith(os.extsep):\n ext = ext[1:]\n name = self.filename(p+infix, ext=ext)\n return name", "response": "Returns the filename of the infix file."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef strftime(self, fmt=\"%d:%H:%M:%S\"):\n substitutions = {\n \"%d\": str(self.days),\n \"%H\": \"{0:02d}\".format(self.dhours),\n \"%h\": str(24*self.days + self.dhours),\n \"%M\": \"{0:02d}\".format(self.dminutes),\n \"%S\": \"{0:02d}\".format(self.dseconds),\n }\n s = fmt\n for search, replacement in substitutions.items():\n s = s.replace(search, replacement)\n return s", "response": "Primitive string formatter.\n\n The only directives understood are the following:\n ============ ==========================\n Directive meaning\n ============ ==========================\n %d day as integer\n %H hour [00-23]\n %h hours including days\n %M minute as integer [00-59]\n %S second as integer [00-59]\n ============ =========================="} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef start_logging(logfile=\"gromacs.log\"):\n from . import log\n log.create(\"gromacs\", logfile=logfile)\n logging.getLogger(\"gromacs\").info(\"GromacsWrapper %s STARTED logging to %r\",\n __version__, logfile)", "response": "Start logging of messages to file and console."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nstops logging to logfile and console.", "response": "def stop_logging():\n \"\"\"Stop logging to logfile and console.\"\"\"\n from . import log\n logger = logging.getLogger(\"gromacs\")\n logger.info(\"GromacsWrapper %s STOPPED logging\", get_version())\n log.clear_handlers(logger)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nsets the : meth:`warnings. simplefilter to action.", "response": "def filter_gromacs_warnings(action, categories=None):\n \"\"\"Set the :meth:`warnings.simplefilter` to *action*.\n\n *categories* must be a list of warning classes or strings.\n ``None`` selects the defaults, :data:`gromacs.less_important_warnings`.\n \"\"\"\n\n if categories is None:\n categories = less_important_warnings\n for c in categories:\n try:\n w = globals()[c]\n except KeyError:\n w = c\n if not issubclass(w, Warning):\n raise TypeError(\"{0!r} is neither a Warning nor the name of a Gromacs warning.\".format(c))\n warnings.simplefilter(action, category=w)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef tool_factory(clsname, name, driver, base=GromacsCommand):\n clsdict = {\n 'command_name': name,\n 'driver': driver,\n '__doc__': property(base._get_gmx_docs)\n }\n return type(clsname, (base,), clsdict)", "response": "Factory for GromacsCommand derived types."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef find_executables(path):\n execs = []\n for exe in os.listdir(path):\n fullexe = os.path.join(path, exe)\n if (os.access(fullexe, os.X_OK) and not os.path.isdir(fullexe) and\n exe not in ['GMXRC', 'GMXRC.bash', 'GMXRC.csh', 'GMXRC.zsh',\n 'demux.pl', 'xplor2gmx.pl']):\n execs.append(exe)\n return execs", "response": "Find executables in a directory excluding some know commands."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_v5_tools():\n logger.debug(\"Loading 2018/2016/5.x tools...\")\n\n drivers = config.get_tool_names()\n\n if len(drivers) == 0 and 'GMXBIN' in os.environ:\n drivers = find_executables(os.environ['GMXBIN'])\n\n if len(drivers) == 0 or len(drivers) > 4:\n drivers = ['gmx', 'gmx_d', 'gmx_mpi', 'gmx_mpi_d']\n\n append = config.cfg.getboolean('Gromacs', 'append_suffix', fallback=True)\n\n tools = {}\n for driver in drivers:\n suffix = driver.partition('_')[2]\n try:\n out = subprocess.check_output([driver, '-quiet', 'help',\n 'commands'])\n for line in out.splitlines()[5:-1]:\n line = str(line.decode('ascii')) # Python 3: byte string -> str, Python 2: normal string\n if line[4] != ' ':\n name = line[4:line.index(' ', 4)]\n fancy = make_valid_identifier(name)\n if suffix and append:\n fancy = '{0!s}_{1!s}'.format(fancy, suffix)\n tools[fancy] = tool_factory(fancy, name, driver)\n except (subprocess.CalledProcessError, OSError):\n pass\n\n if not tools:\n errmsg = \"Failed to load 2018/2016/5.x tools (tried drivers: {})\".format(drivers)\n logger.debug(errmsg)\n raise GromacsToolLoadingError(errmsg)\n logger.debug(\"Loaded {0} v5 tools successfully!\".format(len(tools)))\n return tools", "response": "Load Gromacs 2018 - 2017 - 5. x tools."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nloading Gromacs 4. x tools automatically using some heuristic.", "response": "def load_v4_tools():\n \"\"\" Load Gromacs 4.x tools automatically using some heuristic.\n\n Tries to load tools (1) in configured tool groups (2) and fails back to\n automatic detection from ``GMXBIN`` (3) then to a prefilled list.\n\n Also load any extra tool configured in ``~/.gromacswrapper.cfg``\n\n :return: dict mapping tool names to GromacsCommand classes\n \"\"\"\n logger.debug(\"Loading v4 tools...\")\n\n names = config.get_tool_names()\n\n if len(names) == 0 and 'GMXBIN' in os.environ:\n names = find_executables(os.environ['GMXBIN'])\n\n if len(names) == 0 or len(names) > len(V4TOOLS) * 4:\n names = list(V4TOOLS)\n\n names.extend(config.get_extra_tool_names())\n\n tools = {}\n for name in names:\n fancy = make_valid_identifier(name)\n tools[fancy] = tool_factory(fancy, name, None)\n\n if not tools:\n errmsg = \"Failed to load v4 tools\"\n logger.debug(errmsg)\n raise GromacsToolLoadingError(errmsg)\n logger.debug(\"Loaded {0} v4 tools successfully!\".format(len(tools)))\n return tools"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntaking one or more index files and optionally one structure file and returns a new merged index file.", "response": "def merge_ndx(*args):\n \"\"\" Takes one or more index files and optionally one structure file and\n returns a path for a new merged index file.\n\n :param args: index files and zero or one structure file\n :return: path for the new merged index file\n \"\"\"\n ndxs = []\n struct = None\n for fname in args:\n if fname.endswith('.ndx'):\n ndxs.append(fname)\n else:\n if struct is not None:\n raise ValueError(\"only one structure file supported\")\n struct = fname\n\n fd, multi_ndx = tempfile.mkstemp(suffix='.ndx', prefix='multi_')\n os.close(fd)\n atexit.register(os.unlink, multi_ndx)\n\n if struct:\n make_ndx = registry['Make_ndx'](f=struct, n=ndxs, o=multi_ndx)\n else:\n make_ndx = registry['Make_ndx'](n=ndxs, o=multi_ndx)\n\n _, _, _ = make_ndx(input=['q'], stdout=False, stderr=False)\n return multi_ndx"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef read(self, filename=None):\n self._init_filename(filename)\n\n data = odict()\n with open(self.real_filename) as ndx:\n current_section = None\n for line in ndx:\n line = line.strip()\n if len(line) == 0:\n continue\n m = self.SECTION.match(line)\n if m:\n current_section = m.group('name')\n data[current_section] = [] # can fail if name not legal python key\n continue\n if current_section is not None:\n data[current_section].extend(map(int, line.split()))\n\n super(NDX,self).update(odict([(name, self._transform(atomnumbers))\n for name, atomnumbers in data.items()]))", "response": "Read and parse index file filename *filename*."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nwriting the index file to filename or overwrite the file that the index was read from.", "response": "def write(self, filename=None, ncol=ncol, format=format):\n \"\"\"Write index file to *filename* (or overwrite the file that the index was read from)\"\"\"\n with open(self.filename(filename, ext='ndx'), 'w') as ndx:\n for name in self:\n atomnumbers = self._getarray(name) # allows overriding\n ndx.write('[ {0!s} ]\\n'.format(name))\n for k in range(0, len(atomnumbers), ncol):\n line = atomnumbers[k:k+ncol].astype(int) # nice formatting in ncol-blocks\n n = len(line)\n ndx.write((\" \".join(n*[format])+'\\n') % tuple(line))\n ndx.write('\\n')"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef ndxlist(self):\n return [{'name': name, 'natoms': len(atomnumbers), 'nr': nr+1} for\n nr,(name,atomnumbers) in enumerate(self.items())]", "response": "Return a list of all the n - dimensional group names and numbers in the current cbook."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an index group that contains atoms from all groupnames.", "response": "def join(self, *groupnames):\n \"\"\"Return an index group that contains atoms from all *groupnames*.\n\n The method will silently ignore any groups that are not in the\n index.\n\n **Example**\n\n Always make a solvent group from water and ions, even if not\n all ions are present in all simulations::\n\n I['SOLVENT'] = I.join('SOL', 'NA+', 'K+', 'CL-')\n \"\"\"\n return self._sum([self[k] for k in groupnames if k in self])"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a 1D array where masks jumps > threshold and other is None.", "response": "def break_array(a, threshold=numpy.pi, other=None):\n \"\"\"Create a array which masks jumps >= threshold.\n\n Extra points are inserted between two subsequent values whose\n absolute difference differs by more than threshold (default is\n pi).\n\n Other can be a secondary array which is also masked according to\n *a*.\n\n Returns (*a_masked*, *other_masked*) (where *other_masked* can be\n ``None``)\n \"\"\"\n assert len(a.shape) == 1, \"Only 1D arrays supported\"\n\n if other is not None and a.shape != other.shape:\n raise ValueError(\"arrays must be of identical shape\")\n\n # jump occurs after the index in break\n breaks = numpy.where(numpy.abs(numpy.diff(a)) >= threshold)[0]\n # insert a blank after\n breaks += 1\n\n # is this needed?? -- no, but leave it here as a reminder\n #f2 = numpy.diff(a, 2)\n #up = (f2[breaks - 1] >= 0) # >0: up, <0: down\n # sort into up and down breaks:\n #breaks_up = breaks[up]\n #breaks_down = breaks[~up]\n\n # new array b including insertions for all the breaks\n m = len(breaks)\n b = numpy.empty((len(a) + m))\n # calculate new indices for breaks in b, taking previous insertions into account\n b_breaks = breaks + numpy.arange(m)\n mask = numpy.zeros_like(b, dtype=numpy.bool)\n mask[b_breaks] = True\n b[~mask] = a\n b[mask] = numpy.NAN\n\n if other is not None:\n c = numpy.empty_like(b)\n c[~mask] = other\n c[mask] = numpy.NAN\n ma_c = numpy.ma.array(c, mask=mask)\n else:\n ma_c = None\n\n return numpy.ma.array(b, mask=mask), ma_c"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nwrite array to xvg file filename in NXY format.", "response": "def write(self, filename=None):\n \"\"\"Write array to xvg file *filename* in NXY format.\n\n .. Note:: Only plain files working at the moment, not compressed.\n \"\"\"\n self._init_filename(filename)\n with utilities.openany(self.real_filename, 'w') as xvg:\n xvg.write(\"# xmgrace compatible NXY data file\\n\"\n \"# Written by gromacs.formats.XVG()\\n\")\n xvg.write(\"# :columns: {0!r}\\n\".format(self.names))\n for xyy in self.array.T:\n xyy.tofile(xvg, sep=\" \", format=\"%-8s\") # quick and dirty ascii output...--no compression!\n xvg.write('\\n')"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nrepresents data as a masked array.", "response": "def ma(self):\n \"\"\"Represent data as a masked array.\n\n The array is returned with column-first indexing, i.e. for a data file with\n columns X Y1 Y2 Y3 ... the array a will be a[0] = X, a[1] = Y1, ... .\n\n inf and nan are filtered via :func:`numpy.isfinite`.\n \"\"\"\n a = self.array\n return numpy.ma.MaskedArray(a, mask=numpy.logical_not(numpy.isfinite(a)))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting and change the parameters for calculations with \u00b5s correlation functions.", "response": "def set_correlparameters(self, **kwargs):\n \"\"\"Set and change the parameters for calculations with correlation functions.\n\n The parameters persist until explicitly changed.\n\n :Keywords:\n *nstep*\n only process every *nstep* data point to speed up the FFT; if\n left empty a default is chosen that produces roughly 25,000 data\n points (or whatever is set in *ncorrel*)\n *ncorrel*\n If no *nstep* is supplied, aim at using *ncorrel* data points for\n the FFT; sets :attr:`XVG.ncorrel` [25000]\n *force*\n force recalculating correlation data even if cached values are\n available\n *kwargs*\n see :func:`numkit.timeseries.tcorrel` for other options\n\n .. SeeAlso: :attr:`XVG.error` for details and references.\n \"\"\"\n self.ncorrel = kwargs.pop('ncorrel', self.ncorrel) or 25000\n nstep = kwargs.pop('nstep', None)\n if nstep is None:\n # good step size leads to ~25,000 data points\n nstep = len(self.array[0])/float(self.ncorrel)\n nstep = int(numpy.ceil(nstep)) # catch small data sets\n kwargs['nstep'] = nstep\n self.__correlkwargs.update(kwargs) # only contains legal kw for numkit.timeseries.tcorrel or force\n return self.__correlkwargs"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef parse(self, stride=None):\n if stride is None:\n stride = self.stride\n self.corrupted_lineno = []\n irow = 0 # count rows of data\n # cannot use numpy.loadtxt() because xvg can have two types of 'comment' lines\n with utilities.openany(self.real_filename) as xvg:\n rows = []\n ncol = None\n for lineno,line in enumerate(xvg):\n line = line.strip()\n if len(line) == 0:\n continue\n if \"label\" in line and \"xaxis\" in line:\n self.xaxis = line.split('\"')[-2]\n if \"label\" in line and \"yaxis\" in line:\n self.yaxis = line.split('\"')[-2]\n if line.startswith(\"@ legend\"):\n if not \"legend\" in self.metadata: self.metadata[\"legend\"] = []\n self.metadata[\"legend\"].append(line.split(\"legend \")[-1])\n if line.startswith(\"@ s\") and \"subtitle\" not in line:\n name = line.split(\"legend \")[-1].replace('\"','').strip()\n self.names.append(name)\n if line.startswith(('#', '@')) :\n continue\n if line.startswith('&'):\n raise NotImplementedError('{0!s}: Multi-data not supported, only simple NXY format.'.format(self.real_filename))\n # parse line as floats\n try:\n row = [float(el) for el in line.split()]\n except:\n if self.permissive:\n self.logger.warn(\"%s: SKIPPING unparsable line %d: %r\",\n self.real_filename, lineno+1, line)\n self.corrupted_lineno.append(lineno+1)\n continue\n self.logger.error(\"%s: Cannot parse line %d: %r\",\n self.real_filename, lineno+1, line)\n raise\n # check for same number of columns as in previous step\n if ncol is not None and len(row) != ncol:\n if self.permissive:\n self.logger.warn(\"%s: SKIPPING line %d with wrong number of columns: %r\",\n self.real_filename, lineno+1, line)\n self.corrupted_lineno.append(lineno+1)\n continue\n errmsg = \"{0!s}: Wrong number of columns in line {1:d}: {2!r}\".format(self.real_filename, lineno+1, line)\n self.logger.error(errmsg)\n raise IOError(errno.ENODATA, errmsg, self.real_filename)\n # finally: a good line\n if irow % stride == 0:\n ncol = len(row)\n rows.append(row)\n irow += 1\n try:\n self.__array = numpy.array(rows).transpose() # cache result\n except:\n self.logger.error(\"%s: Failed reading XVG file, possibly data corrupted. \"\n \"Check the last line of the file...\", self.real_filename)\n raise\n finally:\n del rows", "response": "Read and cache the file as a numpy array."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nplotting the xvg file data.", "response": "def plot(self, **kwargs):\n \"\"\"Plot xvg file data.\n\n The first column of the data is always taken as the abscissa\n X. Additional columns are plotted as ordinates Y1, Y2, ...\n\n In the special case that there is only a single column then this column\n is plotted against the index, i.e. (N, Y).\n\n :Keywords:\n *columns* : list\n Select the columns of the data to be plotted; the list\n is used as a numpy.array extended slice. The default is\n to use all columns. Columns are selected *after* a transform.\n *transform* : function\n function ``transform(array) -> array`` which transforms\n the original array; must return a 2D numpy array of\n shape [X, Y1, Y2, ...] where X, Y1, ... are column\n vectors. By default the transformation is the\n identity [``lambda x: x``].\n *maxpoints* : int\n limit the total number of data points; matplotlib has issues processing\n png files with >100,000 points and pdfs take forever to display. Set to\n ``None`` if really all data should be displayed. At the moment we simply\n decimate the data at regular intervals. [10000]\n *method*\n method to decimate the data to *maxpoints*, see :meth:`XVG.decimate`\n for details\n *color*\n single color (used for all plots); sequence of colors\n (will be repeated as necessary); or a matplotlib\n colormap (e.g. \"jet\", see :mod:`matplotlib.cm`). The\n default is to use the :attr:`XVG.default_color_cycle`.\n *ax*\n plot into given axes or create new one if ``None`` [``None``]\n *kwargs*\n All other keyword arguments are passed on to :func:`matplotlib.pyplot.plot`.\n\n :Returns:\n *ax*\n axes instance\n \"\"\"\n columns = kwargs.pop('columns', Ellipsis) # slice for everything\n maxpoints = kwargs.pop('maxpoints', self.maxpoints_default)\n transform = kwargs.pop('transform', lambda x: x) # default is identity transformation\n method = kwargs.pop('method', \"mean\")\n ax = kwargs.pop('ax', None)\n\n if columns is Ellipsis or columns is None:\n columns = numpy.arange(self.array.shape[0])\n if len(columns) == 0:\n raise MissingDataError(\"plot() needs at least one column of data\")\n\n if len(self.array.shape) == 1 or self.array.shape[0] == 1:\n # special case: plot against index; plot would do this automatically but\n # we'll just produce our own xdata and pretend that this was X all along\n a = numpy.ravel(self.array)\n X = numpy.arange(len(a))\n a = numpy.vstack((X, a))\n columns = [0] + [c+1 for c in columns]\n else:\n a = self.array\n\n color = kwargs.pop('color', self.default_color_cycle)\n try:\n cmap = matplotlib.cm.get_cmap(color)\n colors = cmap(matplotlib.colors.Normalize()(numpy.arange(len(columns[1:]), dtype=float)))\n except TypeError:\n colors = cycle(utilities.asiterable(color))\n\n if ax is None:\n ax = plt.gca()\n\n # (decimate/smooth o slice o transform)(array)\n a = self.decimate(method, numpy.asarray(transform(a))[columns], maxpoints=maxpoints)\n\n # now deal with infs, nans etc AFTER all transformations (needed for plotting across inf/nan)\n ma = numpy.ma.MaskedArray(a, mask=numpy.logical_not(numpy.isfinite(a)))\n\n # finally plot (each column separately to catch empty sets)\n for column, color in zip(range(1,len(columns)), colors):\n if len(ma[column]) == 0:\n warnings.warn(\"No data to plot for column {column:d}\".format(**vars()), category=MissingDataWarning)\n kwargs['color'] = color\n ax.plot(ma[0], ma[column], **kwargs) # plot all other columns in parallel\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplots data like XVG. plot with the range of all data shown.", "response": "def plot_coarsened(self, **kwargs):\n \"\"\"Plot data like :meth:`XVG.plot` with the range of **all** data shown.\n\n Data are reduced to *maxpoints* (good results are obtained\n with low values such as 100) and the actual range of observed\n data is plotted as a translucent error band around the mean.\n\n Each column in *columns* (except the abscissa, i.e. the first\n column) is decimated (with :meth:`XVG.decimate`) and the range\n of data is plotted alongside the mean using\n :meth:`XVG.errorbar` (see for arguments). Additional\n arguments:\n\n :Kewords:\n *maxpoints*\n number of points (bins) to coarsen over\n *color*\n single color (used for all plots); sequence of colors\n (will be repeated as necessary); or a matplotlib\n colormap (e.g. \"jet\", see :mod:`matplotlib.cm`). The\n default is to use the :attr:`XVG.default_color_cycle`.\n *method*\n Method to coarsen the data. See :meth:`XVG.decimate`\n\n The *demean* keyword has no effect as it is required to be ``True``.\n\n .. SeeAlso:: :meth:`XVG.plot`, :meth:`XVG.errorbar` and :meth:`XVG.decimate`\n \"\"\"\n ax = kwargs.pop('ax', None)\n columns = kwargs.pop('columns', Ellipsis) # slice for everything\n if columns is Ellipsis or columns is None:\n columns = numpy.arange(self.array.shape[0])\n if len(columns) < 2:\n raise MissingDataError(\"plot_coarsened() assumes that there is at least one column \"\n \"of data for the abscissa and one or more for the ordinate.\")\n\n color = kwargs.pop('color', self.default_color_cycle)\n try:\n cmap = matplotlib.cm.get_cmap(color)\n colors = cmap(matplotlib.colors.Normalize()(numpy.arange(len(columns[1:]), dtype=float)))\n except TypeError:\n colors = cycle(utilities.asiterable(color))\n\n if ax is None:\n ax = plt.gca()\n\n t = columns[0]\n kwargs['demean'] = True\n kwargs['ax'] = ax\n for column, color in zip(columns[1:], colors):\n kwargs['color'] = color\n self.errorbar(columns=[t, column, column], **kwargs)\n return ax"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nplots an errorbar plot for a single time series with errors.", "response": "def errorbar(self, **kwargs):\n \"\"\"errorbar plot for a single time series with errors.\n\n Set *columns* keyword to select [x, y, dy] or [x, y, dx, dy],\n e.g. ``columns=[0,1,2]``. See :meth:`XVG.plot` for\n details. Only a single timeseries can be plotted and the user\n needs to select the appropriate columns with the *columns*\n keyword.\n\n By default, the data are decimated (see :meth:`XVG.plot`) for\n the default of *maxpoints* = 10000 by averaging data in\n *maxpoints* bins.\n\n x,y,dx,dy data can plotted with error bars in the x- and\n y-dimension (use *filled* = ``False``).\n\n For x,y,dy use *filled* = ``True`` to fill the region between\n y\u00b1dy. *fill_alpha* determines the transparency of the fill\n color. *filled* = ``False`` will draw lines for the error\n bars. Additional keywords are passed to\n :func:`pylab.errorbar`.\n\n By default, the errors are decimated by plotting the 5% and\n 95% percentile of the data in each bin. The percentile can be\n changed with the *percentile* keyword; e.g. *percentile* = 1\n will plot the 1% and 99% perentile (as will *percentile* =\n 99).\n\n The *error_method* keyword can be used to compute errors as\n the root mean square sum (*error_method* = \"rms\") across each\n bin instead of percentiles (\"percentile\"). The value of the\n keyword *demean* is applied to the decimation of error data\n alone.\n\n .. SeeAlso::\n\n :meth:`XVG.plot` lists keywords common to both methods.\n \"\"\"\n ax = kwargs.pop('ax', None)\n color = kwargs.pop('color', 'black')\n filled = kwargs.pop('filled', True)\n fill_alpha = kwargs.pop('fill_alpha', 0.2)\n\n kwargs.setdefault('capsize', 0)\n kwargs.setdefault('elinewidth', 1)\n kwargs.setdefault('ecolor', color)\n kwargs.setdefault('alpha', 0.3)\n kwargs.setdefault('fmt', None)\n\n columns = kwargs.pop('columns', Ellipsis) # slice for everything\n maxpoints = kwargs.pop('maxpoints', self.maxpoints_default)\n transform = kwargs.pop('transform', lambda x: x) # default is identity transformation\n method = kwargs.pop('method', \"mean\")\n if method != \"mean\":\n raise NotImplementedError(\"For errors only method == 'mean' is supported.\")\n error_method = kwargs.pop('error_method', \"percentile\") # can also use 'rms' and 'error'\n percentile = numpy.abs(kwargs.pop('percentile', 95.))\n demean = kwargs.pop('demean', False)\n\n # order: (decimate/smooth o slice o transform)(array)\n try:\n data = numpy.asarray(transform(self.array))[columns]\n except IndexError:\n raise MissingDataError(\"columns {0!r} are not suitable to index the transformed array, possibly not eneough data\".format(columns))\n if data.shape[-1] == 0:\n raise MissingDataError(\"There is no data to be plotted.\")\n a = numpy.zeros((data.shape[0], maxpoints), dtype=numpy.float64)\n a[0:2] = self.decimate(\"mean\", data[0:2], maxpoints=maxpoints)\n error_data = numpy.vstack((data[0], data[2:]))\n if error_method == \"percentile\":\n if percentile > 50:\n upper_per = percentile\n lower_per = 100 - percentile\n else:\n upper_per = 100 - percentile\n lower_per = percentile\n # demean generally does not make sense with the percentiles (but for analysing\n # the regularised data itself we use this as a flag --- see below!)\n upper = a[2:] = self.decimate(\"percentile\", error_data, maxpoints=maxpoints,\n per=upper_per, demean=False)[1:]\n lower = self.decimate(\"percentile\", error_data, maxpoints=maxpoints,\n per=lower_per, demean=False)[1:]\n else:\n a[2:] = self.decimate(error_method, error_data, maxpoints=maxpoints, demean=demean)[1:]\n lower = None\n\n # now deal with infs, nans etc AFTER all transformations (needed for plotting across inf/nan)\n ma = numpy.ma.MaskedArray(a, mask=numpy.logical_not(numpy.isfinite(a)))\n if lower is not None:\n mlower = numpy.ma.MaskedArray(lower, mask=numpy.logical_not(numpy.isfinite(lower)))\n\n # finally plot\n X = ma[0] # abscissa set separately\n Y = ma[1]\n try:\n kwargs['yerr'] = ma[3]\n kwargs['xerr'] = ma[2]\n except IndexError:\n try:\n kwargs['yerr'] = ma[2]\n except IndexError:\n raise TypeError(\"Either too few columns selected or data does not have a error column\")\n\n if ax is None:\n ax = plt.gca()\n\n if filled:\n # can only plot dy\n if error_method == \"percentile\":\n if demean:\n # signal that we are looking at percentiles of an observable and not error\n y1 = mlower[-1]\n y2 = kwargs['yerr']\n else:\n # percentiles of real errors (>0)\n y1 = Y - mlower[-1]\n y2 = Y + kwargs['yerr']\n else:\n y1 = Y - kwargs['yerr']\n y2 = Y + kwargs['yerr']\n ax.fill_between(X, y1, y2, color=color, alpha=fill_alpha)\n else:\n if error_method == \"percentile\":\n # errorbars extend to different lengths;\n if demean:\n kwargs['yerr'] = numpy.vstack((mlower[-1], kwargs['yerr']))\n else:\n kwargs['yerr'] = numpy.vstack((Y - mlower[-1], Y + kwargs['yerr']))\n try:\n # xerr only makes sense when the data is a real\n # error so we don't even bother with demean=?\n kwargs['xerr'] = numpy.vstack((X - mlower[0], X + kwargs['xerr']))\n except (KeyError, IndexError):\n pass\n ax.errorbar(X, Y, **kwargs)\n\n # clean up args for plot\n for kw in \"yerr\", \"xerr\", \"capsize\", \"ecolor\", \"elinewidth\", \"fmt\":\n kwargs.pop(kw, None)\n kwargs['alpha'] = 1.0\n\n ax.plot(X, Y, color=color, **kwargs)\n\n return ax"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndecimates data in a given number of data points.", "response": "def decimate(self, method, a, maxpoints=10000, **kwargs):\n \"\"\"Decimate data *a* to *maxpoints* using *method*.\n\n If *a* is a 1D array then it is promoted to a (2, N) array\n where the first column simply contains the index.\n\n If the array contains fewer than *maxpoints* points or if\n *maxpoints* is ``None`` then it is returned as it is. The\n default for *maxpoints* is 10000.\n\n Valid values for the reduction *method*:\n\n * \"mean\", uses :meth:`XVG.decimate_mean` to coarse grain by\n averaging the data in bins along the time axis\n\n * \"circmean\", uses :meth:`XVG.decimate_circmean` to coarse\n grain by calculating the circular mean of the data in bins\n along the time axis. Use additional keywords *low* and\n *high* to set the limits. Assumes that the data are in\n degrees.\n\n * \"min\" and \"max* select the extremum in each bin\n\n * \"rms\", uses :meth:`XVG.decimate_rms` to coarse grain by\n computing the root mean square sum of the data in bins\n along the time axis (for averaging standard deviations and\n errors)\n\n * \"percentile\" with keyword *per*: :meth:`XVG.decimate_percentile`\n reduces data in each bin to the percentile *per*\n\n * \"smooth\", uses :meth:`XVG.decimate_smooth` to subsample\n from a smoothed function (generated with a running average\n of the coarse graining step size derived from the original\n number of data points and *maxpoints*)\n\n :Returns: numpy array ``(M', N')`` from a ``(M', N)`` array\n with ``M' == M`` (except when ``M == 1``, see above)\n and ``N' <= N`` (``N'`` is *maxpoints*).\n \"\"\"\n methods = {'mean': self.decimate_mean,\n 'min': self.decimate_min,\n 'max': self.decimate_max,\n 'smooth': self.decimate_smooth,\n 'rms': self.decimate_rms,\n 'percentile': self.decimate_percentile,\n 'error': self.decimate_error, # undocumented, not working well\n 'circmean': self.decimate_circmean,\n }\n if len(a.shape) == 1:\n # add first column as index\n # (probably should do this in class/init anyway...)\n X = numpy.arange(len(a))\n a = numpy.vstack([X, a])\n ny = a.shape[-1] # assume 1D or 2D array with last dimension varying fastest\n if maxpoints is None or ny <= maxpoints:\n return a\n return methods[method](a, maxpoints, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning data a * mean - decimated on maxpoints * bins and calculates the weighted average of each bin in each column into maxpoints * bins and returns a new array of the same dimensions as a.", "response": "def decimate_mean(self, a, maxpoints, **kwargs):\n \"\"\"Return data *a* mean-decimated on *maxpoints*.\n\n Histograms each column into *maxpoints* bins and calculates\n the weighted average in each bin as the decimated data, using\n :func:`numkit.timeseries.mean_histogrammed_function`. The coarse grained\n time in the first column contains the centers of the histogram\n time.\n\n If *a* contains <= *maxpoints* then *a* is simply returned;\n otherwise a new array of the same dimensions but with a\n reduced number of *maxpoints* points is returned.\n\n .. Note::\n\n Assumes that the first column is time.\n\n \"\"\"\n return self._decimate(numkit.timeseries.mean_histogrammed_function, a, maxpoints, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns data a * circmean - decimated on maxpoints * bins and calculates the weighted circular mean in each bin.", "response": "def decimate_circmean(self, a, maxpoints, **kwargs):\n \"\"\"Return data *a* circmean-decimated on *maxpoints*.\n\n Histograms each column into *maxpoints* bins and calculates\n the weighted circular mean in each bin as the decimated data,\n using\n :func:`numkit.timeseries.circmean_histogrammed_function`. The\n coarse grained time in the first column contains the centers\n of the histogram time.\n\n If *a* contains <= *maxpoints* then *a* is simply returned;\n otherwise a new array of the same dimensions but with a\n reduced number of *maxpoints* points is returned.\n\n Keywords *low* and *high* can be used to set the\n boundaries. By default they are [-pi, +pi].\n\n This method returns a **masked** array where jumps are flagged\n by an insertion of a masked point.\n\n .. Note::\n\n Assumes that the first column is time and that the data are\n in **degrees**.\n\n .. Warning::\n\n Breaking of arrays only works properly with a two-column\n array because breaks are only inserted in the x-column\n (a[0]) where y1 = a[1] has a break.\n\n \"\"\"\n a_rad = numpy.vstack((a[0], numpy.deg2rad(a[1:])))\n b = self._decimate(numkit.timeseries.circmean_histogrammed_function, a_rad, maxpoints, **kwargs)\n y_ma, x_ma = break_array(b[1], threshold=numpy.pi, other=b[0])\n v = [y_ma]\n for y in b[2:]:\n v.append(break_array(y, threshold=numpy.pi)[0])\n if v[-1].shape != v[0].shape:\n raise ValueError(\"y dimensions have different breaks: you MUST deal with them separately\")\n return numpy.vstack((x_ma, numpy.rad2deg(v)))"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decimate_min(self, a, maxpoints, **kwargs):\n return self._decimate(numkit.timeseries.min_histogrammed_function, a, maxpoints, **kwargs)", "response": "Return data *a* min - decimated on maxpoints."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef decimate_max(self, a, maxpoints, **kwargs):\n return self._decimate(numkit.timeseries.max_histogrammed_function, a, maxpoints, **kwargs)", "response": "Return data *a* max - decimated on maxpoints."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning data a * rms - decimated on maxpoints * bins and calculates the root mean square sum of each bin in each column into maxpoints * bins and returns a new array of the same dimensions as a.", "response": "def decimate_rms(self, a, maxpoints, **kwargs):\n \"\"\"Return data *a* rms-decimated on *maxpoints*.\n\n Histograms each column into *maxpoints* bins and calculates\n the root mean square sum in each bin as the decimated data,\n using :func:`numkit.timeseries.rms_histogrammed_function`. The coarse\n grained time in the first column contains the centers of the\n histogram time.\n\n If *a* contains <= *maxpoints* then *a* is simply returned;\n otherwise a new array of the same dimensions but with a\n reduced number of *maxpoints* points is returned.\n\n .. Note::\n\n Assumes that the first column is time.\n\n \"\"\"\n return self._decimate(numkit.timeseries.rms_histogrammed_function, a, maxpoints, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef decimate_percentile(self, a, maxpoints, **kwargs):\n return self._decimate(numkit.timeseries.percentile_histogrammed_function, a, maxpoints, **kwargs)", "response": "Return data *a* percentile - decimated on maxpoints."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning data a * error - decimated on maxpoints * bins and calculates an error estimate in each bin.", "response": "def decimate_error(self, a, maxpoints, **kwargs):\n \"\"\"Return data *a* error-decimated on *maxpoints*.\n\n Histograms each column into *maxpoints* bins and calculates an\n error estimate in each bin as the decimated data, using\n :func:`numkit.timeseries.error_histogrammed_function`. The\n coarse grained time in the first column contains the centers\n of the histogram time.\n\n If *a* contains <= *maxpoints* then *a* is simply returned;\n otherwise a new array of the same dimensions but with a\n reduced number of *maxpoints* points is returned.\n\n .. SeeAlso:: :func:`numkit.timeseries.tcorrel`\n\n .. Note::\n\n Assumes that the first column is time.\n\n Does not work very well because often there are too few\n datapoints to compute a good autocorrelation function.\n\n \"\"\"\n warnings.warn(\"Using undocumented decimate_error() is highly EXPERIMENTAL\",\n category=gromacs.exceptions.LowAccuracyWarning)\n return self._decimate(numkit.timeseries.error_histogrammed_function, a,\n maxpoints, **kwargs)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef decimate_smooth(self, a, maxpoints, window=\"flat\"):\n ny = a.shape[-1] # assume 1D or 2D array with last dimension varying fastest\n # reduce size by averaging oover stepsize steps and then just\n # picking every stepsize data points. (primitive --- can\n # leave out bits at the end or end up with almost twice of\n # maxpoints)\n stepsize = int(ny / maxpoints)\n if stepsize % 2 == 0:\n stepsize += 1 # must be odd for the running average/smoothing window\n out = numpy.empty_like(a)\n\n # smoothed\n out[0,:] = a[0]\n for i in range(1, a.shape[0]):\n # process columns because smooth() only handles 1D arrays :-p\n out[i,:] = numkit.timeseries.smooth(a[i], stepsize, window=window)\n\n if maxpoints == self.maxpoints_default: # only warn if user did not set maxpoints\n warnings.warn(\"Plot had %d datapoints > maxpoints = %d; decimated to %d regularly \"\n \"spaced points with smoothing (%r) over %d steps.\"\n % (ny, maxpoints, ny/stepsize, window, stepsize),\n category=AutoCorrectionWarning)\n return out[..., ::stepsize]", "response": "Return a smoothed version of a."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nparse a processed. top GROMACS topology file and return a dict containing the current section and molecule names and the CMAP parameters.", "response": "def _parse(self, fname):\n \"\"\"Parse a processed.top GROMACS topology file\n\n The function reads in the file line-by-line, and it's a bunch of 'elif' statements,\n writing parameter/atom line to current section/molecule.\n\n ParamTypes are added to self.xyztypes (AtomType goes to self.atomtypes).\n Params are added to current molecule (Atom goes to mol.atoms.append(atom))\n\n MoleculeTypes and Molecules are odd, and are added to\n * MoleculeType to :attr:`self.dict_molname_mol[mol.name] = mol`\n * Molecule to :attr:`self.molecules.append(self.dict_molname_mol[mname])`\n\n :obj:`curr_sec` variable stores to current section being read-in\n :obj:`mol` variable stores the current molecule being read-in\n :obj:`cmap_lines` are a little odd, since CMAP parameters are stored on multiple lines\n\n :Arguments:\n *fname*\n name of the processed.top file\n\n :Returns: None\n \"\"\"\n def _find_section(line):\n return line.strip('[').strip(']').strip()\n\n def _add_info(sys_or_mol, section, container):\n # like (mol, 'atomtypes', mol.atomtypes)\n if sys_or_mol.information.get(section, False) is False:\n sys_or_mol.information[section] = container\n\n mol = None # to hold the current mol\n curr_sec = None\n cmap_lines = []\n\n with open(fname) as f:\n for i_line, line in enumerate(f):\n\n # trimming\n if ';' in line:\n line = line[0:line.index(';')]\n line = line.strip()\n\n if line == '':\n continue\n\n if line[0] == '*':\n continue\n\n # the topology must be stand-alone (i.e. no includes)\n if line.startswith('#include'):\n msg = 'The topology file has \"#include\" statements.'\n msg+= ' You must provide a processed topology file that grompp creates.'\n raise ValueError(msg)\n\n # find sections\n if line[0] == '[':\n curr_sec = _find_section(line)\n self.found_sections.append(curr_sec)\n continue\n\n fields = line.split()\n\n if curr_sec == 'defaults':\n '''\n # ; nbfunc comb-rule gen-pairs fudgeLJ fudgeQQ\n #1 2 yes 0.5 0.8333\n '''\n assert len(fields) in [2, 5]\n self.defaults['nbfunc'] = int(fields[0])\n self.defaults['comb-rule'] = int(fields[1])\n if len(fields) == 5:\n\n self.defaults['gen-pairs'] = fields[2]\n self.defaults['fudgeLJ'] = float(fields[3])\n self.defaults['fudgeQQ'] = float(fields[4])\n\n elif curr_sec == 'atomtypes':\n '''\n # ;name at.num mass charge ptype sigma epsilon\n # ;name bond_type at.num mass charge ptype sigma epsilon\n # ;name mass charge ptype c6 c12\n\n '''\n if len(fields) not in (6,7,8):\n self.logger.warning('skipping atomtype line with neither 7 or 8 fields: \\n {0:s}'.format(line))\n continue\n\n #shift = 0 if len(fields) == 7 else 1\n shift = len(fields) - 7\n at = blocks.AtomType('gromacs')\n at.atype = fields[0]\n if shift == 1: at.bond_type = fields[1]\n\n at.mass = float(fields[2+shift])\n at.charge= float(fields[3+shift])\n\n particletype = fields[4+shift]\n assert particletype in ('A', 'S', 'V', 'D')\n if particletype not in ('A',):\n self.logger.warning('warning: non-atom particletype: \"{0:s}\"'.format(line))\n\n sig = float(fields[5+shift])\n eps = float(fields[6+shift])\n\n at.gromacs= {'param': {'lje':eps, 'ljl':sig, 'lje14':None, 'ljl14':None} }\n\n self.atomtypes.append(at)\n\n _add_info(self, curr_sec, self.atomtypes)\n\n\n # extend system.molecules\n elif curr_sec == 'moleculetype':\n assert len(fields) == 2\n\n mol = blocks.Molecule()\n\n mol.name = fields[0]\n mol.exclusion_numb = int(fields[1])\n\n self.dict_molname_mol[mol.name] = mol\n\n\n elif curr_sec == 'atoms':\n '''\n #id at_type res_nr residu_name at_name cg_nr charge mass typeB chargeB massB\n # 1 OC 1 OH O1 1 -1.32\n\n OR\n\n [ atoms ]\n ; id at type res nr residu name at name cg nr charge\n 1 OT 1 SOL OW 1 -0.834\n\n '''\n\n aserial = int(fields[0])\n atype = fields[1]\n resnumb = int(fields[2])\n resname = fields[3]\n aname = fields[4]\n cgnr = int(fields[5])\n charge = float(fields[6])\n rest = fields[7:]\n\n atom = blocks.Atom()\n atom.name = aname\n atom.atomtype= atype\n atom.number = aserial\n atom.resname = resname\n atom.resnumb = resnumb\n atom.charge = charge\n\n if rest:\n mass = float(rest[0])\n atom.mass = mass\n\n mol.atoms.append(atom)\n\n _add_info(mol, curr_sec, mol.atoms)\n\n elif curr_sec in ('pairtypes', 'pairs', 'pairs_nb'):\n '''\n section #at fu #param\n ---------------------------------\n pairs 2 1 V,W\n pairs 2 2 fudgeQQ, qi, qj, V, W\n pairs_nb 2 1 qi, qj, V, W\n\n '''\n\n ai, aj = fields[:2]\n fu = int(fields[2])\n assert fu in (1,2)\n\n pair = blocks.InteractionType('gromacs')\n if fu == 1:\n if curr_sec=='pairtypes':\n pair.atype1 = ai\n pair.atype2 = aj\n v, w = list(map(float, fields[3:5]))\n pair.gromacs = {'param': {'lje':None, 'ljl':None, 'lje14':w, 'ljl14':v}, 'func':fu }\n\n self.pairtypes.append(pair)\n _add_info(self, curr_sec, self.pairtypes)\n\n elif curr_sec == 'pairs':\n ai, aj = list( map(int, [ai,aj]) )\n pair.atom1 = mol.atoms[ai-1]\n pair.atom2 = mol.atoms[aj-1]\n pair.gromacs['func'] = fu\n\n mol.pairs.append(pair)\n _add_info(mol, curr_sec, mol.pairs)\n\n else:\n raise ValueError\n\n else:\n raise NotImplementedError('{0:s} with functiontype {1:d} is not supported'.format(curr_sec,fu))\n\n elif curr_sec == 'nonbond_params':\n '''\n ; typei typej f.type sigma epsilon\n ; f.type=1 means LJ (not buckingham)\n ; sigma&eps since mixing-rule = 2\n '''\n\n assert len(fields) == 5\n ai, aj = fields[:2]\n fu = int(fields[2])\n\n assert fu == 1\n sig = float(fields[3])\n eps = float(fields[4])\n\n nonbond_param = blocks.NonbondedParamType('gromacs')\n nonbond_param.atype1 = ai\n nonbond_param.atype2 = aj\n nonbond_param.gromacs['func'] = fu\n nonbond_param.gromacs['param'] = {'eps': eps, 'sig': sig}\n\n self.nonbond_params.append(nonbond_param)\n _add_info(self, curr_sec, self.nonbond_params)\n\n elif curr_sec in ('bondtypes', 'bonds'):\n '''\n section #at fu #param\n ----------------------------------\n bonds 2 1 2\n bonds 2 2 2\n bonds 2 3 3\n bonds 2 4 2\n bonds 2 5 ??\n bonds 2 6 2\n bonds 2 7 2\n bonds 2 8 ??\n bonds 2 9 ??\n bonds 2 10 4\n '''\n\n ai, aj = fields[:2]\n fu = int(fields[2])\n assert fu in (1,2,3,4,5,6,7,8,9,10)\n\n if fu != 1:\n raise NotImplementedError('function {0:d} is not yet supported'.format(fu))\n\n bond = blocks.BondType('gromacs')\n\n if fu == 1:\n if curr_sec == 'bondtypes':\n bond.atype1 = ai\n bond.atype2 = aj\n\n b0, kb = list(map(float, fields[3:5]))\n bond.gromacs = {'param':{'kb':kb, 'b0':b0}, 'func':fu}\n\n self.bondtypes.append(bond)\n _add_info(self, curr_sec, self.bondtypes)\n\n elif curr_sec == 'bonds':\n ai, aj = list(map(int, [ai, aj]))\n bond.atom1 = mol.atoms[ai-1]\n bond.atom2 = mol.atoms[aj-1]\n bond.gromacs['func'] = fu\n\n if len(fields) > 3:\n b0, kb = list(map(float, fields[3:5]))\n bond.gromacs = {'param':{'kb':kb, 'b0':b0}, 'func':fu}\n\n mol.bonds.append(bond)\n _add_info(mol, curr_sec, mol.bonds)\n\n else:\n raise NotImplementedError\n\n elif curr_sec in ('angletypes', 'angles'):\n '''\n section #at fu #param\n ----------------------------------\n angles 3 1 2\n angles 3 2 2\n angles 3 3 3\n angles 3 4 4\n angles 3 5 4\n angles 3 6 6\n angles 3 8 ??\n '''\n\n ai, aj , ak = fields[:3]\n fu = int(fields[3])\n assert fu in (1,2,3,4,5,6,8) # no 7\n\n if fu not in (1,2,5):\n raise NotImplementedError('function {0:d} is not yet supported'.format(fu))\n\n ang = blocks.AngleType('gromacs')\n if fu == 1:\n if curr_sec == 'angletypes':\n ang.atype1 = ai\n ang.atype2 = aj\n ang.atype3 = ak\n\n tetha0, ktetha = list(map(float, fields[4:6]))\n ang.gromacs = {'param':{'ktetha':ktetha, 'tetha0':tetha0, 'kub':None, 's0':None}, 'func':fu}\n\n self.angletypes.append(ang)\n _add_info(self, curr_sec, self.angletypes)\n\n elif curr_sec == 'angles':\n ai, aj, ak = list(map(int, [ai, aj, ak]))\n ang.atom1 = mol.atoms[ai-1]\n ang.atom2 = mol.atoms[aj-1]\n ang.atom3 = mol.atoms[ak-1]\n ang.gromacs['func'] = fu\n\n mol.angles.append(ang)\n _add_info(mol, curr_sec, mol.angles)\n\n else:\n raise ValueError\n\n elif fu == 2:\n if curr_sec == 'angletypes':\n raise NotImplementedError()\n\n elif curr_sec == 'angles':\n ai, aj, ak = list(map(int, [ai, aj, ak]))\n ang.atom1 = mol.atoms[ai-1]\n ang.atom2 = mol.atoms[aj-1]\n ang.atom3 = mol.atoms[ak-1]\n ang.gromacs['func'] = fu\n\n tetha0, ktetha = list(map(float, fields[4:6]))\n ang.gromacs = {'param':{'ktetha':ktetha, 'tetha0':tetha0, 'kub':None, 's0':None}, 'func':fu}\n\n mol.angles.append(ang)\n _add_info(mol, curr_sec, mol.angles)\n\n elif fu == 5:\n if curr_sec == 'angletypes':\n ang.atype1 = ai\n ang.atype2 = aj\n ang.atype3 = ak\n tetha0, ktetha, s0, kub = list(map(float, fields[4:8]))\n\n ang.gromacs = {'param':{'ktetha':ktetha, 'tetha0':tetha0, 'kub':kub, 's0':s0}, 'func':fu}\n\n self.angletypes.append(ang)\n _add_info(self, curr_sec, self.angletypes)\n\n elif curr_sec == 'angles':\n ai, aj, ak = list(map(int, [ai, aj, ak]))\n ang.atom1 = mol.atoms[ai-1]\n ang.atom2 = mol.atoms[aj-1]\n ang.atom3 = mol.atoms[ak-1]\n ang.gromacs['func'] = fu\n\n mol.angles.append(ang)\n _add_info(mol, curr_sec, mol.angles)\n\n else:\n raise ValueError\n\n else:\n raise NotImplementedError\n\n\n elif curr_sec in ('dihedraltypes', 'dihedrals'):\n '''\n section #at fu #param\n ----------------------------------\n dihedrals 4 1 3\n dihedrals 4 2 2\n dihedrals 4 3 6\n dihedrals 4 4 3\n dihedrals 4 5 4\n dihedrals 4 8 ??\n dihedrals 4 9 3\n '''\n\n if curr_sec == 'dihedraltypes' and len(fields) == 6:\n # in oplsaa - quartz parameters\n fields.insert(2, 'X')\n fields.insert(0, 'X')\n\n ai, aj, ak, am = fields[:4]\n fu = int(fields[4])\n assert fu in (1,2,3,4,5,8,9)\n\n if fu not in (1,2,3,4,9):\n raise NotImplementedError('dihedral function {0:d} is not yet supported'.format(fu))\n\n dih = blocks.DihedralType('gromacs')\n imp = blocks.ImproperType('gromacs')\n # proper dihedrals\n if fu in (1,3,9):\n if curr_sec == 'dihedraltypes':\n dih.atype1 = ai\n dih.atype2 = aj\n dih.atype3 = ak\n dih.atype4 = am\n\n dih.line = i_line + 1\n\n if fu == 1:\n delta, kchi, n = list(map(float, fields[5:8]))\n dih.gromacs['param'].append({'kchi':kchi, 'n':n, 'delta':delta})\n elif fu == 3:\n c0, c1, c2, c3, c4, c5 = list(map(float, fields[5:11]))\n m = dict(c0=c0, c1=c1, c2=c2, c3=c3, c4=c4, c5=c5)\n dih.gromacs['param'].append(m)\n elif fu == 4:\n delta, kchi, n = list(map(float, fields[5:8]))\n dih.gromacs['param'].append({'kchi':kchi, 'n':int(n), 'delta':delta})\n elif fu == 9:\n delta, kchi, n = list(map(float, fields[5:8]))\n dih.gromacs['param'].append({'kchi':kchi, 'n':int(n), 'delta':delta})\n else:\n raise ValueError\n\n dih.gromacs['func'] = fu\n self.dihedraltypes.append(dih)\n _add_info(self, curr_sec, self.dihedraltypes)\n\n elif curr_sec == 'dihedrals':\n ai, aj, ak, am = list(map(int, fields[:4]))\n dih.atom1 = mol.atoms[ai-1]\n dih.atom2 = mol.atoms[aj-1]\n dih.atom3 = mol.atoms[ak-1]\n dih.atom4 = mol.atoms[am-1]\n dih.gromacs['func'] = fu\n\n dih.line = i_line + 1\n\n if fu == 1:\n delta, kchi, n = list(map(float, fields[5:8]))\n dih.gromacs['param'].append({'kchi':kchi, 'n': int(n), 'delta':delta})\n elif fu == 3:\n pass\n elif fu == 4:\n pass\n elif fu == 9:\n if len(fields[5:8]) == 3:\n delta, kchi, n = list(map(float, fields[5:8]))\n dih.gromacs['param'].append({'kchi':kchi, 'n':int(n), 'delta':delta})\n else:\n raise ValueError\n\n mol.dihedrals.append(dih)\n _add_info(mol, curr_sec, mol.dihedrals)\n\n else:\n raise ValueError\n # impropers\n elif fu in (2,4):\n if curr_sec == 'dihedraltypes':\n imp.atype1 = ai\n imp.atype2 = aj\n imp.atype3 = ak\n imp.atype4 = am\n\n imp.line = i_line + 1\n\n if fu == 2:\n psi0 , kpsi = list(map(float, fields[5:7]))\n imp.gromacs['param'].append({'kpsi':kpsi, 'psi0': psi0})\n elif fu == 4:\n psi0 , kpsi, n = list(map(float, fields[5:8]))\n imp.gromacs['param'].append({'kpsi':kpsi, 'psi0': psi0, 'n': int(n)})\n else:\n raise ValueError\n\n imp.gromacs['func'] = fu\n self.impropertypes.append(imp)\n _add_info(self, curr_sec, self.impropertypes)\n\n elif curr_sec == 'dihedrals':\n ai, aj, ak, am = list(map(int, fields[:4]))\n imp.atom1 = mol.atoms[ai-1]\n imp.atom2 = mol.atoms[aj-1]\n imp.atom3 = mol.atoms[ak-1]\n imp.atom4 = mol.atoms[am-1]\n imp.gromacs['func'] = fu\n\n imp.line = i_line + 1\n\n if fu == 2:\n pass\n elif fu == 4:\n # in-line override of dihedral parameters\n if len(fields[5:8]) == 3:\n psi0 , kpsi, n = list(map(float, fields[5:8]))\n imp.gromacs['param'].append({'kpsi':kpsi, 'psi0': psi0, 'n': int(n)})\n else:\n raise ValueError\n\n mol.impropers.append(imp)\n _add_info(mol, curr_sec, mol.impropers)\n\n else:\n raise ValueError\n\n else:\n raise NotImplementedError\n\n\n elif curr_sec in ('cmaptypes', 'cmap'):\n\n cmap = blocks.CMapType('gromacs')\n if curr_sec == 'cmaptypes':\n cmap_lines.append(line)\n _add_info(self, curr_sec, self.cmaptypes)\n else:\n ai, aj, ak, am, an = list(map(int, fields[:5]))\n fu = int(fields[5])\n assert fu == 1\n cmap.atom1 = mol.atoms[ai-1]\n cmap.atom2 = mol.atoms[aj-1]\n cmap.atom3 = mol.atoms[ak-1]\n cmap.atom4 = mol.atoms[am-1]\n cmap.atom8 = mol.atoms[an-1]\n cmap.gromacs['func'] = fu\n\n mol.cmaps.append(cmap)\n _add_info(mol, curr_sec, mol.cmaps)\n\n\n elif curr_sec == 'settles':\n '''\n section #at fu #param\n ----------------------------------\n '''\n\n assert len(fields) == 4\n ai = int(fields[0])\n fu = int(fields[1])\n assert fu == 1\n\n settle = blocks.SettleType('gromacs')\n settle.atom = mol.atoms[ai-1]\n settle.dOH = float(fields[2])\n settle.dHH = float(fields[3])\n\n mol.settles.append(settle)\n _add_info(mol, curr_sec, mol.settles)\n\n elif curr_sec == \"virtual_sites3\":\n '''\n ; Dummy from funct a b\n 4 1 2 3 1 0.131937768 0.131937768\n '''\n assert len(fields) == 7\n ai = int(fields[0])\n aj = int(fields[1])\n ak = int(fields[2])\n al = int(fields[3])\n fu = int(fields[4])\n assert fu == 1\n a = float(fields[5])\n b = float(fields[6])\n\n vs3 = blocks.VirtualSites3Type('gromacs')\n vs3.atom1 = ai\n vs3.atom2 = aj\n vs3.atom3 = ak\n vs3.atom4 = al\n vs3.gromacs['func'] = fu\n vs3.gromacs['param'] = { 'a': a, 'b':b }\n mol.virtual_sites3.append(vs3)\n _add_info(mol, curr_sec, mol.virtual_sites3)\n\n\n elif curr_sec in ('exclusions',):\n ai = int(fields[0])\n other = list(map(int, fields[1:]))\n\n exc = blocks.Exclusion()\n exc.main_atom = mol.atoms[ai-1]\n exc.other_atoms= [mol.atoms[k-1] for k in other]\n\n mol.exclusions.append(exc)\n _add_info(mol, curr_sec, mol.exclusions)\n\n\n elif curr_sec in ('constrainttypes', 'constraints'):\n '''\n section #at fu #param\n ----------------------------------\n constraints 2 1 1\n constraints 2 2 1\n '''\n\n ai, aj = fields[:2]\n fu = int(fields[2])\n assert fu in (1,2)\n\n cons = blocks.ConstraintType('gromacs')\n\n # TODO: what's different between 1 and 2\n if fu in [1, 2]:\n if curr_sec == 'constrainttypes':\n cons.atype1 = ai\n cons.atype2 = aj\n b0 = float(fields[3])\n cons.gromacs = {'param':{'b0':b0}, 'func': fu}\n\n self.constrainttypes.append(cons)\n _add_info(self, curr_sec, self.constrainttypes)\n\n elif curr_sec == 'constraints':\n ai, aj = list(map(int, fields[:2]))\n cons.atom1 = mol.atoms[ai-1]\n cons.atom2 = mol.atoms[aj-1]\n cons.gromacs['func'] = fu\n\n mol.constraints.append(cons)\n _add_info(mol, curr_sec, mol.constraints)\n\n else:\n raise ValueError\n else:\n raise ValueError\n\n elif curr_sec in ('position_restraints',\n 'distance_restraints',\n 'dihedral_restraints',\n 'orientation_restraints',\n 'angle_restraints',\n 'angle_restraints_z'):\n pass\n\n\n elif curr_sec in ('implicit_genborn_params',):\n '''\n attype sar st pi gbr hct\n '''\n pass\n\n elif curr_sec == 'system':\n #assert len(fields) == 1\n self.name = fields[0]\n\n\n elif curr_sec == 'molecules':\n assert len(fields) == 2\n mname, nmol = fields[0], int(fields[1])\n\n # if the number of a molecule is more than 1, add copies to system.molecules\n for i in range(nmol):\n self.molecules.append(self.dict_molname_mol[mname])\n\n else:\n raise NotImplementedError('Unknown section in topology: {0}'.format(curr_sec))\n\n # process cmap_lines\n curr_cons = None\n for line in cmap_lines:\n\n # cmaptype opening line\n if len(line.split()) == 8:\n cons = blocks.CMapType('gromacs')\n\n atype1, atype2, atype3, atype4, atype8, func, sizeX, sizeY = line.replace(\"\\\\\",\"\").split()\n func, sizeX, sizeY = int(func), int(sizeX), int(sizeY)\n cons.atype1 = atype1\n cons.atype2 = atype2\n cons.atype3 = atype3\n cons.atype4 = atype4\n cons.atype8 = atype8\n cons.gromacs = {'param':[], 'func': func}\n\n curr_cons = cons\n\n # cmap body\n elif len(line.split()) == 10:\n cmap_param = map(float, line.replace(\"\\\\\",\"\").split())\n cons.gromacs['param'] += cmap_param\n\n # cmaptype cloning line\n elif len(line.split()) == 6:\n cmap_param = map(float, line.replace(\"\\\\\",\"\").split())\n cons.gromacs['param'] += cmap_param\n self.cmaptypes.append(curr_cons)\n else:\n raise ValueError"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef assemble_topology(self):\n self.logger.debug(\"starting to assemble topology...\")\n\n top = ''\n\n self.logger.debug(\"making atom/pair/bond/angle/dihedral/improper types\")\n top += self.toptemplate\n top = top.replace('*DEFAULTS*', ''.join( self._make_defaults(self.system)) )\n top = top.replace('*ATOMTYPES*', ''.join( self._make_atomtypes(self.system)) )\n top = top.replace('*NONBOND_PARAM*', ''.join( self._make_nonbond_param(self.system)) )\n top = top.replace('*PAIRTYPES*', ''.join( self._make_pairtypes(self.system)) )\n top = top.replace('*BONDTYPES*', ''.join( self._make_bondtypes(self.system)) )\n top = top.replace('*CONSTRAINTTYPES*',''.join( self._make_constrainttypes(self.system)))\n top = top.replace('*ANGLETYPES*', ''.join( self._make_angletypes(self.system)))\n top = top.replace('*DIHEDRALTYPES*', ''.join( self._make_dihedraltypes(self.system)) )\n top = top.replace('*IMPROPERTYPES*', ''.join( self._make_impropertypes(self.system)) )\n top = top.replace('*CMAPTYPES*', ''.join( self._make_cmaptypes(self.system)) )\n\n for i,(molname,m) in enumerate(self.system.dict_molname_mol.items()):\n\n itp = self.itptemplate\n itp = itp.replace('*MOLECULETYPE*', ''.join( self._make_moleculetype(m, molname, m.exclusion_numb)) )\n itp = itp.replace('*ATOMS*', ''.join( self._make_atoms(m)) )\n itp = itp.replace('*BONDS*', ''.join( self._make_bonds(m)) )\n itp = itp.replace('*PAIRS*', ''.join( self._make_pairs(m)) )\n itp = itp.replace('*SETTLES*', ''.join( self._make_settles(m)) )\n itp = itp.replace('*VIRTUAL_SITES3*',''.join( self._make_virtual_sites3(m)) )\n itp = itp.replace('*EXCLUSIONS*', ''.join( self._make_exclusions(m)) )\n itp = itp.replace('*ANGLES*', ''.join( self._make_angles(m)) )\n itp = itp.replace('*DIHEDRALS*', ''.join( self._make_dihedrals(m)) )\n itp = itp.replace('*IMPROPERS*', ''.join( self._make_impropers(m)) )\n itp = itp.replace('*CMAPS*', ''.join( self._make_cmaps(m)) )\n if not self.multiple_output:\n top += itp\n else:\n outfile = \"mol_{0}.itp\".format(molname)\n top += '#include \"mol_{0}.itp\" \\n'.format( molname )\n with open(outfile, \"w\") as f:\n f.writelines([itp])\n\n top += '\\n[system] \\nConvertedSystem\\n\\n'\n top += '[molecules] \\n'\n molecules = [(\"\", 0)]\n\n for m in self.system.molecules:\n if (molecules[-1][0] != m.name):\n molecules.append([m.name, 0])\n if molecules[-1][0] == m.name:\n molecules[-1][1] += 1\n\n for molname, n in molecules[1:]:\n top += '{0:s} {1:d}\\n'.format(molname, n)\n top += '\\n'\n\n with open(self.outfile, 'w') as f:\n f.writelines([top])", "response": "Assemble the topology into a string"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nbuilding a new topology file from a structure.", "response": "def topology(struct=None, protein='protein',\n top='system.top', dirname='top',\n posres=\"posres.itp\",\n ff=\"oplsaa\", water=\"tip4p\",\n **pdb2gmx_args):\n \"\"\"Build Gromacs topology files from pdb.\n\n :Keywords:\n *struct*\n input structure (**required**)\n *protein*\n name of the output files\n *top*\n name of the topology file\n *dirname*\n directory in which the new topology will be stored\n *ff*\n force field (string understood by ``pdb2gmx``); default\n \"oplsaa\"\n *water*\n water model (string), default \"tip4p\"\n *pdb2gmxargs*\n other arguments for ``pdb2gmx``\n\n .. note::\n At the moment this function simply runs ``pdb2gmx`` and uses\n the resulting topology file directly. If you want to create\n more complicated topologies and maybe also use additional itp\n files or make a protein itp file then you will have to do this\n manually.\n \"\"\"\n\n structure = realpath(struct)\n\n new_struct = protein + '.pdb'\n if posres is None:\n posres = protein + '_posres.itp'\n\n pdb2gmx_args.update({'f': structure, 'o': new_struct, 'p': top, 'i': posres,\n 'ff': ff, 'water': water})\n\n with in_dir(dirname):\n logger.info(\"[{dirname!s}] Building topology {top!r} from struct = {struct!r}\".format(**vars()))\n # perhaps parse output from pdb2gmx 4.5.x to get the names of the chain itp files?\n gromacs.pdb2gmx(**pdb2gmx_args)\n return { \\\n 'top': realpath(dirname, top), \\\n 'struct': realpath(dirname, new_struct), \\\n 'posres' : realpath(dirname, posres) }"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nmake index file with the special groups. This routine adds the group __main__ and the group __environment__ to the end of the index file. __main__ contains what the user defines as the *central* and *most important* parts of the system. __environment__ is everything else. The template mdp file, for instance, uses these two groups for T-coupling. These groups are mainly useful if the default groups \"Protein\" and \"Non-Protein\" are not appropriate. By using symbolic names such as __main__ one can keep scripts more general. :Returns: *groups* is a list of dictionaries that describe the index groups. See :func:`gromacs.cbook.parse_ndxlist` for details. :Arguments: *struct* : filename structure (tpr, pdb, gro) *selection* : string is a ``make_ndx`` command such as ``\"Protein\"`` or ``r DRG`` which determines what is considered the main group for centering etc. It is passed directly to ``make_ndx``. *ndx* : string name of the final index file *oldndx* : string name of index file that should be used as a basis; if None then the ``make_ndx`` default groups are used. This routine is very dumb at the moment; maybe some heuristics will be added later as could be other symbolic groups such as __membrane__.", "response": "def make_main_index(struct, selection='\"Protein\"', ndx='main.ndx', oldndx=None):\n \"\"\"Make index file with the special groups.\n\n This routine adds the group __main__ and the group __environment__\n to the end of the index file. __main__ contains what the user\n defines as the *central* and *most important* parts of the\n system. __environment__ is everything else.\n\n The template mdp file, for instance, uses these two groups for T-coupling.\n\n These groups are mainly useful if the default groups \"Protein\" and \"Non-Protein\"\n are not appropriate. By using symbolic names such as __main__ one\n can keep scripts more general.\n\n :Returns:\n *groups* is a list of dictionaries that describe the index groups. See\n :func:`gromacs.cbook.parse_ndxlist` for details.\n\n :Arguments:\n *struct* : filename\n structure (tpr, pdb, gro)\n *selection* : string\n is a ``make_ndx`` command such as ``\"Protein\"`` or ``r DRG`` which\n determines what is considered the main group for centering etc. It is\n passed directly to ``make_ndx``.\n *ndx* : string\n name of the final index file\n *oldndx* : string\n name of index file that should be used as a basis; if None\n then the ``make_ndx`` default groups are used.\n\n This routine is very dumb at the moment; maybe some heuristics will be\n added later as could be other symbolic groups such as __membrane__.\n \"\"\"\n\n logger.info(\"Building the main index file {ndx!r}...\".format(**vars()))\n\n # pass 1: select\n # get a list of groups\n # need the first \"\" to get make_ndx to spit out the group list.\n _,out,_ = gromacs.make_ndx(f=struct, n=oldndx, o=ndx, stdout=False,\n input=(\"\", \"q\"))\n groups = cbook.parse_ndxlist(out)\n\n # find the matching groups,\n # there is a nasty bug in GROMACS where make_ndx may have multiple\n # groups, which caused the previous approach to fail big time.\n # this is a work around the make_ndx bug.\n # striping the \"\" allows compatibility with existing make_ndx selection commands.\n selection = selection.strip(\"\\\"\")\n\n selected_groups = [g for g in groups if g['name'].lower() == selection.lower()]\n\n if len(selected_groups) > 1:\n logging.warn(\"make_ndx created duplicated groups, performing work around\")\n\n if len(selected_groups) <= 0:\n msg = \"no groups found for selection {0}, available groups are {1}\".format(selection, groups)\n logging.error(msg)\n raise ValueError(msg)\n\n # Found at least one matching group, we're OK\n\n # index of last group\n last = len(groups) - 1\n assert last == groups[-1]['nr']\n\n group = selected_groups[0]\n\n # pass 2:\n # 1) last group is __main__\n # 2) __environment__ is everything else (eg SOL, ions, ...)\n _,out,_ = gromacs.make_ndx(f=struct, n=ndx, o=ndx,\n stdout=False,\n # make copy selected group, this now has index last + 1\n input=(\"{0}\".format(group['nr']),\n # rename this to __main__\n \"name {0} __main__\".format(last+1),\n # make a complement to this group, it get index last + 2\n \"! \\\"__main__\\\"\",\n # rename this to __environment__\n \"name {0} __environment__\".format(last+2),\n # list the groups\n \"\",\n # quit\n \"q\"))\n return cbook.parse_ndxlist(out)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nfinds vdwradii. dat and add special entries for lipids.", "response": "def get_lipid_vdwradii(outdir=os.path.curdir, libdir=None):\n \"\"\"Find vdwradii.dat and add special entries for lipids.\n\n See :data:`gromacs.setup.vdw_lipid_resnames` for lipid\n resnames. Add more if necessary.\n \"\"\"\n vdwradii_dat = os.path.join(outdir, \"vdwradii.dat\")\n\n if libdir is not None:\n filename = os.path.join(libdir, 'vdwradii.dat') # canonical name\n if not os.path.exists(filename):\n msg = 'No VDW database file found in {filename!r}.'.format(**vars())\n logger.exception(msg)\n raise OSError(msg, errno.ENOENT)\n else:\n try:\n filename = os.path.join(os.environ['GMXLIB'], 'vdwradii.dat')\n except KeyError:\n try:\n filename = os.path.join(os.environ['GMXDATA'], 'top', 'vdwradii.dat')\n except KeyError:\n msg = \"Cannot find vdwradii.dat. Set GMXLIB (point to 'top') or GMXDATA ('share/gromacs').\"\n logger.exception(msg)\n raise OSError(msg, errno.ENOENT)\n if not os.path.exists(filename):\n msg = \"Cannot find {filename!r}; something is wrong with the Gromacs installation.\".format(**vars())\n logger.exception(msg, errno.ENOENT)\n raise OSError(msg)\n\n # make sure to catch 3 and 4 letter resnames\n patterns = vdw_lipid_resnames + list({x[:3] for x in vdw_lipid_resnames})\n # TODO: should do a tempfile...\n with open(vdwradii_dat, 'w') as outfile:\n # write lipid stuff before general\n outfile.write('; Special larger vdw radii for solvating lipid membranes\\n')\n for resname in patterns:\n for atom,radius in vdw_lipid_atom_radii.items():\n outfile.write('{resname:4!s} {atom:<5!s} {radius:5.3f}\\n'.format(**vars()))\n with open(filename, 'r') as infile:\n for line in infile:\n outfile.write(line)\n logger.debug('Created lipid vdW radii file {vdwradii_dat!r}.'.format(**vars()))\n return realpath(vdwradii_dat)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nputting protein into box, add water, add counter-ions. Currently this really only supports solutes in water. If you need to embedd a protein in a membrane then you will require more sophisticated approaches. However, you *can* supply a protein already inserted in a bilayer. In this case you will probably want to set *distance* = ``None`` and also enable *with_membrane* = ``True`` (using extra big vdw radii for typical lipids). .. Note:: The defaults are suitable for solvating a globular protein in a fairly tight (increase *distance*!) dodecahedral box. :Arguments: *struct* : filename pdb or gro input structure *top* : filename Gromacs topology *distance* : float When solvating with water, make the box big enough so that at least *distance* nm water are between the solute *struct* and the box boundary. Set *boxtype* to ``None`` in order to use a box size in the input file (gro or pdb). *boxtype* or *bt*: string Any of the box types supported by :class:`~gromacs.tools.Editconf` (triclinic, cubic, dodecahedron, octahedron). Set the box dimensions either with *distance* or the *box* and *angle* keywords. If set to ``None`` it will ignore *distance* and use the box inside the *struct* file. *bt* overrides the value of *boxtype*. *box* List of three box lengths [A,B,C] that are used by :class:`~gromacs.tools.Editconf` in combination with *boxtype* (``bt`` in :program:`editconf`) and *angles*. Setting *box* overrides *distance*. *angles* List of three angles (only necessary for triclinic boxes). *concentration* : float Concentration of the free ions in mol/l. Note that counter ions are added in excess of this concentration. *cation* and *anion* : string Molecule names of the ions. This depends on the chosen force field. *water* : string Name of the water model; one of \"spc\", \"spce\", \"tip3p\", \"tip4p\". This should be appropriate for the chosen force field. If an alternative solvent is required, simply supply the path to a box with solvent molecules (used by :func:`~gromacs.genbox`'s *cs* argument) and also supply the molecule name via *solvent_name*. *solvent_name* Name of the molecules that make up the solvent (as set in the itp/top). Typically needs to be changed when using non-standard/non-water solvents. [\"SOL\"] *with_membrane* : bool ``True``: use special ``vdwradii.dat`` with 0.1 nm-increased radii on lipids. Default is ``False``. *ndx* : filename How to name the index file that is produced by this function. *mainselection* : string A string that is fed to :class:`~gromacs.tools.Make_ndx` and which should select the solute. *dirname* : directory name Name of the directory in which all files for the solvation stage are stored. *includes* List of additional directories to add to the mdp include path *kwargs* Additional arguments are passed on to :class:`~gromacs.tools.Editconf` or are interpreted as parameters to be changed in the mdp file.", "response": "def solvate(struct='top/protein.pdb', top='top/system.top',\n distance=0.9, boxtype='dodecahedron',\n concentration=0, cation='NA', anion='CL',\n water='tip4p', solvent_name='SOL', with_membrane=False,\n ndx = 'main.ndx', mainselection = '\"Protein\"',\n dirname='solvate',\n **kwargs):\n \"\"\"Put protein into box, add water, add counter-ions.\n\n Currently this really only supports solutes in water. If you need\n to embedd a protein in a membrane then you will require more\n sophisticated approaches.\n\n However, you *can* supply a protein already inserted in a\n bilayer. In this case you will probably want to set *distance* =\n ``None`` and also enable *with_membrane* = ``True`` (using extra\n big vdw radii for typical lipids).\n\n .. Note:: The defaults are suitable for solvating a globular\n protein in a fairly tight (increase *distance*!) dodecahedral\n box.\n\n :Arguments:\n *struct* : filename\n pdb or gro input structure\n *top* : filename\n Gromacs topology\n *distance* : float\n When solvating with water, make the box big enough so that\n at least *distance* nm water are between the solute *struct*\n and the box boundary.\n Set *boxtype* to ``None`` in order to use a box size in the input\n file (gro or pdb).\n *boxtype* or *bt*: string\n Any of the box types supported by :class:`~gromacs.tools.Editconf`\n (triclinic, cubic, dodecahedron, octahedron). Set the box dimensions\n either with *distance* or the *box* and *angle* keywords.\n\n If set to ``None`` it will ignore *distance* and use the box\n inside the *struct* file.\n\n *bt* overrides the value of *boxtype*.\n *box*\n List of three box lengths [A,B,C] that are used by :class:`~gromacs.tools.Editconf`\n in combination with *boxtype* (``bt`` in :program:`editconf`) and *angles*.\n Setting *box* overrides *distance*.\n *angles*\n List of three angles (only necessary for triclinic boxes).\n *concentration* : float\n Concentration of the free ions in mol/l. Note that counter\n ions are added in excess of this concentration.\n *cation* and *anion* : string\n Molecule names of the ions. This depends on the chosen force field.\n *water* : string\n Name of the water model; one of \"spc\", \"spce\", \"tip3p\",\n \"tip4p\". This should be appropriate for the chosen force\n field. If an alternative solvent is required, simply supply the path to a box\n with solvent molecules (used by :func:`~gromacs.genbox`'s *cs* argument)\n and also supply the molecule name via *solvent_name*.\n *solvent_name*\n Name of the molecules that make up the solvent (as set in the itp/top).\n Typically needs to be changed when using non-standard/non-water solvents.\n [\"SOL\"]\n *with_membrane* : bool\n ``True``: use special ``vdwradii.dat`` with 0.1 nm-increased radii on\n lipids. Default is ``False``.\n *ndx* : filename\n How to name the index file that is produced by this function.\n *mainselection* : string\n A string that is fed to :class:`~gromacs.tools.Make_ndx` and\n which should select the solute.\n *dirname* : directory name\n Name of the directory in which all files for the solvation stage are stored.\n *includes*\n List of additional directories to add to the mdp include path\n *kwargs*\n Additional arguments are passed on to\n :class:`~gromacs.tools.Editconf` or are interpreted as parameters to be\n changed in the mdp file.\n\n \"\"\"\n sol = solvate_sol(struct=struct, top=top,\n distance=distance, boxtype=boxtype,\n water=water, solvent_name=solvent_name, \n with_membrane=with_membrane,\n dirname=dirname, **kwargs)\n \n ion = solvate_ion(struct=sol['struct'], top=top,\n concentration=concentration, cation=cation, anion=anion,\n solvent_name=solvent_name, ndx=ndx,\n mainselection=mainselection, dirname=dirname,\n **kwargs)\n return ion"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef check_mdpargs(d):\n if len(d) > 0:\n wmsg = \"Unprocessed mdp option are interpreted as options for grompp:\\n\"+str(d)\n logger.warn(wmsg)\n warnings.warn(wmsg, category=UsageWarning)\n return len(d) == 0", "response": "Check if any arguments remain in dict *d*."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef energy_minimize(dirname='em', mdp=config.templates['em.mdp'],\n struct='solvate/ionized.gro', top='top/system.top',\n output='em.pdb', deffnm=\"em\",\n mdrunner=None, mdrun_args=None,\n **kwargs):\n \"\"\"Energy minimize the system.\n\n This sets up the system (creates run input files) and also runs\n ``mdrun_d``. Thus it can take a while.\n\n Additional itp files should be in the same directory as the top file.\n\n Many of the keyword arguments below already have sensible values.\n\n :Keywords:\n *dirname*\n set up under directory dirname [em]\n *struct*\n input structure (gro, pdb, ...) [solvate/ionized.gro]\n *output*\n output structure (will be put under dirname) [em.pdb]\n *deffnm*\n default name for mdrun-related files [em]\n *top*\n topology file [top/system.top]\n *mdp*\n mdp file (or use the template) [templates/em.mdp]\n *includes*\n additional directories to search for itp files\n *mdrunner*\n :class:`gromacs.run.MDrunner` instance; by default we\n just try :func:`gromacs.mdrun_d` and :func:`gromacs.mdrun` but a\n MDrunner instance gives the user the ability to run mpi jobs\n etc. [None]\n *mdrun_args*\n arguments for *mdrunner* (as a dict), e.g. ``{'nt': 2}``;\n empty by default\n\n .. versionaddedd:: 0.7.0\n\n *kwargs*\n remaining key/value pairs that should be changed in the\n template mdp file, eg ``nstxtcout=250, nstfout=250``.\n\n .. note:: If :func:`~gromacs.mdrun_d` is not found, the function\n falls back to :func:`~gromacs.mdrun` instead.\n \"\"\"\n\n structure = realpath(struct)\n topology = realpath(top)\n mdp_template = config.get_template(mdp)\n deffnm = deffnm.strip()\n\n mdrun_args = {} if mdrun_args is None else mdrun_args\n\n # write the processed topology to the default output\n kwargs.setdefault('pp', 'processed.top')\n\n # filter some kwargs that might come through when feeding output\n # from previous stages such as solvate(); necessary because *all*\n # **kwargs must be *either* substitutions in the mdp file *or* valid\n # command line parameters for ``grompp``.\n kwargs.pop('ndx', None)\n # mainselection is not used but only passed through; right now we\n # set it to the default that is being used in all argument lists\n # but that is not pretty. TODO.\n mainselection = kwargs.pop('mainselection', '\"Protein\"')\n # only interesting when passed from solvate()\n qtot = kwargs.pop('qtot', 0)\n\n # mdp is now the *output* MDP that will be generated from mdp_template\n mdp = deffnm+'.mdp'\n tpr = deffnm+'.tpr'\n\n logger.info(\"[{dirname!s}] Energy minimization of struct={struct!r}, top={top!r}, mdp={mdp!r} ...\".format(**vars()))\n\n cbook.add_mdp_includes(topology, kwargs)\n\n if qtot != 0:\n # At the moment this is purely user-reported and really only here because\n # it might get fed into the function when using the keyword-expansion pipeline\n # usage paradigm.\n wmsg = \"Total charge was reported as qtot = {qtot:g} <> 0; probably a problem.\".format(**vars())\n logger.warn(wmsg)\n warnings.warn(wmsg, category=BadParameterWarning)\n\n with in_dir(dirname):\n unprocessed = cbook.edit_mdp(mdp_template, new_mdp=mdp, **kwargs)\n check_mdpargs(unprocessed)\n gromacs.grompp(f=mdp, o=tpr, c=structure, r=structure, p=topology, **unprocessed)\n mdrun_args.update(v=True, stepout=10, deffnm=deffnm, c=output)\n if mdrunner is None:\n mdrun = run.get_double_or_single_prec_mdrun()\n mdrun(**mdrun_args)\n else:\n if type(mdrunner) is type:\n # class\n # user wants full control and provides simulation.MDrunner **class**\n # NO CHECKING --- in principle user can supply any callback they like\n mdrun = mdrunner(**mdrun_args)\n mdrun.run()\n else:\n # anything with a run() method that takes mdrun arguments...\n try:\n mdrunner.run(mdrunargs=mdrun_args)\n except AttributeError:\n logger.error(\"mdrunner: Provide a gromacs.run.MDrunner class or instance or a callback with a run() method\")\n raise TypeError(\"mdrunner: Provide a gromacs.run.MDrunner class or instance or a callback with a run() method\")\n\n # em.gro --> gives 'Bad box in file em.gro' warning --- why??\n # --> use em.pdb instead.\n if not os.path.exists(output):\n errmsg = \"Energy minimized system NOT produced.\"\n logger.error(errmsg)\n raise GromacsError(errmsg)\n final_struct = realpath(output)\n\n logger.info(\"[{dirname!s}] energy minimized structure {final_struct!r}\".format(**vars()))\n return {'struct': final_struct,\n 'top': topology,\n 'mainselection': mainselection,\n }", "response": "Energy minimize the system."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef em_schedule(**kwargs):\n\n mdrunner = kwargs.pop('mdrunner', None)\n integrators = kwargs.pop('integrators', ['l-bfgs', 'steep'])\n kwargs.pop('integrator', None) # clean input; we set intgerator from integrators\n nsteps = kwargs.pop('nsteps', [100, 1000])\n\n outputs = ['em{0:03d}_{1!s}.pdb'.format(i, integrator) for i,integrator in enumerate(integrators)]\n outputs[-1] = kwargs.pop('output', 'em.pdb')\n\n files = {'struct': kwargs.pop('struct', None)} # fake output from energy_minimize()\n\n for i, integrator in enumerate(integrators):\n struct = files['struct']\n logger.info(\"[em %d] energy minimize with %s for maximum %d steps\", i, integrator, nsteps[i])\n kwargs.update({'struct':struct, 'output':outputs[i],\n 'integrator':integrator, 'nsteps': nsteps[i]})\n if not integrator == 'l-bfgs':\n kwargs['mdrunner'] = mdrunner\n else:\n kwargs['mdrunner'] = None\n logger.warning(\"[em %d] Not using mdrunner for L-BFGS because it cannot \"\n \"do parallel runs.\", i)\n\n files = energy_minimize(**kwargs)\n\n return files", "response": "Run multiple energy minimizations one after each other."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _setup_MD(dirname,\n deffnm='md', mdp=config.templates['md_OPLSAA.mdp'],\n struct=None,\n top='top/system.top', ndx=None,\n mainselection='\"Protein\"',\n qscript=config.qscript_template, qname=None, startdir=None, mdrun_opts=\"\", budget=None, walltime=1/3.,\n dt=0.002, runtime=1e3, **mdp_kwargs):\n \"\"\"Generic function to set up a ``mdrun`` MD simulation.\n\n See the user functions for usage.\n \"\"\"\n\n if struct is None:\n raise ValueError('struct must be set to a input structure')\n structure = realpath(struct)\n topology = realpath(top)\n try:\n index = realpath(ndx)\n except AttributeError: # (that's what realpath(None) throws...)\n index = None # None is handled fine below\n\n qname = mdp_kwargs.pop('sgename', qname) # compatibility for old scripts\n qscript = mdp_kwargs.pop('sge', qscript) # compatibility for old scripts\n qscript_template = config.get_template(qscript)\n mdp_template = config.get_template(mdp)\n\n nsteps = int(float(runtime)/float(dt))\n\n mdp = deffnm + '.mdp'\n tpr = deffnm + '.tpr'\n mainindex = deffnm + '.ndx'\n final_structure = deffnm + '.gro' # guess... really depends on templates,could also be DEFFNM.pdb\n\n # write the processed topology to the default output\n mdp_parameters = {'nsteps':nsteps, 'dt':dt, 'pp': 'processed.top'}\n mdp_parameters.update(mdp_kwargs)\n\n cbook.add_mdp_includes(topology, mdp_parameters)\n\n logger.info(\"[%(dirname)s] input mdp = %(mdp_template)r\", vars())\n with in_dir(dirname):\n if not (mdp_parameters.get('Tcoupl','').lower() == 'no' or mainselection is None):\n logger.info(\"[{dirname!s}] Automatic adjustment of T-coupling groups\".format(**vars()))\n\n # make index file in almost all cases; with mainselection == None the user\n # takes FULL control and also has to provide the template or index\n groups = make_main_index(structure, selection=mainselection,\n oldndx=index, ndx=mainindex)\n natoms = {g['name']: float(g['natoms']) for g in groups}\n tc_group_names = ('__main__', '__environment__') # defined in make_main_index()\n try:\n x = natoms['__main__']/natoms['__environment__']\n except KeyError:\n x = 0 # force using SYSTEM in code below\n wmsg = \"Missing __main__ and/or __environment__ index group.\\n\" \\\n \"This probably means that you have an atypical system. You can \" \\\n \"set mainselection=None and provide your own mdp and index files \" \\\n \"in order to set up temperature coupling.\\n\" \\\n \"If no T-coupling is required then set Tcoupl='no'.\\n\" \\\n \"For now we will just couple everything to 'System'.\"\n logger.warn(wmsg)\n warnings.warn(wmsg, category=AutoCorrectionWarning)\n if x < 0.1:\n # couple everything together\n tau_t = firstof(mdp_parameters.pop('tau_t', 0.1))\n ref_t = firstof(mdp_parameters.pop('ref_t', 300))\n # combine all in one T-coupling group\n mdp_parameters['tc-grps'] = 'System'\n mdp_parameters['tau_t'] = tau_t # this overrides the commandline!\n mdp_parameters['ref_t'] = ref_t # this overrides the commandline!\n mdp_parameters['gen-temp'] = mdp_parameters.pop('gen_temp', ref_t)\n wmsg = \"Size of __main__ is only %.1f%% of __environment__ so \" \\\n \"we use 'System' for T-coupling and ref_t = %g K and \" \\\n \"tau_t = %g 1/ps (can be changed in mdp_parameters).\\n\" \\\n % (x * 100, ref_t, tau_t)\n logger.warn(wmsg)\n warnings.warn(wmsg, category=AutoCorrectionWarning)\n else:\n # couple protein and bath separately\n n_tc_groups = len(tc_group_names)\n tau_t = asiterable(mdp_parameters.pop('tau_t', 0.1))\n ref_t = asiterable(mdp_parameters.pop('ref_t', 300))\n\n if len(tau_t) != n_tc_groups:\n tau_t = n_tc_groups * [tau_t[0]]\n wmsg = \"%d coupling constants should have been supplied for tau_t. \"\\\n \"Using %f 1/ps for all of them.\" % (n_tc_groups, tau_t[0])\n logger.warn(wmsg)\n warnings.warn(wmsg, category=AutoCorrectionWarning)\n if len(ref_t) != n_tc_groups:\n ref_t = n_tc_groups * [ref_t[0]]\n wmsg = \"%d temperatures should have been supplied for ref_t. \"\\\n \"Using %g K for all of them.\" % (n_tc_groups, ref_t[0])\n logger.warn(wmsg)\n warnings.warn(wmsg, category=AutoCorrectionWarning)\n\n mdp_parameters['tc-grps'] = tc_group_names\n mdp_parameters['tau_t'] = tau_t\n mdp_parameters['ref_t'] = ref_t\n mdp_parameters['gen-temp'] = mdp_parameters.pop('gen_temp', ref_t[0])\n index = realpath(mainindex)\n if mdp_parameters.get('Tcoupl','').lower() == 'no':\n logger.info(\"Tcoupl == no: disabling all temperature coupling mdp options\")\n mdp_parameters['tc-grps'] = \"\"\n mdp_parameters['tau_t'] = \"\"\n mdp_parameters['ref_t'] = \"\"\n mdp_parameters['gen-temp'] = \"\"\n if mdp_parameters.get('Pcoupl','').lower() == 'no':\n logger.info(\"Pcoupl == no: disabling all pressure coupling mdp options\")\n mdp_parameters['tau_p'] = \"\"\n mdp_parameters['ref_p'] = \"\"\n mdp_parameters['compressibility'] = \"\"\n\n unprocessed = cbook.edit_mdp(mdp_template, new_mdp=mdp, **mdp_parameters)\n check_mdpargs(unprocessed)\n gromacs.grompp(f=mdp, p=topology, c=structure, n=index, o=tpr, **unprocessed)\n\n runscripts = qsub.generate_submit_scripts(\n qscript_template, deffnm=deffnm, jobname=qname, budget=budget,\n startdir=startdir, mdrun_opts=mdrun_opts, walltime=walltime)\n\n logger.info(\"[%(dirname)s] output mdp = %(mdp)r\", vars())\n logger.info(\"[%(dirname)s] output ndx = %(ndx)r\", vars())\n logger.info(\"[%(dirname)s] output tpr = %(tpr)r\", vars())\n logger.info(\"[%(dirname)s] output runscripts = %(runscripts)r\", vars())\n logger.info(\"[%(dirname)s] All files set up for a run time of %(runtime)g ps \"\n \"(dt=%(dt)g, nsteps=%(nsteps)g)\" % vars())\n\n kwargs = {'struct': realpath(os.path.join(dirname, final_structure)), # guess\n 'top': topology,\n 'ndx': index, # possibly mainindex\n 'qscript': runscripts,\n 'mainselection': mainselection,\n 'deffnm': deffnm, # return deffnm (tpr = deffnm.tpr!)\n }\n kwargs.update(mdp_kwargs) # return extra mdp args so that one can use them for prod run\n return kwargs", "response": "This function is used to setup a new MD simulation."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets up MD with position restraints.", "response": "def MD_restrained(dirname='MD_POSRES', **kwargs):\n \"\"\"Set up MD with position restraints.\n\n Additional itp files should be in the same directory as the top file.\n\n Many of the keyword arguments below already have sensible values. Note that\n setting *mainselection* = ``None`` will disable many of the automated\n choices and is often recommended when using your own mdp file.\n\n :Keywords:\n *dirname*\n set up under directory dirname [MD_POSRES]\n *struct*\n input structure (gro, pdb, ...) [em/em.pdb]\n *top*\n topology file [top/system.top]\n *mdp*\n mdp file (or use the template) [templates/md.mdp]\n *ndx*\n index file (supply when using a custom mdp)\n *includes*\n additional directories to search for itp files\n *mainselection*\n :program:`make_ndx` selection to select main group [\"Protein\"]\n (If ``None`` then no canonical index file is generated and\n it is the user's responsibility to set *tc_grps*,\n *tau_t*, and *ref_t* as keyword arguments, or provide the mdp template\n with all parameter pre-set in *mdp* and probably also your own *ndx*\n index file.)\n *deffnm*\n default filename for Gromacs run [md]\n *runtime*\n total length of the simulation in ps [1000]\n *dt*\n integration time step in ps [0.002]\n *qscript*\n script to submit to the queuing system; by default\n uses the template :data:`gromacs.config.qscript_template`, which can\n be manually set to another template from :data:`gromacs.config.templates`;\n can also be a list of template names.\n *qname*\n name to be used for the job in the queuing system [PR_GMX]\n *mdrun_opts*\n option flags for the :program:`mdrun` command in the queuing system\n scripts such as \"-stepout 100\". [\"\"]\n *kwargs*\n remaining key/value pairs that should be changed in the template mdp\n file, eg ``nstxtcout=250, nstfout=250`` or command line options for\n ``grompp` such as ``maxwarn=1``.\n\n In particular one can also set **define** and activate\n whichever position restraints have been coded into the itp\n and top file. For instance one could have\n\n *define* = \"-DPOSRES_MainChain -DPOSRES_LIGAND\"\n\n if these preprocessor constructs exist. Note that there\n **must not be any space between \"-D\" and the value.**\n\n By default *define* is set to \"-DPOSRES\".\n\n :Returns: a dict that can be fed into :func:`gromacs.setup.MD`\n (but check, just in case, especially if you want to\n change the ``define`` parameter in the mdp file)\n\n .. Note:: The output frequency is drastically reduced for position\n restraint runs by default. Set the corresponding ``nst*``\n variables if you require more output. The `pressure coupling`_\n option *refcoord_scaling* is set to \"com\" by default (but can\n be changed via *kwargs*) and the pressure coupling\n algorithm itself is set to *Pcoupl* = \"Berendsen\" to\n run a stable simulation.\n\n .. _`pressure coupling`: http://manual.gromacs.org/online/mdp_opt.html#pc\n \"\"\"\n\n logger.info(\"[{dirname!s}] Setting up MD with position restraints...\".format(**vars()))\n kwargs.setdefault('struct', 'em/em.pdb')\n kwargs.setdefault('qname', 'PR_GMX')\n kwargs.setdefault('define', '-DPOSRES')\n # reduce size of output files\n kwargs.setdefault('nstxout', '50000') # trr pos\n kwargs.setdefault('nstvout', '50000') # trr veloc\n kwargs.setdefault('nstfout', '0') # trr forces\n kwargs.setdefault('nstlog', '500') # log file\n kwargs.setdefault('nstenergy', '2500') # edr energy\n kwargs.setdefault('nstxtcout', '5000') # xtc pos\n # try to get good pressure equilibration\n kwargs.setdefault('refcoord_scaling', 'com')\n kwargs.setdefault('Pcoupl', \"Berendsen\")\n\n new_kwargs = _setup_MD(dirname, **kwargs)\n\n # clean up output kwargs\n new_kwargs.pop('define', None) # but make sure that -DPOSRES does not stay...\n new_kwargs.pop('refcoord_scaling', None)\n new_kwargs.pop('Pcoupl', None)\n return new_kwargs"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef MD(dirname='MD', **kwargs):\n\n logger.info(\"[{dirname!s}] Setting up MD...\".format(**vars()))\n kwargs.setdefault('struct', 'MD_POSRES/md.gro')\n kwargs.setdefault('qname', 'MD_GMX')\n return _setup_MD(dirname, **kwargs)", "response": "Set up equilibrium MD."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating the submit scripts for a queuing system.", "response": "def generate_submit_scripts(templates, prefix=None, deffnm='md', jobname='MD', budget=None,\n mdrun_opts=None, walltime=1.0, jobarray_string=None, startdir=None,\n npme=None, **kwargs):\n \"\"\"Write scripts for queuing systems.\n\n\n This sets up queuing system run scripts with a simple search and replace in\n templates. See :func:`gromacs.cbook.edit_txt` for details. Shell scripts\n are made executable.\n\n :Arguments:\n *templates*\n Template file or list of template files. The \"files\" can also be names\n or symbolic names for templates in the templates directory. See\n :mod:`gromacs.config` for details and rules for writing templates.\n *prefix*\n Prefix for the final run script filename; by default the filename will be\n the same as the template. [None]\n *dirname*\n Directory in which to place the submit scripts. [.]\n *deffnm*\n Default filename prefix for :program:`mdrun` ``-deffnm`` [md]\n *jobname*\n Name of the job in the queuing system. [MD]\n *budget*\n Which budget to book the runtime on [None]\n *startdir*\n Explicit path on the remote system (for run scripts that need to `cd`\n into this directory at the beginning of execution) [None]\n *mdrun_opts*\n String of additional options for :program:`mdrun`.\n *walltime*\n Maximum runtime of the job in hours. [1]\n *npme*\n number of PME nodes\n *jobarray_string*\n Multi-line string that is spliced in for job array functionality\n (see :func:`gromacs.qsub.generate_submit_array`; do not use manually)\n *kwargs*\n all other kwargs are ignored\n\n :Returns: list of generated run scripts\n \"\"\"\n if not jobname[0].isalpha():\n jobname = 'MD_'+jobname\n wmsg = \"To make the jobname legal it must start with a letter: changed to {0!r}\".format(jobname)\n logger.warn(wmsg)\n warnings.warn(wmsg, category=AutoCorrectionWarning)\n if prefix is None:\n prefix = \"\"\n if mdrun_opts is not None:\n mdrun_opts = '\"'+str(mdrun_opts)+'\"' # TODO: could test if quotes already present\n\n dirname = kwargs.pop('dirname', os.path.curdir)\n\n wt = Timedelta(hours=walltime)\n walltime = wt.strftime(\"%h:%M:%S\")\n wall_hours = wt.ashours\n\n def write_script(template):\n submitscript = os.path.join(dirname, prefix + os.path.basename(template))\n logger.info(\"Setting up queuing system script {submitscript!r}...\".format(**vars()))\n # These substitution rules are documented for the user in the module doc string\n qsystem = detect_queuing_system(template)\n if qsystem is not None and (qsystem.name == 'Slurm'):\n cbook.edit_txt(template,\n [('^ *DEFFNM=','(?<==)(.*)', deffnm),\n ('^#.*(-J)', '((?<=-J\\s))\\s*\\w+', jobname),\n ('^#.*(-A|account_no)', '((?<=-A\\s)|(?<=account_no\\s))\\s*\\w+', budget),\n ('^#.*(-t)', '(?<=-t\\s)(\\d+:\\d+:\\d+)', walltime),\n ('^ *WALL_HOURS=', '(?<==)(.*)', wall_hours),\n ('^ *STARTDIR=', '(?<==)(.*)', startdir),\n ('^ *NPME=', '(?<==)(.*)', npme),\n ('^ *MDRUN_OPTS=', '(?<==)(\"\")', mdrun_opts), # only replace literal \"\"\n ('^# JOB_ARRAY_PLACEHOLDER', '^.*$', jobarray_string),\n ],\n newname=submitscript)\n ext = os.path.splitext(submitscript)[1]\n else:\n cbook.edit_txt(template,\n [('^ *DEFFNM=','(?<==)(.*)', deffnm),\n ('^#.*(-N|job_name)', '((?<=-N\\s)|(?<=job_name\\s))\\s*\\w+', jobname),\n ('^#.*(-A|account_no)', '((?<=-A\\s)|(?<=account_no\\s))\\s*\\w+', budget),\n ('^#.*(-l walltime|wall_clock_limit)', '(?<==)(\\d+:\\d+:\\d+)', walltime),\n ('^ *WALL_HOURS=', '(?<==)(.*)', wall_hours),\n ('^ *STARTDIR=', '(?<==)(.*)', startdir),\n ('^ *NPME=', '(?<==)(.*)', npme),\n ('^ *MDRUN_OPTS=', '(?<==)(\"\")', mdrun_opts), # only replace literal \"\"\n ('^# JOB_ARRAY_PLACEHOLDER', '^.*$', jobarray_string),\n ],\n newname=submitscript)\n ext = os.path.splitext(submitscript)[1]\n if ext in ('.sh', '.csh', '.bash'):\n os.chmod(submitscript, 0o755)\n return submitscript\n\n return [write_script(template) for template in config.get_templates(templates)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ngenerating a submit script for a single array job.", "response": "def generate_submit_array(templates, directories, **kwargs):\n \"\"\"Generate a array job.\n\n For each ``work_dir`` in *directories*, the array job will\n 1. cd into ``work_dir``\n 2. run the job as detailed in the template\n It will use all the queuing system directives found in the\n template. If more complicated set ups are required, then this\n function cannot be used.\n\n :Arguments:\n *templates*\n Basic template for a single job; the job array logic is spliced into\n the position of the line ::\n # JOB_ARRAY_PLACEHOLDER\n The appropriate commands for common queuing systems (Sun Gridengine, PBS)\n are hard coded here. The queuing system is detected from the suffix of\n the template.\n *directories*\n List of directories under *dirname*. One task is set up for each\n directory.\n *dirname*\n The array script will be placed in this directory. The *directories*\n **must** be located under *dirname*.\n *kwargs*\n See :func:`gromacs.setup.generate_submit_script` for details.\n \"\"\"\n dirname = kwargs.setdefault('dirname', os.path.curdir)\n reldirs = [relpath(p, start=dirname) for p in asiterable(directories)]\n missing = [p for p in (os.path.join(dirname, subdir) for subdir in reldirs)\n if not os.path.exists(p)]\n if len(missing) > 0:\n logger.debug(\"template=%(template)r: dirname=%(dirname)r reldirs=%(reldirs)r\", vars())\n logger.error(\"Some directories are not accessible from the array script: \"\n \"%(missing)r\", vars())\n def write_script(template):\n qsystem = detect_queuing_system(template)\n if qsystem is None or not qsystem.has_arrays():\n logger.warning(\"Not known how to make a job array for %(template)r; skipping...\", vars())\n return None\n kwargs['jobarray_string'] = qsystem.array(reldirs)\n return generate_submit_scripts(template, **kwargs)[0] # returns list of length 1\n\n # must use config.get_templates() because we need to access the file for detecting\n return [write_script(template) for template in config.get_templates(templates)]"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef array(self, directories):\n if not self.has_arrays():\n raise NotImplementedError('Not known how make array jobs for '\n 'queuing system %(name)s' % vars(self))\n hrule = '#'+60*'-'\n lines = [\n '',\n hrule,\n '# job array:',\n self.array_flag(directories),\n hrule,\n '# directories for job tasks',\n 'declare -a jobdirs']\n for i,dirname in enumerate(asiterable(directories)):\n idx = i+1 # job array indices are 1-based\n lines.append('jobdirs[{idx:d}]={dirname!r}'.format(**vars()))\n lines.extend([\n '# Switch to the current tasks directory:',\n 'wdir=\"${{jobdirs[${{{array_variable!s}}}]}}\"'.format(**vars(self)),\n 'cd \"$wdir\" || { echo \"ERROR: failed to enter $wdir.\"; exit 1; }',\n hrule,\n ''\n ])\n return \"\\n\".join(lines)", "response": "Return multiline string for simple array jobs over directories."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef isMine(self, scriptname):\n suffix = os.path.splitext(scriptname)[1].lower()\n if suffix.startswith('.'):\n suffix = suffix[1:]\n return self.suffix == suffix", "response": "Primitive queuing system detection ; only looks at suffix at the moment."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the atom object corresponding to an anumb number", "response": "def anumb_to_atom(self, anumb):\n '''Returns the atom object corresponding to an atom number'''\n\n assert isinstance(anumb, int), \"anumb must be integer\"\n\n if not self._anumb_to_atom: # empty dictionary\n\n if self.atoms:\n for atom in self.atoms:\n self._anumb_to_atom[atom.number] = atom\n return self._anumb_to_atom[anumb]\n else:\n self.logger(\"no atoms in the molecule\")\n return False\n\n else:\n if anumb in self._anumb_to_atom:\n return self._anumb_to_atom[anumb]\n else:\n self.logger(\"no such atom number ({0:d}) in the molecule\".format(anumb))\n return False"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef renumber_atoms(self):\n if self.atoms:\n\n # reset the mapping\n self._anumb_to_atom = {}\n\n for i,atom in enumerate(self.atoms):\n atom.number = i+1 # starting from 1\n\n else:\n self.logger(\"the number of atoms is zero - no renumbering\")", "response": "Reset the molecule s atoms : attr : number to be 1 - indexed"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing the total number of unmasked regular pixels in a masks.", "response": "def total_regular_pixels_from_mask(mask):\n \"\"\"Compute the total number of unmasked regular pixels in a masks.\"\"\"\n\n total_regular_pixels = 0\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n if not mask[y, x]:\n total_regular_pixels += 1\n\n return total_regular_pixels"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef total_sparse_pixels_from_mask(mask, unmasked_sparse_grid_pixel_centres):\n\n total_sparse_pixels = 0\n\n for unmasked_sparse_pixel_index in range(unmasked_sparse_grid_pixel_centres.shape[0]):\n\n y = unmasked_sparse_grid_pixel_centres[unmasked_sparse_pixel_index, 0]\n x = unmasked_sparse_grid_pixel_centres[unmasked_sparse_pixel_index, 1]\n\n if not mask[y,x]:\n total_sparse_pixels += 1\n\n return total_sparse_pixels", "response": "This function calculates the total number of pixels which are within the regular - masks of the unmasked pixelization grid and the unmasked pixelization grid s pixel centers."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef mask_circular_from_shape_pixel_scale_and_radius(shape, pixel_scale, radius_arcsec, centre=(0.0, 0.0)):\n\n mask = np.full(shape, True)\n\n centres_arcsec = mask_centres_from_shape_pixel_scale_and_centre(shape=mask.shape, pixel_scale=pixel_scale, centre=centre)\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n\n y_arcsec = (y - centres_arcsec[0]) * pixel_scale\n x_arcsec = (x - centres_arcsec[1]) * pixel_scale\n\n r_arcsec = np.sqrt(x_arcsec ** 2 + y_arcsec ** 2)\n\n if r_arcsec <= radius_arcsec:\n mask[y, x] = False\n\n return mask", "response": "Compute a circular mask from an input masks radius and regular shape."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute an annular mask from an input inner and outer masks radius and regular shape.", "response": "def mask_circular_annular_from_shape_pixel_scale_and_radii(shape, pixel_scale, inner_radius_arcsec, outer_radius_arcsec,\n centre=(0.0, 0.0)):\n \"\"\"Compute an annular masks from an input inner and outer masks radius and regular shape.\"\"\"\n\n mask = np.full(shape, True)\n\n centres_arcsec = mask_centres_from_shape_pixel_scale_and_centre(shape=mask.shape, pixel_scale=pixel_scale, centre=centre)\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n\n y_arcsec = (y - centres_arcsec[0]) * pixel_scale\n x_arcsec = (x - centres_arcsec[1]) * pixel_scale\n\n r_arcsec = np.sqrt(x_arcsec ** 2 + y_arcsec ** 2)\n\n if outer_radius_arcsec >= r_arcsec >= inner_radius_arcsec:\n mask[y, x] = False\n\n return mask"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mask_elliptical_from_shape_pixel_scale_and_radius(shape, pixel_scale, major_axis_radius_arcsec, axis_ratio, phi,\n centre=(0.0, 0.0)):\n \"\"\"Compute a circular masks from an input masks radius and regular shape.\"\"\"\n\n mask = np.full(shape, True)\n\n centres_arcsec = mask_centres_from_shape_pixel_scale_and_centre(shape=mask.shape, pixel_scale=pixel_scale,\n centre=centre)\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n\n y_arcsec = (y - centres_arcsec[0]) * pixel_scale\n x_arcsec = (x - centres_arcsec[1]) * pixel_scale\n\n r_arcsec_elliptical = elliptical_radius_from_y_x_phi_and_axis_ratio(y_arcsec, x_arcsec, phi, axis_ratio)\n\n if r_arcsec_elliptical <= major_axis_radius_arcsec:\n mask[y, x] = False\n\n return mask", "response": "Compute a circular mask from an input masks radius and regular shape."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing a circular mask from an input masks radius and regular shape.", "response": "def mask_elliptical_annular_from_shape_pixel_scale_and_radius(shape, pixel_scale,\n inner_major_axis_radius_arcsec, inner_axis_ratio, inner_phi,\n outer_major_axis_radius_arcsec, outer_axis_ratio, outer_phi,\n centre=(0.0, 0.0)):\n \"\"\"Compute a circular masks from an input masks radius and regular shape.\"\"\"\n\n mask = np.full(shape, True)\n\n centres_arcsec = mask_centres_from_shape_pixel_scale_and_centre(shape=mask.shape, pixel_scale=pixel_scale,\n centre=centre)\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n\n y_arcsec = (y - centres_arcsec[0]) * pixel_scale\n x_arcsec = (x - centres_arcsec[1]) * pixel_scale\n\n inner_r_arcsec_elliptical = elliptical_radius_from_y_x_phi_and_axis_ratio(y_arcsec, x_arcsec,\n inner_phi, inner_axis_ratio)\n\n outer_r_arcsec_elliptical = elliptical_radius_from_y_x_phi_and_axis_ratio(y_arcsec, x_arcsec,\n outer_phi, outer_axis_ratio)\n\n if inner_r_arcsec_elliptical >= inner_major_axis_radius_arcsec and \\\n outer_r_arcsec_elliptical <= outer_major_axis_radius_arcsec:\n mask[y, x] = False\n\n return mask"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute a blurring mask from an input masks and psf shape.", "response": "def mask_blurring_from_mask_and_psf_shape(mask, psf_shape):\n \"\"\"Compute a blurring masks from an input masks and psf shape.\n\n The blurring masks corresponds to all pixels which are outside of the masks but will have a fraction of their \\\n light blur into the masked region due to PSF convolution.\"\"\"\n\n blurring_mask = np.full(mask.shape, True)\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n if not mask[y, x]:\n for y1 in range((-psf_shape[0] + 1) // 2, (psf_shape[0] + 1) // 2):\n for x1 in range((-psf_shape[1] + 1) // 2, (psf_shape[1] + 1) // 2):\n if 0 <= x + x1 <= mask.shape[1] - 1 and 0 <= y + y1 <= mask.shape[0] - 1:\n if mask[y + y1, x + x1]:\n blurring_mask[y + y1, x + x1] = False\n else:\n raise exc.MaskException(\n \"setup_blurring_mask extends beyond the sub_grid_size of the masks - pad the \"\n \"datas array before masking\")\n\n return blurring_mask"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef masked_grid_1d_index_to_2d_pixel_index_from_mask(mask):\n\n total_regular_pixels = total_regular_pixels_from_mask(mask)\n grid_to_pixel = np.zeros(shape=(total_regular_pixels, 2))\n pixel_count = 0\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n if not mask[y, x]:\n grid_to_pixel[pixel_count, :] = y, x\n pixel_count += 1\n\n return grid_to_pixel", "response": "Compute a 1D array that maps every unmasked pixel to its corresponding 2d pixel using its ( y x ) pixel indexes."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute a 1D array that maps every unmasked pixel to its corresponding 2d pixel using its ( y x ) pixel indexes.", "response": "def masked_sub_grid_1d_index_to_2d_sub_pixel_index_from_mask(mask, sub_grid_size):\n \"\"\"Compute a 1D array that maps every unmasked pixel to its corresponding 2d pixel using its (y,x) pixel indexes.\n\n For howtolens if pixel [2,5] corresponds to the second pixel on the 1D array, grid_to_pixel[1] = [2,5]\"\"\"\n\n total_sub_pixels = total_sub_pixels_from_mask_and_sub_grid_size(mask=mask, sub_grid_size=sub_grid_size)\n sub_grid_to_sub_pixel = np.zeros(shape=(total_sub_pixels, 2))\n sub_pixel_count = 0\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n if not mask[y, x]:\n for y1 in range(sub_grid_size):\n for x1 in range(sub_grid_size):\n sub_grid_to_sub_pixel[sub_pixel_count, :] = (y*sub_grid_size)+y1, (x*sub_grid_size)+x1\n sub_pixel_count += 1\n\n return sub_grid_to_sub_pixel"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef total_edge_pixels_from_mask(mask):\n\n border_pixel_total = 0\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n if not mask[y, x]:\n if mask[y + 1, x] or mask[y - 1, x] or mask[y, x + 1] or mask[y, x - 1] or \\\n mask[y + 1, x + 1] or mask[y + 1, x - 1] or mask[y - 1, x + 1] or mask[y - 1, x - 1]:\n border_pixel_total += 1\n\n return border_pixel_total", "response": "Compute the total number of borders - pixels in a masks."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncomputes a 1D array listing all edge pixels in the masks.", "response": "def edge_pixels_from_mask(mask):\n \"\"\"Compute a 1D array listing all edge pixel indexes in the masks. An edge pixel is a pixel which is not fully \\\n surrounding by False masks values i.e. it is on an edge.\"\"\"\n\n edge_pixel_total = total_edge_pixels_from_mask(mask)\n\n edge_pixels = np.zeros(edge_pixel_total)\n edge_index = 0\n regular_index = 0\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n if not mask[y, x]:\n if mask[y + 1, x] or mask[y - 1, x] or mask[y, x + 1] or mask[y, x - 1] or \\\n mask[y + 1, x + 1] or mask[y + 1, x - 1] or mask[y - 1, x + 1] or mask[y - 1, x - 1]:\n edge_pixels[edge_index] = regular_index\n edge_index += 1\n\n regular_index += 1\n\n return edge_pixels"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef total_border_pixels_from_mask_and_edge_pixels(mask, edge_pixels, masked_grid_index_to_pixel):\n\n border_pixel_total = 0\n\n for i in range(edge_pixels.shape[0]):\n\n if check_if_border_pixel(mask, edge_pixels[i], masked_grid_index_to_pixel):\n border_pixel_total += 1\n\n return border_pixel_total", "response": "Compute the total number of borders - pixels in a masks."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute a 1D array listing all borders pixels in a masked pixel.", "response": "def border_pixels_from_mask(mask):\n \"\"\"Compute a 1D array listing all borders pixel indexes in the masks. A borders pixel is a pixel which:\n\n 1) is not fully surrounding by False masks values.\n 2) Can reach the edge of the array without hitting a masked pixel in one of four directions (upwards, downwards,\n left, right).\n\n The borders pixels are thus pixels which are on the exterior edge of the masks. For example, the inner ring of edge \\\n pixels in an annular masks are edge pixels but not borders pixels.\"\"\"\n\n edge_pixels = edge_pixels_from_mask(mask)\n masked_grid_index_to_pixel = masked_grid_1d_index_to_2d_pixel_index_from_mask(mask)\n\n border_pixel_total = total_border_pixels_from_mask_and_edge_pixels(mask, edge_pixels, masked_grid_index_to_pixel)\n\n border_pixels = np.zeros(border_pixel_total)\n\n border_pixel_index = 0\n\n for edge_pixel_index in range(edge_pixels.shape[0]):\n\n if check_if_border_pixel(mask, edge_pixels[edge_pixel_index], masked_grid_index_to_pixel):\n border_pixels[border_pixel_index] = edge_pixels[edge_pixel_index]\n border_pixel_index += 1\n\n return border_pixels"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nbins up an array to coarser resolution by using the sum value of the new pixel.", "response": "def bin_up_mask_2d(mask_2d, bin_up_factor):\n \"\"\"Bin up an array to coarser resolution, by binning up groups of pixels and using their sum value to determine \\\n the value of the new pixel.\n\n If an array of shape (8,8) is input and the bin up size is 2, this would return a new array of size (4,4) where \\\n every pixel was the sum of each collection of 2x2 pixels on the (8,8) array.\n\n If binning up the array leads to an edge being cut (e.g. a (9,9) array binned up by 2), an array is first \\\n extracted around the centre of that array.\n\n\n Parameters\n ----------\n mask_2d : ndarray\n The 2D array that is resized.\n new_shape : (int, int)\n The (y,x) new pixel dimension of the trimmed array.\n origin : (int, int)\n The oigin of the resized array, e.g. the central pixel around which the array is extracted.\n\n Returns\n -------\n ndarray\n The resized 2D array from the input 2D array.\n\n Examples\n --------\n array_2d = np.ones((5,5))\n resize_array = resize_array_2d(array_2d=array_2d, new_shape=(2,2), origin=(2, 2))\n \"\"\"\n\n padded_array_2d = array_util.pad_2d_array_for_binning_up_with_bin_up_factor(\n array_2d=mask_2d, bin_up_factor=bin_up_factor, pad_value=True)\n\n binned_array_2d = np.zeros(shape=(padded_array_2d.shape[0] // bin_up_factor,\n padded_array_2d.shape[1] // bin_up_factor))\n\n for y in range(binned_array_2d.shape[0]):\n for x in range(binned_array_2d.shape[1]):\n value = True\n for y1 in range(bin_up_factor):\n for x1 in range(bin_up_factor):\n padded_y = y*bin_up_factor + y1\n padded_x = x*bin_up_factor + x1\n if padded_array_2d[padded_y, padded_x] == False:\n value = False\n\n binned_array_2d[y,x] = value\n\n return binned_array_2d"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nplotting the model datas_ of an analysis using the lens fit.", "response": "def plot_fit_subplot_lens_plane_only(\n fit, should_plot_mask=True, extract_array_from_mask=False, zoom_around_mask=False, positions=None,\n should_plot_image_plane_pix=False,\n units='arcsec', figsize=None, aspect='square',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n titlesize=10, xlabelsize=10, ylabelsize=10, xyticksize=10,\n mask_pointsize=10, position_pointsize=10, grid_pointsize=1,\n output_path=None, output_filename='lens_fit', output_format='show'):\n \"\"\"Plot the model datas_ of an analysis, using the *Fitter* class object.\n\n The visualization and output type can be fully customized.\n\n Parameters\n -----------\n fit : autolens.lens.fitting.Fitter\n Class containing fit between the model datas_ and observed lens datas_ (including residual_map, chi_squared_map etc.)\n output_path : str\n The path where the datas_ is output if the output_type is a file format (e.g. png, fits)\n output_filename : str\n The name of the file that is output, if the output_type is a file format (e.g. png, fits)\n output_format : str\n How the datas_ is output. File formats (e.g. png, fits) output the datas_ to harddisk. 'show' displays the datas_ \\\n in the python interpreter window.\n \"\"\"\n\n rows, columns, figsize_tool = plotter_util.get_subplot_rows_columns_figsize(number_subplots=6)\n\n mask = lens_plotter_util.get_mask(fit=fit, should_plot_mask=should_plot_mask)\n\n if figsize is None:\n figsize = figsize_tool\n\n plt.figure(figsize=figsize)\n plt.subplot(rows, columns, 1)\n\n kpc_per_arcsec = fit.tracer.image_plane.kpc_per_arcsec\n\n image_plane_pix_grid = lens_plotter_util.get_image_plane_pix_grid(should_plot_image_plane_pix, fit)\n\n lens_plotter_util.plot_image(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n positions=positions, image_plane_pix_grid=image_plane_pix_grid, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n grid_pointsize=grid_pointsize, position_pointsize=position_pointsize, mask_pointsize=mask_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 2)\n\n lens_plotter_util.plot_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n positions=positions, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n position_pointsize=position_pointsize, mask_pointsize=mask_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 3)\n\n lens_plotter_util.plot_signal_to_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n positions=positions, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n position_pointsize=position_pointsize, mask_pointsize=mask_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 4)\n\n lens_plotter_util.plot_model_data(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 5)\n\n lens_plotter_util.plot_residual_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 6)\n\n lens_plotter_util.plot_chi_squared_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plotter_util.output_subplot_array(output_path=output_path, output_filename=output_filename,\n output_format=output_format)\n\n plt.close()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nplotting the model datas_ of an analysis and observed lens datas_ of an analysis.", "response": "def plot_fit_subplot_lens_and_source_planes(\n fit, should_plot_mask=True, extract_array_from_mask=False, zoom_around_mask=False,\n should_plot_source_grid=False, positions=None, should_plot_image_plane_pix=False, plot_mass_profile_centres=True,\n units='arcsec', figsize=None, aspect='square',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n titlesize=10, xlabelsize=10, ylabelsize=10, xyticksize=10,\n mask_pointsize=10, position_pointsize=10, grid_pointsize=1,\n output_path=None, output_filename='lens_fit', output_format='show'):\n \"\"\"Plot the model datas_ of an analysis, using the *Fitter* class object.\n\n The visualization and output type can be fully customized.\n\n Parameters\n -----------\n fit : autolens.lens.fitting.Fitter\n Class containing fit between the model datas_ and observed lens datas_ (including residual_map, chi_squared_map etc.)\n output_path : str\n The path where the datas_ is output if the output_type is a file format (e.g. png, fits)\n output_filename : str\n The name of the file that is output, if the output_type is a file format (e.g. png, fits)\n output_format : str\n How the datas_ is output. File formats (e.g. png, fits) output the datas_ to harddisk. 'show' displays the datas_ \\\n in the python interpreter window.\n \"\"\"\n\n rows, columns, figsize_tool = plotter_util.get_subplot_rows_columns_figsize(number_subplots=9)\n\n mask = lens_plotter_util.get_mask(fit=fit, should_plot_mask=should_plot_mask)\n\n if figsize is None:\n figsize = figsize_tool\n\n plt.figure(figsize=figsize)\n\n kpc_per_arcsec = fit.tracer.image_plane.kpc_per_arcsec\n\n image_plane_pix_grid = lens_plotter_util.get_image_plane_pix_grid(should_plot_image_plane_pix, fit)\n\n plt.subplot(rows, columns, 1)\n\n lens_plotter_util.plot_image(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, positions=positions,\n image_plane_pix_grid=image_plane_pix_grid, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n grid_pointsize=grid_pointsize, position_pointsize=position_pointsize, mask_pointsize=mask_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 2)\n\n lens_plotter_util.plot_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, positions=None, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n position_pointsize=position_pointsize, mask_pointsize=mask_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 3)\n\n lens_plotter_util.plot_signal_to_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, positions=None, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n position_pointsize=position_pointsize, mask_pointsize=mask_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 4)\n\n lens_plotter_util.plot_lens_subtracted_image(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, positions=positions, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n title='Fit Lens Subtracted Image', titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize,\n position_pointsize=position_pointsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n if fit.tracer.image_plane.has_light_profile:\n\n plt.subplot(rows, columns, 5)\n\n lens_plotter_util.plot_model_image_of_planes(\n fit=fit, plot_foreground=True, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, plot_mass_profile_centres=plot_mass_profile_centres, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n title='Fit Image-Plane Model Image', titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize,\n xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 6)\n\n lens_plotter_util.plot_model_image_of_planes(\n fit=fit, plot_source=True, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, plot_mass_profile_centres=plot_mass_profile_centres, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n title='Fit Source-Plane Model Image', titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize,\n xyticksize=xyticksize,\n output_path=output_path, output_filename='lens_subtracted_image', output_format=output_format)\n\n if fit.total_inversions == 0:\n\n plt.subplot(rows, columns, 7)\n\n plane_plotters.plot_plane_image(\n plane=fit.tracer.source_plane, positions=None, plot_grid=should_plot_source_grid, as_subplot=True,\n units=units, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad,\n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n grid_pointsize=grid_pointsize, position_pointsize=position_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n elif fit.total_inversions == 1:\n \n ratio = float((fit.inversion.mapper.geometry.arc_second_maxima[1] - fit.inversion.mapper.geometry.arc_second_minima[1]) / \\\n (fit.inversion.mapper.geometry.arc_second_maxima[0] - fit.inversion.mapper.geometry.arc_second_minima[0]))\n\n if aspect is 'square':\n aspect_inv = ratio\n elif aspect is 'auto':\n aspect_inv = 1.0 / ratio\n elif aspect is 'equal':\n aspect_inv = 1.0\n\n plt.subplot(rows, columns, 7, aspect=float(aspect_inv))\n\n inversion_plotters.plot_reconstructed_pixelization(\n inversion=fit.inversion, positions=None, should_plot_grid=False, should_plot_centres=False, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=None,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad,\n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename=None, output_format=output_format)\n\n plt.subplot(rows, columns, 8)\n\n lens_plotter_util.plot_residual_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad,\n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 9)\n\n lens_plotter_util.plot_chi_squared_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad,\n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plotter_util.output_subplot_array(output_path=output_path, output_filename=output_filename,\n output_format=output_format)\n\n plt.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef plot_fit_individuals_lens_plane_only(\n fit, should_plot_mask=True, extract_array_from_mask=False, zoom_around_mask=False, positions=None,\n should_plot_image_plane_pix=False,\n should_plot_image=False,\n should_plot_noise_map=False,\n should_plot_signal_to_noise_map=False,\n should_plot_model_image=False,\n should_plot_residual_map=False,\n should_plot_chi_squared_map=False,\n units='arcsec',\n output_path=None, output_format='show'):\n \"\"\"Plot the model datas_ of an analysis, using the *Fitter* class object.\n\n The visualization and output type can be fully customized.\n\n Parameters\n -----------\n fit : autolens.lens.fitting.Fitter\n Class containing fit between the model datas_ and observed lens datas_ (including residual_map, chi_squared_map etc.)\n output_path : str\n The path where the datas_ is output if the output_type is a file format (e.g. png, fits)\n output_format : str\n How the datas_ is output. File formats (e.g. png, fits) output the datas_ to harddisk. 'show' displays the datas_ \\\n in the python interpreter window.\n \"\"\"\n\n mask = lens_plotter_util.get_mask(fit=fit, should_plot_mask=should_plot_mask)\n\n kpc_per_arcsec = fit.tracer.image_plane.kpc_per_arcsec\n\n if should_plot_image:\n\n image_plane_pix_grid = lens_plotter_util.get_image_plane_pix_grid(should_plot_image_plane_pix, fit)\n\n lens_plotter_util.plot_image(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n image_plane_pix_grid=image_plane_pix_grid,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_noise_map:\n\n lens_plotter_util.plot_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_signal_to_noise_map:\n\n lens_plotter_util.plot_signal_to_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_model_image:\n\n lens_plotter_util.plot_model_data(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask, positions=positions,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_residual_map:\n\n lens_plotter_util.plot_residual_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_chi_squared_map:\n\n lens_plotter_util.plot_chi_squared_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)", "response": "Plot the model datas_ of an analysis using the lens - plane only plotter."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nplotting the model datas_ of an analysis using the lens fitter.", "response": "def plot_fit_individuals_lens_and_source_planes(\n fit, should_plot_mask=True, extract_array_from_mask=False, zoom_around_mask=False, positions=None,\n should_plot_image_plane_pix=False,\n should_plot_image=False,\n should_plot_noise_map=False,\n should_plot_signal_to_noise_map=False,\n should_plot_lens_subtracted_image=False,\n should_plot_model_image=False,\n should_plot_lens_model_image=False,\n should_plot_source_model_image=False,\n should_plot_source_plane_image=False,\n should_plot_residual_map=False,\n should_plot_chi_squared_map=False,\n units='arcsec',\n output_path=None, output_format='show'):\n \"\"\"Plot the model datas_ of an analysis, using the *Fitter* class object.\n\n The visualization and output type can be fully customized.\n\n Parameters\n -----------\n fit : autolens.lens.fitting.Fitter\n Class containing fit between the model datas_ and observed lens datas_ (including residual_map, chi_squared_map etc.)\n output_path : str\n The path where the datas_ is output if the output_type is a file format (e.g. png, fits)\n output_format : str\n How the datas_ is output. File formats (e.g. png, fits) output the datas_ to harddisk. 'show' displays the datas_ \\\n in the python interpreter window.\n \"\"\"\n\n mask = lens_plotter_util.get_mask(fit=fit, should_plot_mask=should_plot_mask)\n\n kpc_per_arcsec = fit.tracer.image_plane.kpc_per_arcsec\n\n if should_plot_image:\n\n image_plane_pix_grid = lens_plotter_util.get_image_plane_pix_grid(should_plot_image_plane_pix, fit)\n\n lens_plotter_util.plot_image(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n image_plane_pix_grid=image_plane_pix_grid,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_noise_map:\n\n lens_plotter_util.plot_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_signal_to_noise_map:\n\n lens_plotter_util.plot_signal_to_noise_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_lens_subtracted_image:\n\n lens_plotter_util.plot_lens_subtracted_image(\n fit=fit, mask=mask, positions=positions, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_model_image:\n lens_plotter_util.plot_model_data(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_lens_model_image:\n\n lens_plotter_util.plot_model_image_of_planes(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n plot_foreground=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_filename='fit_lens_plane_model_image', output_format=output_format)\n\n if should_plot_source_model_image:\n\n lens_plotter_util.plot_model_image_of_planes(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n plot_source=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_filename='fit_source_plane_model_image', output_format=output_format)\n\n if should_plot_source_plane_image:\n\n if fit.total_inversions == 0:\n\n plane_plotters.plot_plane_image(\n plane=fit.tracer.source_plane, plot_grid=True,\n units=units, figsize=(20, 20),\n output_path=output_path, output_filename='fit_source_plane', output_format=output_format)\n\n elif fit.total_inversions == 1:\n\n inversion_plotters.plot_reconstructed_pixelization(\n inversion=fit.inversion, should_plot_grid=True,\n units=units, figsize=(20, 20),\n output_path=output_path, output_filename='fit_source_plane', output_format=output_format)\n\n if should_plot_residual_map:\n\n lens_plotter_util.plot_residual_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)\n\n if should_plot_chi_squared_map:\n\n lens_plotter_util.plot_chi_squared_map(\n fit=fit, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n units=units, kpc_per_arcsec=kpc_per_arcsec,\n output_path=output_path, output_format=output_format)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef setup_figure(figsize, as_subplot):\n if not as_subplot:\n fig = plt.figure(figsize=figsize)\n return fig", "response": "Setup a figure for plotting an image."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef output_figure(array, as_subplot, output_path, output_filename, output_format):\n if not as_subplot:\n\n if output_format is 'show':\n plt.show()\n elif output_format is 'png':\n plt.savefig(output_path + output_filename + '.png', bbox_inches='tight')\n elif output_format is 'fits':\n array_util.numpy_array_2d_to_fits(array_2d=array, file_path=output_path + output_filename + '.fits',\n overwrite=True)", "response": "Output the figure as an image on the computer screen or as a. fits file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\noutputting a figure which consists of a set of subplot,, either as an image on the computer screen or to the hard - disk as a png file.", "response": "def output_subplot_array(output_path, output_filename, output_format):\n \"\"\"Output a figure which consists of a set of subplot,, either as an image on the screen or to the hard-disk as a \\\n .png file.\n\n Parameters\n -----------\n output_path : str\n The path on the hard-disk where the figure is output.\n output_filename : str\n The filename of the figure that is output.\n output_format : str\n The format the figue is output:\n 'show' - display on computer screen.\n 'png' - output to hard-disk as a png.\n \"\"\"\n if output_format is 'show':\n plt.show()\n elif output_format is 'png':\n plt.savefig(output_path + output_filename + '.png', bbox_inches='tight')\n elif output_format is 'fits':\n raise exc.PlottingException('You cannot output a subplots with format .fits')"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating an image psf shape tag from image PSF shape.", "response": "def image_psf_shape_tag_from_image_psf_shape(image_psf_shape):\n \"\"\"Generate an image psf shape tag, to customize phase names based on size of the image PSF that the original PSF \\\n is trimmed to for faster run times.\n\n This changes the phase name 'phase_name' as follows:\n\n image_psf_shape = 1 -> phase_name\n image_psf_shape = 2 -> phase_name_image_psf_shape_2\n image_psf_shape = 2 -> phase_name_image_psf_shape_2\n \"\"\"\n if image_psf_shape is None:\n return ''\n else:\n y = str(image_psf_shape[0])\n x = str(image_psf_shape[1])\n return ('_image_psf_' + y + 'x' + x)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ngenerates an inversion psf shape tag from the size of the inversion PSF.", "response": "def inversion_psf_shape_tag_from_inversion_psf_shape(inversion_psf_shape):\n \"\"\"Generate an inversion psf shape tag, to customize phase names based on size of the inversion PSF that the \\\n original PSF is trimmed to for faster run times.\n\n This changes the phase name 'phase_name' as follows:\n\n inversion_psf_shape = 1 -> phase_name\n inversion_psf_shape = 2 -> phase_name_inversion_psf_shape_2\n inversion_psf_shape = 2 -> phase_name_inversion_psf_shape_2\n \"\"\"\n if inversion_psf_shape is None:\n return ''\n else:\n y = str(inversion_psf_shape[0])\n x = str(inversion_psf_shape[1])\n return ('_inv_psf_' + y + 'x' + x)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a tag for the bulge - disk of a system", "response": "def bulge_disk_tag_from_align_bulge_disks(align_bulge_disk_centre, align_bulge_disk_axis_ratio, align_bulge_disk_phi):\n \"\"\"Generate a tag for the alignment of the geometry of the bulge and disk of a bulge-disk system, to customize \\ \n phase names based on the bulge-disk model. This adds together the bulge_disk tags generated in the 3 functions\n above\n \"\"\"\n align_bulge_disk_centre_tag = align_bulge_disk_centre_tag_from_align_bulge_disk_centre(\n align_bulge_disk_centre=align_bulge_disk_centre)\n align_bulge_disk_axis_ratio_tag = align_bulge_disk_axis_ratio_tag_from_align_bulge_disk_axis_ratio(\n align_bulge_disk_axis_ratio=align_bulge_disk_axis_ratio)\n align_bulge_disk_phi_tag = align_bulge_disk_phi_tag_from_align_bulge_disk_phi(\n align_bulge_disk_phi=align_bulge_disk_phi)\n\n return align_bulge_disk_centre_tag + align_bulge_disk_axis_ratio_tag + align_bulge_disk_phi_tag"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives a list of galaxies return a list of the redshifts in ascending order.", "response": "def ordered_plane_redshifts_from_galaxies(galaxies):\n \"\"\"Given a list of galaxies (with redshifts), return a list of the redshifts in ascending order.\n\n If two or more galaxies have the same redshift that redshift is not double counted.\n\n Parameters\n -----------\n galaxies : [Galaxy]\n The list of galaxies in the ray-tracing calculation.\n \"\"\"\n ordered_galaxies = sorted(galaxies, key=lambda galaxy: galaxy.redshift, reverse=False)\n\n # Ideally we'd extract the planes_red_Shfit order from the list above. However, I dont know how to extract it\n # Using a list of class attributes so make a list of redshifts for now.\n\n galaxy_redshifts = list(map(lambda galaxy: galaxy.redshift, ordered_galaxies))\n return [redshift for i, redshift in enumerate(galaxy_redshifts) if redshift not in galaxy_redshifts[:i]]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngive a set of lens plane redshifts, the source-plane redshift and the number of planes between each, setup the \\ plane redshifts using these values. A lens redshift corresponds to the 'main' lens galaxy(s), whereas the slices collect line-of-sight halos over a range of redshifts. The source-plane redshift is removed from the ordered plane redshifts that are returned, so that galaxies are not \\ planed at the source-plane redshift. For example, if the main plane redshifts are [1.0, 2.0], and the bin sizes are [1,3], the following redshift \\ slices for planes will be used: z=0.5 z=1.0 z=1.25 z=1.5 z=1.75 z=2.0 Parameters ----------- lens_redshifts : [float] The redshifts of the main-planes (e.g. the lens galaxy), which determine where redshift intervals are placed. planes_between_lenses : [int] The number of slices between each main plane. The first entry in this list determines the number of slices \\ between Earth (redshift 0.0) and main plane 0, the next between main planes 0 and 1, etc. source_plane_redshift : float The redshift of the source-plane, which is input explicitly to ensure galaxies are not placed in the \\ source-plane.", "response": "def ordered_plane_redshifts_from_lens_and_source_plane_redshifts_and_slice_sizes(lens_redshifts, planes_between_lenses,\n source_plane_redshift):\n \"\"\"Given a set of lens plane redshifts, the source-plane redshift and the number of planes between each, setup the \\\n plane redshifts using these values. A lens redshift corresponds to the 'main' lens galaxy(s),\n whereas the slices collect line-of-sight halos over a range of redshifts.\n\n The source-plane redshift is removed from the ordered plane redshifts that are returned, so that galaxies are not \\\n planed at the source-plane redshift.\n\n For example, if the main plane redshifts are [1.0, 2.0], and the bin sizes are [1,3], the following redshift \\\n slices for planes will be used:\n\n z=0.5\n z=1.0\n z=1.25\n z=1.5\n z=1.75\n z=2.0\n\n Parameters\n -----------\n lens_redshifts : [float]\n The redshifts of the main-planes (e.g. the lens galaxy), which determine where redshift intervals are placed.\n planes_between_lenses : [int]\n The number of slices between each main plane. The first entry in this list determines the number of slices \\\n between Earth (redshift 0.0) and main plane 0, the next between main planes 0 and 1, etc.\n source_plane_redshift : float\n The redshift of the source-plane, which is input explicitly to ensure galaxies are not placed in the \\\n source-plane.\n \"\"\"\n\n # Check that the number of slices between lens planes is equal to the number of intervals between the lens planes.\n if len(lens_redshifts) != len(planes_between_lenses)-1:\n raise exc.RayTracingException('The number of lens_plane_redshifts input is not equal to the number of '\n 'slices_between_lens_planes+1.')\n\n plane_redshifts = []\n\n # Add redshift 0.0 and the source plane redshifit to the lens plane redshifts, so that calculation below can use\n # them when dividing slices. These will be removed by the return function at the end from the plane redshifts.\n\n lens_redshifts.insert(0, 0.0)\n lens_redshifts.append(source_plane_redshift)\n\n for lens_plane_index in range(1, len(lens_redshifts)):\n\n previous_plane_redshift = lens_redshifts[lens_plane_index - 1]\n plane_redshift = lens_redshifts[lens_plane_index]\n slice_total = planes_between_lenses[lens_plane_index - 1]\n plane_redshifts += list(np.linspace(previous_plane_redshift, plane_redshift, slice_total+2))[1:]\n\n return plane_redshifts[0:-1]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngiving a list of galaxies return a list of the galaxies where each entry contains a list of galaxies at the same redshift in ascending redshift order.", "response": "def galaxies_in_redshift_ordered_planes_from_galaxies(galaxies, plane_redshifts):\n \"\"\"Given a list of galaxies (with redshifts), return a list of the galaxies where each entry contains a list \\\n of galaxies at the same redshift in ascending redshift order.\n\n Parameters\n -----------\n galaxies : [Galaxy]\n The list of galaxies in the ray-tracing calculation.\n \"\"\"\n\n galaxies_in_redshift_ordered_planes = [[] for i in range(len(plane_redshifts))]\n\n for galaxy in galaxies:\n\n index = (np.abs(np.asarray(plane_redshifts) - galaxy.redshift)).argmin()\n\n galaxies_in_redshift_ordered_planes[index].append(galaxy)\n\n return galaxies_in_redshift_ordered_planes"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef compute_deflections_at_next_plane(plane_index, total_planes):\n\n if plane_index < total_planes - 1:\n return True\n elif plane_index == total_planes - 1:\n return False\n else:\n raise exc.RayTracingException('A galaxy was not correctly allocated its previous / next redshifts')", "response": "This function determines whether the tracer should compute the deflections at the next plane."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a set of scaled deflections from a plane and a scaling factor.", "response": "def scaled_deflection_stack_from_plane_and_scaling_factor(plane, scaling_factor):\n \"\"\"Given a plane and scaling factor, compute a set of scaled deflections.\n\n Parameters\n -----------\n plane : plane.Plane\n The plane whose deflection stack is scaled.\n scaling_factor : float\n The factor the deflection angles are scaled by, which is typically the scaling factor between redshifts for \\\n multi-plane lensing.\n \"\"\"\n\n def scale(grid):\n return np.multiply(scaling_factor, grid)\n\n if plane.deflection_stack is not None:\n return plane.deflection_stack.apply_function(scale)\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef plot_ray_tracing_subplot(\n tracer, mask=None, extract_array_from_mask=False, zoom_around_mask=False, positions=None,\n units='arcsec', figsize=None, aspect='square',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n titlesize=10, xlabelsize=10, ylabelsize=10, xyticksize=10,\n mask_pointsize=10, position_pointsize=10.0, grid_pointsize=1.0,\n output_path=None, output_filename='tracer', output_format='show'):\n \"\"\"Plot the observed _tracer of an analysis, using the *CCD* class object.\n\n The visualization and output type can be fully customized.\n\n Parameters\n -----------\n tracer : autolens.ccd.tracer.CCD\n Class containing the _tracer, noise_map-mappers and PSF that are to be plotted.\n The font size of the figure ylabel.\n output_path : str\n The path where the _tracer is output if the output_type is a file format (e.g. png, fits)\n output_format : str\n How the _tracer is output. File formats (e.g. png, fits) output the _tracer to harddisk. 'show' displays the _tracer \\\n in the python interpreter window.\n \"\"\"\n\n rows, columns, figsize_tool = plotter_util.get_subplot_rows_columns_figsize(number_subplots=6)\n\n if figsize is None:\n figsize = figsize_tool\n\n plt.figure(figsize=figsize)\n plt.subplot(rows, columns, 1)\n\n plot_image_plane_image(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask, zoom_around_mask=zoom_around_mask,\n positions=positions, as_subplot=True,\n units=units, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize, position_pointsize=position_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n if tracer.has_mass_profile:\n\n plt.subplot(rows, columns, 2)\n\n plot_convergence(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 3)\n\n plot_potential(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 4)\n\n plane_plotters.plot_plane_image(\n plane=tracer.source_plane, as_subplot=True, positions=None, plot_grid=False,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n grid_pointsize=grid_pointsize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n if tracer.has_mass_profile:\n\n plt.subplot(rows, columns, 5)\n\n plot_deflections_y(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plt.subplot(rows, columns, 6)\n\n plot_deflections_x(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_filename='', output_format=output_format)\n\n plotter_util.output_subplot_array(output_path=output_path, output_filename=output_filename,\n output_format=output_format)\n\n plt.close()", "response": "Plots the observed _tracer of an analysis."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef plot_ray_tracing_individual(\n tracer, mask=None, extract_array_from_mask=False, zoom_around_mask=False, positions=None,\n should_plot_image_plane_image=False,\n should_plot_source_plane=False,\n should_plot_convergence=False,\n should_plot_potential=False,\n should_plot_deflections=False,\n units='arcsec',\n output_path=None, output_format='show'):\n \"\"\"Plot the observed _tracer of an analysis, using the *CCD* class object.\n\n The visualization and output type can be fully customized.\n\n Parameters\n -----------\n tracer : autolens.ccd.tracer.CCD\n Class containing the _tracer, noise_map-mappers and PSF that are to be plotted.\n The font size of the figure ylabel.\n output_path : str\n The path where the _tracer is output if the output_type is a file format (e.g. png, fits)\n output_format : str\n How the _tracer is output. File formats (e.g. png, fits) output the _tracer to harddisk. 'show' displays the _tracer \\\n in the python interpreter window.\n \"\"\"\n\n if should_plot_image_plane_image:\n\n plot_image_plane_image(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, positions=positions,\n units=units,\n output_path=output_path, output_format=output_format)\n\n if should_plot_convergence:\n\n plot_convergence(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)\n\n if should_plot_potential:\n\n plot_potential(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)\n\n if should_plot_source_plane:\n\n plane_plotters.plot_plane_image(\n plane=tracer.source_plane, positions=None, plot_grid=False,\n units=units,\n output_path=output_path, output_filename='tracer_source_plane', output_format=output_format)\n\n if should_plot_deflections:\n\n plot_deflections_y(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)\n\n if should_plot_deflections:\n\n plot_deflections_x(\n tracer=tracer, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)", "response": "Plots the observed _tracer of an analysis."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompute the weighted signal in each pixel from the given image.", "response": "def weighted_pixel_signals_from_images(pixels, signal_scale, regular_to_pix, galaxy_image):\n \"\"\"Compute the (scaled) signal in each pixel, where the signal is the sum of its datas_-pixel fluxes. \\\n These pixel-signals are used to compute the effective regularization weight of each pixel.\n\n The pixel signals are scaled in the following ways:\n\n 1) Divided by the number of datas_-pixels in the pixel, to ensure all pixels have the same \\\n 'relative' signal (i.e. a pixel with 10 regular-pixels doesn't have x2 the signal of one with 5).\n\n 2) Divided by the maximum pixel-signal, so that all signals vary between 0 and 1. This ensures that the \\\n regularizations weights are defined identically for any datas_ units or signal-to-noise_map ratio.\n\n 3) Raised to the power of the hyper-parameter *signal_scale*, so the method can control the relative \\\n contribution regularization in different regions of pixelization.\n\n Parameters\n -----------\n pixels : int\n The total number of pixels in the pixelization the regularization scheme is applied to.\n signal_scale : float\n A factor which controls how rapidly the smoothness of regularization varies from high signal regions to \\\n low signal regions.\n regular_to_pix : ndarray\n A 1D array mapping every pixel on the regular-grid to a pixel on the pixelization.\n galaxy_image : ndarray\n The image of the galaxy which is used to compute the weigghted pixel signals.\n \"\"\"\n\n pixel_signals = np.zeros((pixels,))\n pixel_sizes = np.zeros((pixels,))\n\n for regular_index in range(galaxy_image.shape[0]):\n pixel_signals[regular_to_pix[regular_index]] += galaxy_image[regular_index]\n pixel_sizes[regular_to_pix[regular_index]] += 1\n\n pixel_signals /= pixel_sizes\n pixel_signals /= np.max(pixel_signals)\n\n return pixel_signals ** signal_scale"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef plot_array(array, origin=None, mask=None, extract_array_from_mask=False, zoom_around_mask=False,\n should_plot_border=False, positions=None, centres=None, axis_ratios=None, phis=None, grid=None,\n as_subplot=False,\n units='arcsec', kpc_per_arcsec=None, figsize=(7, 7), aspect='equal',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n title='Array', titlesize=16, xlabelsize=16, ylabelsize=16, xyticksize=16,\n mask_pointsize=10, border_pointsize=2, position_pointsize=30, grid_pointsize=1,\n xticks_manual=None, yticks_manual=None,\n output_path=None, output_format='show', output_filename='array'):\n \"\"\"Plot an array of data as a figure.\n\n Parameters\n -----------\n array : data.array.scaled_array.ScaledArray\n The 2D array of data which is plotted.\n origin : (float, float).\n The origin of the coordinate system of the array, which is plotted as an 'x' on the image if input.\n mask : data.array.mask.Mask\n The mask applied to the array, the edge of which is plotted as a set of points over the plotted array.\n extract_array_from_mask : bool\n The plotter array is extracted using the mask, such that masked values are plotted as zeros. This ensures \\\n bright features outside the mask do not impact the color map of the plot.\n zoom_around_mask : bool\n If True, the 2D region of the array corresponding to the rectangle encompassing all unmasked values is \\\n plotted, thereby zooming into the region of interest.\n should_plot_border : bool\n If a mask is supplied, its borders pixels (e.g. the exterior edge) is plotted if this is *True*.\n positions : [[]]\n Lists of (y,x) coordinates on the image which are plotted as colored dots, to highlight specific pixels.\n grid : data.array.grids.RegularGrid\n A grid of (y,x) coordinates which may be plotted over the plotted array.\n as_subplot : bool\n Whether the array is plotted as part of a subplot, in which case the grid figure is not opened / closed.\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float or None\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n figsize : (int, int)\n The size of the figure in (rows, columns).\n aspect : str\n The aspect ratio of the array, specifically whether it is forced to be square ('equal') or adapts its size to \\\n the figure size ('auto').\n cmap : str\n The colormap the array is plotted using, which may be chosen from the standard matplotlib colormaps.\n norm : str\n The normalization of the colormap used to plot the image, specifically whether it is linear ('linear'), log \\\n ('log') or a symmetric log normalization ('symmetric_log').\n norm_min : float or None\n The minimum array value the colormap map spans (all values below this value are plotted the same color).\n norm_max : float or None\n The maximum array value the colormap map spans (all values above this value are plotted the same color).\n linthresh : float\n For the 'symmetric_log' colormap normalization ,this specifies the range of values within which the colormap \\\n is linear.\n linscale : float\n For the 'symmetric_log' colormap normalization, this allowws the linear range set by linthresh to be stretched \\\n relative to the logarithmic range.\n cb_ticksize : int\n The size of the tick labels on the colorbar.\n cb_fraction : float\n The fraction of the figure that the colorbar takes up, which resizes the colorbar relative to the figure.\n cb_pad : float\n Pads the color bar in the figure, which resizes the colorbar relative to the figure.\n xlabelsize : int\n The fontsize of the x axes label.\n ylabelsize : int\n The fontsize of the y axes label.\n xyticksize : int\n The font size of the x and y ticks on the figure axes.\n mask_pointsize : int\n The size of the points plotted to show the mask.\n border_pointsize : int\n The size of the points plotted to show the borders.\n positions_pointsize : int\n The size of the points plotted to show the input positions.\n grid_pointsize : int\n The size of the points plotted to show the grid.\n xticks_manual : [] or None\n If input, the xticks do not use the array's default xticks but instead overwrite them as these values.\n yticks_manual : [] or None\n If input, the yticks do not use the array's default yticks but instead overwrite them as these values.\n output_path : str\n The path on the hard-disk where the figure is output.\n output_filename : str\n The filename of the figure that is output.\n output_format : str\n The format the figue is output:\n 'show' - display on computer screen.\n 'png' - output to hard-disk as a png.\n 'fits' - output to hard-disk as a fits file.'\n\n Returns\n --------\n None\n\n Examples\n --------\n array_plotters.plot_array(\n array=image, origin=(0.0, 0.0), mask=circular_mask, extract_array_from_mask=True, zoom_around_mask=True,\n should_plot_border=False, positions=[[1.0, 1.0], [2.0, 2.0]], grid=None, as_subplot=False,\n units='arcsec', kpc_per_arcsec=None, figsize=(7,7), aspect='auto',\n cmap='jet', norm='linear, norm_min=None, norm_max=None, linthresh=None, linscale=None,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n title='Image', titlesize=16, xlabelsize=16, ylabelsize=16, xyticksize=16,\n mask_pointsize=10, border_pointsize=2, position_pointsize=10, grid_pointsize=10,\n xticks_manual=None, yticks_manual=None,\n output_path='/path/to/output', output_format='png', output_filename='image')\n \"\"\"\n\n if array is None:\n return\n\n if extract_array_from_mask and mask is not None:\n array = np.add(array, 0.0, out=np.zeros_like(array), where=np.asarray(mask) == 0)\n\n if zoom_around_mask and mask is not None:\n array = array.zoomed_scaled_array_around_mask(mask=mask, buffer=2)\n zoom_offset_pixels = np.asarray(mask.zoom_offset_pixels)\n zoom_offset_arcsec = np.asarray(mask.zoom_offset_arcsec)\n else:\n zoom_offset_pixels = None\n zoom_offset_arcsec = None\n\n if aspect is 'square':\n aspect = float(array.shape_arcsec[1]) / float(array.shape_arcsec[0])\n\n fig = plot_figure(array=array, as_subplot=as_subplot, units=units, kpc_per_arcsec=kpc_per_arcsec,\n figsize=figsize, aspect=aspect, cmap=cmap, norm=norm,\n norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n xticks_manual=xticks_manual, yticks_manual=yticks_manual)\n\n plotter_util.set_title(title=title, titlesize=titlesize)\n set_xy_labels_and_ticksize(units=units, kpc_per_arcsec=kpc_per_arcsec, xlabelsize=xlabelsize, ylabelsize=ylabelsize,\n xyticksize=xyticksize)\n\n set_colorbar(cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad,\n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels)\n plot_origin(array=array, origin=origin, units=units, kpc_per_arcsec=kpc_per_arcsec,\n zoom_offset_arcsec=zoom_offset_arcsec)\n plot_mask(mask=mask, units=units, kpc_per_arcsec=kpc_per_arcsec, pointsize=mask_pointsize,\n zoom_offset_pixels=zoom_offset_pixels)\n plot_border(mask=mask, should_plot_border=should_plot_border, units=units, kpc_per_arcsec=kpc_per_arcsec,\n pointsize=border_pointsize, zoom_offset_pixels=zoom_offset_pixels)\n plot_points(points_arcsec=positions, array=array, units=units, kpc_per_arcsec=kpc_per_arcsec,\n pointsize=position_pointsize, zoom_offset_arcsec=zoom_offset_arcsec)\n plot_grid(grid_arcsec=grid, array=array, units=units, kpc_per_arcsec=kpc_per_arcsec, pointsize=grid_pointsize,\n zoom_offset_arcsec=zoom_offset_arcsec)\n plot_centres(array=array, centres=centres, units=units, kpc_per_arcsec=kpc_per_arcsec,\n zoom_offset_arcsec=zoom_offset_arcsec)\n plot_ellipses(fig=fig, array=array, centres=centres, axis_ratios=axis_ratios, phis=phis, units=units,\n kpc_per_arcsec=kpc_per_arcsec, zoom_offset_arcsec=zoom_offset_arcsec)\n plotter_util.output_figure(array, as_subplot=as_subplot, output_path=output_path, output_filename=output_filename,\n output_format=output_format)\n plotter_util.close_figure(as_subplot=as_subplot)", "response": "Plot an array of data."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef plot_figure(array, as_subplot, units, kpc_per_arcsec, figsize, aspect, cmap, norm, norm_min, norm_max,\n linthresh, linscale, xticks_manual, yticks_manual):\n \"\"\"Open a matplotlib figure and plot the array of data on it.\n\n Parameters\n -----------\n array : data.array.scaled_array.ScaledArray\n The 2D array of data which is plotted.\n as_subplot : bool\n Whether the array is plotted as part of a subplot, in which case the grid figure is not opened / closed.\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float or None\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n figsize : (int, int)\n The size of the figure in (rows, columns).\n aspect : str\n The aspect ratio of the array, specifically whether it is forced to be square ('equal') or adapts its size to \\\n the figure size ('auto').\n cmap : str\n The colormap the array is plotted using, which may be chosen from the standard matplotlib colormaps.\n norm : str\n The normalization of the colormap used to plot the image, specifically whether it is linear ('linear'), log \\\n ('log') or a symmetric log normalization ('symmetric_log').\n norm_min : float or None\n The minimum array value the colormap map spans (all values below this value are plotted the same color).\n norm_max : float or None\n The maximum array value the colormap map spans (all values above this value are plotted the same color).\n linthresh : float\n For the 'symmetric_log' colormap normalization ,this specifies the range of values within which the colormap \\\n is linear.\n linscale : float\n For the 'symmetric_log' colormap normalization, this allowws the linear range set by linthresh to be stretched \\\n relative to the logarithmic range.\n xticks_manual : [] or None\n If input, the xticks do not use the array's default xticks but instead overwrite them as these values.\n yticks_manual : [] or None\n If input, the yticks do not use the array's default yticks but instead overwrite them as these values.\n \"\"\"\n\n fig = plotter_util.setup_figure(figsize=figsize, as_subplot=as_subplot)\n\n norm_min, norm_max = get_normalization_min_max(array=array, norm_min=norm_min, norm_max=norm_max)\n norm_scale = get_normalization_scale(norm=norm, norm_min=norm_min, norm_max=norm_max,\n linthresh=linthresh, linscale=linscale)\n\n extent = get_extent(array=array, units=units, kpc_per_arcsec=kpc_per_arcsec,\n xticks_manual=xticks_manual, yticks_manual=yticks_manual)\n\n plt.imshow(array, aspect=aspect, cmap=cmap, norm=norm_scale, extent=extent)\n return fig", "response": "Open a matplotlib figure and plot the array of data on it."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the extent of the dimensions of the array in the units of the figure.", "response": "def get_extent(array, units, kpc_per_arcsec, xticks_manual, yticks_manual):\n \"\"\"Get the extent of the dimensions of the array in the units of the figure (e.g. arc-seconds or kpc).\n\n This is used to set the extent of the array and thus the y / x axis limits.\n\n Parameters\n -----------\n array : data.array.scaled_array.ScaledArray\n The 2D array of data which is plotted.\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n xticks_manual : [] or None\n If input, the xticks do not use the array's default xticks but instead overwrite them as these values.\n yticks_manual : [] or None\n If input, the yticks do not use the array's default yticks but instead overwrite them as these values.\n \"\"\"\n if xticks_manual is not None and yticks_manual is not None:\n return np.asarray([xticks_manual[0], xticks_manual[3], yticks_manual[0], yticks_manual[3]])\n\n if units in 'pixels':\n return np.asarray([0, array.shape[1], 0, array.shape[0]])\n elif units in 'arcsec' or kpc_per_arcsec is None:\n return np.asarray([array.arc_second_minima[1], array.arc_second_maxima[1],\n array.arc_second_minima[0], array.arc_second_maxima[0]])\n elif units in 'kpc':\n return list(map(lambda tick : tick*kpc_per_arcsec,\n np.asarray([array.arc_second_minima[1], array.arc_second_maxima[1],\n array.arc_second_minima[0], array.arc_second_maxima[0]])))\n else:\n raise exc.PlottingException('The units supplied to the plotted are not a valid string (must be pixels | '\n 'arcsec | kpc)')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef get_normalization_min_max(array, norm_min, norm_max):\n if norm_min is None:\n norm_min = array.min()\n if norm_max is None:\n norm_max = array.max()\n\n return norm_min, norm_max", "response": "Get the minimum and maximum of the normalization of the array which sets the lower and upper limits of the colormap map."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_normalization_scale(norm, norm_min, norm_max, linthresh, linscale):\n if norm is 'linear':\n return colors.Normalize(vmin=norm_min, vmax=norm_max)\n elif norm is 'log':\n if norm_min == 0.0:\n norm_min = 1.e-4\n return colors.LogNorm(vmin=norm_min, vmax=norm_max)\n elif norm is 'symmetric_log':\n return colors.SymLogNorm(linthresh=linthresh, linscale=linscale, vmin=norm_min, vmax=norm_max)\n else:\n raise exc.PlottingException('The normalization (norm) supplied to the plotter is not a valid string (must be '\n 'linear | log | symmetric_log')", "response": "Returns the normalization scale of the colormap."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convert_grid_units(array, grid_arcsec, units, kpc_per_arcsec):\n if units in 'pixels':\n return array.grid_arcsec_to_grid_pixels(grid_arcsec=grid_arcsec)\n elif units in 'arcsec' or kpc_per_arcsec is None:\n return grid_arcsec\n elif units in 'kpc':\n return grid_arcsec * kpc_per_arcsec\n else:\n raise exc.PlottingException('The units supplied to the plotter are not a valid string (must be pixels | '\n 'arcsec | kpc)')", "response": "Convert the grid from its input units ( arc - seconds or convert to the input units of pixels or kilo parsecs."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef plot_origin(array, origin, units, kpc_per_arcsec, zoom_offset_arcsec):\n if origin is not None:\n\n origin_grid = np.asarray(origin)\n\n if zoom_offset_arcsec is not None:\n origin_grid -= zoom_offset_arcsec\n\n origin_units = convert_grid_units(array=array, grid_arcsec=origin_grid, units=units,\n kpc_per_arcsec=kpc_per_arcsec)\n plt.scatter(y=origin_units[0], x=origin_units[1], s=80, c='k', marker='x')", "response": "Plots the y x coordinates of the array in a series of kilometers."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nplotting the centres of the array on the array as an x.", "response": "def plot_centres(array, centres, units, kpc_per_arcsec, zoom_offset_arcsec):\n \"\"\"Plot the (y,x) centres (e.g. of a mass profile) on the array as an 'x'.\n\n Parameters\n -----------\n array : data.array.scaled_array.ScaledArray\n The 2D array of data which is plotted.\n centres : [[tuple]]\n The list of centres; centres in the same list entry are colored the same.\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float or None\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n \"\"\"\n if centres is not None:\n\n colors = itertools.cycle([\"m\", \"y\", \"r\", \"w\", \"c\", \"b\", \"g\", \"k\"])\n\n for centres_of_galaxy in centres:\n color = next(colors)\n for centre in centres_of_galaxy:\n\n if zoom_offset_arcsec is not None:\n centre -= zoom_offset_arcsec\n\n centre_units = convert_grid_units(array=array, grid_arcsec=centre, units=units,\n kpc_per_arcsec=kpc_per_arcsec)\n plt.scatter(y=centre_units[0], x=centre_units[1], s=300, c=color, marker='x')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef plot_ellipses(fig, array, centres, axis_ratios, phis, units, kpc_per_arcsec, zoom_offset_arcsec):\n if centres is not None and axis_ratios is not None and phis is not None:\n\n colors = itertools.cycle([\"m\", \"y\", \"r\", \"w\", \"c\", \"b\", \"g\", \"k\"])\n\n for set_index in range(len(centres)):\n color = next(colors)\n for geometry_index in range(len(centres[set_index])):\n\n centre = centres[set_index][geometry_index]\n axis_ratio = axis_ratios[set_index][geometry_index]\n phi = phis[set_index][geometry_index]\n\n if zoom_offset_arcsec is not None:\n centre -= zoom_offset_arcsec\n\n centre_units = convert_grid_units(array=array, grid_arcsec=centre, units=units,\n kpc_per_arcsec=kpc_per_arcsec)\n\n y = 1.0\n x = 1.0*axis_ratio\n\n t = np.linspace(0, 2*np.pi, 100)\n plt.plot(centre_units[0] + y*np.cos(t), centre_units[1] + x*np.sin(t), color=color)", "response": "Plots the centres of the array as an x."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef plot_mask(mask, units, kpc_per_arcsec, pointsize, zoom_offset_pixels):\n\n if mask is not None:\n\n plt.gca()\n edge_pixels = mask.masked_grid_index_to_pixel[mask.edge_pixels] + 0.5\n if zoom_offset_pixels is not None:\n edge_pixels -= zoom_offset_pixels\n edge_arcsec = mask.grid_pixels_to_grid_arcsec(grid_pixels=edge_pixels)\n edge_units = convert_grid_units(array=mask, grid_arcsec=edge_arcsec, units=units,\n kpc_per_arcsec=kpc_per_arcsec)\n\n plt.scatter(y=edge_units[:,0], x=edge_units[:,1], s=pointsize, c='k')", "response": "Plots the mask of the array on the figure."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nplots the borders of the mask or array on the figure.", "response": "def plot_border(mask, should_plot_border, units, kpc_per_arcsec, pointsize, zoom_offset_pixels):\n \"\"\"Plot the borders of the mask or the array on the figure.\n\n Parameters\n -----------t.\n mask : ndarray of data.array.mask.Mask\n The mask applied to the array, the edge of which is plotted as a set of points over the plotted array.\n should_plot_border : bool\n If a mask is supplied, its borders pixels (e.g. the exterior edge) is plotted if this is *True*.\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float or None\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n border_pointsize : int\n The size of the points plotted to show the borders.\n \"\"\"\n if should_plot_border and mask is not None:\n\n plt.gca()\n border_pixels = mask.masked_grid_index_to_pixel[mask.border_pixels]\n\n if zoom_offset_pixels is not None:\n border_pixels -= zoom_offset_pixels\n\n border_arcsec = mask.grid_pixels_to_grid_arcsec(grid_pixels=border_pixels)\n border_units = convert_grid_units(array=mask, grid_arcsec=border_arcsec, units=units,\n kpc_per_arcsec=kpc_per_arcsec)\n\n plt.scatter(y=border_units[:,0], x=border_units[:,1], s=pointsize, c='y')"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef plot_points(points_arcsec, array, units, kpc_per_arcsec, pointsize, zoom_offset_arcsec):\n if points_arcsec is not None:\n points_arcsec = list(map(lambda position_set: np.asarray(position_set), points_arcsec))\n point_colors = itertools.cycle([\"m\", \"y\", \"r\", \"w\", \"c\", \"b\", \"g\", \"k\"])\n for point_set_arcsec in points_arcsec:\n\n if zoom_offset_arcsec is not None:\n point_set_arcsec -= zoom_offset_arcsec\n\n point_set_units = convert_grid_units(array=array, grid_arcsec=point_set_arcsec, units=units,\n kpc_per_arcsec=kpc_per_arcsec)\n plt.scatter(y=point_set_units[:,0], x=point_set_units[:,1], color=next(point_colors), s=pointsize)", "response": "Plots a set of points over the array of data."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nplot a grid of points over the array of data.", "response": "def plot_grid(grid_arcsec, array, units, kpc_per_arcsec, pointsize, zoom_offset_arcsec):\n \"\"\"Plot a grid of points over the array of data on the figure.\n\n Parameters\n -----------.\n grid_arcsec : ndarray or data.array.grids.RegularGrid\n A grid of (y,x) coordinates in arc-seconds which may be plotted over the array.\n array : data.array.scaled_array.ScaledArray\n The 2D array of data which is plotted.\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float or None\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n grid_pointsize : int\n The size of the points plotted to show the grid.\n \"\"\"\n if grid_arcsec is not None:\n\n if zoom_offset_arcsec is not None:\n grid_arcsec -= zoom_offset_arcsec\n\n grid_units = convert_grid_units(grid_arcsec=grid_arcsec, array=array, units=units,\n kpc_per_arcsec=kpc_per_arcsec)\n\n plt.scatter(y=np.asarray(grid_units[:, 0]), x=np.asarray(grid_units[:, 1]), s=pointsize, c='k')"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef mapping_matrix(self):\n return mapper_util.mapping_matrix_from_sub_to_pix(sub_to_pix=self.sub_to_pix, pixels=self.pixels,\n regular_pixels=self.grid_stack.regular.shape[0],\n sub_to_regular=self.grid_stack.sub.sub_to_regular,\n sub_grid_fraction=self.grid_stack.sub.sub_grid_fraction)", "response": "The mapping matrix for the unmasked pixels of a pixelization."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the mappings between a pixelization s pixels and the unmasked regular - grid pixels.", "response": "def pix_to_regular(self):\n \"\"\"Compute the mappings between a pixelization's pixels and the unmasked regular-grid pixels. These mappings \\\n are determined after the regular-grid is used to determine the pixelization.\n\n The pixelization's pixels map to different number of regular-grid pixels, thus a list of lists is used to \\\n represent these mappings\"\"\"\n pix_to_regular = [[] for _ in range(self.pixels)]\n\n for regular_pixel, pix_pixel in enumerate(self.regular_to_pix):\n\n pix_to_regular[pix_pixel].append(regular_pixel)\n\n return pix_to_regular"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the mappings between a pixelization s pixels and the unmasked sub - grid pixels.", "response": "def pix_to_sub(self):\n \"\"\"Compute the mappings between a pixelization's pixels and the unmasked sub-grid pixels. These mappings \\\n are determined after the regular-grid is used to determine the pixelization.\n\n The pixelization's pixels map to different number of sub-grid pixels, thus a list of lists is used to \\\n represent these mappings\"\"\"\n pix_to_sub = [[] for _ in range(self.pixels)]\n\n for regular_pixel, pix_pixel in enumerate(self.sub_to_pix):\n pix_to_sub[pix_pixel].append(regular_pixel)\n\n return pix_to_sub"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngives a solution vector determine the reconstructed \\ object.", "response": "def reconstructed_pixelization_from_solution_vector(self, solution_vector):\n \"\"\"Given the solution vector of an inversion (see *inversions.Inversion*), determine the reconstructed \\\n pixelization of the rectangular pixelization by using the mapper.\"\"\"\n recon = mapping_util.map_unmasked_1d_array_to_2d_array_from_array_1d_and_shape(array_1d=solution_vector,\n shape=self.shape)\n return scaled_array.ScaledRectangularPixelArray(array=recon, pixel_scales=self.geometry.pixel_scales,\n origin=self.geometry.origin)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef regular_to_pix(self):\n return mapper_util.voronoi_regular_to_pix_from_grids_and_geometry(regular_grid=self.grid_stack.regular,\n regular_to_nearest_pix=self.grid_stack.pix.regular_to_nearest_pix,\n pixel_centres=self.geometry.pixel_centres, pixel_neighbors=self.geometry.pixel_neighbors,\n pixel_neighbors_size=self.geometry.pixel_neighbors_size).astype('int')", "response": "The 1D index mappings between the regular pixels and Voronoi pixelization pixels."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sub_to_pix(self):\n return mapper_util.voronoi_sub_to_pix_from_grids_and_geometry(sub_grid=self.grid_stack.sub,\n regular_to_nearest_pix=self.grid_stack.pix.regular_to_nearest_pix,\n sub_to_regular=self.grid_stack.sub.sub_to_regular, pixel_centres=self.geometry.pixel_centres,\n pixel_neighbors=self.geometry.pixel_neighbors,\n pixel_neighbors_size=self.geometry.pixel_neighbors_size).astype('int')", "response": "The 1D index mappings between the sub pixels and Voronoi pixelization pixels."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ndetermine the centres of an array from its shape pixel - scales and origin.", "response": "def centres_from_shape_pixel_scales_and_origin(shape, pixel_scales, origin):\n \"\"\"Determine the (y,x) arc-second central coordinates of an array from its shape, pixel-scales and origin.\n\n The coordinate system is defined such that the positive y axis is up and positive x axis is right.\n\n Parameters\n ----------\n shape : (int, int)\n The (y,x) shape of the 2D array the arc-second centre is computed for.\n pixel_scales : (float, float)\n The (y,x) arc-second to pixel scales of the 2D array.\n origin : (float, flloat)\n The (y,x) origin of the 2D array, which the centre is shifted to.\n\n Returns\n --------\n tuple (float, float)\n The (y,x) arc-second central coordinates of the input array.\n\n Examples\n --------\n centres_arcsec = centres_from_shape_pixel_scales_and_origin(shape=(5,5), pixel_scales=(0.5, 0.5), origin=(0.0, 0.0))\n \"\"\"\n\n y_centre_arcsec = float(shape[0] - 1) / 2 + (origin[0] / pixel_scales[0])\n x_centre_arcsec = float(shape[1] - 1) / 2 - (origin[1] / pixel_scales[1])\n\n return (y_centre_arcsec, x_centre_arcsec)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a 2D array of shape with the given pixel scales and origin.", "response": "def regular_grid_2d_from_shape_pixel_scales_and_origin(shape, pixel_scales, origin=(0.0, 0.0)):\n \"\"\"Compute the (y,x) arc second coordinates at the centre of every pixel of an array of shape (rows, columns).\n\n Coordinates are defined from the top-left corner, such that the first pixel at location [0, 0] has negative x \\\n and y values in arc seconds.\n\n The regular grid is returned on an array of shape (total_pixels, total_pixels, 2) where coordinate indexes match \\\n those of the original 2D array. y coordinates are stored in the 0 index of the third dimension, x coordinates in \\\n the 1 index.\n\n Parameters\n ----------\n shape : (int, int)\n The (y,x) shape of the 2D array the regular grid of coordinates is computed for.\n pixel_scales : (float, float)\n The (y,x) arc-second to pixel scales of the 2D array.\n origin : (float, flloat)\n The (y,x) origin of the 2D array, which the regular grid is shifted around.\n\n Returns\n --------\n ndarray\n A regular grid of (y,x) arc-second coordinates at the centre of every pixel on a 2D array. The regular grid \\\n array has dimensions (total_pixels, total_pixels, 2).\n\n Examples\n --------\n regular_grid_1d = regular_grid_2d_from_shape_pixel_scales_and_origin(shape=(5,5), pixel_scales=(0.5, 0.5), \\\n origin=(0.0, 0.0))\n \"\"\"\n\n regular_grid_2d = np.zeros((shape[0], shape[1], 2))\n\n centres_arcsec = centres_from_shape_pixel_scales_and_origin(shape=shape, pixel_scales=pixel_scales, origin=origin)\n\n for y in range(shape[0]):\n for x in range(shape[1]):\n\n regular_grid_2d[y, x, 0] = -(y - centres_arcsec[0]) * pixel_scales[0]\n regular_grid_2d[y, x, 1] = (x - centres_arcsec[1]) * pixel_scales[1]\n\n return regular_grid_2d"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef regular_grid_1d_from_shape_pixel_scales_and_origin(shape, pixel_scales, origin=(0.0, 0.0)):\n\n regular_grid_1d = np.zeros((shape[0]*shape[1], 2))\n\n centres_arcsec = centres_from_shape_pixel_scales_and_origin(shape=shape, pixel_scales=pixel_scales, origin=origin)\n\n i=0\n for y in range(shape[0]):\n for x in range(shape[1]):\n\n regular_grid_1d[i, 0] = -(y - centres_arcsec[0]) * pixel_scales[0]\n regular_grid_1d[i, 1] = (x - centres_arcsec[1]) * pixel_scales[1]\n i += 1\n\n return regular_grid_1d", "response": "Returns a 2D array of shape with the centres of every pixel at the centre of every pixel."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomputing the 1D masked grid from a 2D mask array.", "response": "def regular_grid_1d_masked_from_mask_pixel_scales_and_origin(mask, pixel_scales, origin=(0.0, 0.0)):\n \"\"\"Compute the (y,x) arc second coordinates at the centre of every pixel of a 2D mask array of shape (rows, columns).\n\n Coordinates are defined from the top-left corner, where the first unmasked pixel corresponds to index 0. The pixel \\\n at the top-left of the array has negative x and y values in arc seconds.\n\n The regular grid is returned on an array of shape (total_unmasked_pixels, 2). y coordinates are stored in the 0 \\\n index of the second dimension, x coordinates in the 1 index.\n\n Parameters\n ----------\n mask : ndarray\n A 2D array of bools, where *False* values mean unmasked and are therefore included as part of the calculated \\\n regular grid.\n pixel_scales : (float, float)\n The (y,x) arc-second to pixel scales of the 2D mask array.\n origin : (float, flloat)\n The (y,x) origin of the 2D array, which the regular grid is shifted around.\n\n Returns\n --------\n ndarray\n A regular grid of (y,x) arc-second coordinates at the centre of every pixel unmasked pixel on the 2D mask \\\n array. The regular grid array has dimensions (total_unmasked_pixels, 2).\n\n Examples\n --------\n mask = np.array([[True, False, True],\n [False, False, False]\n [True, False, True]])\n regular_grid_1d = regular_grid_1d_masked_from_mask_pixel_scales_and_origin(mask=mask, pixel_scales=(0.5, 0.5),\n origin=(0.0, 0.0))\n \"\"\"\n\n grid_2d = regular_grid_2d_from_shape_pixel_scales_and_origin(mask.shape, pixel_scales, origin)\n\n total_regular_pixels = mask_util.total_regular_pixels_from_mask(mask)\n regular_grid_1d = np.zeros(shape=(total_regular_pixels, 2))\n pixel_count = 0\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n if not mask[y, x]:\n regular_grid_1d[pixel_count, :] = grid_2d[y, x]\n pixel_count += 1\n\n return regular_grid_1d"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef sub_grid_1d_masked_from_mask_pixel_scales_and_sub_grid_size(mask, pixel_scales, sub_grid_size, origin=(0.0, 0.0)):\n\n total_sub_pixels = mask_util.total_sub_pixels_from_mask_and_sub_grid_size(mask, sub_grid_size)\n\n sub_grid_1d = np.zeros(shape=(total_sub_pixels, 2))\n\n centres_arcsec = centres_from_shape_pixel_scales_and_origin(shape=mask.shape, pixel_scales=pixel_scales,\n origin=origin)\n\n sub_index = 0\n\n y_sub_half = pixel_scales[0] / 2\n y_sub_step = pixel_scales[0] / (sub_grid_size + 1)\n\n x_sub_half = pixel_scales[1] / 2\n x_sub_step = pixel_scales[1] / (sub_grid_size + 1)\n\n for y in range(mask.shape[0]):\n for x in range(mask.shape[1]):\n\n if not mask[y, x]:\n\n y_arcsec = (y - centres_arcsec[0]) * pixel_scales[0]\n x_arcsec = (x - centres_arcsec[1]) * pixel_scales[1]\n\n for y1 in range(sub_grid_size):\n for x1 in range(sub_grid_size):\n\n sub_grid_1d[sub_index, 0] = -(y_arcsec - y_sub_half + (y1 + 1) * y_sub_step)\n sub_grid_1d[sub_index, 1] = x_arcsec - x_sub_half + (x1 + 1) * x_sub_step\n sub_index += 1\n\n return sub_grid_1d", "response": "This routine computes the sub - grid of the given 2D mask array and returns the 1D array of sub - grids."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts a 1D grid of arc - second coordinates to a grid of pixel values.", "response": "def grid_arcsec_1d_to_grid_pixel_centres_1d(grid_arcsec_1d, shape, pixel_scales, origin=(0.0, 0.0)):\n \"\"\" Convert a grid of (y,x) arc second coordinates to a grid of (y,x) pixel values. Pixel coordinates are \\\n returned as integers such that they map directly to the pixel they are contained within.\n\n The pixel coordinate origin is at the top left corner of the grid, such that the pixel [0,0] corresponds to \\\n higher y arc-second coordinate value and lowest x arc-second coordinate.\n\n The arc-second coordinate grid is defined by the class attribute origin, and coordinates are shifted to this \\\n origin before computing their 1D grid pixel indexes.\n\n The input and output grids are both of shape (total_pixels, 2).\n\n Parameters\n ----------\n grid_arcsec_1d: ndarray\n The grid of (y,x) coordinates in arc seconds which is converted to pixel indexes.\n shape : (int, int)\n The (y,x) shape of the original 2D array the arc-second coordinates were computed on.\n pixel_scales : (float, float)\n The (y,x) arc-second to pixel scales of the original 2D array.\n origin : (float, flloat)\n The (y,x) origin of the grid, which the arc-second grid is shifted\n\n Returns\n --------\n ndarray\n A grid of (y,x) pixel indexes with dimensions (total_pixels, 2).\n\n Examples\n --------\n grid_arcsec_1d = np.array([[1.0, 1.0], [2.0, 2.0], [3.0, 3.0], [4.0, 4.0]])\n grid_pixels_1d = grid_arcsec_1d_to_grid_pixel_centres_1d(grid_arcsec_1d=grid_arcsec_1d, shape=(2,2),\n pixel_scales=(0.5, 0.5), origin=(0.0, 0.0))\n \"\"\"\n\n grid_pixels_1d = np.zeros((grid_arcsec_1d.shape[0], 2))\n\n centres_arcsec = centres_from_shape_pixel_scales_and_origin(shape=shape, pixel_scales=pixel_scales, origin=origin)\n\n for i in range(grid_arcsec_1d.shape[0]):\n\n grid_pixels_1d[i, 0] = int((-grid_arcsec_1d[i, 0] / pixel_scales[0]) + centres_arcsec[0] + 0.5)\n grid_pixels_1d[i, 1] = int((grid_arcsec_1d[i, 1] / pixel_scales[1]) + centres_arcsec[1] + 0.5)\n\n return grid_pixels_1d"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef grid_arcsec_1d_to_grid_pixel_indexes_1d(grid_arcsec_1d, shape, pixel_scales, origin=(0.0, 0.0)):\n\n grid_pixels_1d = grid_arcsec_1d_to_grid_pixel_centres_1d(grid_arcsec_1d=grid_arcsec_1d, shape=shape,\n pixel_scales=pixel_scales, origin=origin)\n\n grid_pixel_indexes_1d = np.zeros(grid_pixels_1d.shape[0])\n\n for i in range(grid_pixels_1d.shape[0]):\n\n grid_pixel_indexes_1d[i] = int(grid_pixels_1d[i,0] * shape[1] + grid_pixels_1d[i,1])\n\n return grid_pixel_indexes_1d", "response": "Convert a 2D grid of arc - second coordinates to 1D grid of pixel indexes."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef grid_pixels_1d_to_grid_arcsec_1d(grid_pixels_1d, shape, pixel_scales, origin=(0.0, 0.0)):\n\n grid_arcsec_1d = np.zeros((grid_pixels_1d.shape[0], 2))\n\n centres_arcsec = centres_from_shape_pixel_scales_and_origin(shape=shape, pixel_scales=pixel_scales, origin=origin)\n\n for i in range(grid_arcsec_1d.shape[0]):\n\n grid_arcsec_1d[i, 0] = -(grid_pixels_1d[i, 0] - centres_arcsec[0] - 0.5) * pixel_scales[0]\n grid_arcsec_1d[i, 1] = (grid_pixels_1d[i, 1] - centres_arcsec[1] - 0.5) * pixel_scales[1]\n\n return grid_arcsec_1d", "response": "Convert a 1D grid of pixels to a 2D grid of arc - second values."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef grid_arcsec_2d_to_grid_pixel_centres_2d(grid_arcsec_2d, shape, pixel_scales, origin=(0.0, 0.0)):\n\n grid_pixels_2d = np.zeros((grid_arcsec_2d.shape[0], grid_arcsec_2d.shape[1], 2))\n\n centres_arcsec = centres_from_shape_pixel_scales_and_origin(shape=shape, pixel_scales=pixel_scales, origin=origin)\n\n for y in range(grid_arcsec_2d.shape[0]):\n for x in range(grid_arcsec_2d.shape[1]):\n grid_pixels_2d[y, x, 0] = int((-grid_arcsec_2d[y, x, 0] / pixel_scales[0]) + centres_arcsec[0] + 0.5)\n grid_pixels_2d[y, x, 1] = int((grid_arcsec_2d[y, x, 1] / pixel_scales[1]) + centres_arcsec[1] + 0.5)\n\n return grid_pixels_2d", "response": "Convert a 2D grid of arc - second coordinates to a 2D grid of pixel values."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a two - dimensional Poisson noise_maps - mappers from an image.", "response": "def generate_poisson_noise(image, exposure_time_map, seed=-1):\n \"\"\"\n Generate a two-dimensional poisson noise_maps-mappers from an image.\n\n Values are computed from a Poisson distribution using the image's input values in units of counts.\n\n Parameters\n ----------\n image : ndarray\n The 2D image, whose values in counts are used to draw Poisson noise_maps values.\n exposure_time_map : Union(ndarray, int)\n 2D array of the exposure time in each pixel used to convert to / from counts and electrons per second.\n seed : int\n The seed of the random number generator, used for the random noise_maps maps.\n\n Returns\n -------\n poisson_noise_map: ndarray\n An array describing simulated poisson noise_maps\n \"\"\"\n setup_random_seed(seed)\n image_counts = np.multiply(image, exposure_time_map)\n return image - np.divide(np.random.poisson(image_counts, image.shape), exposure_time_map)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload the ccd - data from a. fits file.", "response": "def load_ccd_data_from_fits(image_path, pixel_scale, image_hdu=0,\n resized_ccd_shape=None, resized_ccd_origin_pixels=None,\n resized_ccd_origin_arcsec=None,\n psf_path=None, psf_hdu=0, resized_psf_shape=None, renormalize_psf=True,\n noise_map_path=None, noise_map_hdu=0,\n noise_map_from_image_and_background_noise_map=False,\n convert_noise_map_from_weight_map=False,\n convert_noise_map_from_inverse_noise_map=False,\n background_noise_map_path=None, background_noise_map_hdu=0,\n convert_background_noise_map_from_weight_map=False,\n convert_background_noise_map_from_inverse_noise_map=False,\n poisson_noise_map_path=None, poisson_noise_map_hdu=0,\n poisson_noise_map_from_image=False,\n convert_poisson_noise_map_from_weight_map=False,\n convert_poisson_noise_map_from_inverse_noise_map=False,\n exposure_time_map_path=None, exposure_time_map_hdu=0,\n exposure_time_map_from_single_value=None,\n exposure_time_map_from_inverse_noise_map=False,\n background_sky_map_path=None, background_sky_map_hdu=0,\n convert_from_electrons=False,\n gain=None, convert_from_adus=False, lens_name=None):\n \"\"\"Factory for loading the ccd data from .fits files, as well as computing properties like the noise-map,\n exposure-time map, etc. from the ccd-data.\n\n This factory also includes a number of routines for converting the ccd-data from units not supported by PyAutoLens \\\n (e.g. adus, electrons) to electrons per second.\n\n Parameters\n ----------\n lens_name\n image_path : str\n The path to the image .fits file containing the image (e.g. '/path/to/image.fits')\n pixel_scale : float\n The size of each pixel in arc seconds.\n image_hdu : int\n The hdu the image is contained in the .fits file specified by *image_path*. \n image_hdu : int\n The hdu the image is contained in the .fits file that *image_path* points too.\n resized_ccd_shape : (int, int) | None\n If input, the ccd arrays that are image sized, e.g. the image, noise-maps) are resized to these dimensions.\n resized_ccd_origin_pixels : (int, int) | None\n If the ccd arrays are resized, this defines a new origin (in pixels) around which recentering occurs.\n resized_ccd_origin_arcsec : (float, float) | None\n If the ccd arrays are resized, this defines a new origin (in arc-seconds) around which recentering occurs.\n psf_path : str\n The path to the psf .fits file containing the psf (e.g. '/path/to/psf.fits') \n psf_hdu : int\n The hdu the psf is contained in the .fits file specified by *psf_path*.\n resized_psf_shape : (int, int) | None\n If input, the psf is resized to these dimensions.\n renormalize_psf : bool\n If True, the PSF is renoralized such that all elements sum to 1.0.\n noise_map_path : str\n The path to the noise_map .fits file containing the noise_map (e.g. '/path/to/noise_map.fits') \n noise_map_hdu : int\n The hdu the noise_map is contained in the .fits file specified by *noise_map_path*.\n noise_map_from_image_and_background_noise_map : bool\n If True, the noise-map is computed from the observed image and background noise-map \\\n (see NoiseMap.from_image_and_background_noise_map).\n convert_noise_map_from_weight_map : bool\n If True, the noise-map loaded from the .fits file is converted from a weight-map to a noise-map (see \\\n *NoiseMap.from_weight_map).\n convert_noise_map_from_inverse_noise_map : bool\n If True, the noise-map loaded from the .fits file is converted from an inverse noise-map to a noise-map (see \\\n *NoiseMap.from_inverse_noise_map).\n background_noise_map_path : str\n The path to the background_noise_map .fits file containing the background noise-map \\ \n (e.g. '/path/to/background_noise_map.fits') \n background_noise_map_hdu : int\n The hdu the background_noise_map is contained in the .fits file specified by *background_noise_map_path*.\n convert_background_noise_map_from_weight_map : bool\n If True, the bacground noise-map loaded from the .fits file is converted from a weight-map to a noise-map (see \\\n *NoiseMap.from_weight_map).\n convert_background_noise_map_from_inverse_noise_map : bool\n If True, the background noise-map loaded from the .fits file is converted from an inverse noise-map to a \\\n noise-map (see *NoiseMap.from_inverse_noise_map).\n poisson_noise_map_path : str\n The path to the poisson_noise_map .fits file containing the Poisson noise-map \\\n (e.g. '/path/to/poisson_noise_map.fits') \n poisson_noise_map_hdu : int\n The hdu the poisson_noise_map is contained in the .fits file specified by *poisson_noise_map_path*.\n poisson_noise_map_from_image : bool\n If True, the Poisson noise-map is estimated using the image.\n convert_poisson_noise_map_from_weight_map : bool\n If True, the Poisson noise-map loaded from the .fits file is converted from a weight-map to a noise-map (see \\\n *NoiseMap.from_weight_map).\n convert_poisson_noise_map_from_inverse_noise_map : bool\n If True, the Poisson noise-map loaded from the .fits file is converted from an inverse noise-map to a \\\n noise-map (see *NoiseMap.from_inverse_noise_map).\n exposure_time_map_path : str\n The path to the exposure_time_map .fits file containing the exposure time map \\ \n (e.g. '/path/to/exposure_time_map.fits') \n exposure_time_map_hdu : int\n The hdu the exposure_time_map is contained in the .fits file specified by *exposure_time_map_path*.\n exposure_time_map_from_single_value : float\n The exposure time of the ccd imaging, which is used to compute the exposure-time map as a single value \\\n (see *ExposureTimeMap.from_single_value*).\n exposure_time_map_from_inverse_noise_map : bool\n If True, the exposure-time map is computed from the background noise_map map \\\n (see *ExposureTimeMap.from_background_noise_map*)\n background_sky_map_path : str\n The path to the background_sky_map .fits file containing the background sky map \\\n (e.g. '/path/to/background_sky_map.fits').\n background_sky_map_hdu : int\n The hdu the background_sky_map is contained in the .fits file specified by *background_sky_map_path*.\n convert_from_electrons : bool\n If True, the input unblurred_image_1d are in units of electrons and all converted to electrons / second using the exposure \\\n time map.\n gain : float\n The image gain, used for convert from ADUs.\n convert_from_adus : bool\n If True, the input unblurred_image_1d are in units of adus and all converted to electrons / second using the exposure \\\n time map and gain.\n \"\"\"\n\n image = load_image(image_path=image_path, image_hdu=image_hdu, pixel_scale=pixel_scale)\n\n background_noise_map = load_background_noise_map(background_noise_map_path=background_noise_map_path,\n background_noise_map_hdu=background_noise_map_hdu,\n pixel_scale=pixel_scale,\n convert_background_noise_map_from_weight_map=convert_background_noise_map_from_weight_map,\n convert_background_noise_map_from_inverse_noise_map=convert_background_noise_map_from_inverse_noise_map)\n\n if background_noise_map is not None:\n inverse_noise_map = 1.0 / background_noise_map\n else:\n inverse_noise_map = None\n\n exposure_time_map = load_exposure_time_map(exposure_time_map_path=exposure_time_map_path,\n exposure_time_map_hdu=exposure_time_map_hdu,\n pixel_scale=pixel_scale, shape=image.shape,\n exposure_time=exposure_time_map_from_single_value,\n exposure_time_map_from_inverse_noise_map=exposure_time_map_from_inverse_noise_map,\n inverse_noise_map=inverse_noise_map)\n\n poisson_noise_map = load_poisson_noise_map(poisson_noise_map_path=poisson_noise_map_path,\n poisson_noise_map_hdu=poisson_noise_map_hdu,\n pixel_scale=pixel_scale,\n convert_poisson_noise_map_from_weight_map=convert_poisson_noise_map_from_weight_map,\n convert_poisson_noise_map_from_inverse_noise_map=convert_poisson_noise_map_from_inverse_noise_map,\n image=image, exposure_time_map=exposure_time_map,\n poisson_noise_map_from_image=poisson_noise_map_from_image,\n convert_from_electrons=convert_from_electrons, gain=gain,\n convert_from_adus=convert_from_adus)\n\n noise_map = load_noise_map(noise_map_path=noise_map_path, noise_map_hdu=noise_map_hdu, pixel_scale=pixel_scale,\n image=image, background_noise_map=background_noise_map,\n exposure_time_map=exposure_time_map,\n convert_noise_map_from_weight_map=convert_noise_map_from_weight_map,\n convert_noise_map_from_inverse_noise_map=convert_noise_map_from_inverse_noise_map,\n noise_map_from_image_and_background_noise_map=noise_map_from_image_and_background_noise_map,\n convert_from_electrons=convert_from_electrons, gain=gain,\n convert_from_adus=convert_from_adus)\n\n psf = load_psf(psf_path=psf_path, psf_hdu=psf_hdu, pixel_scale=pixel_scale, renormalize=renormalize_psf)\n\n background_sky_map = load_background_sky_map(background_sky_map_path=background_sky_map_path,\n background_sky_map_hdu=background_sky_map_hdu,\n pixel_scale=pixel_scale)\n\n image = CCDData(image=image, pixel_scale=pixel_scale, psf=psf, noise_map=noise_map,\n background_noise_map=background_noise_map, poisson_noise_map=poisson_noise_map,\n exposure_time_map=exposure_time_map, background_sky_map=background_sky_map, gain=gain,\n name=lens_name)\n\n if resized_ccd_shape is not None:\n image = image.new_ccd_data_with_resized_arrays(new_shape=resized_ccd_shape,\n new_centre_pixels=resized_ccd_origin_pixels,\n new_centre_arcsec=resized_ccd_origin_arcsec)\n\n if resized_psf_shape is not None:\n image = image.new_ccd_data_with_resized_psf(new_shape=resized_psf_shape)\n\n if convert_from_electrons:\n image = image.new_ccd_data_converted_from_electrons()\n elif convert_from_adus:\n image = image.new_ccd_data_converted_from_adus(gain=gain)\n\n return image"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload a noise - map from a. fits file.", "response": "def load_noise_map(noise_map_path, noise_map_hdu, pixel_scale, image, background_noise_map, exposure_time_map,\n convert_noise_map_from_weight_map, convert_noise_map_from_inverse_noise_map,\n noise_map_from_image_and_background_noise_map, convert_from_electrons, gain, convert_from_adus):\n \"\"\"Factory for loading the noise-map from a .fits file.\n\n This factory also includes a number of routines for converting the noise-map from from other units (e.g. \\\n a weight map) or computing the noise-map from other unblurred_image_1d (e.g. the ccd image and background noise-map).\n\n Parameters\n ----------\n noise_map_path : str\n The path to the noise_map .fits file containing the noise_map (e.g. '/path/to/noise_map.fits')\n noise_map_hdu : int\n The hdu the noise_map is contained in the .fits file specified by *noise_map_path*.\n pixel_scale : float\n The size of each pixel in arc seconds.\n image : ndarray\n The image-image, which the noise-map can be calculated using.\n background_noise_map : ndarray\n The background noise-map, which the noise-map can be calculated using.\n exposure_time_map : ndarray\n The exposure-time map, which the noise-map can be calculated using.\n convert_noise_map_from_weight_map : bool\n If True, the noise-map loaded from the .fits file is converted from a weight-map to a noise-map (see \\\n *NoiseMap.from_weight_map).\n convert_noise_map_from_inverse_noise_map : bool\n If True, the noise-map loaded from the .fits file is converted from an inverse noise-map to a noise-map (see \\\n *NoiseMap.from_inverse_noise_map).\n background_noise_map_path : str\n The path and filename of the .fits image containing the background noise-map.\n background_noise_map_hdu : int\n The hdu the background noise-map is contained in the .fits file that *background_noise_map_path* points too.\n convert_background_noise_map_from_weight_map : bool\n If True, the bacground noise-map loaded from the .fits file is converted from a weight-map to a noise-map (see \\\n *NoiseMap.from_weight_map).\n convert_background_noise_map_from_inverse_noise_map : bool\n If True, the background noise-map loaded from the .fits file is converted from an inverse noise-map to a \\\n noise-map (see *NoiseMap.from_inverse_noise_map).\n noise_map_from_image_and_background_noise_map : bool\n If True, the noise-map is computed from the observed image and background noise-map \\\n (see NoiseMap.from_image_and_background_noise_map).\n convert_from_electrons : bool\n If True, the input unblurred_image_1d are in units of electrons and all converted to electrons / second using the exposure \\\n time map.\n gain : float\n The image gain, used for convert from ADUs.\n convert_from_adus : bool\n If True, the input unblurred_image_1d are in units of adus and all converted to electrons / second using the exposure \\\n time map and gain.\n \"\"\"\n noise_map_options = sum([convert_noise_map_from_weight_map,\n convert_noise_map_from_inverse_noise_map,\n noise_map_from_image_and_background_noise_map])\n\n if noise_map_options > 1:\n raise exc.DataException('You have specified more than one method to load the noise_map map, e.g.:'\n 'convert_noise_map_from_weight_map | '\n 'convert_noise_map_from_inverse_noise_map |'\n 'noise_map_from_image_and_background_noise_map')\n\n if noise_map_options == 0 and noise_map_path is not None:\n return NoiseMap.from_fits_with_pixel_scale(file_path=noise_map_path, hdu=noise_map_hdu, pixel_scale=pixel_scale)\n elif convert_noise_map_from_weight_map and noise_map_path is not None:\n weight_map = Array.from_fits(file_path=noise_map_path, hdu=noise_map_hdu)\n return NoiseMap.from_weight_map(weight_map=weight_map, pixel_scale=pixel_scale)\n elif convert_noise_map_from_inverse_noise_map and noise_map_path is not None:\n inverse_noise_map = Array.from_fits(file_path=noise_map_path, hdu=noise_map_hdu)\n return NoiseMap.from_inverse_noise_map(inverse_noise_map=inverse_noise_map, pixel_scale=pixel_scale)\n elif noise_map_from_image_and_background_noise_map:\n\n if background_noise_map is None:\n raise exc.DataException('Cannot compute the noise-map from the image and background noise_map map if a '\n 'background noise_map map is not supplied.')\n\n if not (convert_from_electrons or convert_from_adus) and exposure_time_map is None:\n raise exc.DataException('Cannot compute the noise-map from the image and background noise_map map if an '\n 'exposure-time (or exposure time map) is not supplied to convert to adus')\n\n if convert_from_adus and gain is None:\n raise exc.DataException('Cannot compute the noise-map from the image and background noise_map map if a'\n 'gain is not supplied to convert from adus')\n\n return NoiseMap.from_image_and_background_noise_map(pixel_scale=pixel_scale, image=image,\n background_noise_map=background_noise_map,\n exposure_time_map=exposure_time_map,\n convert_from_electrons=convert_from_electrons,\n gain=gain, convert_from_adus=convert_from_adus)\n else:\n raise exc.DataException(\n 'A noise_map map was not loaded, specify a noise_map_path or option to compute a noise_map map.')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_background_noise_map(background_noise_map_path, background_noise_map_hdu, pixel_scale,\n convert_background_noise_map_from_weight_map,\n convert_background_noise_map_from_inverse_noise_map):\n \"\"\"Factory for loading the background noise-map from a .fits file.\n\n This factory also includes a number of routines for converting the background noise-map from from other units (e.g. \\\n a weight map).\n\n Parameters\n ----------\n background_noise_map_path : str\n The path to the background_noise_map .fits file containing the background noise-map \\\n (e.g. '/path/to/background_noise_map.fits')\n background_noise_map_hdu : int\n The hdu the background_noise_map is contained in the .fits file specified by *background_noise_map_path*.\n pixel_scale : float\n The size of each pixel in arc seconds.\n convert_background_noise_map_from_weight_map : bool\n If True, the bacground noise-map loaded from the .fits file is converted from a weight-map to a noise-map (see \\\n *NoiseMap.from_weight_map).\n convert_background_noise_map_from_inverse_noise_map : bool\n If True, the background noise-map loaded from the .fits file is converted from an inverse noise-map to a \\\n noise-map (see *NoiseMap.from_inverse_noise_map).\n \"\"\"\n background_noise_map_options = sum([convert_background_noise_map_from_weight_map,\n convert_background_noise_map_from_inverse_noise_map])\n\n if background_noise_map_options == 0 and background_noise_map_path is not None:\n return NoiseMap.from_fits_with_pixel_scale(file_path=background_noise_map_path, hdu=background_noise_map_hdu,\n pixel_scale=pixel_scale)\n elif convert_background_noise_map_from_weight_map and background_noise_map_path is not None:\n weight_map = Array.from_fits(file_path=background_noise_map_path, hdu=background_noise_map_hdu)\n return NoiseMap.from_weight_map(weight_map=weight_map, pixel_scale=pixel_scale)\n elif convert_background_noise_map_from_inverse_noise_map and background_noise_map_path is not None:\n inverse_noise_map = Array.from_fits(file_path=background_noise_map_path, hdu=background_noise_map_hdu)\n return NoiseMap.from_inverse_noise_map(inverse_noise_map=inverse_noise_map, pixel_scale=pixel_scale)\n else:\n return None", "response": "Load the background noise - map from a. fits file."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef load_poisson_noise_map(poisson_noise_map_path, poisson_noise_map_hdu, pixel_scale,\n convert_poisson_noise_map_from_weight_map,\n convert_poisson_noise_map_from_inverse_noise_map,\n poisson_noise_map_from_image,\n image, exposure_time_map, convert_from_electrons, gain, convert_from_adus):\n \"\"\"Factory for loading the Poisson noise-map from a .fits file.\n\n This factory also includes a number of routines for converting the Poisson noise-map from from other units (e.g. \\\n a weight map) or computing the Poisson noise_map from other unblurred_image_1d (e.g. the ccd image).\n\n Parameters\n ----------\n poisson_noise_map_path : str\n The path to the poisson_noise_map .fits file containing the Poisson noise-map \\\n (e.g. '/path/to/poisson_noise_map.fits')\n poisson_noise_map_hdu : int\n The hdu the poisson_noise_map is contained in the .fits file specified by *poisson_noise_map_path*.\n pixel_scale : float\n The size of each pixel in arc seconds.\n convert_poisson_noise_map_from_weight_map : bool\n If True, the Poisson noise-map loaded from the .fits file is converted from a weight-map to a noise-map (see \\\n *NoiseMap.from_weight_map).\n convert_poisson_noise_map_from_inverse_noise_map : bool\n If True, the Poisson noise-map loaded from the .fits file is converted from an inverse noise-map to a \\\n noise-map (see *NoiseMap.from_inverse_noise_map).\n poisson_noise_map_from_image : bool\n If True, the Poisson noise-map is estimated using the image.\n image : ndarray\n The image, which the Poisson noise-map can be calculated using.\n background_noise_map : ndarray\n The background noise-map, which the Poisson noise-map can be calculated using.\n exposure_time_map : ndarray\n The exposure-time map, which the Poisson noise-map can be calculated using.\n convert_from_electrons : bool\n If True, the input unblurred_image_1d are in units of electrons and all converted to electrons / second using the exposure \\\n time map.\n gain : float\n The image gain, used for convert from ADUs.\n convert_from_adus : bool\n If True, the input unblurred_image_1d are in units of adus and all converted to electrons / second using the exposure \\\n time map and gain.\n \"\"\"\n poisson_noise_map_options = sum([convert_poisson_noise_map_from_weight_map,\n convert_poisson_noise_map_from_inverse_noise_map,\n poisson_noise_map_from_image])\n\n if poisson_noise_map_options == 0 and poisson_noise_map_path is not None:\n return PoissonNoiseMap.from_fits_with_pixel_scale(file_path=poisson_noise_map_path, hdu=poisson_noise_map_hdu,\n pixel_scale=pixel_scale)\n elif poisson_noise_map_from_image:\n\n if not (convert_from_electrons or convert_from_adus) and exposure_time_map is None:\n raise exc.DataException('Cannot compute the Poisson noise-map from the image if an '\n 'exposure-time (or exposure time map) is not supplied to convert to adus')\n\n if convert_from_adus and gain is None:\n raise exc.DataException('Cannot compute the Poisson noise-map from the image if a'\n 'gain is not supplied to convert from adus')\n\n return PoissonNoiseMap.from_image_and_exposure_time_map(pixel_scale=pixel_scale, image=image,\n exposure_time_map=exposure_time_map,\n convert_from_electrons=convert_from_electrons,\n gain=gain,\n convert_from_adus=convert_from_adus)\n\n elif convert_poisson_noise_map_from_weight_map and poisson_noise_map_path is not None:\n weight_map = Array.from_fits(file_path=poisson_noise_map_path, hdu=poisson_noise_map_hdu)\n return PoissonNoiseMap.from_weight_map(weight_map=weight_map, pixel_scale=pixel_scale)\n elif convert_poisson_noise_map_from_inverse_noise_map and poisson_noise_map_path is not None:\n inverse_noise_map = Array.from_fits(file_path=poisson_noise_map_path, hdu=poisson_noise_map_hdu)\n return PoissonNoiseMap.from_inverse_noise_map(inverse_noise_map=inverse_noise_map, pixel_scale=pixel_scale)\n else:\n return None", "response": "Load the Poisson noise - map from a. fits file."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_background_sky_map(background_sky_map_path, background_sky_map_hdu, pixel_scale):\n if background_sky_map_path is not None:\n return ScaledSquarePixelArray.from_fits_with_pixel_scale(file_path=background_sky_map_path,\n hdu=background_sky_map_hdu, pixel_scale=pixel_scale)\n else:\n return None", "response": "Factory for loading the background sky map from a. fits file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load_positions(positions_path):\n with open(positions_path) as f:\n position_string = f.readlines()\n\n positions = []\n\n for line in position_string:\n position_list = ast.literal_eval(line)\n positions.append(position_list)\n\n return positions", "response": "Loads the positions of an image from a. dat file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\noutput the positions of an image to a positions. dat file.", "response": "def output_positions(positions, positions_path):\n \"\"\"Output the positions of an image to a positions.dat file.\n\n Positions correspond to a set of pixels in the lensed source galaxy that are anticipated to come from the same \\\n multiply-imaged region of the source-plane. Mass models which do not trace the pixels within a threshold value of \\\n one another are resampled during the non-linear search.\n\n Positions are stored in a .dat file, where each line of the file gives a list of list of (y,x) positions which \\\n correspond to the same region of the source-plane. Thus, multiple source-plane regions can be input over multiple \\\n lines of the same positions file.\n\n Parameters\n ----------\n positions : [[[]]]\n The lists of positions (e.g. [[[1.0, 1.0], [2.0, 2.0]], [[3.0, 3.0], [4.0, 4.0]]])\n positions_path : str\n The path to the positions .dat file containing the positions (e.g. '/path/to/positions.dat')\n \"\"\"\n with open(positions_path, 'w') as f:\n for position in positions:\n f.write(\"%s\\n\" % position)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nsimulate a set of variable arrays.", "response": "def simulate_variable_arrays(cls, array, pixel_scale, exposure_time_map, psf=None, background_sky_map=None,\n add_noise=True, noise_if_add_noise_false=0.1, noise_seed=-1, name=None):\n \"\"\"\n Create a realistic simulated image by applying effects to a plain simulated image.\n\n Parameters\n ----------\n name\n array : ndarray\n The image before simulating (e.g. the lens and source galaxies before optics blurring and CCD read-out).\n pixel_scale: float\n The scale of each pixel in arc seconds\n exposure_time_map : ndarray\n An array representing the effective exposure time of each pixel.\n psf: PSF\n An array describing the PSF the simulated image is blurred with.\n background_sky_map : ndarray\n The value of background sky in every image pixel (electrons per second).\n add_noise: Bool\n If True poisson noise_maps is simulated and added to the image, based on the total counts in each image\n pixel\n noise_seed: int\n A seed for random noise_maps generation\n \"\"\"\n\n if background_sky_map is not None:\n array += background_sky_map\n\n if psf is not None:\n array = psf.convolve(array)\n array = cls.trim_psf_edges(array, psf)\n exposure_time_map = cls.trim_psf_edges(exposure_time_map, psf)\n if background_sky_map is not None:\n background_sky_map = cls.trim_psf_edges(background_sky_map, psf)\n\n if add_noise is True:\n array += generate_poisson_noise(array, exposure_time_map, noise_seed)\n array_counts = np.multiply(array, exposure_time_map)\n noise_map = np.divide(np.sqrt(array_counts), exposure_time_map)\n else:\n noise_map = noise_if_add_noise_false * np.ones(array.shape)\n\n if np.isnan(noise_map).any():\n raise exc.DataException('The noise-map has NaN values in it. This suggests your exposure time and / or'\n 'background sky levels are too low, create signal counts at or close to 0.0.')\n\n if background_sky_map is not None:\n array -= background_sky_map\n\n # ESTIMATE THE BACKGROUND NOISE MAP FROM THE IMAGE\n\n if background_sky_map is not None:\n background_noise_map_counts = np.sqrt(np.multiply(background_sky_map, exposure_time_map))\n background_noise_map = np.divide(background_noise_map_counts, exposure_time_map)\n else:\n background_noise_map = None\n\n # ESTIMATE THE POISSON NOISE MAP FROM THE IMAGE\n\n array_counts = np.multiply(array, exposure_time_map)\n poisson_noise_map = np.divide(np.sqrt(np.abs(array_counts)), exposure_time_map)\n\n array = ScaledSquarePixelArray(array=array, pixel_scale=pixel_scale)\n noise_map = NoiseMap(array=noise_map, pixel_scale=pixel_scale)\n\n if background_noise_map is not None:\n background_noise_map = NoiseMap(array=background_noise_map, pixel_scale=pixel_scale)\n\n if poisson_noise_map is not None:\n poisson_noise_map = PoissonNoiseMap(array=poisson_noise_map, pixel_scale=pixel_scale)\n\n return CCDData(array, pixel_scale=pixel_scale, psf=psf, noise_map=noise_map,\n background_noise_map=background_noise_map, poisson_noise_map=poisson_noise_map,\n exposure_time_map=exposure_time_map, background_sky_map=background_sky_map, name=name)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a realistic simulated image by applying effects to the target signal to noise.", "response": "def simulate_to_target_signal_to_noise(cls, array, pixel_scale, target_signal_to_noise, exposure_time_map,\n psf=None, background_sky_map=None, seed=-1):\n \"\"\"\n Create a realistic simulated image by applying effects to a plain simulated image.\n\n Parameters\n ----------\n target_signal_to_noise\n array : ndarray\n The image before simulating (e.g. the lens and source galaxies before optics blurring and CCD read-out).\n pixel_scale: float\n The scale of each pixel in arc seconds\n exposure_time_map : ndarray\n An array representing the effective exposure time of each pixel.\n psf: PSF\n An array describing the PSF the simulated image is blurred with.\n background_sky_map : ndarray\n The value of background sky in every image pixel (electrons per second).\n seed: int\n A seed for random noise_maps generation\n \"\"\"\n\n max_index = np.unravel_index(array.argmax(), array.shape)\n max_image = array[max_index]\n max_effective_exposure_time = exposure_time_map[max_index]\n max_array_counts = np.multiply(max_image, max_effective_exposure_time)\n if background_sky_map is not None:\n max_background_sky_map = background_sky_map[max_index]\n max_background_sky_map_counts = np.multiply(max_background_sky_map, max_effective_exposure_time)\n else:\n max_background_sky_map_counts = None\n\n scale_factor = 1.\n\n if background_sky_map is None:\n scale_factor = target_signal_to_noise ** 2.0 / max_array_counts\n elif background_sky_map is not None:\n scale_factor = (max_array_counts + max_background_sky_map_counts) * target_signal_to_noise ** 2.0 \\\n / max_array_counts ** 2.0\n\n scaled_effective_exposure_time = np.multiply(scale_factor, exposure_time_map)\n\n return cls.simulate_variable_arrays(array=array, pixel_scale=pixel_scale,\n exposure_time_map=scaled_effective_exposure_time,\n psf=psf, background_sky_map=background_sky_map,\n add_noise=True, noise_seed=seed)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts the array from adus to electrons per second.", "response": "def array_from_adus_to_electrons_per_second(self, array, gain):\n \"\"\"\n For an array (in counts) and an exposure time mappers, convert the array to units electrons per second\n\n Parameters\n ----------\n array : ndarray\n The array the values are to be converted from counts to electrons per second.\n \"\"\"\n if array is not None:\n return np.divide(gain * array, self.exposure_time_map)\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef estimated_noise_map_counts(self):\n return np.sqrt((np.abs(self.image_counts) + np.square(self.background_noise_map_counts)))", "response": "The estimated noise_maps mappers of the image using its background noise_maps mappers and image values\n in counts."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef background_noise_from_edges(self, no_edges):\n\n edges = []\n\n for edge_no in range(no_edges):\n top_edge = self.image[edge_no, edge_no:self.image.shape[1] - edge_no]\n bottom_edge = self.image[self.image.shape[0] - 1 - edge_no, edge_no:self.image.shape[1] - edge_no]\n left_edge = self.image[edge_no + 1:self.image.shape[0] - 1 - edge_no, edge_no]\n right_edge = self.image[edge_no + 1:self.image.shape[0] - 1 - edge_no, self.image.shape[1] - 1 - edge_no]\n\n edges = np.concatenate((edges, top_edge, bottom_edge, right_edge, left_edge))\n\n return norm.fit(edges)[1]", "response": "Estimate the background signal_to_noise_ratio by binning data_to_image located at the edge of an image and fitting a Gaussian profile to this histogram."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsimulates the PSF as an elliptical Gaussian profile.", "response": "def simulate_as_gaussian(cls, shape, pixel_scale, sigma, centre=(0.0, 0.0), axis_ratio=1.0, phi=0.0):\n \"\"\"Simulate the PSF as an elliptical Gaussian profile.\"\"\"\n from autolens.model.profiles.light_profiles import EllipticalGaussian\n gaussian = EllipticalGaussian(centre=centre, axis_ratio=axis_ratio, phi=phi, intensity=1.0, sigma=sigma)\n grid_1d = grid_util.regular_grid_1d_masked_from_mask_pixel_scales_and_origin(mask=np.full(shape, False),\n pixel_scales=(\n pixel_scale, pixel_scale))\n gaussian_1d = gaussian.intensities_from_grid(grid=grid_1d)\n gaussian_2d = mapping_util.map_unmasked_1d_array_to_2d_array_from_array_1d_and_shape(array_1d=gaussian_1d,\n shape=shape)\n return PSF(array=gaussian_2d, pixel_scale=pixel_scale, renormalize=True)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nloads a PSF from a. fits file and renormalizes it.", "response": "def from_fits_renormalized(cls, file_path, hdu, pixel_scale):\n \"\"\"Loads a PSF from fits and renormalizes it\n\n Parameters\n ----------\n pixel_scale\n file_path: String\n The path to the file containing the PSF\n hdu : int\n The HDU the PSF is stored in the .fits file.\n\n Returns\n -------\n psf: PSF\n A renormalized PSF instance\n \"\"\"\n psf = PSF.from_fits_with_scale(file_path, hdu, pixel_scale)\n psf[:, :] = np.divide(psf, np.sum(psf))\n return psf"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_fits_with_scale(cls, file_path, hdu, pixel_scale):\n return cls(array=array_util.numpy_array_2d_from_fits(file_path, hdu), pixel_scale=pixel_scale)", "response": "Loads the PSF from a. fits file."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef new_psf_with_renormalized_array(self):\n return PSF(array=self, pixel_scale=self.pixel_scale, renormalize=True)", "response": "Return a new PSF with the data_vector values sum to unity."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef convolve(self, array):\n if self.shape[0] % 2 == 0 or self.shape[1] % 2 == 0:\n raise exc.KernelException(\"PSF Kernel must be odd\")\n\n return scipy.signal.convolve2d(array, self, mode='same')", "response": "Convolve an array with this PSF"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncalculates the image - plane pixelization from a regular - grid of coordinates and its mask.", "response": "def image_plane_pix_grid_from_regular_grid(self, regular_grid):\n \"\"\"Calculate the image-plane pixelization from a regular-grid of coordinates (and its mask).\n\n See *grid_stacks.SparseToRegularGrid* for details on how this grid is calculated.\n\n Parameters\n -----------\n regular_grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates at the centre of every image value (e.g. image-pixels).\n \"\"\"\n\n pixel_scale = regular_grid.mask.pixel_scale\n\n pixel_scales = ((regular_grid.masked_shape_arcsec[0] + pixel_scale) / (self.shape[0]),\n (regular_grid.masked_shape_arcsec[1] + pixel_scale) / (self.shape[1]))\n\n return grids.SparseToRegularGrid(unmasked_sparse_grid_shape=self.shape, pixel_scales=pixel_scales,\n regular_grid=regular_grid, origin=regular_grid.mask.centre)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef geometry_from_grid(self, grid, buffer=1e-8):\n y_min = np.min(grid[:, 0]) - buffer\n y_max = np.max(grid[:, 0]) + buffer\n x_min = np.min(grid[:, 1]) - buffer\n x_max = np.max(grid[:, 1]) + buffer\n pixel_scales = (float((y_max - y_min) / self.shape[0]), float((x_max - x_min) / self.shape[1]))\n origin = ((y_max + y_min) / 2.0, (x_max + x_min) / 2.0)\n pixel_neighbors, pixel_neighbors_size = self.neighbors_from_pixelization()\n return self.Geometry(shape=self.shape, pixel_scales=pixel_scales, origin=origin,\n pixel_neighbors=pixel_neighbors, pixel_neighbors_size=pixel_neighbors_size)", "response": "Determine the geometry of the rectangular grid."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef mapper_from_grid_stack_and_border(self, grid_stack, border):\n\n if border is not None:\n relocated_grid_stack = border.relocated_grid_stack_from_grid_stack(grid_stack)\n else:\n relocated_grid_stack = grid_stack\n\n geometry = self.geometry_from_grid(grid=relocated_grid_stack.sub)\n\n return mappers.RectangularMapper(pixels=self.pixels, grid_stack=relocated_grid_stack, border=border,\n shape=self.shape, geometry=geometry)", "response": "Setup a rectangular mapper from a grid - stack and a border."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndetermine the geometry of the Voronoi pixelization, by alligning it with the outer-most coordinates on a \\ grid plus a small buffer. Parameters ----------- grid : ndarray The (y,x) grid of coordinates which determine the Voronoi pixelization's geometry. pixel_centres : ndarray The (y,x) centre of every Voronoi pixel in arc-seconds. origin : (float, float) The arc-second origin of the Voronoi pixelization's coordinate system. pixel_neighbors : ndarray An array of length (voronoi_pixels) which provides the index of all neighbors of every pixel in \\ the Voronoi grid (entries of -1 correspond to no neighbor). pixel_neighbors_size : ndarrayy An array of length (voronoi_pixels) which gives the number of neighbors of every pixel in the \\ Voronoi grid.", "response": "def geometry_from_grid(self, grid, pixel_centres, pixel_neighbors, pixel_neighbors_size, buffer=1e-8):\n \"\"\"Determine the geometry of the Voronoi pixelization, by alligning it with the outer-most coordinates on a \\\n grid plus a small buffer.\n\n Parameters\n -----------\n grid : ndarray\n The (y,x) grid of coordinates which determine the Voronoi pixelization's geometry.\n pixel_centres : ndarray\n The (y,x) centre of every Voronoi pixel in arc-seconds.\n origin : (float, float)\n The arc-second origin of the Voronoi pixelization's coordinate system.\n pixel_neighbors : ndarray\n An array of length (voronoi_pixels) which provides the index of all neighbors of every pixel in \\\n the Voronoi grid (entries of -1 correspond to no neighbor).\n pixel_neighbors_size : ndarrayy\n An array of length (voronoi_pixels) which gives the number of neighbors of every pixel in the \\\n Voronoi grid.\n \"\"\"\n y_min = np.min(grid[:, 0]) - buffer\n y_max = np.max(grid[:, 0]) + buffer\n x_min = np.min(grid[:, 1]) - buffer\n x_max = np.max(grid[:, 1]) + buffer\n shape_arcsec = (y_max - y_min, x_max - x_min)\n origin = ((y_max + y_min) / 2.0, (x_max + x_min) / 2.0)\n return self.Geometry(shape_arcsec=shape_arcsec, pixel_centres=pixel_centres, origin=origin,\n pixel_neighbors=pixel_neighbors, pixel_neighbors_size=pixel_neighbors_size)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef voronoi_from_pixel_centers(pixel_centers):\n return scipy.spatial.Voronoi(np.asarray([pixel_centers[:, 1], pixel_centers[:, 0]]).T,\n qhull_options='Qbb Qc Qx Qm')", "response": "Compute the Voronoi grid of the pixelization using the pixel centers."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef neighbors_from_pixelization(self, pixels, ridge_points):\n return pixelization_util.voronoi_neighbors_from_pixels_and_ridge_points(pixels=pixels,\n ridge_points=np.asarray(ridge_points))", "response": "Compute the neighbors of every Voronoi pixel and the ridge points of each Voronoi vertex with."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef mapper_from_grid_stack_and_border(self, grid_stack, border):\n\n if border is not None:\n relocated_grids = border.relocated_grid_stack_from_grid_stack(grid_stack)\n else:\n relocated_grids = grid_stack\n\n pixel_centres = relocated_grids.pix\n pixels = pixel_centres.shape[0]\n\n voronoi = self.voronoi_from_pixel_centers(pixel_centres)\n\n pixel_neighbors, pixel_neighbors_size = self.neighbors_from_pixelization(pixels=pixels,\n ridge_points=voronoi.ridge_points)\n geometry = self.geometry_from_grid(grid=relocated_grids.sub, pixel_centres=pixel_centres,\n pixel_neighbors=pixel_neighbors,\n pixel_neighbors_size=pixel_neighbors_size)\n\n return mappers.VoronoiMapper(pixels=pixels, grid_stack=relocated_grids, border=border,\n voronoi=voronoi, geometry=geometry)", "response": "Setup a Voronoi mapper from a grid - stack and a border."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplots a grid of y and x coordinates.", "response": "def plot_grid(grid, axis_limits=None, points=None, as_subplot=False,\n units='arcsec', kpc_per_arcsec=None,\n figsize=(12, 8), pointsize=5, pointcolor='k', xyticksize=16,\n title='Grid', titlesize=16, xlabelsize=16, ylabelsize=16,\n output_path=None, output_format='show', output_filename='grid'):\n \"\"\"Plot a grid of (y,x) Cartesian coordinates as a scatter plot of points.\n\n Parameters\n -----------\n grid : data.array.grids.RegularGrid\n The (y,x) coordinates of the grid, in an array of shape (total_coordinates, 2).\n axis_limits : []\n The axis limits of the figure on which the grid is plotted, following [xmin, xmax, ymin, ymax].\n points : []\n A set of points that are plotted in a different colour for emphasis (e.g. to show the mappings between \\\n different planes).\n as_subplot : bool\n Whether the grid is plotted as part of a subplot, in which case the grid figure is not opened / closed.\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n figsize : (int, int)\n The size of the figure in (rows, columns).\n pointsize : int\n The size of the points plotted on the grid.\n xyticksize : int\n The font size of the x and y ticks on the figure axes.\n title : str\n The text of the title.\n titlesize : int\n The size of of the title of the figure.\n xlabelsize : int\n The fontsize of the x axes label.\n ylabelsize : int\n The fontsize of the y axes label.\n output_path : str\n The path on the hard-disk where the figure is output.\n output_filename : str\n The filename of the figure that is output.\n output_format : str\n The format the figue is output:\n 'show' - display on computer screen.\n 'png' - output to hard-disk as a png.\n \"\"\"\n\n plotter_util.setup_figure(figsize=figsize, as_subplot=as_subplot)\n grid = convert_grid_units(grid_arcsec=grid, units=units, kpc_per_arcsec=kpc_per_arcsec)\n plt.scatter(y=np.asarray(grid[:, 0]), x=np.asarray(grid[:, 1]), s=pointsize, marker='.')\n plotter_util.set_title(title=title, titlesize=titlesize)\n set_xy_labels(units, kpc_per_arcsec, xlabelsize, ylabelsize, xyticksize)\n\n set_axis_limits(axis_limits)\n plot_points(grid, points, pointcolor)\n\n plt.tick_params(labelsize=xyticksize)\n plotter_util.output_figure(None, as_subplot, output_path, output_filename, output_format)\n plotter_util.close_figure(as_subplot=as_subplot)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting the grid from its input units ( arc - seconds or convert to the input units ( kpc_per_arcsec.", "response": "def convert_grid_units(grid_arcsec, units, kpc_per_arcsec):\n \"\"\"Convert the grid from its input units (arc-seconds) to the input unit (e.g. retain arc-seconds) or convert to \\\n another set of units (kiloparsecs).\n\n Parameters\n -----------\n grid_arcsec : ndarray or data.array.grids.RegularGrid\n The (y,x) coordinates of the grid in arc-seconds, in an array of shape (total_coordinates, 2).\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n \"\"\"\n\n if units in 'arcsec' or kpc_per_arcsec is None:\n return grid_arcsec\n elif units in 'kpc':\n return grid_arcsec * kpc_per_arcsec"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsets the x and y labels of the base class of the base class of the base class of the base class of the base class of the base class of the base class.", "response": "def set_xy_labels(units, kpc_per_arcsec, xlabelsize, ylabelsize, xyticksize):\n \"\"\"Set the x and y labels of the figure, and set the fontsize of those labels.\n\n The x and y labels are always the distance scales, thus the labels are either arc-seconds or kpc and depend on the \\\n units the figure is plotted in.\n\n Parameters\n -----------\n units : str\n The units of the y / x axis of the plots, in arc-seconds ('arcsec') or kiloparsecs ('kpc').\n kpc_per_arcsec : float\n The conversion factor between arc-seconds and kiloparsecs, required to plot the units in kpc.\n xlabelsize : int\n The fontsize of the x axes label.\n ylabelsize : int\n The fontsize of the y axes label.\n xyticksize : int\n The font size of the x and y ticks on the figure axes.\n \"\"\"\n if units in 'arcsec' or kpc_per_arcsec is None:\n\n plt.xlabel('x (arcsec)', fontsize=xlabelsize)\n plt.ylabel('y (arcsec)', fontsize=ylabelsize)\n\n elif units in 'kpc':\n\n plt.xlabel('x (kpc)', fontsize=xlabelsize)\n plt.ylabel('y (kpc)', fontsize=ylabelsize)\n\n else:\n raise exc.PlottingException('The units supplied to the plotted are not a valid string (must be pixels | '\n 'arcsec | kpc)')\n\n plt.tick_params(labelsize=xyticksize)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nplots a subset of points in a different color for emphasis.", "response": "def plot_points(grid, points, pointcolor):\n \"\"\"Plot a subset of points in a different color, to highlight a specifc region of the grid (e.g. how certain \\\n pixels map between different planes).\n\n Parameters\n -----------\n grid : ndarray or data.array.grids.RegularGrid\n The (y,x) coordinates of the grid, in an array of shape (total_coordinates, 2).\n points : []\n A set of points that are plotted in a different colour for emphasis (e.g. to show the mappings between \\\n different planes).\n pointcolor : str or None\n The color the points should be plotted. If None, the points are iterated through a cycle of colors.\n \"\"\"\n if points is not None:\n\n if pointcolor is None:\n\n point_colors = itertools.cycle([\"y\", \"r\", \"k\", \"g\", \"m\"])\n for point_set in points:\n plt.scatter(y=np.asarray(grid[point_set, 0]),\n x=np.asarray(grid[point_set, 1]), s=8, color=next(point_colors))\n\n else:\n\n for point_set in points:\n plt.scatter(y=np.asarray(grid[point_set, 0]),\n x=np.asarray(grid[point_set, 1]), s=8, color=pointcolor)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sub_to_image_grid(func):\n\n @wraps(func)\n def wrapper(grid, galaxies, *args, **kwargs):\n \"\"\"\n\n Parameters\n ----------\n grid : RegularGrid\n The (y,x) coordinates of the grid, in an array of shape (total_coordinates, 2).\n galaxies : [Galaxy]\n The list of galaxies a profile quantity (e.g. intensities) is computed for which is rebinned if it \\\n is a sub-grid.\n\n Returns\n -------\n ndarray\n If a RegularGrid is input, the profile quantity of the galaxy (e.g. intensities) evaluated using this grid.\n\n If a SubGrid is input, the profile quantity of the galaxy (e.g. intensities) evaluated using the sub-grid \\\n and rebinned to the regular-grids array dimensions.\n\n Examples\n ---------\n @grids.sub_to_image_grid\n def intensities_of_galaxies_from_grid(grid, galaxies):\n return sum(map(lambda g: g.intensities_from_grid(grid), galaxies))\n\n galaxy_util.intensities_of_galaxies_from_grid(grid=grid_stack.sub, galaxies=galaxies)\n \"\"\"\n\n result = func(grid, galaxies, *args, *kwargs)\n\n if isinstance(grid, SubGrid):\n return grid.regular_data_1d_from_sub_data_1d(result)\n else:\n return result\n\n return wrapper", "response": "Decorator that converts a function that requires the sub - grid and galaxies and returns the summed sum of each set of sub - gridded values."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef grid_interpolate(func):\n\n @wraps(func)\n def wrapper(profile, grid, grid_radial_minimum=None, *args, **kwargs):\n if hasattr(grid, \"interpolator\"):\n interpolator = grid.interpolator\n if grid.interpolator is not None:\n values = func(profile, interpolator.interp_grid, grid_radial_minimum, *args, **kwargs)\n if values.ndim == 1:\n return interpolator.interpolated_values_from_values(values=values)\n elif values.ndim == 2:\n y_values = interpolator.interpolated_values_from_values(values=values[:, 0])\n x_values = interpolator.interpolated_values_from_values(values=values[:, 1])\n return np.asarray([y_values, x_values]).T\n return func(profile, grid, grid_radial_minimum, *args, **kwargs)\n\n return wrapper", "response": "Decorate a profile method that accepts a coordinate grid and returns a data grid."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef unmasked_blurred_image_from_psf_and_unmasked_image(self, psf, unmasked_image_1d):\n blurred_image_1d = self.regular.convolve_array_1d_with_psf(padded_array_1d=unmasked_image_1d,\n psf=psf)\n\n return self.regular.scaled_array_2d_from_array_1d(array_1d=blurred_image_1d)", "response": "For a padded grid - stack and psf compute an unmasked blurred image."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef grid_stack_for_simulation(cls, shape, pixel_scale, psf_shape, sub_grid_size=2):\n return cls.padded_grid_stack_from_mask_sub_grid_size_and_psf_shape(mask=msk.Mask(array=np.full(shape, False),\n pixel_scale=pixel_scale),\n sub_grid_size=sub_grid_size,\n psf_shape=psf_shape)", "response": "Setup a grid - stack of an image of a strong lens whereby the grid s use the PSF blurring in the simulation routine."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef new_grid_stack_with_pix_grid_added(self, pix_grid, regular_to_nearest_pix):\n pix = PixGrid(arr=pix_grid, regular_to_nearest_pix=regular_to_nearest_pix)\n return GridStack(regular=self.regular, sub=self.sub, blurring=self.blurring, pix=pix)", "response": "Setup a new grid - stack with a grid - stack of pixels added."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\napplies a function to all grid - stacks in the grid - stack.", "response": "def apply_function(self, func):\n \"\"\"Apply a function to all grid_stack in the grid-stack.\n \n This is used by the *ray-tracing* module to easily apply tracing operations to all grid_stack.\"\"\"\n if self.blurring is not None and self.pix is not None:\n return GridStack(func(self.regular), func(self.sub), func(self.blurring), func(self.pix))\n elif self.blurring is None and self.pix is not None:\n return GridStack(func(self.regular), func(self.sub), self.blurring, func(self.pix))\n elif self.blurring is not None and self.pix is None:\n return GridStack(func(self.regular), func(self.sub), func(self.blurring), self.pix)\n else:\n return GridStack(func(self.regular), func(self.sub), self.blurring, self.pix)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmapping a function to all grid_stacks in a grid - stack", "response": "def map_function(self, func, *arg_lists):\n \"\"\"Map a function to all grid_stack in a grid-stack\"\"\"\n return GridStack(*[func(*args) for args in zip(self, *arg_lists)])"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef convolve_array_1d_with_psf(self, padded_array_1d, psf):\n padded_array_2d = mapping_util.map_unmasked_1d_array_to_2d_array_from_array_1d_and_shape(\n array_1d=padded_array_1d, shape=self.mask.shape)\n # noinspection PyUnresolvedReferences\n blurred_padded_array_2d = psf.convolve(array=padded_array_2d)\n return mapping_util.map_2d_array_to_masked_1d_array_from_array_2d_and_mask(array_2d=blurred_padded_array_2d,\n mask=np.full(self.mask.shape, False))", "response": "Convolve a 1d padded array of values with a PSF."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef blurring_grid_from_mask_and_psf_shape(cls, mask, psf_shape):\n blurring_mask = mask.blurring_mask_for_psf_shape(psf_shape)\n return RegularGrid.from_mask(blurring_mask)", "response": "Setup a blurring - grid from a mask and a PSF shape."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef array_2d_from_array_1d(self, array_1d):\n return mapping_util.map_masked_1d_array_to_2d_array_from_array_1d_shape_and_one_to_two(\n array_1d=array_1d, shape=self.mask.shape, one_to_two=self.mask.masked_grid_index_to_pixel)", "response": "Maps a 1D array to its original masked 2D array."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nmaps a 1D array to its original masked 2D array and return a scaled 2D array.", "response": "def scaled_array_2d_from_array_1d(self, array_1d):\n \"\"\" Map a 1D array the same dimension as the grid to its original masked 2D array and return it as a scaled \\\n array.\n\n Parameters\n -----------\n array_1d : ndarray\n The 1D array of which is mapped to a 2D scaled array.\n \"\"\"\n return scaled_array.ScaledSquarePixelArray(array=self.array_2d_from_array_1d(array_1d),\n pixel_scale=self.mask.pixel_scale,\n origin=self.mask.origin)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef yticks(self):\n return np.linspace(np.min(self[:, 0]), np.max(self[:, 0]), 4)", "response": "Compute the y - axis ticks of this grid used for plotting the y - axis ticks when visualizing a regular grid"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute the x - axis ticks labels of this grid used for plotting the x - axis ticks when visualizing a regular grid", "response": "def xticks(self):\n \"\"\"Compute the xticks labels of this grid, used for plotting the x-axis ticks when visualizing a regular\"\"\"\n return np.linspace(np.min(self[:, 1]), np.max(self[:, 1]), 4)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef from_mask_and_sub_grid_size(cls, mask, sub_grid_size=1):\n sub_grid_masked = grid_util.sub_grid_1d_masked_from_mask_pixel_scales_and_sub_grid_size(\n mask=mask,\n pixel_scales=mask.pixel_scales,\n sub_grid_size=sub_grid_size)\n return SubGrid(sub_grid_masked, mask, sub_grid_size)", "response": "Setup a sub - grid of the unmasked pixels using a mask and a sub - grid size."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sub_array_2d_from_sub_array_1d(self, sub_array_1d):\n sub_shape = (self.mask.shape[0] * self.sub_grid_size, self.mask.shape[1] * self.sub_grid_size)\n sub_one_to_two = self.mask.masked_sub_grid_index_to_sub_pixel(sub_grid_size=self.sub_grid_size)\n return mapping_util.map_masked_1d_array_to_2d_array_from_array_1d_shape_and_one_to_two(\n array_1d=sub_array_1d, shape=sub_shape, one_to_two=sub_one_to_two)", "response": "Maps a 1D sub - array to its original 2D sub - array."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef scaled_array_2d_with_sub_dimensions_from_sub_array_1d(self, sub_array_1d):\n return scaled_array.ScaledSquarePixelArray(array=self.sub_array_2d_from_sub_array_1d(sub_array_1d=sub_array_1d),\n pixel_scale=self.mask.pixel_scale / self.sub_grid_size,\n origin=self.mask.origin)", "response": "Returns a 2D sub - array with the same dimensions as the sub - grid and the dimensions of the original 1D sub - array."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nmap a 1D sub - array to its original masked 2D sub - array and return it as a scaled array.", "response": "def scaled_array_2d_with_regular_dimensions_from_binned_up_sub_array_1d(self, sub_array_1d):\n \"\"\" Map a 1D sub-array the same dimension as the sub-grid to its original masked 2D sub-array and return it as\n a scaled array.\n\n Parameters\n -----------\n sub_array_1d : ndarray\n The 1D sub-array of which is mapped to a 2D scaled sub-array the dimensions.\n \"\"\"\n array_1d = self.regular_data_1d_from_sub_data_1d(sub_array_1d=sub_array_1d)\n return scaled_array.ScaledSquarePixelArray(array=self.array_2d_from_array_1d(array_1d=array_1d),\n pixel_scale=self.mask.pixel_scale,\n origin=self.mask.origin)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unmasked_sparse_to_sparse(self):\n\n return mapping_util.unmasked_sparse_to_sparse_from_mask_and_pixel_centres(\n mask=self.regular_grid.mask,\n unmasked_sparse_grid_pixel_centres=self.unmasked_sparse_grid_pixel_centres,\n total_sparse_pixels=self.total_sparse_pixels).astype(\n 'int')", "response": "The 1D index mappings between the unmasked sparse - grid and masked sparse - grid."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sparse_to_unmasked_sparse(self):\n return mapping_util.sparse_to_unmasked_sparse_from_mask_and_pixel_centres(\n total_sparse_pixels=self.total_sparse_pixels, mask=self.regular_grid.mask,\n unmasked_sparse_grid_pixel_centres=self.unmasked_sparse_grid_pixel_centres).astype('int')", "response": "The 1D index mappings between the masked sparse - grid and unmasked sparse - grid."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sparse_grid(self):\n return mapping_util.sparse_grid_from_unmasked_sparse_grid(\n unmasked_sparse_grid=self.unmasked_sparse_grid,\n sparse_to_unmasked_sparse=self.sparse_to_unmasked_sparse)", "response": "The y - second coordinates of the masked sparse - grid."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes a 2D padded blurred image from a 1D padded image and a PSF.", "response": "def padded_blurred_image_2d_from_padded_image_1d_and_psf(self, padded_image_1d, psf):\n \"\"\"Compute a 2D padded blurred image from a 1D padded image.\n\n Parameters\n ----------\n padded_image_1d : ndarray\n A 1D unmasked image which is blurred with the PSF.\n psf : ndarray\n An array describing the PSF kernel of the image.\n \"\"\"\n padded_model_image_1d = self.convolve_array_1d_with_psf(padded_array_1d=padded_image_1d, psf=psf)\n return self.scaled_array_2d_from_array_1d(array_1d=padded_model_image_1d)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef array_2d_from_array_1d(self, padded_array_1d):\n padded_array_2d = self.map_to_2d_keep_padded(padded_array_1d)\n pad_size_0 = self.mask.shape[0] - self.image_shape[0]\n pad_size_1 = self.mask.shape[1] - self.image_shape[1]\n return (padded_array_2d[pad_size_0 // 2:self.mask.shape[0] - pad_size_0 // 2,\n pad_size_1 // 2:self.mask.shape[1] - pad_size_1 // 2])", "response": "Map a padded 1D array of values to its original 2D array trimming all edge values."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef map_to_2d_keep_padded(self, padded_array_1d):\n return mapping_util.map_unmasked_1d_array_to_2d_array_from_array_1d_and_shape(array_1d=padded_array_1d,\n shape=self.mask.shape)", "response": "Map a padded 1D array of values to its padded 2D array."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef padded_grid_from_mask_sub_grid_size_and_psf_shape(cls, mask, sub_grid_size, psf_shape):\n\n padded_shape = (mask.shape[0] + psf_shape[0] - 1, mask.shape[1] + psf_shape[1] - 1)\n\n padded_sub_grid = grid_util.sub_grid_1d_masked_from_mask_pixel_scales_and_sub_grid_size(\n mask=np.full(padded_shape, False), pixel_scales=mask.pixel_scales, sub_grid_size=sub_grid_size)\n\n padded_mask = msk.Mask.unmasked_for_shape_and_pixel_scale(shape=padded_shape, pixel_scale=mask.pixel_scale)\n\n return PaddedSubGrid(arr=padded_sub_grid, mask=padded_mask, image_shape=mask.shape,\n sub_grid_size=sub_grid_size)", "response": "Setup an * PaddedSubGrid* for an input mask sub - grid size and psf - shape."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef relocated_grid_stack_from_grid_stack(self, grid_stack):\n border_grid = grid_stack.regular[self]\n return GridStack(regular=self.relocated_grid_from_grid_jit(grid=grid_stack.regular, border_grid=border_grid),\n sub=self.relocated_grid_from_grid_jit(grid=grid_stack.sub, border_grid=border_grid),\n blurring=None,\n pix=self.relocated_grid_from_grid_jit(grid=grid_stack.pix, border_grid=border_grid))", "response": "Return a grid_stack that contains the pixels relocated based on the borders of the input grid_stack."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nrelocates the coordinates of a grid to its border if they are outside the border.", "response": "def relocated_grid_from_grid_jit(grid, border_grid):\n \"\"\" Relocate the coordinates of a grid to its border if they are outside the border. This is performed as \\\n follows:\n\n 1) Use the mean value of the grid's y and x coordinates to determine the origin of the grid.\n 2) Compute the radial distance of every grid coordinate from the origin.\n 3) For every coordinate, find its nearest pixel in the border.\n 4) Determine if it is outside the border, by comparing its radial distance from the origin to its paid \\\n border pixel's radial distance.\n 5) If its radial distance is larger, use the ratio of radial distances to move the coordinate to the border \\\n (if its inside the border, do nothing).\n \"\"\"\n border_origin = np.zeros(2)\n border_origin[0] = np.mean(border_grid[:, 0])\n border_origin[1] = np.mean(border_grid[:, 1])\n border_grid_radii = np.sqrt(np.add(np.square(np.subtract(border_grid[:, 0], border_origin[0])),\n np.square(np.subtract(border_grid[:, 1], border_origin[1]))))\n border_min_radii = np.min(border_grid_radii)\n\n grid_radii = np.sqrt(np.add(np.square(np.subtract(grid[:, 0], border_origin[0])),\n np.square(np.subtract(grid[:, 1], border_origin[1]))))\n\n for pixel_index in range(grid.shape[0]):\n\n if grid_radii[pixel_index] > border_min_radii:\n\n closest_pixel_index = np.argmin(np.square(grid[pixel_index, 0] - border_grid[:, 0]) +\n np.square(grid[pixel_index, 1] - border_grid[:, 1]))\n\n move_factor = border_grid_radii[closest_pixel_index] / grid_radii[pixel_index]\n if move_factor < 1.0:\n grid[pixel_index, :] = move_factor * (grid[pixel_index, :] - border_origin[:]) + border_origin[:]\n\n return grid"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run(self, positions, pixel_scale, results=None):\n analysis = self.make_analysis(positions=positions, pixel_scale=pixel_scale, results=results)\n result = self.run_analysis(analysis)\n return self.make_result(result, analysis)", "response": "Runs the next phase and returns the result."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncreate an analysis object for this object.", "response": "def make_analysis(self, positions, pixel_scale, results=None):\n \"\"\"\n Create an lens object. Also calls the prior passing and lens_data modifying functions to allow child\n classes to change the behaviour of the phase.\n\n Parameters\n ----------\n pixel_scale\n positions\n results: autofit.tools.pipeline.ResultsCollection\n The result from the previous phase\n\n Returns\n -------\n lens: Analysis\n An lens object that the non-linear optimizer calls to determine the fit of a set of values\n \"\"\"\n self.pass_priors(results)\n analysis = self.__class__.Analysis(positions=positions, pixel_scale=pixel_scale, cosmology=self.cosmology,\n results=results)\n return analysis"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef run(self, data, results=None, mask=None, positions=None):\n analysis = self.make_analysis(data=data, results=results, mask=mask, positions=positions)\n\n result = self.run_analysis(analysis)\n\n return self.make_result(result, analysis)", "response": "Runs the next phase."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates an analysis object for the child phase.", "response": "def make_analysis(self, data, results=None, mask=None, positions=None):\n \"\"\"\n Create an lens object. Also calls the prior passing and lens_data modifying functions to allow child\n classes to change the behaviour of the phase.\n\n Parameters\n ----------\n positions\n mask: Mask\n The default masks passed in by the pipeline\n data: im.CCD\n An lens_data that has been masked\n results: autofit.tools.pipeline.ResultsCollection\n The result from the previous phase\n\n Returns\n -------\n lens : Analysis\n An lens object that the non-linear optimizer calls to determine the fit of a set of values\n \"\"\"\n\n mask = setup_phase_mask(data=data, mask=mask, mask_function=self.mask_function,\n inner_mask_radii=self.inner_mask_radii)\n\n if self.positions_threshold is not None and positions is not None:\n positions = list(map(lambda position_set: np.asarray(position_set), positions))\n elif self.positions_threshold is None:\n positions = None\n elif self.positions_threshold is not None and positions is None:\n raise exc.PhaseException('You have specified for a phase to use positions, but not input positions to the '\n 'pipeline when you ran it.')\n\n lens_data = li.LensData(ccd_data=data, mask=mask, sub_grid_size=self.sub_grid_size,\n image_psf_shape=self.image_psf_shape, positions=positions,\n interp_pixel_scale=self.interp_pixel_scale)\n\n modified_image = self.modify_image(image=lens_data.image, results=results)\n lens_data = lens_data.new_lens_data_with_modified_image(modified_image=modified_image)\n\n if self.bin_up_factor is not None:\n lens_data = lens_data.new_lens_data_with_binned_up_ccd_data_and_mask(bin_up_factor=self.bin_up_factor)\n\n self.pass_priors(results)\n\n self.output_phase_info()\n\n analysis = self.__class__.Analysis(lens_data=lens_data, cosmology=self.cosmology,\n positions_threshold=self.positions_threshold, results=results)\n return analysis"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrunning the next phase and returns the result object.", "response": "def run(self, galaxy_data, results=None, mask=None):\n \"\"\"\n Run this phase.\n\n Parameters\n ----------\n galaxy_data\n mask: Mask\n The default masks passed in by the pipeline\n results: autofit.tools.pipeline.ResultsCollection\n An object describing the results of the last phase or None if no phase has been executed\n\n Returns\n -------\n result: AbstractPhase.Result\n A result object comprising the best fit model and other hyper.\n \"\"\"\n analysis = self.make_analysis(galaxy_data=galaxy_data, results=results, mask=mask)\n result = self.run_analysis(analysis)\n\n return self.make_result(result, analysis)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef run(self, data, results=None, mask=None, positions=None):\n model_image = results.last.unmasked_model_image\n galaxy_tuples = results.last.constant.name_instance_tuples_for_class(g.Galaxy)\n\n results_copy = copy.copy(results.last)\n\n for name, galaxy in galaxy_tuples:\n optimizer = self.optimizer.copy_with_name_extension(name)\n optimizer.variable.hyper_galaxy = g.HyperGalaxy\n galaxy_image = results.last.unmasked_image_for_galaxy(galaxy)\n optimizer.fit(self.__class__.Analysis(data, model_image, galaxy_image))\n\n getattr(results_copy.variable, name).hyper_galaxy = optimizer.variable.hyper_galaxy\n getattr(results_copy.constant, name).hyper_galaxy = optimizer.constant.hyper_galaxy\n\n return results_copy", "response": "Run a fit for each galaxy from the previous phase."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nmapping 1D pixel indexes to 2D pixel indexes.", "response": "def map_1d_indexes_to_2d_indexes_for_shape(indexes_1d, shape):\n \"\"\"For pixels on a 2D array of shape (rows, colums), map an array of 1D pixel indexes to 2D pixel indexes.\n\n Indexing is defined from the top-left corner rightwards and downwards, whereby the top-left pixel on the 2D array\n corresponds to index 0, the pixel to its right pixel 1, and so on.\n\n For a 2D array of shape (3,3), 1D pixel indexes are converted as follows:\n\n - 1D Pixel index 0 maps -> 2D pixel index [0,0].\n - 1D Pixel index 1 maps -> 2D pixel index [0,1].\n - 1D Pixel index 4 maps -> 2D pixel index [1,0].\n - 1D Pixel index 8 maps -> 2D pixel index [2,2].\n\n Parameters\n ----------\n indexes_1d : ndarray\n The 1D pixel indexes which are mapped to 2D indexes.\n shape : (int, int)\n The shape of the 2D array which the pixels are defined on.\n\n Returns\n --------\n ndarray\n An array of 2d pixel indexes with dimensions (total_indexes, 2).\n\n Examples\n --------\n indexes_1d = np.array([0, 1, 2, 5])\n indexes_2d = map_1d_indexes_to_2d_indexes_for_shape(indexes_1d=indexes_1d, shape=(3,3))\n \"\"\"\n indexes_2d = np.zeros((indexes_1d.shape[0], 2))\n\n for i, index_1d in enumerate(indexes_1d):\n indexes_2d[i, 0] = int(index_1d / shape[1])\n indexes_2d[i, 1] = int(index_1d % shape[1])\n\n return indexes_2d"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef map_2d_indexes_to_1d_indexes_for_shape(indexes_2d, shape):\n indexes_1d = np.zeros(indexes_2d.shape[0])\n\n for i in range(indexes_2d.shape[0]):\n indexes_1d[i] = int((indexes_2d[i, 0]) * shape[1] + indexes_2d[i, 1])\n\n return indexes_1d", "response": "Maps a 2D array of indexes to 1D indexes."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef map_masked_1d_array_to_2d_array_from_array_1d_shape_and_one_to_two(array_1d, shape, one_to_two):\n\n array_2d = np.zeros(shape)\n\n for index in range(len(one_to_two)):\n array_2d[one_to_two[index, 0], one_to_two[index, 1]] = array_1d[index]\n\n return array_2d", "response": "This function maps unmasked values from a 1D array to a 2D array where masked values are set to zero."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef map_unmasked_1d_array_to_2d_array_from_array_1d_and_shape(array_1d, shape):\n\n array_2d = np.zeros(shape)\n\n index = 0\n for y in range(shape[0]):\n for x in range(shape[1]):\n array_2d[y, x] = array_1d[index]\n index += 1\n\n return array_2d", "response": "This function maps a 1D array to a 2D array."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sparse_to_unmasked_sparse_from_mask_and_pixel_centres(total_sparse_pixels, mask,\n unmasked_sparse_grid_pixel_centres):\n \"\"\"Determine the mapping between every masked pixelization-grid pixel and pixelization-grid pixel. This is\n performed by checking whether each pixelization-grid pixel is within the regular-masks, and mapping the indexes.\n\n Parameters\n -----------\n total_sparse_pixels : int\n The total number of pixels in the pixelization grid which fall within the regular-masks.\n mask : ccd.masks.Mask\n The regular-masks within which pixelization pixels must be inside\n unmasked_sparse_grid_pixel_centres : ndarray\n The centres of the unmasked pixelization grid pixels.\n \"\"\"\n\n pix_to_full_pix = np.zeros(total_sparse_pixels)\n\n pixel_index = 0\n\n for full_pixel_index in range(unmasked_sparse_grid_pixel_centres.shape[0]):\n\n y = unmasked_sparse_grid_pixel_centres[full_pixel_index, 0]\n x = unmasked_sparse_grid_pixel_centres[full_pixel_index, 1]\n\n if not mask[y, x]:\n pix_to_full_pix[pixel_index] = full_pixel_index\n pixel_index += 1\n\n return pix_to_full_pix", "response": "This function returns a sparse matrix that maps every pixel in the unmasked pixelization - grid pixel to the full - pixels of the unmasked pixelization - grid pixel."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndetermining the mapping between every pixelization-grid pixel and masked pixelization-grid pixel. This is performed by checking whether each pixelization-grid pixel is within the regular-masks, and mapping the indexes. Pixelization pixels are paired with the next masked pixel index. This may mean that a pixel is not paired with a pixel near it, if the next pixel is on the next row of the grid. This is not a problem, as it is only unmasked pixels that are referened when computing image_to_pix, which is what this array is used for. Parameters ----------- total_sparse_pixels : int The total number of pixels in the pixelization grid which fall within the regular-masks. mask : ccd.masks.Mask The regular-masks within which pixelization pixels must be inside unmasked_sparse_grid_pixel_centres : ndarray The centres of the unmasked pixelization grid pixels.", "response": "def unmasked_sparse_to_sparse_from_mask_and_pixel_centres(mask, unmasked_sparse_grid_pixel_centres,\n total_sparse_pixels):\n \"\"\"Determine the mapping between every pixelization-grid pixel and masked pixelization-grid pixel. This is\n performed by checking whether each pixelization-grid pixel is within the regular-masks, and mapping the indexes.\n\n Pixelization pixels are paired with the next masked pixel index. This may mean that a pixel is not paired with a\n pixel near it, if the next pixel is on the next row of the grid. This is not a problem, as it is only\n unmasked pixels that are referened when computing image_to_pix, which is what this array is used for.\n\n Parameters\n -----------\n total_sparse_pixels : int\n The total number of pixels in the pixelization grid which fall within the regular-masks.\n mask : ccd.masks.Mask\n The regular-masks within which pixelization pixels must be inside\n unmasked_sparse_grid_pixel_centres : ndarray\n The centres of the unmasked pixelization grid pixels.\n \"\"\"\n\n total_unmasked_sparse_pixels = unmasked_sparse_grid_pixel_centres.shape[0]\n\n unmasked_sparse_to_sparse = np.zeros(total_unmasked_sparse_pixels)\n pixel_index = 0\n\n for unmasked_sparse_pixel_index in range(total_unmasked_sparse_pixels):\n\n y = unmasked_sparse_grid_pixel_centres[unmasked_sparse_pixel_index, 0]\n x = unmasked_sparse_grid_pixel_centres[unmasked_sparse_pixel_index, 1]\n\n unmasked_sparse_to_sparse[unmasked_sparse_pixel_index] = pixel_index\n\n if not mask[y, x]:\n if pixel_index < total_sparse_pixels - 1:\n pixel_index += 1\n\n return unmasked_sparse_to_sparse"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef regular_to_sparse_from_sparse_mappings(regular_to_unmasked_sparse, unmasked_sparse_to_sparse):\n total_regular_pixels = regular_to_unmasked_sparse.shape[0]\n\n regular_to_sparse = np.zeros(total_regular_pixels)\n\n for regular_index in range(total_regular_pixels):\n # print(regular_index, regular_to_unmasked_sparse[regular_index], unmasked_sparse_to_sparse.shape[0])\n regular_to_sparse[regular_index] = unmasked_sparse_to_sparse[regular_to_unmasked_sparse[regular_index]]\n\n\n return regular_to_sparse", "response": "This function computes the index mapping between the regular and unmasked pixelizations grid and the unmasked sparse grid."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sparse_grid_from_unmasked_sparse_grid(unmasked_sparse_grid, sparse_to_unmasked_sparse):\n total_pix_pixels = sparse_to_unmasked_sparse.shape[0]\n\n pix_grid = np.zeros((total_pix_pixels, 2))\n\n for pixel_index in range(total_pix_pixels):\n pix_grid[pixel_index, 0] = unmasked_sparse_grid[sparse_to_unmasked_sparse[pixel_index], 0]\n pix_grid[pixel_index, 1] = unmasked_sparse_grid[sparse_to_unmasked_sparse[pixel_index], 1]\n\n return pix_grid", "response": "This function returns a sparse grid from the unmasked pixelization grid."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nbin up an array to coarser resolution using the mean value of the new pixel.", "response": "def bin_up_array_2d_using_mean(array_2d, bin_up_factor):\n \"\"\"Bin up an array to coarser resolution, by binning up groups of pixels and using their mean value to determine \\\n the value of the new pixel.\n\n If an array of shape (8,8) is input and the bin up size is 2, this would return a new array of size (4,4) where \\\n every pixel was the mean of each collection of 2x2 pixels on the (8,8) array.\n\n If binning up the array leads to an edge being cut (e.g. a (9,9) array binned up by 2), an array is first \\\n extracted around the centre of that array.\n\n\n Parameters\n ----------\n array_2d : ndarray\n The 2D array that is resized.\n new_shape : (int, int)\n The (y,x) new pixel dimension of the trimmed array.\n origin : (int, int)\n The oigin of the resized array, e.g. the central pixel around which the array is extracted.\n\n Returns\n -------\n ndarray\n The resized 2D array from the input 2D array.\n\n Examples\n --------\n array_2d = np.ones((5,5))\n resize_array = resize_array_2d(array_2d=array_2d, new_shape=(2,2), origin=(2, 2))\n \"\"\"\n\n padded_array_2d = pad_2d_array_for_binning_up_with_bin_up_factor(array_2d=array_2d, bin_up_factor=bin_up_factor)\n\n binned_array_2d = np.zeros(shape=(padded_array_2d.shape[0] // bin_up_factor,\n padded_array_2d.shape[1] // bin_up_factor))\n\n for y in range(binned_array_2d.shape[0]):\n for x in range(binned_array_2d.shape[1]):\n value = 0.0\n for y1 in range(bin_up_factor):\n for x1 in range(bin_up_factor):\n padded_y = y*bin_up_factor + y1\n padded_x = x*bin_up_factor + x1\n value += padded_array_2d[padded_y, padded_x]\n\n binned_array_2d[y,x] = value / (bin_up_factor ** 2.0)\n\n return binned_array_2d"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nwriting a 2D NumPy array to a. fits file.", "response": "def numpy_array_2d_to_fits(array_2d, file_path, overwrite=False):\n \"\"\"Write a 2D NumPy array to a .fits file.\n\n Before outputting a NumPy array, the array is flipped upside-down using np.flipud. This is so that the arrays \\\n appear the same orientation as .fits files loaded in DS9.\n\n Parameters\n ----------\n array_2d : ndarray\n The 2D array that is written to fits.\n file_path : str\n The full path of the file that is output, including the file name and '.fits' extension.\n overwrite : bool\n If True and a file already exists with the input file_path the .fits file is overwritten. If False, an error \\\n will be raised.\n\n Returns\n -------\n None\n\n Examples\n --------\n array_2d = np.ones((5,5))\n numpy_array_to_fits(array=array_2d, file_path='/path/to/file/filename.fits', overwrite=True)\n \"\"\"\n if overwrite and os.path.exists(file_path):\n os.remove(file_path)\n\n new_hdr = fits.Header()\n hdu = fits.PrimaryHDU(np.flipud(array_2d), new_hdr)\n hdu.writeto(file_path)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef numpy_array_2d_from_fits(file_path, hdu):\n hdu_list = fits.open(file_path)\n return np.flipud(np.array(hdu_list[hdu].data))", "response": "Read a 2D NumPy array to a. fits file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nplotting the lens subtracted image of a specific fit.", "response": "def plot_lens_subtracted_image(\n fit, mask=None, extract_array_from_mask=False, zoom_around_mask=False, positions=None, as_subplot=False,\n units='arcsec', kpc_per_arcsec=None, figsize=(7, 7), aspect='square',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n title='Fit Model Image', titlesize=16, xlabelsize=16, ylabelsize=16, xyticksize=16,\n mask_pointsize=10, position_pointsize=10,\n output_path=None, output_format='show', output_filename='fit_lens_subtracted_image'):\n \"\"\"Plot the model image of a specific plane of a lens fit.\n\n Set *autolens.datas.array.plotters.array_plotters* for a description of all input parameters not described below.\n\n Parameters\n -----------\n fit : datas.fitting.fitting.AbstractFitter\n The fit to the datas, which includes a list of every model image, residual_map, chi-squareds, etc.\n image_index : int\n The index of the datas in the datas-set of which the model image is plotted.\n plane_indexes : int\n The plane from which the model image is generated.\n \"\"\"\n\n if fit.tracer.total_planes == 2:\n if fit.tracer.image_plane.has_light_profile:\n lens_subtracted_image = fit.image - fit.model_image_of_planes[0]\n else:\n lens_subtracted_image = fit.image\n else:\n lens_subtracted_image = fit.image - sum(fit.model_image_of_planes[0:-2])\n\n array_plotters.plot_array(\n array=lens_subtracted_image, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, positions=positions, as_subplot=as_subplot,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n title=title, titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize, position_pointsize=position_pointsize,\n output_path=output_path, output_format=output_format, output_filename=output_filename)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef plot_model_image_of_planes(\n fit, plot_foreground=False, plot_source=False, mask=None, extract_array_from_mask=False, zoom_around_mask=False,\n positions=None, plot_mass_profile_centres=True, as_subplot=False,\n units='arcsec', kpc_per_arcsec=None, figsize=(7, 7), aspect='square',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n title='Fit Model Image', titlesize=16, xlabelsize=16, ylabelsize=16, xyticksize=16,\n mask_pointsize=10, position_pointsize=10,\n output_path=None, output_format='show', output_filename='fit_model_image_of_plane'):\n \"\"\"Plot the model image of a specific plane of a lens fit.\n\n Set *autolens.datas.array.plotters.array_plotters* for a description of all input parameters not described below.\n\n Parameters\n -----------\n fit : datas.fitting.fitting.AbstractFitter\n The fit to the datas, which includes a list of every model image, residual_map, chi-squareds, etc.\n plane_indexes : [int]\n The plane from which the model image is generated.\n \"\"\"\n\n centres = get_mass_profile_centes(plot_mass_profile_centres=plot_mass_profile_centres, fit=fit)\n\n if plot_foreground:\n\n if fit.tracer.total_planes == 2:\n model_image = fit.model_image_of_planes[0]\n else:\n model_image = sum(fit.model_image_of_planes[0:-2])\n\n elif plot_source:\n\n model_image = fit.model_image_of_planes[-1]\n\n else:\n\n raise exc.PlottingException('Both plot_foreground and plot_source were False, one must be True')\n\n array_plotters.plot_array(\n array=model_image, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, positions=positions, centres=centres, as_subplot=as_subplot,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n title=title, titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize, position_pointsize=position_pointsize,\n output_path=output_path, output_format=output_format, output_filename=output_filename)", "response": "Plot the model image of a specific plane of a lens fit."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef check_tracer_for_mass_profile(func):\n\n @wraps(func)\n def wrapper(self):\n \"\"\"\n\n Parameters\n ----------\n self\n\n Returns\n -------\n A value or coordinate in the same coordinate system as those passed in.\n \"\"\"\n\n if self.has_mass_profile is True:\n return func(self)\n else:\n return None\n\n return wrapper", "response": "A property decorator that checks if the tracer s galaxies have a mass profile."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convolve_mapping_matrix(self, mapping_matrix):\n return self.convolve_matrix_jit(mapping_matrix, self.image_frame_indexes,\n self.image_frame_psfs, self.image_frame_lengths)", "response": "Given a mapping matrix convolves every entry in the given inversion mapping matrix and returns a new set of entries."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the total mass within a circle in the specified units.", "response": "def mass_within_circle_in_units(self, radius: dim.Length, unit_mass='angular', kpc_per_arcsec=None,\n critical_surface_density=None):\n \"\"\" Integrate the mass profiles's convergence profile to compute the total mass within a circle of \\\n specified radius. This is centred on the mass profile.\n\n The following units for mass can be specified and output:\n\n - Dimensionless angular units (default) - 'angular'.\n - Solar masses - 'angular' (multiplies the angular mass by the critical surface mass density).\n\n Parameters\n ----------\n radius : dim.Length\n The radius of the circle to compute the dimensionless mass within.\n unit_mass : str\n The units the mass is returned in (angular | angular).\n critical_surface_density : float or None\n The critical surface mass density of the strong lens configuration, which converts mass from angulalr \\\n units to phsical units (e.g. solar masses).\n \"\"\"\n\n self.check_units_of_radius_and_critical_surface_density(\n radius=radius, critical_surface_density=critical_surface_density)\n\n profile = self.new_profile_with_units_converted(\n unit_length=radius.unit_length, unit_mass='angular',\n kpc_per_arcsec=kpc_per_arcsec, critical_surface_density=critical_surface_density)\n\n mass_angular = dim.Mass(value=quad(profile.mass_integral, a=0.0, b=radius, args=(1.0,))[0], unit_mass='angular')\n return mass_angular.convert(unit_mass=unit_mass, critical_surface_density=critical_surface_density)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the total angular mass within an ellipse of a given major axis.", "response": "def mass_within_ellipse_in_units(self, major_axis, unit_mass='angular', kpc_per_arcsec=None,\n critical_surface_density=None):\n \"\"\" Integrate the mass profiles's convergence profile to compute the total angular mass within an ellipse of \\\n specified major axis. This is centred on the mass profile.\n\n The following units for mass can be specified and output:\n\n - Dimensionless angular units (default) - 'angular'.\n - Solar masses - 'angular' (multiplies the angular mass by the critical surface mass density)\n\n Parameters\n ----------\n major_axis : float\n The major-axis radius of the ellipse.\n unit_mass : str\n The units the mass is returned in (angular | angular).\n critical_surface_density : float or None\n The critical surface mass density of the strong lens configuration, which converts mass from angular \\\n units to phsical units (e.g. solar masses).\n \"\"\"\n\n self.check_units_of_radius_and_critical_surface_density(\n radius=major_axis, critical_surface_density=critical_surface_density)\n\n profile = self.new_profile_with_units_converted(\n unit_length=major_axis.unit_length, unit_mass='angular',\n kpc_per_arcsec=kpc_per_arcsec, critical_surface_density=critical_surface_density)\n\n mass_angular = dim.Mass(value=quad(profile.mass_integral, a=0.0, b=major_axis, args=(self.axis_ratio,))[0],\n unit_mass='angular')\n return mass_angular.convert(unit_mass=unit_mass, critical_surface_density=critical_surface_density)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nintegrates an elliptical light profile into mass", "response": "def mass_integral(self, x, axis_ratio):\n \"\"\"Routine to integrate an elliptical light profiles - set axis ratio to 1 to compute the luminosity within a \\\n circle\"\"\"\n r = x * axis_ratio\n return 2 * np.pi * r * self.convergence_func(x)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncalculate the density between two circular annuli in angular units.", "response": "def density_between_circular_annuli_in_angular_units(self, inner_annuli_radius, outer_annuli_radius):\n \"\"\"Calculate the mass between two circular annuli and compute the density by dividing by the annuli surface\n area.\n\n The value returned by the mass integral is dimensionless, therefore the density between annuli is returned in \\\n units of inverse radius squared. A conversion factor can be specified to convert this to a physical value \\\n (e.g. the critical surface mass density).\n\n Parameters\n -----------\n inner_annuli_radius : float\n The radius of the inner annulus outside of which the density are estimated.\n outer_annuli_radius : float\n The radius of the outer annulus inside of which the density is estimated.\n \"\"\"\n annuli_area = (np.pi * outer_annuli_radius ** 2.0) - (np.pi * inner_annuli_radius ** 2.0)\n return (self.mass_within_circle_in_units(radius=outer_annuli_radius) -\n self.mass_within_circle_in_units(radius=inner_annuli_radius)) \\\n / annuli_area"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef average_convergence_of_1_radius_in_units(self, unit_length='arcsec', kpc_per_arcsec=None):\n\n def func(radius):\n radius = dim.Length(radius, unit_length=unit_length)\n return self.mass_within_circle_in_units(radius=radius) - np.pi * radius ** 2.0\n\n radius = self.ellipticity_rescale * root_scalar(func, bracket=[1e-4, 1000.0]).root\n radius = dim.Length(radius, unit_length)\n return radius.convert(unit_length=unit_length, kpc_per_arcsec=kpc_per_arcsec)", "response": "The average convergence of 1 radius in units."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef einstein_radius_rescaled(self):\n return ((3 - self.slope) / (1 + self.axis_ratio)) * self.einstein_radius ** (self.slope - 1)", "response": "Rescale the einstein radius by slope and axis_ratio to reduce its degeneracy with other mass - profiles\n parameters"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the projected convergence at a given set of arc - second gridded coordinates.", "response": "def convergence_from_grid(self, grid):\n \"\"\" Calculate the projected convergence at a given set of arc-second gridded coordinates.\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the surface density is computed on.\n \"\"\"\n\n surface_density_grid = np.zeros(grid.shape[0])\n\n grid_eta = self.grid_to_elliptical_radii(grid)\n\n for i in range(grid.shape[0]):\n surface_density_grid[i] = self.convergence_func(grid_eta[i])\n\n return surface_density_grid"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef potential_from_grid(self, grid):\n\n potential_grid = quad_grid(self.potential_func, 0.0, 1.0, grid,\n args=(self.axis_ratio, self.slope, self.core_radius))[0]\n\n return self.einstein_radius_rescaled * self.axis_ratio * potential_grid", "response": "Calculates the potential at a given set of arc - second gridded coordinates."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef deflections_from_grid(self, grid):\n\n def calculate_deflection_component(npow, index):\n einstein_radius_rescaled = self.einstein_radius_rescaled\n\n deflection_grid = self.axis_ratio * grid[:, index]\n deflection_grid *= quad_grid(self.deflection_func, 0.0, 1.0,\n grid, args=(npow, self.axis_ratio,\n einstein_radius_rescaled, self.slope,\n self.core_radius))[0]\n\n return deflection_grid\n\n deflection_y = calculate_deflection_component(1.0, 0)\n deflection_x = calculate_deflection_component(0.0, 1)\n\n return self.rotate_grid_from_profile(np.multiply(1.0, np.vstack((deflection_y, deflection_x)).T))", "response": "Calculates the deflection angles at a given set of arc - second coordinates."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the deflection angles at a given set of arc - second gridded coordinates.", "response": "def deflections_from_grid(self, grid):\n \"\"\"\n Calculate the deflection angles at a given set of arc-second gridded coordinates.\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the deflection angles are computed on.\n \"\"\"\n eta = self.grid_to_grid_radii(grid=grid)\n deflection = np.multiply(2. * self.einstein_radius_rescaled, np.divide(\n np.add(np.power(np.add(self.core_radius ** 2, np.square(eta)), (3. - self.slope) / 2.),\n -self.core_radius ** (3 - self.slope)), np.multiply((3. - self.slope), eta)))\n return self.grid_to_grid_cartesian(grid=grid, radius=deflection)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the deflection angles at a given set of arc - second coordinates.", "response": "def deflections_from_grid(self, grid):\n \"\"\"\n Calculate the deflection angles at a given set of arc-second gridded coordinates.\n\n For coordinates (0.0, 0.0) the analytic calculation of the deflection angle gives a NaN. Therefore, \\\n coordinates at (0.0, 0.0) are shifted slightly to (1.0e-8, 1.0e-8).\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the deflection angles are computed on.\n \"\"\"\n factor = 2.0 * self.einstein_radius_rescaled * self.axis_ratio / np.sqrt(1 - self.axis_ratio ** 2)\n\n psi = np.sqrt(np.add(np.multiply(self.axis_ratio ** 2, np.square(grid[:, 1])), np.square(grid[:, 0])))\n\n deflection_y = np.arctanh(np.divide(np.multiply(np.sqrt(1 - self.axis_ratio ** 2), grid[:, 0]), psi))\n deflection_x = np.arctan(np.divide(np.multiply(np.sqrt(1 - self.axis_ratio ** 2), grid[:, 1]), psi))\n return self.rotate_grid_from_profile(np.multiply(factor, np.vstack((deflection_y, deflection_x)).T))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the potential at a given set of arc - second gridded coordinates.", "response": "def potential_from_grid(self, grid):\n \"\"\"\n Calculate the potential at a given set of arc-second gridded coordinates.\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the deflection angles are computed on.\n \"\"\"\n eta = self.grid_to_elliptical_radii(grid)\n return 2.0 * self.einstein_radius_rescaled * eta"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef deflections_from_grid(self, grid):\n return self.grid_to_grid_cartesian(grid=grid,\n radius=np.full(grid.shape[0], 2.0 * self.einstein_radius_rescaled))", "response": "Calculate the deflection angles at a given set of arc - second gridded coordinates."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ntabulate an integral over the surface density of deflection potential of a mass profile.", "response": "def tabulate_integral(self, grid, tabulate_bins):\n \"\"\"Tabulate an integral over the surface density of deflection potential of a mass profile. This is used in \\\n the GeneralizedNFW profile classes to speed up the integration procedure.\n\n Parameters\n -----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the potential / deflection_stacks are computed on.\n tabulate_bins : int\n The number of bins to tabulate the inner integral of this profile.\n \"\"\"\n eta_min = 1.0e-4\n eta_max = 1.05 * np.max(self.grid_to_elliptical_radii(grid))\n\n minimum_log_eta = np.log10(eta_min)\n maximum_log_eta = np.log10(eta_max)\n bin_size = (maximum_log_eta - minimum_log_eta) / (tabulate_bins - 1)\n\n return eta_min, eta_max, minimum_log_eta, maximum_log_eta, bin_size"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncalculate the potential at a given set of arc - second coordinates.", "response": "def potential_from_grid(self, grid, tabulate_bins=1000):\n \"\"\"\n Calculate the potential at a given set of arc-second gridded coordinates.\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the deflection angles are computed on.\n tabulate_bins : int\n The number of bins to tabulate the inner integral of this profile.\n \"\"\"\n\n @jit_integrand\n def deflection_integrand(x, kappa_radius, scale_radius, inner_slope):\n return (x + kappa_radius / scale_radius) ** (inner_slope - 3) * ((1 - np.sqrt(1 - x ** 2)) / x)\n\n eta_min, eta_max, minimum_log_eta, maximum_log_eta, bin_size = self.tabulate_integral(grid, tabulate_bins)\n\n potential_grid = np.zeros(grid.shape[0])\n\n deflection_integral = np.zeros((tabulate_bins,))\n\n for i in range(tabulate_bins):\n eta = 10. ** (minimum_log_eta + (i - 1) * bin_size)\n\n integral = \\\n quad(deflection_integrand, a=0.0, b=1.0, args=(eta, self.scale_radius, self.inner_slope),\n epsrel=EllipticalGeneralizedNFW.epsrel)[0]\n\n deflection_integral[i] = ((eta / self.scale_radius) ** (2 - self.inner_slope)) * (\n (1.0 / (3 - self.inner_slope)) *\n special.hyp2f1(3 - self.inner_slope, 3 - self.inner_slope, 4 - self.inner_slope,\n - (eta / self.scale_radius)) + integral)\n\n for i in range(grid.shape[0]):\n potential_grid[i] = (2.0 * self.kappa_s * self.axis_ratio) * \\\n quad(self.potential_func, a=0.0, b=1.0, args=(grid[i, 0], grid[i, 1],\n self.axis_ratio, minimum_log_eta,\n maximum_log_eta, tabulate_bins,\n deflection_integral),\n epsrel=EllipticalGeneralizedNFW.epsrel)[0]\n\n return potential_grid"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef deflections_from_grid(self, grid, tabulate_bins=1000):\n\n @jit_integrand\n def surface_density_integrand(x, kappa_radius, scale_radius, inner_slope):\n return (3 - inner_slope) * (x + kappa_radius / scale_radius) ** (inner_slope - 4) * (1 - np.sqrt(1 - x * x))\n\n def calculate_deflection_component(npow, index):\n deflection_grid = 2.0 * self.kappa_s * self.axis_ratio * grid[:, index]\n deflection_grid *= quad_grid(self.deflection_func, 0.0, 1.0,\n grid, args=(npow, self.axis_ratio, minimum_log_eta, maximum_log_eta,\n tabulate_bins, surface_density_integral),\n epsrel=EllipticalGeneralizedNFW.epsrel)[0]\n\n return deflection_grid\n\n eta_min, eta_max, minimum_log_eta, maximum_log_eta, bin_size = self.tabulate_integral(grid, tabulate_bins)\n\n surface_density_integral = np.zeros((tabulate_bins,))\n\n for i in range(tabulate_bins):\n eta = 10. ** (minimum_log_eta + (i - 1) * bin_size)\n\n integral = quad(surface_density_integrand, a=0.0, b=1.0, args=(eta, self.scale_radius,\n self.inner_slope),\n epsrel=EllipticalGeneralizedNFW.epsrel)[0]\n\n surface_density_integral[i] = ((eta / self.scale_radius) ** (1 - self.inner_slope)) * \\\n (((1 + eta / self.scale_radius) ** (self.inner_slope - 3)) + integral)\n\n deflection_y = calculate_deflection_component(1.0, 0)\n deflection_x = calculate_deflection_component(0.0, 1)\n\n return self.rotate_grid_from_profile(np.multiply(1.0, np.vstack((deflection_y, deflection_x)).T))", "response": "Calculates the deflection angles at a given set of arc - second coordinates."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef deflections_from_grid(self, grid, **kwargs):\n\n eta = np.multiply(1. / self.scale_radius, self.grid_to_grid_radii(grid))\n\n deflection_grid = np.zeros(grid.shape[0])\n\n for i in range(grid.shape[0]):\n deflection_grid[i] = np.multiply(4. * self.kappa_s * self.scale_radius, self.deflection_func_sph(eta[i]))\n\n return self.grid_to_grid_cartesian(grid, deflection_grid)", "response": "Calculate the deflection angles at a given set of arc - second gridded coordinates."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef deflections_from_grid(self, grid, **kwargs):\n\n eta = np.multiply(1. / self.scale_radius, self.grid_to_grid_radii(grid))\n\n deflection_grid = np.multiply((4. * self.kappa_s * self.scale_radius / eta), self.deflection_func_sph(eta))\n\n return self.grid_to_grid_cartesian(grid, deflection_grid)", "response": "Calculate the deflection angles at a given set of arc - second gridded coordinates."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef potential_from_grid(self, grid):\n potential_grid = quad_grid(self.potential_func, 0.0, 1.0, grid,\n args=(self.axis_ratio, self.kappa_s, self.scale_radius),\n epsrel=1.49e-5)[0]\n\n return potential_grid", "response": "Calculate the potential at a given set of arc - second gridded coordinates."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates the deflection angles at a given set of arc - second coordinates.", "response": "def deflections_from_grid(self, grid):\n \"\"\"\n Calculate the deflection angles at a given set of arc-second gridded coordinates.\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the deflection angles are computed on.\n \"\"\"\n\n def calculate_deflection_component(npow, index):\n deflection_grid = self.axis_ratio * grid[:, index]\n deflection_grid *= quad_grid(self.deflection_func, 0.0, 1.0, grid,\n args=(npow, self.axis_ratio, self.kappa_s,\n self.scale_radius))[0]\n\n return deflection_grid\n\n deflection_y = calculate_deflection_component(1.0, 0)\n deflection_x = calculate_deflection_component(0.0, 1)\n\n return self.rotate_grid_from_profile(np.multiply(1.0, np.vstack((deflection_y, deflection_x)).T))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef potential_from_grid(self, grid):\n eta = (1.0 / self.scale_radius) * self.grid_to_grid_radii(grid) + 0j\n return np.real(2.0 * self.scale_radius * self.kappa_s * self.potential_func_sph(eta))", "response": "Calculates the potential at a given set of arc - second gridded coordinates."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate the deflection angles at a given set of arc - second gridded coordinates.", "response": "def deflections_from_grid(self, grid):\n \"\"\"\n Calculate the deflection angles at a given set of arc-second gridded coordinates.\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the deflection angles are computed on.\n \"\"\"\n eta = np.multiply(1. / self.scale_radius, self.grid_to_grid_radii(grid=grid))\n deflection_r = np.multiply(4. * self.kappa_s * self.scale_radius, self.deflection_func_sph(eta))\n\n return self.grid_to_grid_cartesian(grid, deflection_r)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the intensity of the sersic at a given radius.", "response": "def intensity_at_radius(self, radius):\n \"\"\" Compute the intensity of the profile at a given radius.\n\n Parameters\n ----------\n radius : float\n The distance from the centre of the profile.\n \"\"\"\n return self.intensity * np.exp(\n -self.sersic_constant * (((radius / self.effective_radius) ** (1. / self.sersic_index)) - 1))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sersic_constant(self):\n return (2 * self.sersic_index) - (1. / 3.) + (4. / (405. * self.sersic_index)) + (\n 46. / (25515. * self.sersic_index ** 2)) + (131. / (1148175. * self.sersic_index ** 3)) - (\n 2194697. / (30690717750. * self.sersic_index ** 4))", "response": "A constant value derived from Sersic index which ensures that effective radius contains 50% of the profile s\n total integrated light."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncalculate the deflection angles at a given set of arc - second coordinates.", "response": "def deflections_from_grid(self, grid):\n \"\"\"\n Calculate the deflection angles at a given set of arc-second gridded coordinates.\n\n Parameters\n ----------\n grid : grids.RegularGrid\n The grid of (y,x) arc-second coordinates the deflection angles are computed on.\n \"\"\"\n\n def calculate_deflection_component(npow, index):\n sersic_constant = self.sersic_constant\n\n deflection_grid = self.axis_ratio * grid[:, index]\n deflection_grid *= quad_grid(self.deflection_func, 0.0, 1.0, grid,\n args=(npow, self.axis_ratio, self.intensity,\n self.sersic_index, self.effective_radius,\n self.mass_to_light_ratio, self.mass_to_light_gradient,\n sersic_constant))[0]\n return deflection_grid\n\n deflection_y = calculate_deflection_component(1.0, 0)\n deflection_x = calculate_deflection_component(0.0, 1)\n\n return self.rotate_grid_from_profile(np.multiply(1.0, np.vstack((deflection_y, deflection_x)).T))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef deflections_from_grid(self, grid):\n deflection_y = -np.multiply(self.magnitude, grid[:, 0])\n deflection_x = np.multiply(self.magnitude, grid[:, 1])\n return self.rotate_grid_from_profile(np.vstack((deflection_y, deflection_x)).T)", "response": "Calculate the deflection angles at a given set of arc - second gridded coordinates."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates the summed intensities of all of the galaxy s light profiles using a grid of Cartesian y x coordinates.", "response": "def intensities_from_grid(self, grid):\n \"\"\"Calculate the summed intensities of all of the galaxy's light profiles using a grid of Cartesian (y,x) \\\n coordinates.\n \n If the galaxy has no light profiles, a grid of zeros is returned.\n \n See *profiles.light_profiles* for a description of how light profile intensities are computed.\n\n Parameters\n ----------\n grid : ndarray\n The (y, x) coordinates in the original reference frame of the grid.\n \"\"\"\n if self.has_light_profile:\n return sum(map(lambda p: p.intensities_from_grid(grid), self.light_profiles))\n else:\n return np.zeros((grid.shape[0],))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute the total luminosity of the galaxy s light profiles within a circle of specified radius in units.", "response": "def luminosity_within_circle_in_units(self, radius : dim.Length, unit_luminosity='eps', kpc_per_arcsec=None, exposure_time=None):\n \"\"\"Compute the total luminosity of the galaxy's light profiles within a circle of specified radius.\n\n See *light_profiles.luminosity_within_circle* for details of how this is performed.\n\n Parameters\n ----------\n radius : float\n The radius of the circle to compute the dimensionless mass within.\n unit_luminosity : str\n The units the luminosity is returned in (eps | counts).\n exposure_time : float\n The exposure time of the observation, which converts luminosity from electrons per second units to counts.\n \"\"\"\n if self.has_light_profile:\n return sum(map(lambda p: p.luminosity_within_circle_in_units(radius=radius, unit_luminosity=unit_luminosity,\n kpc_per_arcsec=kpc_per_arcsec, exposure_time=exposure_time),\n self.light_profiles))\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncomputes the total luminosity of the galaxy s light profiles within an ellipse of specified major axis.", "response": "def luminosity_within_ellipse_in_units(self, major_axis : dim.Length, unit_luminosity='eps', kpc_per_arcsec=None, exposure_time=None):\n \"\"\"Compute the total luminosity of the galaxy's light profiles, within an ellipse of specified major axis. This \n is performed via integration of each light profile and is centred, oriented and aligned with each light\n model's individual geometry.\n\n See *light_profiles.luminosity_within_ellipse* for details of how this is performed.\n\n Parameters\n ----------\n major_axis : float\n The major-axis radius of the ellipse.\n unit_luminosity : str\n The units the luminosity is returned in (eps | counts).\n exposure_time : float\n The exposure time of the observation, which converts luminosity from electrons per second units to counts.\n \"\"\"\n if self.has_light_profile:\n return sum(map(lambda p: p.luminosity_within_ellipse_in_units(major_axis=major_axis, unit_luminosity=unit_luminosity,\n kpc_per_arcsec=kpc_per_arcsec, exposure_time=exposure_time),\n self.light_profiles))\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncomputes the summed convergence of the galaxy s mass profiles using a grid of Cartesian y x coordinates.", "response": "def convergence_from_grid(self, grid):\n \"\"\"Compute the summed convergence of the galaxy's mass profiles using a grid of Cartesian (y,x) \\\n coordinates.\n\n If the galaxy has no mass profiles, a grid of zeros is returned.\n \n See *profiles.mass_profiles* module for details of how this is performed.\n\n Parameters\n ----------\n grid : ndarray\n The (y, x) coordinates in the original reference frame of the grid.\n \"\"\"\n if self.has_mass_profile:\n return sum(map(lambda p: p.convergence_from_grid(grid), self.mass_profiles))\n else:\n return np.zeros((grid.shape[0],))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef deflections_from_grid(self, grid):\n if self.has_mass_profile:\n return sum(map(lambda p: p.deflections_from_grid(grid), self.mass_profiles))\n else:\n return np.full((grid.shape[0], 2), 0.0)", "response": "Compute the summed deflection angles of the galaxy s mass profiles using a grid of Cartesian y x coordinates."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef mass_within_circle_in_units(self, radius, unit_mass='angular', kpc_per_arcsec=None, critical_surface_density=None):\n if self.has_mass_profile:\n return sum(map(lambda p: p.mass_within_circle_in_units(radius=radius, unit_mass=unit_mass,\n kpc_per_arcsec=kpc_per_arcsec,\n critical_surface_density=critical_surface_density),\n self.mass_profiles))\n else:\n return None", "response": "Compute the total angular mass of the galaxy s mass profiles within a circle of specified radius in units."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the total angular mass of the galaxy s mass profiles within an ellipse of specified major_axis.", "response": "def mass_within_ellipse_in_units(self, major_axis, unit_mass='angular', kpc_per_arcsec=None, critical_surface_density=None):\n \"\"\"Compute the total angular mass of the galaxy's mass profiles within an ellipse of specified major_axis.\n\n See *profiles.mass_profiles.angualr_mass_within_ellipse* for details of how this is performed.\n\n Parameters\n ----------\n major_axis : float\n The major-axis radius of the ellipse.\n units_luminosity : str\n The units the luminosity is returned in (eps | counts).\n exposure_time : float\n The exposure time of the observation, which converts luminosity from electrons per second units to counts.\n \"\"\"\n if self.has_mass_profile:\n return sum(map(lambda p: p.mass_within_ellipse_in_units(major_axis=major_axis, unit_mass=unit_mass,\n kpc_per_arcsec=kpc_per_arcsec,\n critical_surface_density=critical_surface_density),\n self.mass_profiles))\n else:\n return None"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef einstein_mass_in_units(self, unit_mass='angular', critical_surface_density=None):\n\n if self.has_mass_profile:\n return sum(\n map(lambda p: p.einstein_mass_in_units(unit_mass=unit_mass,\n critical_surface_density=critical_surface_density),\n self.mass_profiles))\n else:\n return None", "response": "The sum of Einstein Mass of this galaxy in units."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncompute the contribution map of a model image and a galaxy image.", "response": "def contributions_from_model_image_and_galaxy_image(self, model_image, galaxy_image, minimum_value=0.0):\n \"\"\"Compute the contribution map of a galaxy, which represents the fraction of flux in each pixel that the \\\n galaxy is attributed to contain, scaled to the *contribution_factor* hyper-parameter.\n\n This is computed by dividing that galaxy's flux by the total flux in that pixel, and then scaling by the \\\n maximum flux such that the contribution map ranges between 0 and 1.\n\n Parameters\n -----------\n model_image : ndarray\n The best-fit model image to the observed image from a previous analysis phase. This provides the \\\n total light attributed to each image pixel by the model.\n galaxy_image : ndarray\n A model image of the galaxy (from light profiles or an inversion) from a previous analysis phase.\n minimum_value : float\n The minimum contribution value a pixel must contain to not be rounded to 0.\n \"\"\"\n contributions = np.divide(galaxy_image, np.add(model_image, self.contribution_factor))\n contributions = np.divide(contributions, np.max(contributions))\n contributions[contributions < minimum_value] = 0.0\n return contributions"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef hyper_noise_from_contributions(self, noise_map, contributions):\n return self.noise_factor * (noise_map * contributions) ** self.noise_power", "response": "Compute a scaled galaxy hyper noise - map from a baseline noise - map and a list of contributions."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef frame_at_coordinates_jit(coordinates, mask, mask_index_array, psf):\n\n psf_shape = psf.shape\n psf_max_size = psf_shape[0] * psf_shape[1]\n\n half_x = int(psf_shape[0] / 2)\n half_y = int(psf_shape[1] / 2)\n\n frame = -1 * np.ones((psf_max_size))\n psf_frame = -1.0 * np.ones((psf_max_size))\n\n count = 0\n for i in range(psf_shape[0]):\n for j in range(psf_shape[1]):\n x = coordinates[0] - half_x + i\n y = coordinates[1] - half_y + j\n if 0 <= x < mask_index_array.shape[0] and 0 <= y < mask_index_array.shape[1]:\n value = mask_index_array[x, y]\n if value >= 0 and not mask[x, y]:\n frame[count] = value\n psf_frame[count] = psf[i, j]\n count += 1\n\n return frame, psf_frame", "response": "Compute the frame and psf_frame for a given coordinate in a mask and a PSF."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convolve_image(self, image_array, blurring_array):\n return self.convolve_jit(image_array, self.image_frame_indexes, self.image_frame_psfs, self.image_frame_lengths,\n blurring_array, self.blurring_frame_indexes, self.blurring_frame_psfs,\n self.blurring_frame_lengths)", "response": "This method is used to convolve the two regular arrays and blurring arrays."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef intensities_of_galaxies_from_grid(grid, galaxies):\n if galaxies:\n return sum(map(lambda g: g.intensities_from_grid(grid), galaxies))\n else:\n return np.full((grid.shape[0]), 0.0)", "response": "Compute the intensities of a list of galaxies from a regular grid."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef convergence_of_galaxies_from_grid(grid, galaxies):\n if galaxies:\n return sum(map(lambda g: g.convergence_from_grid(grid), galaxies))\n else:\n return np.full((grid.shape[0]), 0.0)", "response": "Compute the convergence of a list of galaxies from a regular grid."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncompute the potential of a list of galaxies from a regular grid.", "response": "def potential_of_galaxies_from_grid(grid, galaxies):\n \"\"\"Compute the potential of a list of galaxies from an input grid, by summing the individual potential \\\n of each galaxy's mass profile.\n\n If the input grid is a *grids.SubGrid*, the surface-density is calculated on the sub-grid and binned-up to the \\\n original regular grid by taking the mean value of every set of sub-pixels.\n\n If no galaxies are entered into the function, an array of all zeros is returned.\n\n Parameters\n -----------\n grid : RegularGrid\n The grid (regular or sub) of (y,x) arc-second coordinates at the centre of every unmasked pixel which the \\\n potential is calculated on.\n galaxies : [galaxy.Galaxy]\n The galaxies whose mass profiles are used to compute the surface densities.\n \"\"\"\n if galaxies:\n return sum(map(lambda g: g.potential_from_grid(grid), galaxies))\n else:\n return np.full((grid.shape[0]), 0.0)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef deflections_of_galaxies_from_grid(grid, galaxies):\n if len(galaxies) > 0:\n deflections = sum(map(lambda galaxy: galaxy.deflections_from_grid(grid), galaxies))\n else:\n deflections = np.full((grid.shape[0], 2), 0.0)\n\n if isinstance(grid, grids.SubGrid):\n return np.asarray([grid.regular_data_1d_from_sub_data_1d(deflections[:, 0]),\n grid.regular_data_1d_from_sub_data_1d(deflections[:, 1])]).T\n\n return deflections", "response": "Compute the deflections of a list of galaxies from a regular grid."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef deflections_of_galaxies_from_sub_grid(sub_grid, galaxies):\n if galaxies:\n return sum(map(lambda galaxy: galaxy.deflections_from_grid(sub_grid), galaxies))\n else:\n return np.full((sub_grid.shape[0], 2), 0.0)", "response": "Compute the deflections of a list of galaxies from a regular grid."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef evidence_from_inversion_terms(chi_squared, regularization_term, log_curvature_regularization_term,\n log_regularization_term, noise_normalization):\n \"\"\"Compute the evidence of an inversion's fit to the datas, where the evidence includes a number of \\\n terms which quantify the complexity of an inversion's reconstruction (see the *inversion* module):\n\n Likelihood = -0.5*[Chi_Squared_Term + Regularization_Term + Log(Covariance_Regularization_Term) -\n Log(Regularization_Matrix_Term) + Noise_Term]\n\n Parameters\n ----------\n chi_squared : float\n The chi-squared term of the inversion's fit to the observed datas.\n regularization_term : float\n The regularization term of the inversion, which is the sum of the difference between reconstructed \\\n flux of every pixel multiplied by the regularization coefficient.\n log_curvature_regularization_term : float\n The log of the determinant of the sum of the curvature and regularization matrices.\n log_regularization_term : float\n The log of the determinant o the regularization matrix.\n noise_normalization : float\n The normalization noise_map-term for the observed datas's noise-map.\n \"\"\"\n return -0.5 * (chi_squared + regularization_term + log_curvature_regularization_term - log_regularization_term\n + noise_normalization)", "response": "Compute the evidence of an inversion s fit to the observed datas."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef unmasked_blurred_image_of_planes_from_padded_grid_stack_and_psf(planes, padded_grid_stack, psf):\n unmasked_blurred_image_of_planes = []\n\n for plane in planes:\n\n if plane.has_pixelization:\n unmasked_blurred_image_of_plane = None\n else:\n unmasked_blurred_image_of_plane = \\\n padded_grid_stack.unmasked_blurred_image_from_psf_and_unmasked_image(\n\n psf=psf, unmasked_image_1d=plane.image_plane_image_1d)\n\n unmasked_blurred_image_of_planes.append(unmasked_blurred_image_of_plane)\n\n return unmasked_blurred_image_of_planes", "response": "For lens data compute the unmasked blurred image of every plane and the PSF of the image."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef contribution_maps_1d_from_hyper_images_and_galaxies(hyper_model_image_1d, hyper_galaxy_images_1d, hyper_galaxies,\n hyper_minimum_values):\n \"\"\"For a fitting hyper_galaxy_image, hyper_galaxy model image, list of hyper galaxies images and model hyper galaxies, compute\n their contribution maps, which are used to compute a scaled-noise_map map. All quantities are masked 1D arrays.\n\n The reason this is separate from the *contributions_from_fitting_hyper_images_and_hyper_galaxies* function is that\n each hyper_galaxy image has a list of hyper galaxies images and associated hyper galaxies (one for each galaxy). Thus,\n this function breaks down the calculation of each 1D masked contribution map and returns them in the same datas\n structure (2 lists with indexes [image_index][contribution_map_index].\n\n Parameters\n ----------\n hyper_model_image_1d : ndarray\n The best-fit model image to the datas (e.g. from a previous analysis phase).\n hyper_galaxy_images_1d : [ndarray]\n The best-fit model image of each hyper galaxy to the datas (e.g. from a previous analysis phase).\n hyper_galaxies : [galaxy.Galaxy]\n The hyper galaxies which represent the model components used to scale the noise_map, which correspond to\n individual galaxies in the image.\n hyper_minimum_values : [float]\n The minimum value of each hyper_galaxy-image contribution map, which ensure zero's don't impact the scaled noise-map.\n \"\"\"\n # noinspection PyArgumentList\n return list(map(lambda hyper_galaxy, hyper_galaxy_image_1d, hyper_minimum_value:\n hyper_galaxy.contributions_from_model_image_and_galaxy_image(model_image=hyper_model_image_1d,\n galaxy_image=hyper_galaxy_image_1d,\n minimum_value=hyper_minimum_value),\n hyper_galaxies, hyper_galaxy_images_1d, hyper_minimum_values))", "response": "This function computes the contribution maps for a single model image and hyper - galaxies and returns the corresponding 1D masked array."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef rectangular_neighbors_from_shape(shape):\n\n\n pixels = shape[0]*shape[1]\n\n pixel_neighbors = -1 * np.ones(shape=(pixels, 4))\n pixel_neighbors_size = np.zeros(pixels)\n\n pixel_neighbors, pixel_neighbors_size = compute_corner_neighbors(pixel_neighbors, pixel_neighbors_size,\n shape, pixels)\n pixel_neighbors, pixel_neighbors_size = compute_top_edge_neighbors(pixel_neighbors, pixel_neighbors_size,\n shape, pixels)\n pixel_neighbors, pixel_neighbors_size = compute_left_edge_neighbors(pixel_neighbors, pixel_neighbors_size,\n shape, pixels)\n pixel_neighbors, pixel_neighbors_size = compute_right_edge_neighbors(pixel_neighbors, pixel_neighbors_size,\n shape, pixels)\n pixel_neighbors, pixel_neighbors_size = compute_bottom_edge_neighbors(pixel_neighbors, pixel_neighbors_size,\n shape, pixels)\n pixel_neighbors, pixel_neighbors_size = compute_central_neighbors(pixel_neighbors, pixel_neighbors_size,\n shape, pixels)\n\n return pixel_neighbors, pixel_neighbors_size", "response": "Compute the neighbors of every pixel as a list of the pixel index s each pixel shares a vertex with."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef voronoi_neighbors_from_pixels_and_ridge_points(pixels, ridge_points):\n\n pixel_neighbors_size = np.zeros(shape=(pixels))\n\n for ridge_index in range(ridge_points.shape[0]):\n pair0 = ridge_points[ridge_index, 0]\n pair1 = ridge_points[ridge_index, 1]\n pixel_neighbors_size[pair0] += 1\n pixel_neighbors_size[pair1] += 1\n\n pixel_neighbors_index = np.zeros(shape=(pixels))\n pixel_neighbors = -1 * np.ones(shape=(pixels, int(np.max(pixel_neighbors_size))))\n\n for ridge_index in range(ridge_points.shape[0]):\n pair0 = ridge_points[ridge_index, 0]\n pair1 = ridge_points[ridge_index, 1]\n pixel_neighbors[pair0, int(pixel_neighbors_index[pair0])] = pair1\n pixel_neighbors[pair1, int(pixel_neighbors_index[pair1])] = pair0\n pixel_neighbors_index[pair0] += 1\n pixel_neighbors_index[pair1] += 1\n\n return pixel_neighbors, pixel_neighbors_size", "response": "Compute the neighbors of every pixel in a list of the pixel index s each pixel shares a vertex with."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef for_data_and_tracer(cls, lens_data, tracer, padded_tracer=None):\n\n if tracer.has_light_profile and not tracer.has_pixelization:\n return LensProfileFit(lens_data=lens_data, tracer=tracer, padded_tracer=padded_tracer)\n elif not tracer.has_light_profile and tracer.has_pixelization:\n return LensInversionFit(lens_data=lens_data, tracer=tracer, padded_tracer=None)\n elif tracer.has_light_profile and tracer.has_pixelization:\n return LensProfileInversionFit(lens_data=lens_data, tracer=tracer, padded_tracer=None)\n else:\n raise exc.FittingException('The fit routine did not call a Fit class - check the '\n 'properties of the tracer')", "response": "Returns a new instance of the fit class for the given lens data and a tracer."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef transform_grid(func):\n\n @wraps(func)\n def wrapper(profile, grid, *args, **kwargs):\n \"\"\"\n\n Parameters\n ----------\n profile : GeometryProfile\n The profiles that owns the function\n grid : ndarray\n PlaneCoordinates in either cartesian or profiles coordinate system\n args\n kwargs\n\n Returns\n -------\n A value or coordinate in the same coordinate system as those passed in.\n \"\"\"\n if not isinstance(grid, TransformedGrid):\n return func(profile, profile.transform_grid_to_reference_frame(grid), *args, **kwargs)\n else:\n return func(profile, grid, *args, **kwargs)\n\n return wrapper", "response": "Decorator that transforms the coordinates of the given grid to the corresponding reference frame."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncache results of a call to a grid function. If a grid that evaluates to the same byte value is passed into the same function of the same instance as previously then the cached result is returned. Parameters ---------- func Some instance method that takes a grid as its argument Returns ------- result Some result, either newly calculated or recovered from the cache", "response": "def cache(func):\n \"\"\"\n Caches results of a call to a grid function. If a grid that evaluates to the same byte value is passed into the same\n function of the same instance as previously then the cached result is returned.\n\n Parameters\n ----------\n func\n Some instance method that takes a grid as its argument\n\n Returns\n -------\n result\n Some result, either newly calculated or recovered from the cache\n \"\"\"\n\n def wrapper(instance: GeometryProfile, grid: np.ndarray, *args, **kwargs):\n if not hasattr(instance, \"cache\"):\n instance.cache = {}\n key = (func.__name__, grid.tobytes())\n if key not in instance.cache:\n instance.cache[key] = func(instance, grid)\n return instance.cache[key]\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncheck whether any coordinates in the grid are radially near (0.0, 0.0), which can lead to numerical faults in \\ the evaluation of a light or mass profiles. If any coordinates are radially within the the radial minimum \\ threshold, their (y,x) coordinates are shifted to that value to ensure they are evaluated correctly. By default this radial minimum is not used, and users should be certain they use a value that does not impact \\ results. Parameters ---------- func : (profile, *args, **kwargs) -> Object A function that takes a grid of coordinates which may have a singularity as (0.0, 0.0) Returns ------- A function that can except cartesian or transformed coordinates", "response": "def move_grid_to_radial_minimum(func):\n \"\"\" Checks whether any coordinates in the grid are radially near (0.0, 0.0), which can lead to numerical faults in \\\n the evaluation of a light or mass profiles. If any coordinates are radially within the the radial minimum \\\n threshold, their (y,x) coordinates are shifted to that value to ensure they are evaluated correctly.\n\n By default this radial minimum is not used, and users should be certain they use a value that does not impact \\\n results.\n\n Parameters\n ----------\n func : (profile, *args, **kwargs) -> Object\n A function that takes a grid of coordinates which may have a singularity as (0.0, 0.0)\n\n Returns\n -------\n A function that can except cartesian or transformed coordinates\n \"\"\"\n\n @wraps(func)\n def wrapper(profile, grid, *args, **kwargs):\n \"\"\"\n\n Parameters\n ----------\n profile : SphericalProfile\n The profiles that owns the function\n grid : ndarray\n PlaneCoordinates in either cartesian or profiles coordinate system\n args\n kwargs\n\n Returns\n -------\n A value or coordinate in the same coordinate system as those passed in.\n \"\"\"\n radial_minimum_config = conf.NamedConfig(f\"{conf.instance.config_path}/radial_minimum.ini\")\n grid_radial_minimum = radial_minimum_config.get(\"radial_minimum\", profile.__class__.__name__, float)\n with np.errstate(all='ignore'): # Division by zero fixed via isnan\n grid_radii = profile.grid_to_grid_radii(grid=grid)\n grid_radial_scale = np.where(grid_radii < grid_radial_minimum, grid_radial_minimum / grid_radii, 1.0)\n grid = np.multiply(grid, grid_radial_scale[:, None])\n grid[np.isnan(grid)] = grid_radial_minimum\n return func(profile, grid, *args, **kwargs)\n\n return wrapper"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nconvert a grid of y x coordinates to a grid of their circular radii.", "response": "def grid_to_grid_radii(self, grid):\n \"\"\"Convert a grid of (y, x) coordinates to a grid of their circular radii.\n\n If the coordinates have not been transformed to the profile's centre, this is performed automatically.\n\n Parameters\n ----------\n grid : TransformedGrid(ndarray)\n The (y, x) coordinates in the reference frame of the profile.\n \"\"\"\n return np.sqrt(np.add(np.square(grid[:, 0]), np.square(grid[:, 1])))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a grid of y x coordinates with their specified circular radii to their original y x Cartesian coordinates.", "response": "def grid_to_grid_cartesian(self, grid, radius):\n \"\"\"\n Convert a grid of (y,x) coordinates with their specified circular radii to their original (y,x) Cartesian \n coordinates.\n\n Parameters\n ----------\n grid : TransformedGrid(ndarray)\n The (y, x) coordinates in the reference frame of the profile.\n radius : ndarray\n The circular radius of each coordinate from the profile center.\n \"\"\"\n grid_thetas = np.arctan2(grid[:, 0], grid[:, 1])\n cos_theta, sin_theta = self.grid_angle_to_profile(grid_thetas=grid_thetas)\n return np.multiply(radius[:, None], np.vstack((sin_theta, cos_theta)).T)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ntransforming a grid of y x coordinates to the reference frame of the profile.", "response": "def transform_grid_to_reference_frame(self, grid):\n \"\"\"Transform a grid of (y,x) coordinates to the reference frame of the profile, including a translation to \\\n its centre.\n\n Parameters\n ----------\n grid : ndarray\n The (y, x) coordinates in the original reference frame of the grid.\n \"\"\"\n transformed = np.subtract(grid, self.centre)\n return transformed.view(TransformedGrid)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef transform_grid_from_reference_frame(self, grid):\n transformed = np.add(grid, self.centre)\n return transformed.view(TransformedGrid)", "response": "Transform a grid of y x coordinates from the original observer \\\n reference frame to the original observer \\\n reference frame including a translation from the centre."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ndetermining the cos and sin of the angle between the positive x - axis and the profile s ellipse.", "response": "def cos_and_sin_from_x_axis(self):\n \"\"\" Determine the sin and cosine of the angle between the profile's ellipse and the positive x-axis, \\\n counter-clockwise. \"\"\"\n phi_radians = np.radians(self.phi)\n return np.cos(phi_radians), np.sin(phi_radians)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the angle between each angle theta on the grid and the profile.", "response": "def grid_angle_to_profile(self, grid_thetas):\n \"\"\"The angle between each angle theta on the grid and the profile, in radians.\n\n Parameters\n -----------\n grid_thetas : ndarray\n The angle theta counter-clockwise from the positive x-axis to each coordinate in radians.\n \"\"\"\n theta_coordinate_to_profile = np.add(grid_thetas, - self.phi_radians)\n return np.cos(theta_coordinate_to_profile), np.sin(theta_coordinate_to_profile)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrotating a grid of elliptical coordinates from the reference frame of the profile to the unrotated coordinate grid.", "response": "def rotate_grid_from_profile(self, grid_elliptical):\n \"\"\" Rotate a grid of elliptical (y,x) coordinates from the reference frame of the profile back to the \\\n unrotated coordinate grid reference frame (coordinates are not shifted back to their original centre).\n\n This routine is used after computing deflection angles in the reference frame of the profile, so that the \\\n deflection angles can be re-rotated to the frame of the original coordinates before performing ray-tracing.\n\n Parameters\n ----------\n grid_elliptical : TransformedGrid(ndarray)\n The (y, x) coordinates in the reference frame of an elliptical profile.\n \"\"\"\n y = np.add(np.multiply(grid_elliptical[:, 1], self.sin_phi), np.multiply(grid_elliptical[:, 0], self.cos_phi))\n x = np.add(np.multiply(grid_elliptical[:, 1], self.cos_phi), - np.multiply(grid_elliptical[:, 0], self.sin_phi))\n return np.vstack((y, x)).T"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a grid of y x coordinates to an elliptical radius.", "response": "def grid_to_elliptical_radii(self, grid):\n \"\"\" Convert a grid of (y,x) coordinates to an elliptical radius.\n\n If the coordinates have not been transformed to the profile's geometry, this is performed automatically.\n\n Parameters\n ----------\n grid : TransformedGrid(ndarray)\n The (y, x) coordinates in the reference frame of the elliptical profile.\n \"\"\"\n return np.sqrt(np.add(np.square(grid[:, 1]), np.square(np.divide(grid[:, 0], self.axis_ratio))))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nconvert a grid of y x coordinates to an eccentric radius which is used to define light profile half - light radii using circular radii.", "response": "def grid_to_eccentric_radii(self, grid):\n \"\"\"Convert a grid of (y,x) coordinates to an eccentric radius, which is (1.0/axis_ratio) * elliptical radius \\\n and used to define light profile half-light radii using circular radii.\n\n If the coordinates have not been transformed to the profile's geometry, this is performed automatically.\n\n Parameters\n ----------\n grid : TransformedGrid(ndarray)\n The (y, x) coordinates in the reference frame of the elliptical profile.\n \"\"\"\n return np.multiply(np.sqrt(self.axis_ratio), self.grid_to_elliptical_radii(grid)).view(np.ndarray)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef transform_grid_to_reference_frame(self, grid):\n if self.__class__.__name__.startswith(\"Spherical\"):\n return super().transform_grid_to_reference_frame(grid)\n shifted_coordinates = np.subtract(grid, self.centre)\n radius = np.sqrt(np.sum(shifted_coordinates ** 2.0, 1))\n theta_coordinate_to_profile = np.arctan2(shifted_coordinates[:, 0],\n shifted_coordinates[:, 1]) - self.phi_radians\n transformed = np.vstack(\n (radius * np.sin(theta_coordinate_to_profile), radius * np.cos(theta_coordinate_to_profile))).T\n return transformed.view(TransformedGrid)", "response": "Transform a grid of y x coordinates to the reference frame of the profile."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef transform_grid_from_reference_frame(self, grid):\n if self.__class__.__name__.startswith(\"Spherical\"):\n return super().transform_grid_from_reference_frame(grid)\n\n y = np.add(np.add(np.multiply(grid[:, 1], self.sin_phi), np.multiply(grid[:, 0], self.cos_phi)), self.centre[0])\n x = np.add(np.add(np.multiply(grid[:, 1], self.cos_phi), - np.multiply(grid[:, 0], self.sin_phi)),\n self.centre[1])\n return np.vstack((y, x)).T", "response": "Transform a grid of y x coordinates from the original observer \\\n reference frame to the original observer \\\n reference frame including a rotation to its original orientation and a translation from the profile s centre."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef mapping_matrix_from_sub_to_pix(sub_to_pix, pixels, regular_pixels, sub_to_regular, sub_grid_fraction):\n\n mapping_matrix = np.zeros((regular_pixels, pixels))\n\n for sub_index in range(sub_to_regular.shape[0]):\n mapping_matrix[sub_to_regular[sub_index], sub_to_pix[sub_index]] += sub_grid_fraction\n\n return mapping_matrix", "response": "Computes the mapping matrix from the observed sub - grid to the observed regular sub - grid."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef voronoi_regular_to_pix_from_grids_and_geometry(regular_grid, regular_to_nearest_pix, pixel_centres,\n pixel_neighbors, pixel_neighbors_size):\n \"\"\" Compute the mappings between a set of regular-grid pixels and pixelization pixels, using information on \\\n how regular pixels map to their closest pixelization pixel on the image-plane pix-grid and the pixelization's \\\n pixel centres.\n\n To determine the complete set of regular-pixel to pixelization pixel mappings, we must pair every regular-pixel to \\\n its nearest pixel. Using a full nearest neighbor search to do this is slow, thus the pixel neighbors (derived via \\\n the Voronoi grid) are used to localize each nearest neighbor search via a graph search.\n\n Parameters\n ----------\n regular_grid : RegularGrid\n The grid of (y,x) arc-second coordinates at the centre of every unmasked pixel, which has been traced to \\\n to an irregular grid via lens.\n regular_to_nearest_pix : ndarray\n A 1D array that maps every regular-grid pixel to its nearest pix-grid pixel (as determined on the unlensed \\\n 2D array).\n pixel_centres : ndarray\n The (y,x) centre of every Voronoi pixel in arc-seconds.\n pixel_neighbors : ndarray\n An array of length (voronoi_pixels) which provides the index of all neighbors of every pixel in \\\n the Voronoi grid (entries of -1 correspond to no neighbor).\n pixel_neighbors_size : ndarray\n An array of length (voronoi_pixels) which gives the number of neighbors of every pixel in the \\\n Voronoi grid.\n \"\"\"\n\n regular_to_pix = np.zeros((regular_grid.shape[0]))\n\n for regular_index in range(regular_grid.shape[0]):\n\n nearest_pix_pixel_index = regular_to_nearest_pix[regular_index]\n\n while True:\n\n nearest_pix_pixel_center = pixel_centres[nearest_pix_pixel_index]\n\n sub_to_nearest_pix_distance = (regular_grid[regular_index, 0] - nearest_pix_pixel_center[0]) ** 2 + \\\n (regular_grid[regular_index, 1] - nearest_pix_pixel_center[1]) ** 2\n\n closest_separation_from_pix_neighbor = 1.0e8\n\n for neighbor_index in range(pixel_neighbors_size[nearest_pix_pixel_index]):\n\n neighbor = pixel_neighbors[nearest_pix_pixel_index, neighbor_index]\n\n separation_from_neighbor = (regular_grid[regular_index, 0] - pixel_centres[neighbor, 0]) ** 2 + \\\n (regular_grid[regular_index, 1] - pixel_centres[neighbor, 1]) ** 2\n\n if separation_from_neighbor < closest_separation_from_pix_neighbor:\n\n closest_separation_from_pix_neighbor = separation_from_neighbor\n closest_neighbor_index = neighbor_index\n\n neighboring_pix_pixel_index = pixel_neighbors[nearest_pix_pixel_index, closest_neighbor_index]\n sub_to_neighboring_pix_distance = closest_separation_from_pix_neighbor\n\n if sub_to_nearest_pix_distance <= sub_to_neighboring_pix_distance:\n regular_to_pix[regular_index] = nearest_pix_pixel_index\n break\n else:\n nearest_pix_pixel_index = neighboring_pix_pixel_index\n\n return regular_to_pix", "response": "This function calculates the mappings between a set of regular pixels and a set of pixelization pixels and a geometry of the nearest pixel."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncomputing the mappings between a set of sub-grid pixels and pixelization pixels, using information on \\ how the regular pixels hosting each sub-pixel map to their closest pixelization pixel on the image-plane pix-grid \\ and the pixelization's pixel centres. To determine the complete set of sub-pixel to pixelization pixel mappings, we must pair every sub-pixel to \\ its nearest pixel. Using a full nearest neighbor search to do this is slow, thus the pixel neighbors (derived via \\ the Voronoi grid) are used to localize each nearest neighbor search by using a graph search. Parameters ---------- regular_grid : RegularGrid The grid of (y,x) arc-second coordinates at the centre of every unmasked pixel, which has been traced to \\ to an irregular grid via lens. regular_to_nearest_pix : ndarray A 1D array that maps every regular-grid pixel to its nearest pix-grid pixel (as determined on the unlensed \\ 2D array). pixel_centres : (float, float) The (y,x) centre of every Voronoi pixel in arc-seconds. pixel_neighbors : ndarray An array of length (voronoi_pixels) which provides the index of all neighbors of every pixel in \\ the Voronoi grid (entries of -1 correspond to no neighbor). pixel_neighbors_size : ndarray An array of length (voronoi_pixels) which gives the number of neighbors of every pixel in the \\ Voronoi grid.", "response": "def voronoi_sub_to_pix_from_grids_and_geometry(sub_grid, regular_to_nearest_pix, sub_to_regular, pixel_centres,\n pixel_neighbors, pixel_neighbors_size):\n \"\"\" Compute the mappings between a set of sub-grid pixels and pixelization pixels, using information on \\\n how the regular pixels hosting each sub-pixel map to their closest pixelization pixel on the image-plane pix-grid \\\n and the pixelization's pixel centres.\n\n To determine the complete set of sub-pixel to pixelization pixel mappings, we must pair every sub-pixel to \\\n its nearest pixel. Using a full nearest neighbor search to do this is slow, thus the pixel neighbors (derived via \\\n the Voronoi grid) are used to localize each nearest neighbor search by using a graph search.\n\n Parameters\n ----------\n regular_grid : RegularGrid\n The grid of (y,x) arc-second coordinates at the centre of every unmasked pixel, which has been traced to \\\n to an irregular grid via lens.\n regular_to_nearest_pix : ndarray\n A 1D array that maps every regular-grid pixel to its nearest pix-grid pixel (as determined on the unlensed \\\n 2D array).\n pixel_centres : (float, float)\n The (y,x) centre of every Voronoi pixel in arc-seconds.\n pixel_neighbors : ndarray\n An array of length (voronoi_pixels) which provides the index of all neighbors of every pixel in \\\n the Voronoi grid (entries of -1 correspond to no neighbor).\n pixel_neighbors_size : ndarray\n An array of length (voronoi_pixels) which gives the number of neighbors of every pixel in the \\\n Voronoi grid.\n \"\"\"\n\n sub_to_pix = np.zeros((sub_grid.shape[0]))\n\n for sub_index in range(sub_grid.shape[0]):\n\n nearest_pix_pixel_index = regular_to_nearest_pix[sub_to_regular[sub_index]]\n\n while True:\n\n nearest_pix_pixel_center = pixel_centres[nearest_pix_pixel_index]\n\n sub_to_nearest_pix_distance = (sub_grid[sub_index, 0] - nearest_pix_pixel_center[0]) ** 2 + \\\n (sub_grid[sub_index, 1] - nearest_pix_pixel_center[1]) ** 2\n\n closest_separation_from_pix_to_neighbor = 1.0e8\n\n for neighbor_index in range(pixel_neighbors_size[nearest_pix_pixel_index]):\n\n neighbor = pixel_neighbors[nearest_pix_pixel_index, neighbor_index]\n\n separation_from_neighbor = (sub_grid[sub_index, 0] - pixel_centres[neighbor, 0]) ** 2 + \\\n (sub_grid[sub_index, 1] - pixel_centres[neighbor, 1]) ** 2\n\n if separation_from_neighbor < closest_separation_from_pix_to_neighbor:\n closest_separation_from_pix_to_neighbor = separation_from_neighbor\n closest_neighbor_index = neighbor_index\n\n neighboring_pix_pixel_index = pixel_neighbors[nearest_pix_pixel_index, closest_neighbor_index]\n sub_to_neighboring_pix_distance = closest_separation_from_pix_to_neighbor\n\n if sub_to_nearest_pix_distance <= sub_to_neighboring_pix_distance:\n sub_to_pix[sub_index] = nearest_pix_pixel_index\n break\n else:\n nearest_pix_pixel_index = neighboring_pix_pixel_index\n\n return sub_to_pix"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nintegrate the light profile to compute the total luminosity within a circle of specified radius in units.", "response": "def luminosity_within_circle_in_units(self, radius: dim.Length, unit_luminosity='eps', kpc_per_arcsec=None,\n exposure_time=None):\n \"\"\"Integrate the light profile to compute the total luminosity within a circle of specified radius. This is \\\n centred on the light profile's centre.\n\n The following units for mass can be specified and output:\n\n - Electrons per second (default) - 'eps'.\n - Counts - 'counts' (multiplies the luminosity in electrons per second by the exposure time).\n\n Parameters\n ----------\n radius : float\n The radius of the circle to compute the dimensionless mass within.\n unit_luminosity : str\n The units the luminosity is returned in (eps | counts).\n exposure_time : float or None\n The exposure time of the observation, which converts luminosity from electrons per second units to counts.\n \"\"\"\n\n if not isinstance(radius, dim.Length):\n radius = dim.Length(value=radius, unit_length='arcsec')\n\n profile = self.new_profile_with_units_converted(unit_length=radius.unit_length, unit_luminosity=unit_luminosity,\n kpc_per_arcsec=kpc_per_arcsec, exposure_time=exposure_time)\n\n luminosity = quad(profile.luminosity_integral, a=0.0, b=radius, args=(1.0,))[0]\n return dim.Luminosity(luminosity, unit_luminosity)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nintegrates the light profiles to compute the total luminosity within an ellipse of specified major axis.", "response": "def luminosity_within_ellipse_in_units(self, major_axis, unit_luminosity='eps', kpc_per_arcsec=None,\n exposure_time=None):\n \"\"\"Integrate the light profiles to compute the total luminosity within an ellipse of specified major axis. \\\n This is centred on the light profile's centre.\n\n The following units for mass can be specified and output:\n\n - Electrons per second (default) - 'eps'.\n - Counts - 'counts' (multiplies the luminosity in electrons per second by the exposure time).\n\n Parameters\n ----------\n major_axis : float\n The major-axis radius of the ellipse.\n unit_luminosity : str\n The units the luminosity is returned in (eps | counts).\n exposure_time : float or None\n The exposure time of the observation, which converts luminosity from electrons per second units to counts.\n \"\"\"\n\n if not isinstance(major_axis, dim.Length):\n major_axis = dim.Length(major_axis, 'arcsec')\n\n profile = self.new_profile_with_units_converted(unit_length=major_axis.unit_length,\n unit_luminosity=unit_luminosity,\n kpc_per_arcsec=kpc_per_arcsec, exposure_time=exposure_time)\n luminosity = quad(profile.luminosity_integral, a=0.0, b=major_axis, args=(self.axis_ratio,))[0]\n return dim.Luminosity(luminosity, unit_luminosity)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nintegrate the luminosity of an elliptical light profile.", "response": "def luminosity_integral(self, x, axis_ratio):\n \"\"\"Routine to integrate the luminosity of an elliptical light profile.\n\n The axis ratio is set to 1.0 for computing the luminosity within a circle\"\"\"\n r = x * axis_ratio\n return 2 * np.pi * r * self.intensities_from_grid_radii(x)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef intensities_from_grid_radii(self, grid_radii):\n return np.multiply(np.divide(self.intensity, self.sigma * np.sqrt(2.0 * np.pi)),\n np.exp(-0.5 * np.square(np.divide(grid_radii, self.sigma))))", "response": "Calculate the intensity of the Gaussian light profile on a grid of radial coordinates."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef intensities_from_grid_radii(self, grid_radii):\n np.seterr(all='ignore')\n return np.multiply(self.intensity, np.exp(\n np.multiply(-self.sersic_constant,\n np.add(np.power(np.divide(grid_radii, self.effective_radius), 1. / self.sersic_index), -1))))", "response": "Calculate the intensity of the Sersic light profile on a grid of radial coordinates."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef intensity_prime(self):\n return self.intensity_break * (2.0 ** (-self.gamma / self.alpha)) * np.exp(\n self.sersic_constant * (((2.0 ** (1.0 / self.alpha)) * self.radius_break) / self.effective_radius) ** (\n 1.0 / self.sersic_index))", "response": "Overall intensity normalisation in the rescaled Core - Sersic light profiles ( electrons per second"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncalculates the intensity of the cored - Sersic light profile on a grid of radial coordinates.", "response": "def intensities_from_grid_radii(self, grid_radii):\n \"\"\"Calculate the intensity of the cored-Sersic light profile on a grid of radial coordinates.\n\n Parameters\n ----------\n grid_radii : float\n The radial distance from the centre of the profile. for each coordinate on the grid.\n \"\"\"\n return np.multiply(np.multiply(self.intensity_prime, np.power(\n np.add(1, np.power(np.divide(self.radius_break, grid_radii), self.alpha)), (self.gamma / self.alpha))),\n np.exp(np.multiply(-self.sersic_constant,\n (np.power(np.divide(np.add(np.power(grid_radii, self.alpha), (\n self.radius_break ** self.alpha)),\n (self.effective_radius ** self.alpha)), (\n 1.0 / (self.alpha * self.sersic_index)))))))"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef luminosities_of_galaxies_within_circles_in_units(self, radius : dim.Length, unit_luminosity='eps', exposure_time=None):\n return list(map(lambda galaxy: galaxy.luminosity_within_circle_in_units(\n radius=radius, unit_luminosity=unit_luminosity, kpc_per_arcsec=self.kpc_per_arcsec,\n exposure_time=exposure_time),\n self.galaxies))", "response": "Compute the total luminosities of all galaxies within a circle of specified radius in units."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef luminosities_of_galaxies_within_ellipses_in_units(self, major_axis : dim.Length, unit_luminosity='eps',\n exposure_time=None):\n \"\"\"\n Compute the total luminosity of all galaxies in this plane within a ellipse of specified major-axis.\n\n The value returned by this integral is dimensionless, and a conversion factor can be specified to convert it \\\n to a physical value (e.g. the photometric zeropoint).\n\n See *galaxy.light_within_ellipse* and *light_profiles.light_within_ellipse* for details\n of how this is performed.\n\n Parameters\n ----------\n major_axis : float\n The major-axis radius of the ellipse.\n units_luminosity : str\n The units the luminosity is returned in (eps | counts).\n exposure_time : float\n The exposure time of the observation, which converts luminosity from electrons per second units to counts.\n \"\"\"\n return list(map(lambda galaxy: galaxy.luminosity_within_ellipse_in_units(\n major_axis=major_axis, unit_luminosity=unit_luminosity, kpc_per_arcsec=self.kpc_per_arcsec,\n exposure_time=exposure_time),\n self.galaxies))", "response": "Returns the total luminosities of all galaxies within a given major - axis."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute the total mass of all galaxies within a circle of specified radius.", "response": "def masses_of_galaxies_within_circles_in_units(self, radius : dim.Length, unit_mass='angular',\n critical_surface_density=None):\n \"\"\"Compute the total mass of all galaxies in this plane within a circle of specified radius.\n\n See *galaxy.angular_mass_within_circle* and *mass_profiles.angular_mass_within_circle* for details\n of how this is performed.\n\n Parameters\n ----------\n radius : float\n The radius of the circle to compute the dimensionless mass within.\n units_mass : str\n The units the mass is returned in (angular | solMass).\n critical_surface_density : float\n The critical surface mass density of the strong lens configuration, which converts mass from angulalr \\\n units to physical units (e.g. solar masses).\n \"\"\"\n return list(map(lambda galaxy: galaxy.mass_within_circle_in_units(\n radius=radius, unit_mass=unit_mass, kpc_per_arcsec=self.kpc_per_arcsec,\n critical_surface_density=critical_surface_density),\n self.galaxies))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncomputing the total mass of all galaxies within a given major - axis.", "response": "def masses_of_galaxies_within_ellipses_in_units(self, major_axis : dim.Length, unit_mass='angular',\n critical_surface_density=None):\n \"\"\"Compute the total mass of all galaxies in this plane within a ellipse of specified major-axis.\n\n See *galaxy.angular_mass_within_ellipse* and *mass_profiles.angular_mass_within_ellipse* for details \\\n of how this is performed.\n\n Parameters\n ----------\n major_axis : float\n The major-axis radius of the ellipse.\n units_luminosity : str\n The units the luminosity is returned in (eps | counts).\n exposure_time : float\n The exposure time of the observation, which converts luminosity from electrons per second units to counts.\n \"\"\"\n return list(map(lambda galaxy: galaxy.mass_within_ellipse_in_units(\n major_axis=major_axis, unit_mass=unit_mass, kpc_per_arcsec=self.kpc_per_arcsec,\n critical_surface_density=critical_surface_density),\n self.galaxies))"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ntraces this plane s grid_stacks to the next plane using its deflection angles.", "response": "def trace_grid_stack_to_next_plane(self):\n \"\"\"Trace this plane's grid_stacks to the next plane, using its deflection angles.\"\"\"\n\n def minus(grid, deflections):\n return grid - deflections\n\n return self.grid_stack.map_function(minus, self.deflection_stack)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef yticks(self):\n return np.linspace(np.amin(self.grid_stack.regular[:, 0]), np.amax(self.grid_stack.regular[:, 0]), 4)", "response": "Compute the y - axis ticks of this grid_stack used for plotting the y - axis ticks when visualizing an image \\"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef xticks(self):\n return np.linspace(np.amin(self.grid_stack.regular[:, 1]), np.amax(self.grid_stack.regular[:, 1]), 4)", "response": "Compute the x - axis ticks of this grid_stack used for plotting the x - axis ticks when visualizing an image"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ntracing the positions to the next plane.", "response": "def trace_to_next_plane(self):\n \"\"\"Trace the positions to the next plane.\"\"\"\n return list(map(lambda positions, deflections: np.subtract(positions, deflections),\n self.positions, self.deflections))"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nconverts a grid of arc second coordinates to a grid of 1D pixel values.", "response": "def grid_arcsec_to_grid_pixels(self, grid_arcsec):\n \"\"\"Convert a grid of (y,x) arc second coordinates to a grid of (y,x) pixel values. Pixel coordinates are \\\n returned as floats such that they include the decimal offset from each pixel's top-left corner.\n\n The pixel coordinate origin is at the top left corner of the grid, such that the pixel [0,0] corresponds to \\\n highest y arc-second coordinate value and lowest x arc-second coordinate.\n\n The arc-second coordinate origin is defined by the class attribute origin, and coordinates are shifted to this \\\n origin before computing their 1D grid pixel indexes.\n\n Parameters\n ----------\n grid_arcsec: ndarray\n A grid of (y,x) coordinates in arc seconds.\n \"\"\"\n return grid_util.grid_arcsec_1d_to_grid_pixels_1d(grid_arcsec_1d=grid_arcsec, shape=self.shape,\n pixel_scales=self.pixel_scales, origin=self.origin)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nconverts a grid of arc second coordinates to a grid of pixel centroids.", "response": "def grid_arcsec_to_grid_pixel_centres(self, grid_arcsec):\n \"\"\"Convert a grid of (y,x) arc second coordinates to a grid of (y,x) pixel values. Pixel coordinates are \\\n returned as integers such that they map directly to the pixel they are contained within.\n\n The pixel coordinate origin is at the top left corner of the grid, such that the pixel [0,0] corresponds to \\\n higher y arc-second coordinate value and lowest x arc-second coordinate.\n\n The arc-second coordinate origin is defined by the class attribute origin, and coordinates are shifted to this \\\n origin before computing their 1D grid pixel indexes.\n\n Parameters\n ----------\n grid_arcsec: ndarray\n The grid of (y,x) coordinates in arc seconds.\n \"\"\"\n return grid_util.grid_arcsec_1d_to_grid_pixel_centres_1d(grid_arcsec_1d=grid_arcsec,\n shape=self.shape,\n pixel_scales=self.pixel_scales,\n origin=self.origin).astype('int')"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nconverts a grid of arc second coordinates to a grid of 1D pixel indexes.", "response": "def grid_arcsec_to_grid_pixel_indexes(self, grid_arcsec):\n \"\"\"Convert a grid of (y,x) arc second coordinates to a grid of (y,x) pixel 1D indexes. Pixel coordinates are \\\n returned as integers such that they are the pixel from the top-left of the 2D grid going rights and then \\\n downwards.\n\n For example:\n\n The pixel at the top-left, whose 2D index is [0,0], corresponds to 1D index 0.\n The fifth pixel on the top row, whose 2D index is [0,5], corresponds to 1D index 4.\n The first pixel on the second row, whose 2D index is [0,1], has 1D index 10 if a row has 10 pixels.\n\n The arc-second coordinate origin is defined by the class attribute origin, and coordinates are shifted to this \\\n origin before computing their 1D grid pixel indexes.\n\n Parameters\n ----------\n grid_arcsec: ndarray\n The grid of (y,x) coordinates in arc seconds.\n \"\"\"\n return grid_util.grid_arcsec_1d_to_grid_pixel_indexes_1d(grid_arcsec_1d=grid_arcsec,\n shape=self.shape,\n pixel_scales=self.pixel_scales,\n origin=self.origin).astype('int')"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef grid_pixels_to_grid_arcsec(self, grid_pixels):\n return grid_util.grid_pixels_1d_to_grid_arcsec_1d(grid_pixels_1d=grid_pixels, shape=self.shape,\n pixel_scales=self.pixel_scales, origin=self.origin)", "response": "Convert a grid of y x coordinates to a 1D grid of arc second values."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef grid_2d(self):\n return grid_util.regular_grid_2d_from_shape_pixel_scales_and_origin(shape=self.shape,\n pixel_scales=self.pixel_scales,\n origin=self.origin)", "response": "The arc second - grid of every pixel."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a new instance of this class that shares all of this instances attributes with a new array.", "response": "def new_with_array(self, array):\n \"\"\"\n Parameters\n ----------\n array: ndarray\n An ndarray\n\n Returns\n -------\n new_array: ScaledSquarePixelArray\n A new instance of this class that shares all of this instances attributes with a new ndarray.\n \"\"\"\n arguments = vars(self)\n arguments.update({\"array\": array})\n if 'centre' in arguments:\n arguments.pop(\"centre\")\n return self.__class__(**arguments)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef from_fits_with_pixel_scale(cls, file_path, hdu, pixel_scale, origin=(0.0, 0.0)):\n return cls(array_util.numpy_array_2d_from_fits(file_path, hdu), pixel_scale, origin)", "response": "Loads the image from a. fits file."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef single_value(cls, value, shape, pixel_scale, origin=(0.0, 0.0)):\n array = np.ones(shape) * value\n return cls(array, pixel_scale, origin)", "response": "Creates an array filled with a single value"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef flatten(self, order='C'):\n return self.new_with_array(super(ScaledSquarePixelArray, self).flatten(order))", "response": "Returns a copy of this array flattened to 1D\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a new array with the 2D region of an array corresponding to the rectangle encompassing all unmasked values.", "response": "def zoomed_scaled_array_around_mask(self, mask, buffer=1):\n \"\"\"Extract the 2D region of an array corresponding to the rectangle encompassing all unmasked values.\n\n This is used to extract and visualize only the region of an image that is used in an analysis.\n\n Parameters\n ----------\n mask : mask.Mask\n The mask around which the scaled array is extracted.\n buffer : int\n The buffer of pixels around the extraction.\n \"\"\"\n return self.new_with_array(array=array_util.extracted_array_2d_from_array_2d_and_coordinates(\n array_2d=self, y0=mask.zoom_region[0]-buffer, y1=mask.zoom_region[1]+buffer,\n x0=mask.zoom_region[2]-buffer, x1=mask.zoom_region[3]+buffer))"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef resized_scaled_array_from_array(self, new_shape, new_centre_pixels=None, new_centre_arcsec=None):\n if new_centre_pixels is None and new_centre_arcsec is None:\n new_centre = (-1, -1) # In Numba, the input origin must be the same image type as the origin, thus we cannot\n # pass 'None' and instead use the tuple (-1, -1).\n elif new_centre_pixels is not None and new_centre_arcsec is None:\n new_centre = new_centre_pixels\n elif new_centre_pixels is None and new_centre_arcsec is not None:\n new_centre = self.arc_second_coordinates_to_pixel_coordinates(arc_second_coordinates=new_centre_arcsec)\n else:\n raise exc.DataException('You have supplied two centres (pixels and arc-seconds) to the resize scaled'\n 'array function')\n\n return self.new_with_array(array=array_util.resized_array_2d_from_array_2d_and_resized_shape(\n array_2d=self, resized_shape=new_shape, origin=new_centre))", "response": "resized the array to a new shape and at a new origin."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nloads the image from a. fits file.", "response": "def from_fits_with_pixel_scale(cls, file_path, hdu, pixel_scales, origin=(0.0, 0.0)):\n \"\"\"\n Loads the image from a .fits file.\n\n Parameters\n ----------\n file_path : str\n The full path of the fits file.\n hdu : int\n The HDU number in the fits file containing the hyper.\n pixel_scales: (float, float)\n The arc-second to pixel conversion factor of each pixel.\n \"\"\"\n return cls(array_util.numpy_array_2d_from_fits(file_path, hdu), pixel_scales, origin)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nfits lens data with a normal tracer and sensitivity tracer.", "response": "def fit_lens_data_with_sensitivity_tracers(lens_data, tracer_normal, tracer_sensitive):\n \"\"\"Fit lens data with a normal tracer and sensitivity tracer, to determine our sensitivity to a selection of \\ \n galaxy components. This factory automatically determines the type of fit based on the properties of the galaxies \\\n in the tracers.\n\n Parameters\n -----------\n lens_data : lens_data.LensData or lens_data.LensDataHyper\n The lens-images that is fitted.\n tracer_normal : ray_tracing.AbstractTracer\n A tracer whose galaxies have the same model components (e.g. light profiles, mass profiles) as the \\\n lens data that we are fitting.\n tracer_sensitive : ray_tracing.AbstractTracerNonStack\n A tracer whose galaxies have the same model components (e.g. light profiles, mass profiles) as the \\\n lens data that we are fitting, but also addition components (e.g. mass clumps) which we measure \\\n how sensitive we are too.\n \"\"\"\n\n if (tracer_normal.has_light_profile and tracer_sensitive.has_light_profile) and \\\n (not tracer_normal.has_pixelization and not tracer_sensitive.has_pixelization):\n return SensitivityProfileFit(lens_data=lens_data, tracer_normal=tracer_normal,\n tracer_sensitive=tracer_sensitive)\n\n elif (not tracer_normal.has_light_profile and not tracer_sensitive.has_light_profile) and \\\n (tracer_normal.has_pixelization and tracer_sensitive.has_pixelization):\n return SensitivityInversionFit(lens_data=lens_data, tracer_normal=tracer_normal,\n tracer_sensitive=tracer_sensitive)\n else:\n\n raise exc.FittingException('The sensitivity_fit routine did not call a SensitivityFit class - check the '\n 'properties of the tracers')"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef circular_anti_annular(cls, shape, pixel_scale, inner_radius_arcsec, outer_radius_arcsec, outer_radius_2_arcsec,\n centre=(0., 0.), invert=False):\n \"\"\"Setup a mask where unmasked pixels are outside an annulus of input inner and outer arc second radii, but \\\n within a second outer radius, and at a given centre.\n\n This mask there has two distinct unmasked regions (an inner circle and outer annulus), with an inner annulus \\\n of masked pixels.\n\n Parameters\n ----------\n shape : (int, int)\n The (y,x) shape of the mask in units of pixels.\n pixel_scale: float\n The arc-second to pixel conversion factor of each pixel.\n inner_radius_arcsec : float\n The radius (in arc seconds) of the inner circle inside of which pixels are unmasked.\n outer_radius_arcsec : float\n The radius (in arc seconds) of the outer circle within which pixels are masked and outside of which they \\\n are unmasked.\n outer_radius_2_arcsec : float\n The radius (in arc seconds) of the second outer circle within which pixels are unmasked and outside of \\\n which they masked.\n centre: (float, float)\n The centre of the anti-annulus used to mask pixels.\n \"\"\"\n mask = mask_util.mask_circular_anti_annular_from_shape_pixel_scale_and_radii(shape, pixel_scale, inner_radius_arcsec,\n outer_radius_arcsec,\n outer_radius_2_arcsec, centre)\n if invert: mask = np.invert(mask)\n return cls(array=mask.astype('bool'), pixel_scale=pixel_scale)", "response": "Setup a mask where unmasked pixels are within an annulus of input inner and outer arc second radii and at a given centre."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef elliptical_annular(cls, shape, pixel_scale,inner_major_axis_radius_arcsec, inner_axis_ratio, inner_phi,\n outer_major_axis_radius_arcsec, outer_axis_ratio, outer_phi, centre=(0.0, 0.0),\n invert=False):\n \"\"\"Setup a mask where unmasked pixels are within an elliptical annulus of input inner and outer arc second \\\n major-axis and centre.\n\n Parameters\n ----------\n shape: (int, int)\n The (y,x) shape of the mask in units of pixels.\n pixel_scale: float\n The arc-second to pixel conversion factor of each pixel.\n inner_major_axis_radius_arcsec : float\n The major-axis (in arc seconds) of the inner ellipse within which pixels are masked.\n inner_axis_ratio : float\n The axis-ratio of the inner ellipse within which pixels are masked.\n inner_phi : float\n The rotation angle of the inner ellipse within which pixels are masked, (counter-clockwise from the \\\n positive x-axis).\n outer_major_axis_radius_arcsec : float\n The major-axis (in arc seconds) of the outer ellipse within which pixels are unmasked.\n outer_axis_ratio : float\n The axis-ratio of the outer ellipse within which pixels are unmasked.\n outer_phi : float\n The rotation angle of the outer ellipse within which pixels are unmasked, (counter-clockwise from the \\\n positive x-axis).\n centre: (float, float)\n The centre of the elliptical annuli used to mask pixels.\n \"\"\"\n mask = mask_util.mask_elliptical_annular_from_shape_pixel_scale_and_radius(shape, pixel_scale,\n inner_major_axis_radius_arcsec, inner_axis_ratio, inner_phi,\n outer_major_axis_radius_arcsec, outer_axis_ratio, outer_phi, centre)\n if invert: mask = np.invert(mask)\n return cls(array=mask.astype('bool'), pixel_scale=pixel_scale)", "response": "Setup a mask where unmasked pixels are within an elliptical annular shape."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef map_2d_array_to_masked_1d_array(self, array_2d):\n if array_2d is None or isinstance(array_2d, float):\n return array_2d\n return mapping_util.map_2d_array_to_masked_1d_array_from_array_2d_and_mask(self, array_2d)", "response": "Map a 2D array to a masked 1D array."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncomputes a blurring mask for the given shape of the PSF.", "response": "def blurring_mask_for_psf_shape(self, psf_shape):\n \"\"\"Compute a blurring mask, which represents all masked pixels whose light will be blurred into unmasked \\\n pixels via PSF convolution (see grid_stack.RegularGrid.blurring_grid_from_mask_and_psf_shape).\n\n Parameters\n ----------\n psf_shape : (int, int)\n The shape of the psf which defines the blurring region (e.g. the shape of the PSF)\n \"\"\"\n\n if psf_shape[0] % 2 == 0 or psf_shape[1] % 2 == 0:\n raise exc.MaskException(\"psf_size of exterior region must be odd\")\n\n blurring_mask = mask_util.mask_blurring_from_mask_and_psf_shape(self, psf_shape)\n\n return Mask(blurring_mask, self.pixel_scale)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef zoom_region(self):\n\n # Have to convert mask to bool for invert function to work.\n where = np.array(np.where(np.invert(self.astype('bool'))))\n y0, x0 = np.amin(where, axis=1)\n y1, x1 = np.amax(where, axis=1)\n return [y0, y1+1, x0, x1+1]", "response": "Returns the zoomed rectangular region corresponding to the square encompassing all unmasked values."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncompute the hyper vector D from a blurred mapping matrix and the 1D image and noise - map.", "response": "def data_vector_from_blurred_mapping_matrix_and_data(blurred_mapping_matrix, image_1d, noise_map_1d):\n \"\"\"Compute the hyper vector *D* from a blurred mapping matrix *f* and the 1D image *d* and 1D noise-map *\\sigma* \\\n (see Warren & Dye 2003).\n \n Parameters\n -----------\n blurred_mapping_matrix : ndarray\n The matrix representing the blurred mappings between sub-grid pixels and pixelization pixels.\n image_1d : ndarray\n Flattened 1D array of the observed image the inversion is fitting.\n noise_map_1d : ndarray\n Flattened 1D array of the noise-map used by the inversion during the fit.\n \"\"\"\n\n mapping_shape = blurred_mapping_matrix.shape\n\n data_vector = np.zeros(mapping_shape[1])\n\n for image_index in range(mapping_shape[0]):\n for pix_index in range(mapping_shape[1]):\n data_vector[pix_index] += image_1d[image_index] * \\\n blurred_mapping_matrix[image_index, pix_index] / (noise_map_1d[image_index] ** 2.0)\n\n return data_vector"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute the curvature matrix F from a blurred mapping matrix F and the 1D noise - map F.", "response": "def curvature_matrix_from_blurred_mapping_matrix(blurred_mapping_matrix, noise_map_1d):\n \"\"\"Compute the curvature matrix *F* from a blurred mapping matrix *f* and the 1D noise-map *\\sigma* \\\n (see Warren & Dye 2003).\n\n Parameters\n -----------\n blurred_mapping_matrix : ndarray\n The matrix representing the blurred mappings between sub-grid pixels and pixelization pixels.\n noise_map_1d : ndarray\n Flattened 1D array of the noise-map used by the inversion during the fit.\n \"\"\"\n\n flist = np.zeros(blurred_mapping_matrix.shape[0])\n iflist = np.zeros(blurred_mapping_matrix.shape[0], dtype='int')\n return curvature_matrix_from_blurred_mapping_matrix_jit(blurred_mapping_matrix, noise_map_1d, flist, iflist)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing the curvature matrix from a blurred mapping matrix and the 1D noise - map.", "response": "def curvature_matrix_from_blurred_mapping_matrix_jit(blurred_mapping_matrix, noise_map_1d, flist, iflist):\n \"\"\"Compute the curvature matrix *F* from a blurred mapping matrix *f* and the 1D noise-map *\\sigma* \\\n (see Warren & Dye 2003).\n\n Parameters\n -----------\n blurred_mapping_matrix : ndarray\n The matrix representing the blurred mappings between sub-grid pixels and pixelization pixels.\n noise_map_1d : ndarray\n Flattened 1D array of the noise-map used by the inversion during the fit.\n flist : ndarray\n NumPy array of floats used to store mappings for efficienctly calculation.\n iflist : ndarray\n NumPy array of integers used to store mappings for efficienctly calculation.\n \"\"\"\n curvature_matrix = np.zeros((blurred_mapping_matrix.shape[1], blurred_mapping_matrix.shape[1]))\n\n for image_index in range(blurred_mapping_matrix.shape[0]):\n index = 0\n for pixel_index in range(blurred_mapping_matrix.shape[1]):\n if blurred_mapping_matrix[image_index, pixel_index] > 0.0:\n flist[index] = blurred_mapping_matrix[image_index, pixel_index] / noise_map_1d[image_index]\n iflist[index] = pixel_index\n index += 1\n\n if index > 0:\n for i1 in range(index):\n for j1 in range(index):\n ix = iflist[i1]\n iy = iflist[j1]\n curvature_matrix[ix, iy] += flist[i1] * flist[j1]\n\n for i in range(blurred_mapping_matrix.shape[1]):\n for j in range(blurred_mapping_matrix.shape[1]):\n curvature_matrix[i, j] = curvature_matrix[j, i]\n\n return curvature_matrix"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncomputing the reconstructed hyper vector from the blurrred mapping matrix *f* and solution vector *S*. Parameters ----------- blurred_mapping_matrix : ndarray The matrix representing the blurred mappings between sub-grid pixels and pixelization pixels.", "response": "def reconstructed_data_vector_from_blurred_mapping_matrix_and_solution_vector(blurred_mapping_matrix, solution_vector):\n \"\"\" Compute the reconstructed hyper vector from the blurrred mapping matrix *f* and solution vector *S*.\n\n Parameters\n -----------\n blurred_mapping_matrix : ndarray\n The matrix representing the blurred mappings between sub-grid pixels and pixelization pixels.\n\n \"\"\"\n reconstructed_data_vector = np.zeros(blurred_mapping_matrix.shape[0])\n for i in range(blurred_mapping_matrix.shape[0]):\n for j in range(solution_vector.shape[0]):\n reconstructed_data_vector[i] += solution_vector[j] * blurred_mapping_matrix[i, j]\n\n return reconstructed_data_vector"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef regularization_term(self):\n return np.matmul(self.solution_vector.T, np.matmul(self.regularization_matrix, self.solution_vector))", "response": "Compute the regularization term of an inversion."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef log_determinant_of_matrix_cholesky(matrix):\n try:\n return 2.0 * np.sum(np.log(np.diag(np.linalg.cholesky(matrix))))\n except np.linalg.LinAlgError:\n raise exc.InversionException()", "response": "This function computes the log determinant of a matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns a list of all constant light profiles.", "response": "def constant_light_profiles(self):\n \"\"\"\n Returns\n -------\n light_profiles: [light_profiles.LightProfile]\n Light profiles with set variables\n \"\"\"\n return [value for value in self.__dict__.values() if galaxy.is_light_profile(value)]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a list of mass profiles with set variables.", "response": "def constant_mass_profiles(self):\n \"\"\"\n Returns\n -------\n mass_profiles: [mass_profiles.MassProfile]\n Mass profiles with set variables\n \"\"\"\n return [value for value in self.__dict__.values() if galaxy.is_mass_profile(value)]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn a list of the prior models attached to this galaxy.", "response": "def prior_models(self):\n \"\"\"\n Returns\n -------\n prior_models: [model_mapper.PriorModel]\n A list of the prior models (e.g. variable profiles) attached to this galaxy prior\n \"\"\"\n return [value for _, value in\n filter(lambda t: isinstance(t[1], pm.PriorModel), self.__dict__.items())]"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef profile_prior_model_dict(self):\n return {key: value for key, value in\n filter(lambda t: isinstance(t[1], pm.PriorModel) and is_profile_class(t[1].cls),\n self.__dict__.items())}", "response": "Returns a dictionary mapping instance variable names to variable profiles."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef constant_profile_dict(self):\n return {key: value for key, value in self.__dict__.items() if\n galaxy.is_light_profile(value) or galaxy.is_mass_profile(value)}", "response": "Returns a dictionary mapping instance variable names to profile values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef prior_class_dict(self):\n return {prior: cls for prior_model in self.prior_models for prior, cls in\n prior_model.prior_class_dict.items()}", "response": "Returns a dictionary mapping priors to the class associated with the prior model."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating an instance of the associated class for a set of arguments.", "response": "def instance_for_arguments(self, arguments):\n \"\"\"\n Create an instance of the associated class for a set of arguments\n\n Parameters\n ----------\n arguments: {Prior: value}\n Dictionary mapping_matrix priors to attribute analysis_path and value pairs\n\n Returns\n -------\n An instance of the class\n \"\"\"\n profiles = {**{key: value.instance_for_arguments(arguments)\n for key, value\n in self.profile_prior_model_dict.items()}, **self.constant_profile_dict}\n\n try:\n redshift = self.redshift.instance_for_arguments(arguments)\n except AttributeError:\n redshift = self.redshift\n pixelization = self.pixelization.instance_for_arguments(arguments) \\\n if isinstance(self.pixelization, pm.PriorModel) \\\n else self.pixelization\n regularization = self.regularization.instance_for_arguments(arguments) \\\n if isinstance(self.regularization, pm.PriorModel) \\\n else self.regularization\n hyper_galaxy = self.hyper_galaxy.instance_for_arguments(arguments) \\\n if isinstance(self.hyper_galaxy, pm.PriorModel) \\\n else self.hyper_galaxy\n\n return galaxy.Galaxy(redshift=redshift, pixelization=pixelization, regularization=regularization,\n hyper_galaxy=hyper_galaxy, **profiles)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef gaussian_prior_model_for_arguments(self, arguments):\n new_model = copy.deepcopy(self)\n\n for key, value in filter(lambda t: isinstance(t[1], pm.PriorModel), self.__dict__.items()):\n setattr(new_model, key, value.gaussian_prior_model_for_arguments(arguments))\n\n return new_model", "response": "Create a new GalaxyModel from a set of arguments replacing the priors of some of this Galaxy model with new arguments."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nplot the observed image of the ccd data.", "response": "def plot_image(\n image, plot_origin=True, mask=None, extract_array_from_mask=False, zoom_around_mask=False,\n should_plot_border=False, positions=None, as_subplot=False,\n units='arcsec', kpc_per_arcsec=None, figsize=(7, 7), aspect='square',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n title='Image', titlesize=16, xlabelsize=16, ylabelsize=16, xyticksize=16,\n mask_pointsize=10, position_pointsize=30, grid_pointsize=1,\n output_path=None, output_format='show', output_filename='image'):\n \"\"\"Plot the observed image of the ccd data.\n\n Set *autolens.data.array.plotters.array_plotters* for a description of all input parameters not described below.\n\n Parameters\n -----------\n image : ScaledSquarePixelArray\n The image of the data.\n plot_origin : True\n If true, the origin of the data's coordinate system is plotted as a 'x'.\n image_plane_pix_grid : ndarray or data.array.grid_stacks.PixGrid\n If an adaptive pixelization whose pixels are formed by tracing pixels from the data, this plots those pixels \\\n over the immage.\n \"\"\"\n origin = get_origin(array=image, plot_origin=plot_origin)\n\n array_plotters.plot_array(\n array=image, origin=origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n should_plot_border=should_plot_border, positions=positions, as_subplot=as_subplot,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n title=title, titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize, position_pointsize=position_pointsize, grid_pointsize=grid_pointsize,\n output_path=output_path, output_format=output_format, output_filename=output_filename)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef plot_ccd_subplot(\n ccd_data, plot_origin=True, mask=None, extract_array_from_mask=False, zoom_around_mask=False,\n should_plot_border=False, positions=None,\n units='arcsec', kpc_per_arcsec=None, figsize=None, aspect='square',\n cmap='jet', norm='linear', norm_min=None, norm_max=None, linthresh=0.05, linscale=0.01,\n cb_ticksize=10, cb_fraction=0.047, cb_pad=0.01, cb_tick_values=None, cb_tick_labels=None,\n titlesize=10, xlabelsize=10, ylabelsize=10, xyticksize=10,\n mask_pointsize=10, position_pointsize=30, grid_pointsize=1,\n output_path=None, output_filename='ccd_data', output_format='show'):\n \"\"\"Plot the ccd data as a sub-plot of all its quantites (e.g. the data, noise_map-map, PSF, Signal-to_noise-map, \\\n etc).\n\n Set *autolens.data.array.plotters.array_plotters* for a description of all innput parameters not described below.\n\n Parameters\n -----------\n ccd_data : data.CCDData\n The ccd data, which includes the observed data, noise_map-map, PSF, signal-to-noise_map-map, etc.\n plot_origin : True\n If true, the origin of the data's coordinate system is plotted as a 'x'.\n image_plane_pix_grid : ndarray or data.array.grid_stacks.PixGrid\n If an adaptive pixelization whose pixels are formed by tracing pixels from the data, this plots those pixels \\\n over the immage.\n ignore_config : bool\n If *False*, the config file general.ini is used to determine whether the subpot is plotted. If *True*, the \\\n config file is ignored.\n \"\"\"\n\n rows, columns, figsize_tool = plotter_util.get_subplot_rows_columns_figsize(number_subplots=6)\n\n if figsize is None:\n figsize = figsize_tool\n\n plt.figure(figsize=figsize)\n plt.subplot(rows, columns, 1)\n\n plot_image(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, should_plot_border=should_plot_border, positions=positions, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize, position_pointsize=position_pointsize, grid_pointsize=grid_pointsize,\n output_path=output_path, output_format=output_format)\n\n plt.subplot(rows, columns, 2)\n\n plot_noise_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize,\n output_path=output_path, output_format=output_format)\n\n plt.subplot(rows, columns, 3)\n\n plot_psf(\n ccd_data=ccd_data, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n output_path=output_path, output_format=output_format)\n\n plt.subplot(rows, columns, 4)\n\n plot_signal_to_noise_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad, \n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize,\n output_path=output_path, output_format=output_format)\n\n plt.subplot(rows, columns, 5)\n\n plot_absolute_signal_to_noise_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad,\n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize,\n output_path=output_path, output_format=output_format)\n\n plt.subplot(rows, columns, 6)\n\n plot_potential_chi_squared_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, as_subplot=True,\n units=units, kpc_per_arcsec=kpc_per_arcsec, figsize=figsize, aspect=aspect,\n cmap=cmap, norm=norm, norm_min=norm_min, norm_max=norm_max, linthresh=linthresh, linscale=linscale,\n cb_ticksize=cb_ticksize, cb_fraction=cb_fraction, cb_pad=cb_pad,\n cb_tick_values=cb_tick_values, cb_tick_labels=cb_tick_labels,\n titlesize=titlesize, xlabelsize=xlabelsize, ylabelsize=ylabelsize, xyticksize=xyticksize,\n mask_pointsize=mask_pointsize,\n output_path=output_path, output_format=output_format)\n\n plotter_util.output_subplot_array(output_path=output_path, output_filename=output_filename,\n output_format=output_format)\n\n plt.close()", "response": "Plot the ccd data as a sub - plot of all quantites."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nplot the individual of the data for the specified ccd.", "response": "def plot_ccd_individual(\n ccd_data, plot_origin=True, mask=None, extract_array_from_mask=False, zoom_around_mask=False, positions=None,\n should_plot_image=False,\n should_plot_noise_map=False,\n should_plot_psf=False,\n should_plot_signal_to_noise_map=False,\n should_plot_absolute_signal_to_noise_map=False,\n should_plot_potential_chi_squared_map=False,\n units='arcsec',\n output_path=None, output_format='png'):\n \"\"\"Plot each attribute of the ccd data as individual figures one by one (e.g. the data, noise_map-map, PSF, \\\n Signal-to_noise-map, etc).\n\n Set *autolens.data.array.plotters.array_plotters* for a description of all innput parameters not described below.\n\n Parameters\n -----------\n ccd_data : data.CCDData\n The ccd data, which includes the observed data, noise_map-map, PSF, signal-to-noise_map-map, etc.\n plot_origin : True\n If true, the origin of the data's coordinate system is plotted as a 'x'.\n \"\"\"\n\n if should_plot_image:\n\n plot_image(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask, positions=positions,\n units=units,\n output_path=output_path, output_format=output_format)\n\n if should_plot_noise_map:\n\n plot_noise_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)\n\n if should_plot_psf:\n\n plot_psf(\n ccd_data=ccd_data, plot_origin=plot_origin,\n output_path=output_path, output_format=output_format)\n\n if should_plot_signal_to_noise_map:\n\n plot_signal_to_noise_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)\n\n if should_plot_absolute_signal_to_noise_map:\n\n plot_absolute_signal_to_noise_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)\n \n if should_plot_potential_chi_squared_map:\n\n plot_potential_chi_squared_map(\n ccd_data=ccd_data, plot_origin=plot_origin, mask=mask, extract_array_from_mask=extract_array_from_mask,\n zoom_around_mask=zoom_around_mask,\n units=units,\n output_path=output_path, output_format=output_format)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef norm_and_check(source_tree, requested):\n if os.path.isabs(requested):\n raise ValueError(\"paths must be relative\")\n\n abs_source = os.path.abspath(source_tree)\n abs_requested = os.path.normpath(os.path.join(abs_source, requested))\n # We have to use commonprefix for Python 2.7 compatibility. So we\n # normalise case to avoid problems because commonprefix is a character\n # based comparison :-(\n norm_source = os.path.normcase(abs_source)\n norm_requested = os.path.normcase(abs_requested)\n if os.path.commonprefix([norm_source, norm_requested]) != norm_source:\n raise ValueError(\"paths must be inside source tree\")\n\n return abs_requested", "response": "Normalise and check a backend path."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntests if a file is located within the given directory.", "response": "def contained_in(filename, directory):\n \"\"\"Test if a file is located within the given directory.\"\"\"\n filename = os.path.normcase(os.path.abspath(filename))\n directory = os.path.normcase(os.path.abspath(directory))\n return os.path.commonprefix([filename, directory]) == directory"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nfinding and load the build backend", "response": "def _build_backend():\n \"\"\"Find and load the build backend\"\"\"\n # Add in-tree backend directories to the front of sys.path.\n backend_path = os.environ.get('PEP517_BACKEND_PATH')\n if backend_path:\n extra_pathitems = backend_path.split(os.pathsep)\n sys.path[:0] = extra_pathitems\n\n ep = os.environ['PEP517_BUILD_BACKEND']\n mod_path, _, obj_path = ep.partition(':')\n try:\n obj = import_module(mod_path)\n except ImportError:\n raise BackendUnavailable(traceback.format_exc())\n\n if backend_path:\n if not any(\n contained_in(obj.__file__, path)\n for path in extra_pathitems\n ):\n raise BackendInvalid(\"Backend was not loaded from backend-path\")\n\n if obj_path:\n for path_part in obj_path.split('.'):\n obj = getattr(obj, path_part)\n return obj"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ninvoke the mandatory build_sdist hook.", "response": "def build_sdist(sdist_directory, config_settings):\n \"\"\"Invoke the mandatory build_sdist hook.\"\"\"\n backend = _build_backend()\n try:\n return backend.build_sdist(sdist_directory, config_settings)\n except getattr(backend, 'UnsupportedOperation', _DummyException):\n raise GotUnsupportedOperation(traceback.format_exc())"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nprint errors. Stop travis-ci from leaking api keys :param e: The error :return: None", "response": "def log_error(self, e):\r\n \"\"\"\r\n Print errors. Stop travis-ci from leaking api keys\r\n\r\n :param e: The error\r\n :return: None\r\n \"\"\"\r\n\r\n if not environ.get('CI'):\r\n self.log_function(e)\r\n if hasattr(e, 'response') and hasattr(e.response, 'text'):\r\n self.log_function(e.response.text)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nsleeping between requests but don t force asynchronous code to wait", "response": "def _sleep(self, seconds):\r\n \"\"\"\r\n Sleep between requests, but don't force asynchronous code to wait\r\n\r\n :param seconds: The number of seconds to sleep\r\n :return: None\r\n \"\"\"\r\n for _ in range(int(seconds)):\r\n if not self.force_stop:\r\n sleep(1)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get(self, *args, **kwargs):\r\n\r\n \"\"\"\r\n An interface for get requests that handles errors more gracefully to\r\n prevent data loss\r\n \"\"\"\r\n\r\n try:\r\n req_func = self.session.get if self.session else requests.get\r\n req = req_func(*args, **kwargs)\r\n req.raise_for_status()\r\n self.failed_last = False\r\n return req\r\n\r\n except requests.exceptions.RequestException as e:\r\n self.log_error(e)\r\n for i in range(1, self.num_retries):\r\n sleep_time = self.retry_rate * i\r\n self.log_function(\"Retrying in %s seconds\" % sleep_time)\r\n self._sleep(sleep_time)\r\n try:\r\n req = requests.get(*args, **kwargs)\r\n req.raise_for_status()\r\n self.log_function(\"New request successful\")\r\n return req\r\n except requests.exceptions.RequestException:\r\n self.log_function(\"New request failed\")\r\n\r\n # Allows for the api to ignore one potentially bad request\r\n if not self.failed_last:\r\n self.failed_last = True\r\n raise ApiError(e)\r\n else:\r\n raise FatalApiError(e)", "response": "Returns a new instance of the class\r\n with the specified parameters."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the node and edge info for a given node and edge.", "response": "def node_edge(self, node, edge, fields=None, params=None):\r\n\r\n \"\"\"\r\n\r\n :param node:\r\n :param edge:\r\n :param fields:\r\n :param params:\r\n :return:\r\n \"\"\"\r\n if fields:\r\n fields = \",\".join(fields)\r\n\r\n parameters = {\"fields\": fields,\r\n \"access_token\": self.key}\r\n parameters = self.merge_params(parameters, params)\r\n\r\n return self.api_call('%s/%s' % (node, edge), parameters)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef post(self, post_id, fields=None, **params):\r\n\r\n \"\"\"\r\n\r\n :param post_id:\r\n :param fields:\r\n :param params:\r\n :return:\r\n \"\"\"\r\n if fields:\r\n fields = \",\".join(fields)\r\n\r\n parameters = {\"fields\": fields,\r\n \"access_token\": self.key}\r\n parameters = self.merge_params(parameters, params)\r\n\r\n return self.api_call('%s' % post_id, parameters)", "response": "Send a POST request to the server."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef page_posts(self, page_id, after='', post_type=\"posts\",\r\n include_hidden=False, fields=None, **params):\r\n\r\n \"\"\"\r\n\r\n :param page_id:\r\n :param after:\r\n :param post_type: Can be 'posts', 'feed', 'tagged', 'promotable_posts'\r\n :param include_hidden:\r\n :param fields:\r\n :param params:\r\n :return:\r\n \"\"\"\r\n if fields:\r\n fields = \",\".join(fields)\r\n\r\n parameters = {\"access_token\": self.key,\r\n \"after\": after,\r\n \"fields\": fields,\r\n \"include_hidden\": include_hidden}\r\n parameters = self.merge_params(parameters, params)\r\n\r\n return self.api_call('%s/%s' % (page_id, post_type), parameters)", "response": "Returns a list of all the posts in a page."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\npost comments to a specific post.", "response": "def post_comments(self, post_id, after='', order=\"chronological\",\r\n filter=\"stream\", fields=None, **params):\r\n\r\n \"\"\"\r\n\r\n :param post_id:\r\n :param after:\r\n :param order: Can be 'ranked', 'chronological', 'reverse_chronological'\r\n :param filter: Can be 'stream', 'toplevel'\r\n :param fields: Can be 'id', 'application', 'attachment', 'can_comment',\r\n 'can_remove', 'can_hide', 'can_like', 'can_reply_privately', 'comments',\r\n 'comment_count', 'created_time', 'from', 'likes', 'like_count',\r\n 'live_broadcast_timestamp', 'message', 'message_tags', 'object',\r\n 'parent', 'private_reply_conversation', 'user_likes'\r\n :param params:\r\n :return:\r\n \"\"\"\r\n if fields:\r\n fields = \",\".join(fields)\r\n\r\n parameters = {\"access_token\": self.key,\r\n \"after\": after,\r\n \"order\": order,\r\n \"fields\": fields,\r\n \"filter\": filter}\r\n parameters = self.merge_params(parameters, params)\r\n\r\n return self.api_call('%s/comments' % post_id, parameters)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef flatten(dictionary, parent_key=False, separator='.'):\r\n\r\n items = []\r\n for key, value in dictionary.items():\r\n new_key = str(parent_key) + separator + key if parent_key else key\r\n if isinstance(value, collections.MutableMapping):\r\n items.extend(flatten(value, new_key, separator).items())\r\n elif isinstance(value, list):\r\n for k, v in enumerate(value):\r\n items.extend(flatten({str(k): v}, new_key).items())\r\n else:\r\n items.append((new_key, value))\r\n return dict(items)", "response": "Converts a nested dictionary into a flattened dictionary."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fill_gaps(list_dicts):\r\n\r\n field_names = [] # != set bc. preserving order is better for output\r\n for datum in list_dicts:\r\n for key in datum.keys():\r\n if key not in field_names:\r\n field_names.append(key)\r\n for datum in list_dicts:\r\n for key in field_names:\r\n if key not in datum:\r\n datum[key] = ''\r\n return list(field_names), list_dicts", "response": "Fill gaps in a list of dictionaries. Add empty keys to dictionaries in that don t contain other entries"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef to_csv(data, field_names=None, filename='data.csv',\r\n overwrite=True,\r\n write_headers=True, append=False, flat=True,\r\n primary_fields=None, sort_fields=True):\r\n \"\"\"\r\n DEPRECATED Write a list of dicts to a csv file\r\n\r\n :param data: List of dicts\r\n :param field_names: The list column names\r\n :param filename: The name of the file\r\n :param overwrite: Overwrite the file if exists\r\n :param write_headers: Write the headers to the csv file\r\n :param append: Write new rows if the file exists\r\n :param flat: Flatten the dictionary before saving\r\n :param primary_fields: The first columns of the csv file\r\n :param sort_fields: Sort the field names alphabetically\r\n :return: None\r\n \"\"\"\r\n\r\n # Don't overwrite if not specified\r\n if not overwrite and path.isfile(filename):\r\n raise FileExistsError('The file already exists')\r\n\r\n # Replace file if append not specified\r\n write_type = 'w' if not append else 'a'\r\n\r\n # Flatten if flat is specified, or there are no predefined field names\r\n if flat or not field_names:\r\n data = [flatten(datum) for datum in data]\r\n\r\n # Fill in gaps between dicts with empty string\r\n if not field_names:\r\n field_names, data = fill_gaps(data)\r\n\r\n # Sort fields if specified\r\n if sort_fields:\r\n field_names.sort()\r\n\r\n # If there are primary fields, move the field names to the front and sort\r\n # based on first field\r\n if primary_fields:\r\n for key in primary_fields[::-1]:\r\n field_names.insert(0, field_names.pop(field_names.index(key)))\r\n\r\n data = sorted(data, key=lambda k: k[field_names[0]], reverse=True)\r\n\r\n # Write the file\r\n with open(filename, write_type, encoding='utf-8') as f:\r\n writer = csv.DictWriter(f, fieldnames=field_names, lineterminator='\\n')\r\n if not append or write_headers:\r\n writer.writeheader()\r\n\r\n # Write rows containing fields in field names\r\n for datum in data:\r\n for key in list(datum.keys()):\r\n if key not in field_names:\r\n del datum[key]\r\n elif type(datum[key]) is str:\r\n datum[key] = datum[key].strip()\r\n\r\n datum[key] = str(datum[key])\r\n\r\n writer.writerow(datum)", "response": "Write a list of dicts to a CSV file."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites an object to a json file", "response": "def to_json(data, filename='data.json', indent=4):\r\n \"\"\"\r\n Write an object to a json file\r\n\r\n :param data: The object\r\n :param filename: The name of the file\r\n :param indent: The indentation of the file\r\n :return: None\r\n \"\"\"\r\n\r\n with open(filename, 'w') as f:\r\n f.write(json.dumps(data, indent=indent))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ndownload and save a file at path Taxonomy", "response": "def save_file(filename, source, folder=\"Downloads\"):\r\n \"\"\"\r\n Download and save a file at path\r\n\r\n :param filename: The name of the file\r\n :param source: The location of the resource online\r\n :param folder: The directory the file will be saved in\r\n :return: None\r\n \"\"\"\r\n\r\n r = requests.get(source, stream=True)\r\n if r.status_code == 200:\r\n if not path.isdir(folder):\r\n makedirs(folder, exist_ok=True)\r\n with open(\"%s/%s\" % (folder, filename), 'wb') as f:\r\n for chunk in r:\r\n f.write(chunk)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nconverts frames from a tar file to a single video file.", "response": "def convert_frames_to_video(tar_file_path, output_path=\"output.mp4\", framerate=60, overwrite=False):\n \"\"\"\n Try to convert a tar file containing a sequence of frames saved by the\n meshcat viewer into a single video file.\n\n This relies on having `ffmpeg` installed on your system.\n \"\"\"\n output_path = os.path.abspath(output_path)\n if os.path.isfile(output_path) and not overwrite:\n raise ValueError(\"The output path {:s} already exists. To overwrite that file, you can pass overwrite=True to this function.\".format(output_path))\n with tempfile.TemporaryDirectory() as tmp_dir:\n with tarfile.open(tar_file_path) as tar:\n tar.extractall(tmp_dir)\n args = [\"ffmpeg\",\n \"-r\", str(framerate),\n \"-i\", r\"%07d.png\",\n \"-vcodec\", \"libx264\",\n \"-preset\", \"slow\",\n \"-crf\", \"18\"]\n if overwrite:\n args.append(\"-y\")\n args.append(output_path)\n try:\n subprocess.check_call(args, cwd=tmp_dir)\n except subprocess.CalledProcessError as e:\n print(\"\"\"\nCould not call `ffmpeg` to convert your frames into a video.\nIf you want to convert the frames manually, you can extract the\n.tar archive into a directory, cd to that directory, and run:\nffmpeg -r 60 -i %07d.png \\\\\\n\\t -vcodec libx264 \\\\\\n\\t -preset slow \\\\\\n\\t -crf 18 \\\\\\n\\t output.mp4\n \"\"\")\n raise\n print(\"Saved output as {:s}\".format(output_path))\n return output_path"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef toJSON(self):\n return {\"id\": self.id,\n \"compile\": self.compile,\n \"position\": self.position,\n \"version\": self.version}", "response": "Get a json dict of the attributes of this object."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _fix_docs(this_abc, child_class):\n # After python 3.5, this is basically handled automatically\n if sys.version_info >= (3, 5):\n return child_class\n\n if not issubclass(child_class, this_abc):\n raise KappaError('Cannot fix docs of class that is not decendent.')\n\n # This method is modified from solution given in\n # https://stackoverflow.com/a/8101598/8863865\n for name, child_func in vars(child_class).items():\n if callable(child_func) and not child_func.__doc__:\n if name in this_abc.__abstractmethods__:\n parent_func = getattr(this_abc, name)\n child_func.__doc__ = parent_func.__doc__\n return child_class", "response": "Fix the documentation of the api methods."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef add_model_string(self, model_str, position=1, file_id=None):\n if file_id is None:\n file_id = self.make_unique_id('inlined_input')\n ret_data = self.file_create(File.from_string(model_str, position,\n file_id))\n return ret_data", "response": "Add a kappa model given in a string to the project."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef add_model_file(self, model_fpath, position=1, file_id=None):\n if file_id is None:\n file_id = self.make_unique_id('file_input')\n ret_data = self.file_create(File.from_file(model_fpath, position,\n file_id))\n return ret_data", "response": "Add a kappa model from a file at given path to the project."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef set_default_sim_param(self, *args, **kwargs):\n if len(args) is 1 and isinstance(args[0], SimulationParameter):\n self.__default_param = args[0]\n else:\n self.__default_param = SimulationParameter(*args, **kwargs)\n return", "response": "Set the default simulation parameter."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get_is_sim_running(self):\n sim_info = self.simulation_info()\n try:\n progress_info = sim_info['simulation_info_progress']\n ret = progress_info['simulation_progress_is_running']\n except KeyError: # Simulation has not been created.\n ret = False\n return ret", "response": "Check if the current simulation is running."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nblocks until the simulation is done or timeout seconds exceeded.", "response": "def wait_for_simulation_stop(self, timeout=None):\n \"\"\"Block until the simulation is done or timeout seconds exceeded.\n\n If the simulation stops before timeout, siminfo is returned.\n \"\"\"\n start = datetime.now()\n while self.get_is_sim_running():\n sleep(0.5)\n if timeout is not None:\n if (datetime.now() - start).seconds >= timeout:\n ret = None\n break\n else:\n ret = self.simulation_info()\n return ret"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef available_devices():\r\n devices = {}\r\n pA = pyaudio.PyAudio()\r\n device_string = str()\r\n for k in range(pA.get_device_count()):\r\n dev = pA.get_device_info_by_index(k)\r\n devices[k] = {'name': dev['name'], 'inputs': dev['maxInputChannels'], 'outputs': dev['maxOutputChannels']}\r\n device_string += 'Index %d device name = %s, inputs = %d, outputs = %d\\n' % \\\r\n (k,dev['name'],dev['maxInputChannels'],dev['maxOutputChannels'])\r\n logger.debug(device_string)\r\n return devices", "response": "Display available audio devices along with their\r\n port indices."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck the input and output devices to see if they are valid", "response": "def in_out_check(self):\r\n \"\"\"\r\n Checks the input and output to see if they are valid\r\n \r\n \"\"\"\r\n devices = available_devices()\r\n if not self.in_idx in devices:\r\n raise OSError(\"Input device is unavailable\")\r\n in_check = devices[self.in_idx]\r\n if not self.out_idx in devices:\r\n raise OSError(\"Output device is unavailable\")\r\n out_check = devices[self.out_idx]\r\n if((in_check['inputs'] == 0) and (out_check['outputs']==0)):\r\n raise StandardError('Invalid input and output devices')\r\n elif(in_check['inputs'] == 0):\r\n raise ValueError('Selected input device has no inputs')\r\n elif(out_check['outputs'] == 0):\r\n raise ValueError('Selected output device has no outputs')\r\n return True"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef interactive_stream(self,Tsec = 2, numChan = 1):\r\n self.Tsec = Tsec\r\n self.numChan = numChan\r\n self.interactiveFG = 1\r\n self.play = interactive(self.interaction,Stream = ToggleButtons(\r\n options=['Start Streaming', 'Stop Streaming'],\r\n description = ' ',\r\n value = 'Stop Streaming') )\r\n display(self.play)", "response": "A convenience method for interacting with the audio stream."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nstream audio in a thread using callback.", "response": "def thread_stream(self,Tsec = 2,numChan = 1):\r\n \"\"\"\r\n Stream audio in a thread using callback. The stream is threaded, so widgets can be\r\n used simultaneously during stream.\r\n\r\n Parameters\r\n ----------\r\n\r\n Tsec : stream time in seconds if Tsec > 0. If Tsec = 0, then stream goes to infinite \r\n mode. When in infinite mode, Tsec.stop() can be used to stop the stream.\r\n \r\n numChan : number of channels. Use 1 for mono and 2 for stereo.\r\n\r\n \"\"\"\r\n def stream_thread(time,channel):\r\n self.stream(Tsec=time,numChan = channel)\r\n\r\n # Thread the streaming function\r\n t = Thread(target=stream_thread, args=(Tsec,numChan,))\r\n\r\n # Start the stream\r\n t.start()"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stream(self,Tsec = 2,numChan = 1):\r\n self.Tsec = Tsec\r\n self.numChan = numChan\r\n self.N_samples = int(self.fs*Tsec)\r\n self.data_capture = []\r\n self.data_capture_left = []\r\n self.data_capture_right = []\r\n self.capture_sample_count = 0\r\n self.DSP_tic = []\r\n self.DSP_toc = []\r\n self.start_time = time.time()\r\n self.stop_stream = False\r\n # open stream using callback (3)\r\n stream = self.p.open(format=pyaudio.paInt16,\r\n channels=numChan,\r\n rate=self.fs,\r\n input=True,\r\n output=True,\r\n input_device_index = self.in_idx,\r\n output_device_index = self.out_idx,\r\n frames_per_buffer = self.frame_length,\r\n stream_callback=self.stream_callback)\r\n\r\n # start the stream (4)\r\n stream.start_stream()\r\n\r\n # infinite mode\r\n if(Tsec == 0):\r\n while stream.is_active():\r\n if self.stop_stream:\r\n stream.stop_stream()\r\n time.sleep(self.sleep_time)\r\n else:\r\n # wait for stream to finish (5)\r\n while stream.is_active():\r\n if self.capture_sample_count >= self.N_samples:\r\n stream.stop_stream()\r\n if self.stop_stream:\r\n stream.stop_stream()\r\n time.sleep(self.sleep_time)\r\n\r\n # stop stream (6)\r\n stream.stop_stream()\r\n stream.close()\r\n\r\n # close PyAudio (7)\r\n self.p.terminate()\r\n self.stream_data = True\r\n # print('Audio input/output streaming session complete!')\r\n \r\n if(self.interactiveFG):\r\n # Move radio button back to 'Stop Streaming'\r\n self.play.children[0].value = 'Stop Streaming'\r\n else:\r\n if(self.print_when_done == 1):\r\n print('Completed')", "response": "Open audio stream using callback."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nadding new samples to the data_capture array and increment the sample counter.", "response": "def DSP_capture_add_samples(self,new_data):\r\n \"\"\"\r\n Append new samples to the data_capture array and increment the sample counter\r\n If length reaches Tcapture, then the newest samples will be kept. If Tcapture = 0 \r\n then new values are not appended to the data_capture array.\r\n \r\n \"\"\"\r\n self.capture_sample_count += len(new_data)\r\n if self.Tcapture > 0:\r\n self.data_capture = np.hstack((self.data_capture,new_data))\r\n if (self.Tcapture > 0) and (len(self.data_capture) > self.Ncapture):\r\n self.data_capture = self.data_capture[-self.Ncapture:]"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef DSP_capture_add_samples_stereo(self,new_data_left,new_data_right):\r\n self.capture_sample_count = self.capture_sample_count + len(new_data_left) + len(new_data_right)\r\n if self.Tcapture > 0:\r\n self.data_capture_left = np.hstack((self.data_capture_left,new_data_left))\r\n self.data_capture_right = np.hstack((self.data_capture_right,new_data_right))\r\n if (len(self.data_capture_left) > self.Ncapture):\r\n self.data_capture_left = self.data_capture_left[-self.Ncapture:]\r\n if (len(self.data_capture_right) > self.Ncapture):\r\n self.data_capture_right = self.data_capture_right[-self.Ncapture:]", "response": "Add new samples to the data_capture array."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds new tic time to the list of tic times. Will not be called if Tcapture is 0.", "response": "def DSP_callback_tic(self):\r\n \"\"\"\r\n Add new tic time to the DSP_tic list. Will not be called if\r\n Tcapture = 0.\r\n \r\n \"\"\"\r\n if self.Tcapture > 0:\r\n self.DSP_tic.append(time.time()-self.start_time)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding new toc time to the DSP_toc list. Will not be called if Tcapture is 0.", "response": "def DSP_callback_toc(self):\r\n \"\"\"\r\n Add new toc time to the DSP_toc list. Will not be called if\r\n Tcapture = 0.\r\n\r\n \"\"\"\r\n if self.Tcapture > 0:\r\n self.DSP_toc.append(time.time()-self.start_time)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef stream_stats(self):\r\n Tp = self.frame_length/float(self.fs)*1000\r\n print('Delay (latency) in Entering the Callback the First Time = %6.2f (ms)' \\\r\n % (self.DSP_tic[0]*1000,))\r\n print('Ideal Callback period = %1.2f (ms)' % Tp)\r\n Tmp_mean = np.mean(np.diff(np.array(self.DSP_tic))[1:]*1000)\r\n print('Average Callback Period = %1.2f (ms)' % Tmp_mean)\r\n Tprocess_mean = np.mean(np.array(self.DSP_toc)-np.array(self.DSP_tic))*1000\r\n print('Average Callback process time = %1.2f (ms)' % Tprocess_mean)", "response": "Display basic statistics of the callback execution"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nplots timing information of time spent in the callback. This is similar to what a logic analyzer provides when probing an interrupt. cb_active_plot( start_ms,stop_ms,line_color='b')", "response": "def cb_active_plot(self,start_ms,stop_ms,line_color='b'):\r\n \"\"\"\r\n Plot timing information of time spent in the callback. This is similar\r\n to what a logic analyzer provides when probing an interrupt.\r\n\r\n cb_active_plot( start_ms,stop_ms,line_color='b')\r\n \r\n \"\"\"\r\n # Find bounding k values that contain the [start_ms,stop_ms]\r\n k_min_idx = np.nonzero(np.ravel(np.array(self.DSP_tic)*1000 < start_ms))[0]\r\n if len(k_min_idx) < 1:\r\n k_min = 0\r\n else:\r\n k_min = k_min_idx[-1]\r\n k_max_idx = np.nonzero(np.ravel(np.array(self.DSP_tic)*1000 > stop_ms))[0]\r\n if len(k_min_idx) < 1:\r\n k_max= len(self.DSP_tic)\r\n else:\r\n k_max = k_max_idx[0]\r\n for k in range(k_min,k_max):\r\n if k == 0:\r\n plt.plot([0,self.DSP_tic[k]*1000,self.DSP_tic[k]*1000,\r\n self.DSP_toc[k]*1000,self.DSP_toc[k]*1000],\r\n [0,0,1,1,0],'b')\r\n else:\r\n plt.plot([self.DSP_toc[k-1]*1000,self.DSP_tic[k]*1000,self.DSP_tic[k]*1000,\r\n self.DSP_toc[k]*1000,self.DSP_toc[k]*1000],[0,0,1,1,0],'b')\r\n plt.plot([self.DSP_toc[k_max-1]*1000,stop_ms],[0,0],'b')\r\n \r\n plt.xlim([start_ms,stop_ms])\r\n plt.title(r'Time Spent in the callback')\r\n plt.ylabel(r'Timing')\r\n plt.xlabel(r'Time (ms)')\r\n plt.grid();"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_LR(self,in_data):\r\n for i in range(0,self.frame_length*2):\r\n if i % 2:\r\n self.right_in[(int)(i/2)] = in_data[i]\r\n else:\r\n self.left_in[(int)(i/2)] = in_data[i]\r\n return self.left_in, self.right_in", "response": "Returns a list of left and right samples and the list of samples with the highest priority."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\npacks separate left and right channel data into one array to output and returns the output.", "response": "def pack_LR(self,left_out,right_out):\r\n \"\"\"\r\n Packs separate left and right channel data into one array to output\r\n and returns the output.\r\n\r\n Parameters\r\n ----------\r\n left_out : left channel array of samples going to output\r\n right_out : right channel array of samples going to output\r\n\r\n Returns\r\n -------\r\n out : packed left and right channel array of samples\r\n \"\"\"\r\n for i in range(0,self.frame_length*2):\r\n if i % 2:\r\n self.out[i] = right_out[(int)(i/2)]\r\n else:\r\n self.out[i] = left_out[(int)(i/2)]\r\n return self.out"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ndesign an IIR lowpass filter using scipy.signal.iirdesign. The filter order is determined based on f_pass Hz, f_stop Hz, and the desired stopband attenuation d_stop in dB, all relative to a sampling rate of fs Hz. Parameters ---------- f_pass : Passband critical frequency in Hz f_stop : Stopband critical frequency in Hz Ripple_pass : Filter gain in dB at f_pass Atten_stop : Filter attenuation in dB at f_stop fs : Sampling rate in Hz ftype : Analog prototype from 'butter' 'cheby1', 'cheby2', 'ellip', and 'bessel' Returns ------- b : ndarray of the numerator coefficients a : ndarray of the denominator coefficients sos : 2D ndarray of second-order section coefficients Notes ----- Additionally a text string telling the user the filter order is written to the console, e.g., IIR cheby1 order = 8. Examples -------- >>> fs = 48000 >>> f_pass = 5000 >>> f_stop = 8000 >>> b_but,a_but,sos_but = IIR_lpf(f_pass,f_stop,0.5,60,fs,'butter') >>> b_cheb1,a_cheb1,sos_cheb1 = IIR_lpf(f_pass,f_stop,0.5,60,fs,'cheby1') >>> b_cheb2,a_cheb2,sos_cheb2 = IIR_lpf(f_pass,f_stop,0.5,60,fs,'cheby2') >>> b_elli,a_elli,sos_elli = IIR_lpf(f_pass,f_stop,0.5,60,fs,'ellip') Mark Wickert October 2016", "response": "def IIR_lpf(f_pass, f_stop, Ripple_pass, Atten_stop, \r\n fs = 1.00, ftype = 'butter'):\r\n \"\"\"\r\n Design an IIR lowpass filter using scipy.signal.iirdesign. \r\n The filter order is determined based on \r\n f_pass Hz, f_stop Hz, and the desired stopband attenuation\r\n d_stop in dB, all relative to a sampling rate of fs Hz.\r\n\r\n Parameters\r\n ----------\r\n f_pass : Passband critical frequency in Hz\r\n f_stop : Stopband critical frequency in Hz\r\n Ripple_pass : Filter gain in dB at f_pass\r\n Atten_stop : Filter attenuation in dB at f_stop\r\n fs : Sampling rate in Hz\r\n ftype : Analog prototype from 'butter' 'cheby1', 'cheby2',\r\n 'ellip', and 'bessel'\r\n\r\n Returns\r\n -------\r\n b : ndarray of the numerator coefficients\r\n a : ndarray of the denominator coefficients\r\n sos : 2D ndarray of second-order section coefficients\r\n\r\n Notes\r\n -----\r\n Additionally a text string telling the user the filter order is\r\n written to the console, e.g., IIR cheby1 order = 8.\r\n\r\n Examples\r\n --------\r\n >>> fs = 48000\r\n >>> f_pass = 5000\r\n >>> f_stop = 8000\r\n >>> b_but,a_but,sos_but = IIR_lpf(f_pass,f_stop,0.5,60,fs,'butter')\r\n >>> b_cheb1,a_cheb1,sos_cheb1 = IIR_lpf(f_pass,f_stop,0.5,60,fs,'cheby1')\r\n >>> b_cheb2,a_cheb2,sos_cheb2 = IIR_lpf(f_pass,f_stop,0.5,60,fs,'cheby2')\r\n >>> b_elli,a_elli,sos_elli = IIR_lpf(f_pass,f_stop,0.5,60,fs,'ellip')\r\n\r\n\r\n Mark Wickert October 2016\r\n \"\"\"\r\n \r\n b,a = signal.iirdesign(2*float(f_pass)/fs, 2*float(f_stop)/fs,\r\n Ripple_pass, Atten_stop,\r\n ftype = ftype, output='ba')\r\n sos = signal.iirdesign(2*float(f_pass)/fs, 2*float(f_stop)/fs,\r\n Ripple_pass, Atten_stop,\r\n ftype = ftype, output='sos')\r\n tag = 'IIR ' + ftype + ' order'\r\n print('%s = %d.' % (tag,len(a)-1))\r\n return b, a, sos"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef IIR_bsf(f_pass1, f_stop1, f_stop2, f_pass2, Ripple_pass, Atten_stop, \r\n fs = 1.00, ftype = 'butter'):\r\n \"\"\"\r\n Design an IIR bandstop filter using scipy.signal.iirdesign. \r\n The filter order is determined based on \r\n f_pass Hz, f_stop Hz, and the desired stopband attenuation\r\n d_stop in dB, all relative to a sampling rate of fs Hz.\r\n\r\n Mark Wickert October 2016\r\n \"\"\"\r\n \r\n b,a = signal.iirdesign([2*float(f_pass1)/fs, 2*float(f_pass2)/fs],\r\n [2*float(f_stop1)/fs, 2*float(f_stop2)/fs],\r\n Ripple_pass, Atten_stop,\r\n ftype = ftype, output='ba')\r\n sos = signal.iirdesign([2*float(f_pass1)/fs, 2*float(f_pass2)/fs],\r\n [2*float(f_stop1)/fs, 2*float(f_stop2)/fs],\r\n Ripple_pass, Atten_stop,\r\n ftype =ftype, output='sos')\r\n tag = 'IIR ' + ftype + ' order'\r\n print('%s = %d.' % (tag,len(a)-1))\r\n return b, a, sos", "response": "This function is a wrapper for the numpy. array. sort function of the IIR bandstop filter"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef freqz_resp_list(b,a=np.array([1]),mode = 'dB',fs=1.0,Npts = 1024,fsize=(6,4)):\r\n if type(b) == list:\r\n # We have a list of filters\r\n N_filt = len(b)\r\n f = np.arange(0,Npts)/(2.0*Npts)\r\n for n in range(N_filt):\r\n w,H = signal.freqz(b[n],a[n],2*np.pi*f)\r\n if n == 0:\r\n plt.figure(figsize=fsize)\r\n if mode.lower() == 'db':\r\n plt.plot(f*fs,20*np.log10(np.abs(H)))\r\n if n == N_filt-1:\r\n plt.xlabel('Frequency (Hz)')\r\n plt.ylabel('Gain (dB)')\r\n plt.title('Frequency Response - Magnitude')\r\n\r\n elif mode.lower() == 'phase':\r\n plt.plot(f*fs,np.angle(H))\r\n if n == N_filt-1:\r\n plt.xlabel('Frequency (Hz)')\r\n plt.ylabel('Phase (rad)')\r\n plt.title('Frequency Response - Phase')\r\n\r\n elif (mode.lower() == 'groupdelay_s') or (mode.lower() == 'groupdelay_t'):\r\n \"\"\"\r\n Notes\r\n -----\r\n\r\n Since this calculation involves finding the derivative of the\r\n phase response, care must be taken at phase wrapping points \r\n and when the phase jumps by +/-pi, which occurs when the \r\n amplitude response changes sign. Since the amplitude response\r\n is zero when the sign changes, the jumps do not alter the group \r\n delay results.\r\n \"\"\"\r\n theta = np.unwrap(np.angle(H))\r\n # Since theta for an FIR filter is likely to have many pi phase\r\n # jumps too, we unwrap a second time 2*theta and divide by 2\r\n theta2 = np.unwrap(2*theta)/2.\r\n theta_dif = np.diff(theta2)\r\n f_diff = np.diff(f)\r\n Tg = -np.diff(theta2)/np.diff(w)\r\n # For gain almost zero set groupdelay = 0\r\n idx = np.nonzero(np.ravel(20*np.log10(H[:-1]) < -400))[0]\r\n Tg[idx] = np.zeros(len(idx))\r\n max_Tg = np.max(Tg)\r\n #print(max_Tg)\r\n if mode.lower() == 'groupdelay_t':\r\n max_Tg /= fs\r\n plt.plot(f[:-1]*fs,Tg/fs)\r\n plt.ylim([0,1.2*max_Tg])\r\n else:\r\n plt.plot(f[:-1]*fs,Tg)\r\n plt.ylim([0,1.2*max_Tg])\r\n if n == N_filt-1:\r\n plt.xlabel('Frequency (Hz)')\r\n if mode.lower() == 'groupdelay_t':\r\n plt.ylabel('Group Delay (s)')\r\n else:\r\n plt.ylabel('Group Delay (samples)')\r\n plt.title('Frequency Response - Group Delay')\r\n else:\r\n s1 = 'Error, mode must be \"dB\", \"phase, '\r\n s2 = '\"groupdelay_s\", or \"groupdelay_t\"'\r\n print(s1 + s2)", "response": "A list of frequency response methods that can be used to display the frequency response of a single object."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef freqz_cas(sos,w):\r\n Ns,Mcol = sos.shape\r\n w,Hcas = signal.freqz(sos[0,:3],sos[0,3:],w)\r\n for k in range(1,Ns):\r\n w,Htemp = signal.freqz(sos[k,:3],sos[k,3:],w)\r\n Hcas *= Htemp\r\n return w, Hcas", "response": "Returns the frequency of the next non - empty sequence in a sequence."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef unique_cpx_roots(rlist,tol = 0.001):\r\n uniq = [rlist[0]]\r\n mult = [1]\r\n for k in range(1,len(rlist)):\r\n N_uniq = len(uniq)\r\n for m in range(N_uniq):\r\n if abs(rlist[k]-uniq[m]) <= tol:\r\n mult[m] += 1\r\n uniq[m] = (uniq[m]*(mult[m]-1) + rlist[k])/float(mult[m])\r\n break\r\n uniq = np.hstack((uniq,rlist[k]))\r\n mult = np.hstack((mult,[1]))\r\n return np.array(uniq), np.array(mult)", "response": "Returns a list of unique cpx_root entries in the given list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating a z - plane plot from a single sos object.", "response": "def sos_zplane(sos,auto_scale=True,size=2,tol = 0.001):\r\n \"\"\"\r\n Create an z-plane pole-zero plot.\r\n\r\n Create an z-plane pole-zero plot using the numerator\r\n and denominator z-domain system function coefficient\r\n ndarrays b and a respectively. Assume descending powers of z.\r\n\r\n Parameters\r\n ----------\r\n sos : ndarray of the sos coefficients\r\n auto_scale : bool (default True)\r\n size : plot radius maximum when scale = False\r\n\r\n Returns\r\n -------\r\n (M,N) : tuple of zero and pole counts + plot window\r\n \r\n Notes\r\n -----\r\n This function tries to identify repeated poles and zeros and will \r\n place the multiplicity number above and to the right of the pole or zero.\r\n The difficulty is setting the tolerance for this detection. Currently it\r\n is set at 1e-3 via the function signal.unique_roots.\r\n\r\n Examples\r\n --------\r\n >>> # Here the plot is generated using auto_scale\r\n >>> sos_zplane(sos)\r\n >>> # Here the plot is generated using manual scaling\r\n >>> sos_zplane(sos,False,1.5)\r\n \"\"\"\r\n Ns,Mcol = sos.shape\r\n # Extract roots from sos num and den removing z = 0\r\n # roots due to first-order sections\r\n N_roots = []\r\n for k in range(Ns):\r\n N_roots_tmp = np.roots(sos[k,:3])\r\n if N_roots_tmp[1] == 0.:\r\n N_roots = np.hstack((N_roots,N_roots_tmp[0]))\r\n else:\r\n N_roots = np.hstack((N_roots,N_roots_tmp))\r\n D_roots = []\r\n for k in range(Ns):\r\n D_roots_tmp = np.roots(sos[k,3:])\r\n if D_roots_tmp[1] == 0.:\r\n D_roots = np.hstack((D_roots,D_roots_tmp[0]))\r\n else:\r\n D_roots = np.hstack((D_roots,D_roots_tmp))\r\n # Plot labels if multiplicity greater than 1\r\n x_scale = 1.5*size\r\n y_scale = 1.5*size \r\n x_off = 0.02\r\n y_off = 0.01\r\n M = len(N_roots)\r\n N = len(D_roots)\r\n if auto_scale:\r\n if M > 0 and N > 0:\r\n size = max(np.max(np.abs(N_roots)),np.max(np.abs(D_roots)))+.1\r\n elif M > 0:\r\n size = max(np.max(np.abs(N_roots)),1.0)+.1\r\n elif N > 0:\r\n size = max(1.0,np.max(np.abs(D_roots)))+.1\r\n else:\r\n size = 1.1\r\n plt.figure(figsize=(5,5))\r\n plt.axis('equal')\r\n r = np.linspace(0,2*np.pi,200)\r\n plt.plot(np.cos(r),np.sin(r),'r--')\r\n plt.plot([-size,size],[0,0],'k-.')\r\n plt.plot([0,0],[-size,size],'k-.')\r\n if M > 0:\r\n #N_roots = np.roots(b)\r\n N_uniq, N_mult=unique_cpx_roots(N_roots,tol=tol)\r\n plt.plot(np.real(N_uniq),np.imag(N_uniq),'ko',mfc='None',ms=8)\r\n idx_N_mult = np.nonzero(np.ravel(N_mult>1))[0]\r\n for k in range(len(idx_N_mult)):\r\n x_loc = np.real(N_uniq[idx_N_mult[k]]) + x_off*x_scale\r\n y_loc =np.imag(N_uniq[idx_N_mult[k]]) + y_off*y_scale\r\n plt.text(x_loc,y_loc,str(N_mult[idx_N_mult[k]]),\r\n ha='center',va='bottom',fontsize=10)\r\n if N > 0:\r\n #D_roots = np.roots(a)\r\n D_uniq, D_mult=unique_cpx_roots(D_roots,tol=tol)\r\n plt.plot(np.real(D_uniq),np.imag(D_uniq),'kx',ms=8)\r\n idx_D_mult = np.nonzero(np.ravel(D_mult>1))[0]\r\n for k in range(len(idx_D_mult)):\r\n x_loc = np.real(D_uniq[idx_D_mult[k]]) + x_off*x_scale\r\n y_loc =np.imag(D_uniq[idx_D_mult[k]]) + y_off*y_scale\r\n plt.text(x_loc,y_loc,str(D_mult[idx_D_mult[k]]),\r\n ha='center',va='bottom',fontsize=10) \r\n if M - N < 0:\r\n plt.plot(0.0,0.0,'bo',mfc='None',ms=8)\r\n elif M - N > 0:\r\n plt.plot(0.0,0.0,'kx',ms=8)\r\n if abs(M - N) > 1:\r\n plt.text(x_off*x_scale,y_off*y_scale,str(abs(M-N)),\r\n ha='center',va='bottom',fontsize=10) \r\n plt.xlabel('Real Part')\r\n plt.ylabel('Imaginary Part')\r\n plt.title('Pole-Zero Plot')\r\n #plt.grid()\r\n plt.axis([-size,size,-size,size])\r\n return M,N"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef firwin_bpf(N_taps, f1, f2, fs = 1.0, pass_zero=False):\r\n return signal.firwin(N_taps,2*(f1,f2)/fs,pass_zero=pass_zero)", "response": "Returns a new FIR bandpass filter with the given frequency range"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef firwin_kaiser_lpf(f_pass, f_stop, d_stop, fs = 1.0, N_bump=0):\r\n wc = 2*np.pi*(f_pass + f_stop)/2/fs\r\n delta_w = 2*np.pi*(f_stop - f_pass)/fs\r\n # Find the filter order\r\n M = np.ceil((d_stop - 8)/(2.285*delta_w))\r\n # Adjust filter order up or down as needed\r\n M += N_bump\r\n N_taps = M + 1\r\n # Obtain the Kaiser window\r\n beta = signal.kaiser_beta(d_stop)\r\n w_k = signal.kaiser(N_taps,beta)\r\n n = np.arange(N_taps)\r\n b_k = wc/np.pi*np.sinc(wc/np.pi*(n-M/2)) * w_k\r\n b_k /= np.sum(b_k)\r\n print('Kaiser Win filter taps = %d.' % N_taps)\r\n return b_k", "response": "This function returns a lowpass filter based on the sinc function kernel and a Kaiser Win filter."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ndesigning an FIR bandpass filter using the sinc function and the Kaiser window.", "response": "def firwin_kaiser_bpf(f_stop1, f_pass1, f_pass2, f_stop2, d_stop, \r\n fs = 1.0, N_bump=0):\r\n \"\"\"\r\n Design an FIR bandpass filter using the sinc() kernel and\r\n a Kaiser window. The filter order is determined based on \r\n f_stop1 Hz, f_pass1 Hz, f_pass2 Hz, f_stop2 Hz, and the \r\n desired stopband attenuation d_stop in dB for both stopbands,\r\n all relative to a sampling rate of fs Hz.\r\n Note: the passband ripple cannot be set independent of the\r\n stopband attenuation.\r\n\r\n Mark Wickert October 2016 \r\n \"\"\"\r\n # Design BPF starting from simple LPF equivalent\r\n # The upper and lower stopbands are assumed to have \r\n # the same attenuation level. The LPF equivalent critical\r\n # frequencies:\r\n f_pass = (f_pass2 - f_pass1)/2\r\n f_stop = (f_stop2 - f_stop1)/2\r\n # Continue to design equivalent LPF\r\n wc = 2*np.pi*(f_pass + f_stop)/2/fs\r\n delta_w = 2*np.pi*(f_stop - f_pass)/fs\r\n # Find the filter order\r\n M = np.ceil((d_stop - 8)/(2.285*delta_w))\r\n # Adjust filter order up or down as needed\r\n M += N_bump\r\n N_taps = M + 1\r\n # Obtain the Kaiser window\r\n beta = signal.kaiser_beta(d_stop)\r\n w_k = signal.kaiser(N_taps,beta)\r\n n = np.arange(N_taps)\r\n b_k = wc/np.pi*np.sinc(wc/np.pi*(n-M/2)) * w_k\r\n b_k /= np.sum(b_k)\r\n # Transform LPF to BPF\r\n f0 = (f_pass2 + f_pass1)/2\r\n w0 = 2*np.pi*f0/fs\r\n n = np.arange(len(b_k))\r\n b_k_bp = 2*b_k*np.cos(w0*(n-M/2))\r\n print('Kaiser Win filter taps = %d.' % N_taps)\r\n return b_k_bp"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a new order based on the given conditions.", "response": "def lowpass_order(f_pass, f_stop, dpass_dB, dstop_dB, fsamp = 1):\r\n \"\"\"\r\n Optimal FIR (equal ripple) Lowpass Order Determination\r\n \r\n Text reference: Ifeachor, Digital Signal Processing a Practical Approach, \r\n second edition, Prentice Hall, 2002.\r\n Journal paper reference: Herriman et al., Practical Design Rules for Optimum\r\n Finite Imulse Response Digitl Filters, Bell Syst. Tech. J., vol 52, pp. \r\n 769-799, July-Aug., 1973.IEEE, 1973.\r\n \"\"\"\r\n dpass = 1 - 10**(-dpass_dB/20)\r\n dstop = 10**(-dstop_dB/20)\r\n Df = (f_stop - f_pass)/fsamp\r\n a1 = 5.309e-3\r\n a2 = 7.114e-2\r\n a3 = -4.761e-1\r\n a4 = -2.66e-3\r\n a5 = -5.941e-1\r\n a6 = -4.278e-1\r\n \r\n Dinf = np.log10(dstop)*(a1*np.log10(dpass)**2 + a2*np.log10(dpass) + a3) \\\r\n + (a4*np.log10(dpass)**2 + a5*np.log10(dpass) + a6)\r\n f = 11.01217 + 0.51244*(np.log10(dpass) - np.log10(dstop))\r\n N = Dinf/Df - f*Df + 1\r\n ff = 2*np.array([0, f_pass, f_stop, fsamp/2])/fsamp\r\n aa = np.array([1, 1, 0, 0])\r\n wts = np.array([1.0, dpass/dstop])\r\n return int(N), ff, aa, wts"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef bandpass_order(f_stop1, f_pass1, f_pass2, f_stop2, dpass_dB, dstop_dB, fsamp = 1):\r\n dpass = 1 - 10**(-dpass_dB/20)\r\n dstop = 10**(-dstop_dB/20)\r\n Df1 = (f_pass1 - f_stop1)/fsamp\r\n Df2 = (f_stop2 - f_pass2)/fsamp\r\n b1 = 0.01201\r\n b2 = 0.09664\r\n b3 = -0.51325\r\n b4 = 0.00203\r\n b5 = -0.5705\r\n b6 = -0.44314\r\n \r\n Df = min(Df1, Df2)\r\n Cinf = np.log10(dstop)*(b1*np.log10(dpass)**2 + b2*np.log10(dpass) + b3) \\\r\n + (b4*np.log10(dpass)**2 + b5*np.log10(dpass) + b6)\r\n g = -14.6*np.log10(dpass/dstop) - 16.9\r\n N = Cinf/Df + g*Df + 1\r\n ff = 2*np.array([0, f_stop1, f_pass1, f_pass2, f_stop2, fsamp/2])/fsamp\r\n aa = np.array([0, 0, 1, 1, 0, 0])\r\n wts = np.array([dpass/dstop, 1, dpass/dstop])\r\n return int(N), ff, aa, wts", "response": "Returns a new structure for a bandpass order"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef fir_remez_lpf(f_pass, f_stop, d_pass, d_stop, fs = 1.0, N_bump=5):\r\n n, ff, aa, wts = lowpass_order(f_pass, f_stop, d_pass, d_stop, fsamp=fs)\r\n # Bump up the order by N_bump to bring down the final d_pass & d_stop\r\n N_taps = n\r\n N_taps += N_bump\r\n b = signal.remez(N_taps, ff, aa[0::2], wts,Hz=2)\r\n print('Remez filter taps = %d.' % N_taps)\r\n return b", "response": "Returns a FIR lowpass filter with the given filter parameters."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fir_remez_hpf(f_stop, f_pass, d_pass, d_stop, fs = 1.0, N_bump=5):\r\n # Transform HPF critical frequencies to lowpass equivalent\r\n f_pass_eq = fs/2. - f_pass\r\n f_stop_eq = fs/2. - f_stop\r\n # Design LPF equivalent\r\n n, ff, aa, wts = lowpass_order(f_pass_eq, f_stop_eq, d_pass, d_stop, fsamp=fs)\r\n # Bump up the order by N_bump to bring down the final d_pass & d_stop\r\n N_taps = n\r\n N_taps += N_bump\r\n b = signal.remez(N_taps, ff, aa[0::2], wts,Hz=2)\r\n # Transform LPF equivalent to HPF\r\n n = np.arange(len(b))\r\n b *= (-1)**n\r\n print('Remez filter taps = %d.' % N_taps)\r\n return b", "response": "Design an FIR highpass filter using remez with order determined by the desired passband ripple diameter f_pass and stopband attenuation d_stop dB."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef fir_remez_bpf(f_stop1, f_pass1, f_pass2, f_stop2, d_pass, d_stop, \r\n fs = 1.0, N_bump=5):\r\n \"\"\"\r\n Design an FIR bandpass filter using remez with order\r\n determination. The filter order is determined based on \r\n f_stop1 Hz, f_pass1 Hz, f_pass2 Hz, f_stop2 Hz, and the \r\n desired passband ripple d_pass dB and stopband attenuation\r\n d_stop dB all relative to a sampling rate of fs Hz.\r\n\r\n Mark Wickert October 2016, updated October 2018\r\n \"\"\"\r\n n, ff, aa, wts = bandpass_order(f_stop1, f_pass1, f_pass2, f_stop2, \r\n d_pass, d_stop, fsamp=fs)\r\n # Bump up the order by N_bump to bring down the final d_pass & d_stop\r\n N_taps = n\r\n N_taps += N_bump\r\n b = signal.remez(N_taps, ff, aa[0::2], wts,Hz=2)\r\n print('Remez filter taps = %d.' % N_taps)\r\n return b", "response": "Returns a FIR bandpass filter with the given order"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef CIC(M, K):\n\n if K == 1:\n b = np.ones(M)\n else:\n h = np.ones(M)\n b = h\n for i in range(1, K):\n b = signal.convolve(b, h) # cascade by convolving impulse responses\n\n # Make filter have unity gain at DC\n return b / np.sum(b)", "response": "A functional form implementation of a cascade of integrator comb ( CIC )."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ten_band_eq_filt(x,GdB,Q=3.5):\n fs = 44100.0 # Hz\n NB = len(GdB)\n if not NB == 10:\n raise ValueError(\"GdB length not equal to ten\")\n Fc = 31.25*2**np.arange(NB)\n B = np.zeros((NB,3))\n A = np.zeros((NB,3))\n \n # Create matrix of cascade coefficients\n for k in range(NB):\n [b,a] = peaking(GdB[k],Fc[k],Q)\n B[k,:] = b\n A[k,:] = a\n # Pass signal x through the cascade of ten filters\n y = np.zeros(len(x))\n for k in range(NB):\n if k == 0:\n y = signal.lfilter(B[k,:],A[k,:],x)\n else:\n y = signal.lfilter(B[k,:],A[k,:],y)\n return y", "response": "Filter the input signal x with a ten - band equalizer having octave gain values in ndarray GdB and the Q of each filter is set to 3. 5 kHz."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncreating a frequency response magnitude plot in dB of a ten band equalizer.", "response": "def ten_band_eq_resp(GdB,Q=3.5):\n \"\"\"\n Create a frequency response magnitude plot in dB of a ten band equalizer\n using a semilogplot (semilogx()) type plot\n \n \n Parameters\n ----------\n GdB : Gain vector for 10 peaking filters [G0,...,G9]\n Q : Quality factor for each peaking filter (default 3.5)\n \n Returns\n -------\n Nothing : two plots are created\n \n Examples\n --------\n >>> import matplotlib.pyplot as plt\n >>> from sk_dsp_comm import sigsys as ss\n >>> ss.ten_band_eq_resp([0,10.0,0,0,-1,0,5,0,-4,0])\n >>> plt.show()\n \"\"\"\n fs = 44100.0 # Hz\n NB = len(GdB)\n if not NB == 10:\n raise ValueError(\"GdB length not equal to ten\")\n Fc = 31.25*2**np.arange(NB)\n B = np.zeros((NB,3));\n A = np.zeros((NB,3));\n \n # Create matrix of cascade coefficients\n for k in range(NB):\n b,a = peaking(GdB[k],Fc[k],Q,fs)\n B[k,:] = b\n A[k,:] = a\n # Create the cascade frequency response\n F = np.logspace(1,np.log10(20e3),1000)\n H = np.ones(len(F))*np.complex(1.0,0.0)\n for k in range(NB):\n w,Htemp = signal.freqz(B[k,:],A[k,:],2*np.pi*F/fs)\n H *= Htemp\n plt.figure(figsize=(6,4))\n plt.subplot(211)\n plt.semilogx(F,20*np.log10(abs(H)))\n plt.axis([10, fs/2, -12, 12])\n plt.grid()\n plt.title('Ten-Band Equalizer Frequency Response')\n plt.xlabel('Frequency (Hz)')\n plt.ylabel('Gain (dB)')\n plt.subplot(212)\n plt.stem(np.arange(NB),GdB,'b','bs')\n #plt.bar(np.arange(NB)-.1,GdB,0.2)\n plt.axis([0, NB-1, -12, 12])\n plt.xlabel('Equalizer Band Number')\n plt.ylabel('Gain Set (dB)')\n plt.grid()"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ex6_2(n):\n x = np.zeros(len(n))\n for k, nn in enumerate(n):\n if nn >= -2 and nn <= 5:\n x[k] = 8 - nn\n return x", "response": "Generate a triangle pulse as described in Example 6 - 2\n of Chapter 6."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef position_CD(Ka,out_type = 'fb_exact'):\n rs = 10/(2*np.pi)\n # Load b and a ndarrays with the coefficients\n if out_type.lower() == 'open_loop':\n b = np.array([Ka*4000*rs])\n a = np.array([1,1275,31250,0])\n elif out_type.lower() == 'fb_approx':\n b = np.array([3.2*Ka*rs])\n a = np.array([1, 25, 3.2*Ka*rs])\n elif out_type.lower() == 'fb_exact':\n b = np.array([4000*Ka*rs])\n a = np.array([1, 1250+25, 25*1250, 4000*Ka*rs])\n else:\n raise ValueError('out_type must be: open_loop, fb_approx, or fc_exact')\n return b, a", "response": "Returns the CD sled position control case study of Chapter 18."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncruises control with PI controller and hill disturbance. This function returns various system function configurations for a the cruise control Case Study example found in the supplementary article. The plant model is obtained by the linearizing the equations of motion and the controller contains a proportional and integral gain term set via the closed-loop parameters natuarl frequency wn (rad/s) and damping zeta. Parameters ---------- wn : closed-loop natural frequency in rad/s, nominally 0.1 zeta : closed-loop damping factor, nominally 1.0 T : vehicle time constant, nominally 10 s vcruise : cruise velocity set point, nominally 75 mph vmax : maximum vehicle velocity, nominally 120 mph tf_mode : 'H', 'HE', 'HVW', or 'HED' controls the system function returned by the function 'H' : closed-loop system function V(s)/R(s) 'HE' : closed-loop system function E(s)/R(s) 'HVW' : closed-loop system function V(s)/W(s) 'HED' : closed-loop system function E(s)/D(s), where D is the hill disturbance input Returns ------- b : numerator coefficient ndarray a : denominator coefficient ndarray Examples -------- >>> # return the closed-loop system function output/input velocity >>> b,a = cruise_control(wn,zeta,T,vcruise,vmax,tf_mode='H') >>> # return the closed-loop system function loop error/hill disturbance >>> b,a = cruise_control(wn,zeta,T,vcruise,vmax,tf_mode='HED')", "response": "def cruise_control(wn,zeta,T,vcruise,vmax,tf_mode='H'):\n \"\"\"\n Cruise control with PI controller and hill disturbance.\n\n This function returns various system function configurations\n for a the cruise control Case Study example found in \n the supplementary article. The plant model is obtained by the\n linearizing the equations of motion and the controller contains a\n proportional and integral gain term set via the closed-loop parameters\n natuarl frequency wn (rad/s) and damping zeta.\n\n Parameters\n ----------\n wn : closed-loop natural frequency in rad/s, nominally 0.1\n zeta : closed-loop damping factor, nominally 1.0\n T : vehicle time constant, nominally 10 s\n vcruise : cruise velocity set point, nominally 75 mph\n vmax : maximum vehicle velocity, nominally 120 mph\n tf_mode : 'H', 'HE', 'HVW', or 'HED' controls the system function returned by the function \n 'H' : closed-loop system function V(s)/R(s)\n 'HE' : closed-loop system function E(s)/R(s)\n 'HVW' : closed-loop system function V(s)/W(s)\n 'HED' : closed-loop system function E(s)/D(s), where D is the hill disturbance input\n\n Returns\n -------\n b : numerator coefficient ndarray\n a : denominator coefficient ndarray \n\n Examples\n --------\n >>> # return the closed-loop system function output/input velocity\n >>> b,a = cruise_control(wn,zeta,T,vcruise,vmax,tf_mode='H')\n >>> # return the closed-loop system function loop error/hill disturbance\n >>> b,a = cruise_control(wn,zeta,T,vcruise,vmax,tf_mode='HED')\n \"\"\"\n tau = T/2.*vmax/vcruise\n g = 9.8\n g *= 3*60**2/5280. # m/s to mph conversion\n Kp = T*(2*zeta*wn-1/tau)/vmax\n Ki = T*wn**2./vmax\n K = Kp*vmax/T\n print('wn = ', np.sqrt(K/(Kp/Ki)))\n print('zeta = ', (K + 1/tau)/(2*wn))\n a = np.array([1, 2*zeta*wn, wn**2])\n if tf_mode == 'H':\n b = np.array([K, wn**2]) \n elif tf_mode == 'HE':\n b = np.array([1, 2*zeta*wn-K, 0.]) \n elif tf_mode == 'HVW':\n b = np.array([ 1, wn**2/K+1/tau, wn**2/(K*tau)])\n b *= Kp\n elif tf_mode == 'HED':\n b = np.array([g, 0])\n else:\n raise ValueError('tf_mode must be: H, HE, HVU, or HED')\n return b, a"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef splane(b,a,auto_scale=True,size=[-1,1,-1,1]):\n M = len(b) - 1\n N = len(a) - 1\n plt.figure(figsize=(5,5))\n #plt.axis('equal')\n N_roots = np.array([0.0])\n if M > 0:\n N_roots = np.roots(b)\n D_roots = np.array([0.0])\n if N > 0:\n D_roots = np.roots(a)\n if auto_scale:\n size[0] = min(np.min(np.real(N_roots)),np.min(np.real(D_roots)))-0.5\n size[1] = max(np.max(np.real(N_roots)),np.max(np.real(D_roots)))+0.5\n size[1] = max(size[1],0.5)\n size[2] = min(np.min(np.imag(N_roots)),np.min(np.imag(D_roots)))-0.5\n size[3] = max(np.max(np.imag(N_roots)),np.max(np.imag(D_roots)))+0.5\n \n plt.plot([size[0],size[1]],[0,0],'k--')\n plt.plot([0,0],[size[2],size[3]],'r--')\n # Plot labels if multiplicity greater than 1\n x_scale = size[1]-size[0]\n y_scale = size[3]-size[2]\n x_off = 0.03\n y_off = 0.01\n if M > 0:\n #N_roots = np.roots(b)\n N_uniq, N_mult=signal.unique_roots(N_roots,tol=1e-3, rtype='avg')\n plt.plot(np.real(N_uniq),np.imag(N_uniq),'ko',mfc='None',ms=8)\n idx_N_mult = np.nonzero(np.ravel(N_mult>1))[0]\n for k in range(len(idx_N_mult)):\n x_loc = np.real(N_uniq[idx_N_mult[k]]) + x_off*x_scale\n y_loc =np.imag(N_uniq[idx_N_mult[k]]) + y_off*y_scale\n plt.text(x_loc,y_loc,str(N_mult[idx_N_mult[k]]),ha='center',va='bottom',fontsize=10)\n if N > 0:\n #D_roots = np.roots(a)\n D_uniq, D_mult=signal.unique_roots(D_roots,tol=1e-3, rtype='avg')\n plt.plot(np.real(D_uniq),np.imag(D_uniq),'kx',ms=8)\n idx_D_mult = np.nonzero(np.ravel(D_mult>1))[0]\n for k in range(len(idx_D_mult)):\n x_loc = np.real(D_uniq[idx_D_mult[k]]) + x_off*x_scale\n y_loc =np.imag(D_uniq[idx_D_mult[k]]) + y_off*y_scale\n plt.text(x_loc,y_loc,str(D_mult[idx_D_mult[k]]),ha='center',va='bottom',fontsize=10)\n plt.xlabel('Real Part')\n plt.ylabel('Imaginary Part')\n plt.title('Pole-Zero Plot')\n #plt.grid()\n plt.axis(np.array(size))\n return M,N", "response": "Create an s - plane pole - zero plot."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef OS_filter(x,h,N,mode=0):\n \n P = len(h)\n # zero pad start of x so first frame can recover first true samples of x\n x = np.hstack((np.zeros(P-1),x))\n L = N - P + 1\n Nx = len(x)\n Nframe = int(np.ceil(Nx/float(L)))\n # zero pad end of x to full number of frames needed\n x = np.hstack((x,np.zeros(Nframe*L-Nx)))\n y = np.zeros(int(Nframe*N))\n # create an instrumentation matrix to observe the overlap and save behavior\n y_mat = np.zeros((Nframe,int(Nframe*N)))\n\n H = fft.fft(h,N)\n # begin the filtering operation\n for k in range(Nframe):\n xk = x[k*L:k*L+N]\n Xk = fft.fft(xk,N)\n Yk = H*Xk\n yk = np.real(fft.ifft(Yk)) # imag part should be zero\n y[k*L+P-1:k*L+N] = yk[P-1:]\n y_mat[k,k*L:k*L+N] = yk\n if mode == 1:\n return y[P-1:Nx], y_mat[:,P-1:Nx]\n else:\n return y[P-1:Nx]", "response": "This function implements the classical overlap and save method of the transform domain FIR filter."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nadding white Gaussian noise to a single real sinusoid. Input a single sinusoid to this function and it returns a noisy sinusoid at a specific SNR value in dB. Sinusoid power is calculated using np.var. Parameters ---------- x : Input signal as ndarray consisting of a single sinusoid SNRdB : SNR in dB for output sinusoid Returns ------- y : Noisy sinusoid return vector Examples -------- >>> # set the SNR to 10 dB >>> n = arange(0,10000) >>> x = cos(2*pi*0.04*n) >>> y = sinusoidAWGN(x,10.0)", "response": "def sinusoidAWGN(x,SNRdB):\n \"\"\"\n Add white Gaussian noise to a single real sinusoid.\n \n Input a single sinusoid to this function and it returns a noisy\n sinusoid at a specific SNR value in dB. Sinusoid power is calculated\n using np.var.\n \n Parameters\n ----------\n x : Input signal as ndarray consisting of a single sinusoid\n SNRdB : SNR in dB for output sinusoid\n \n Returns\n -------\n y : Noisy sinusoid return vector\n\n Examples\n --------\n >>> # set the SNR to 10 dB\n >>> n = arange(0,10000)\n >>> x = cos(2*pi*0.04*n)\n >>> y = sinusoidAWGN(x,10.0)\n \"\"\"\n # Estimate signal power\n x_pwr = np.var(x)\n\n # Create noise vector\n noise = np.sqrt(x_pwr/10**(SNRdB/10.))*np.random.randn(len(x));\n return x + noise"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the disdata of a given angle modulated signal.", "response": "def discrim(x):\r\n \"\"\"\r\n function disdata = discrim(x)\r\n where x is an angle modulated signal in complex baseband form.\r\n \r\n Mark Wickert\r\n \"\"\"\r\n X=np.real(x) # X is the real part of the received signal\r\n Y=np.imag(x) # Y is the imaginary part of the received signal\r\n b=np.array([1, -1]) # filter coefficients for discrete derivative\r\n a=np.array([1, 0]) # filter coefficients for discrete derivative\r\n derY=signal.lfilter(b,a,Y) # derivative of Y, \r\n derX=signal.lfilter(b,a,X) # \" X,\r\n disdata=(X*derY-Y*derX)/(X**2+Y**2)\r\n return disdata"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef mono_FM(x,fs=2.4e6,file_name='test.wav'):\r\n b = signal.firwin(64,2*200e3/float(fs))\r\n # Filter and decimate (should be polyphase)\r\n y = signal.lfilter(b,1,x)\r\n z = ss.downsample(y,10)\r\n # Apply complex baseband discriminator\r\n z_bb = discrim(z)\r\n # Design 2nd decimation lowpass filter (fc = 12 KHz)\r\n bb = signal.firwin(64,2*12e3/(float(fs)/10))\r\n # Filter and decimate\r\n zz_bb = signal.lfilter(bb,1,z_bb)\r\n # Decimate by 5\r\n z_out = ss.downsample(zz_bb,5)\r\n # Save to wave file\r\n ss.to_wav(file_name, 48000, z_out/2)\r\n print('Done!')\r\n return z_bb, z_out", "response": "Decimate complex baseband input by 10\r\n Design 1st decimation lowpass filter (f_c = 200 KHz)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sccs_bit_sync(y,Ns):\r\n # decimated symbol sequence for SEP\r\n rx_symb_d = np.zeros(int(np.fix(len(y)/Ns)))\r\n track = np.zeros(int(np.fix(len(y)/Ns)))\r\n bit_count = -1\r\n y_abs = np.zeros(len(y))\r\n clk = np.zeros(len(y))\r\n k = Ns+1 #initial 1-of-Ns symbol synch clock phase\r\n # Sample-by-sample processing required\r\n for i in range(len(y)):\r\n #y_abs(i) = abs(round(real(y(i))))\r\n if i >= Ns: # do not process first Ns samples\r\n # Collect timing decision unit (TDU) samples\r\n y_abs[i] = np.abs(np.sum(y[i-Ns+1:i+1]))\r\n # Update sampling instant and take a sample\r\n # For causality reason the early sample is 'i',\r\n # the on-time or prompt sample is 'i-1', and \r\n # the late sample is 'i-2'.\r\n if (k == 0):\r\n # Load the samples into the 3x1 TDU register w_hat.\r\n # w_hat[1] = late, w_hat[2] = on-time; w_hat[3] = early.\r\n w_hat = y_abs[i-2:i+1]\r\n bit_count += 1\r\n if w_hat[1] != 0:\r\n if w_hat[0] < w_hat[2]:\r\n k = Ns-1\r\n clk[i-2] = 1\r\n rx_symb_d[bit_count] = y[i-2-int(np.round(Ns/2))-1]\r\n elif w_hat[0] > w_hat[2]:\r\n k = Ns+1\r\n clk[i] = 1\r\n rx_symb_d[bit_count] = y[i-int(np.round(Ns/2))-1]\r\n else:\r\n k = Ns\r\n clk[i-1] = 1\r\n rx_symb_d[bit_count] = y[i-1-int(np.round(Ns/2))-1]\r\n else:\r\n k = Ns\r\n clk[i-1] = 1\r\n rx_symb_d[bit_count] = y[i-1-int(np.round(Ns/2))]\r\n track[bit_count] = np.mod(i,Ns)\r\n k -= 1\r\n # Trim the final output to bit_count\r\n rx_symb_d = rx_symb_d[:bit_count]\r\n return rx_symb_d, clk, track", "response": "Synthesize a sequence of baseband NRZ data into a single baseband NRZ waveform and a single baseband NRZ waveform."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef fsk_BEP(rx_data,m,flip):\r\n Nbits = len(rx_data)\r\n c = dc.m_seq(m)\r\n if flip == 1:\r\n # Flip the sequence to compenstate for mbed code difference\r\n # First make it a 1xN array\r\n c.shape = (1,len(c))\r\n c = np.fliplr(c).flatten()\r\n L = int(np.ceil(Nbits/float(len(c))))\r\n tx_data = np.dot(c.reshape(len(c),1),np.ones((1,L)))\r\n tx_data = tx_data.T.reshape((1,len(c)*L)).flatten()\r\n tx_data = tx_data[:Nbits]\r\n # Convert to +1/-1 bits\r\n tx_data = 2*tx_data - 1\r\n Bit_count,Bit_errors = dc.BPSK_BEP(rx_data,tx_data)\r\n print('len rx_data = %d, len tx_data = %d' % (len(rx_data),len(tx_data)))\r\n Pe = Bit_errors/float(Bit_count)\r\n print('/////////////////////////////////////')\r\n print('Bit Errors: %d' % Bit_errors)\r\n print('Bits Total: %d' % Bit_count)\r\n print(' BEP: %2.2e' % Pe)\r\n print('/////////////////////////////////////')", "response": "This function is used to reconstruct the BEP of the data bits recovered by the RTL - SDR Based FSK Receiver."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef complex2wav(filename,rate,x):\r\n x_wav = np.hstack((np.array([x.real]).T,np.array([x.imag]).T))\r\n ss.to_wav(filename, rate, x_wav)\r\n print('Saved as binary wav file with (I,Q)<=>(L,R)')", "response": "Write a complex signal vector to a wav file."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning a complex signal vector from a wav file that was used to store the real I and imaginary Q values of a complex signal ndarray.", "response": "def wav2complex(filename):\r\n \"\"\"\r\n Return a complex signal vector from a wav file that was used to store\r\n the real (I) and imaginary (Q) values of a complex signal ndarray. \r\n The rate is included as means of recalling the original signal sample \r\n rate.\r\n\r\n fs,x = wav2complex(filename)\r\n\r\n Mark Wickert April 2014\r\n \"\"\"\r\n fs, x_LR_cols = ss.from_wav(filename)\r\n x = x_LR_cols[:,0] + 1j*x_LR_cols[:,1]\r\n return fs,x"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef FIR_header(fname_out, h):\r\n M = len(h)\r\n N = 3 # Coefficients per line\r\n f = open(fname_out, 'wt')\r\n f.write('//define a FIR coefficient Array\\n\\n')\r\n f.write('#include \\n\\n')\r\n f.write('#ifndef M_FIR\\n')\r\n f.write('#define M_FIR %d\\n' % M)\r\n f.write('#endif\\n')\r\n f.write('/************************************************************************/\\n');\r\n f.write('/* FIR Filter Coefficients */\\n');\r\n f.write('float32_t h_FIR[M_FIR] = {')\r\n kk = 0;\r\n for k in range(M):\r\n # k_mod = k % M\r\n if (kk < N - 1) and (k < M - 1):\r\n f.write('%15.12f,' % h[k])\r\n kk += 1\r\n elif (kk == N - 1) & (k < M - 1):\r\n f.write('%15.12f,\\n' % h[k])\r\n if k < M:\r\n f.write(' ')\r\n kk = 0\r\n else:\r\n f.write('%15.12f' % h[k])\r\n f.write('};\\n')\r\n f.write('/************************************************************************/\\n')\r\n f.close()", "response": "Write the FIR filter header files"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nwrites the fixed - point filter header files.", "response": "def FIR_fix_header(fname_out, h):\r\n \"\"\"\r\n Write FIR Fixed-Point Filter Header Files \r\n \r\n Mark Wickert February 2015\r\n \"\"\"\r\n M = len(h)\r\n hq = int16(rint(h * 2 ** 15))\r\n N = 8 # Coefficients per line\r\n f = open(fname_out, 'wt')\r\n f.write('//define a FIR coefficient Array\\n\\n')\r\n f.write('#include \\n\\n')\r\n f.write('#ifndef M_FIR\\n')\r\n f.write('#define M_FIR %d\\n' % M)\r\n f.write('#endif\\n')\r\n f.write('/************************************************************************/\\n');\r\n f.write('/* FIR Filter Coefficients */\\n');\r\n f.write('int16_t h_FIR[M_FIR] = {')\r\n kk = 0;\r\n for k in range(M):\r\n # k_mod = k % M\r\n if (kk < N - 1) and (k < M - 1):\r\n f.write('%5d,' % hq[k])\r\n kk += 1\r\n elif (kk == N - 1) & (k < M - 1):\r\n f.write('%5d,\\n' % hq[k])\r\n if k < M:\r\n f.write(' ')\r\n kk = 0\r\n else:\r\n f.write('%5d' % hq[k])\r\n f.write('};\\n')\r\n f.write('/************************************************************************/\\n')\r\n f.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef IIR_sos_header(fname_out, SOS_mat):\r\n Ns, Mcol = SOS_mat.shape\r\n f = open(fname_out, 'wt')\r\n f.write('//define a IIR SOS CMSIS-DSP coefficient array\\n\\n')\r\n f.write('#include \\n\\n')\r\n f.write('#ifndef STAGES\\n')\r\n f.write('#define STAGES %d\\n' % Ns)\r\n f.write('#endif\\n')\r\n f.write('/*********************************************************/\\n');\r\n f.write('/* IIR SOS Filter Coefficients */\\n');\r\n f.write('float32_t ba_coeff[%d] = { //b0,b1,b2,a1,a2,... by stage\\n' % (5 * Ns))\r\n for k in range(Ns):\r\n if (k < Ns - 1):\r\n f.write(' %+-13e, %+-13e, %+-13e,\\n' % \\\r\n (SOS_mat[k, 0], SOS_mat[k, 1], SOS_mat[k, 2]))\r\n f.write(' %+-13e, %+-13e,\\n' % \\\r\n (-SOS_mat[k, 4], -SOS_mat[k, 5]))\r\n else:\r\n f.write(' %+-13e, %+-13e, %+-13e,\\n' % \\\r\n (SOS_mat[k, 0], SOS_mat[k, 1], SOS_mat[k, 2]))\r\n f.write(' %+-13e, %+-13e\\n' % \\\r\n (-SOS_mat[k, 4], -SOS_mat[k, 5]))\r\n # for k in range(Ns):\r\n # if (k < Ns-1):\r\n # f.write(' %15.12f, %15.12f, %15.12f,\\n' % \\\r\n # (SOS_mat[k,0],SOS_mat[k,1],SOS_mat[k,2]))\r\n # f.write(' %15.12f, %15.12f,\\n' % \\\r\n # (-SOS_mat[k,4],-SOS_mat[k,5]))\r\n # else:\r\n # f.write(' %15.12f, %15.12f, %15.12f,\\n' % \\\r\n # (SOS_mat[k,0],SOS_mat[k,1],SOS_mat[k,2]))\r\n # f.write(' %15.12f, %15.12f\\n' % \\\r\n # (-SOS_mat[k,4],-SOS_mat[k,5]))\r\n f.write('};\\n')\r\n f.write('/*********************************************************/\\n')\r\n f.close()", "response": "Write the header of the SOS - DSP file."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nwrites the header of a single CA file.", "response": "def CA_code_header(fname_out, Nca):\r\n \"\"\"\r\n Write 1023 bit CA (Gold) Code Header Files\r\n\r\n Mark Wickert February 2015\r\n \"\"\"\r\n dir_path = os.path.dirname(os.path.realpath(__file__))\r\n ca = loadtxt(dir_path + '/ca1thru37.txt', dtype=int16, usecols=(Nca - 1,), unpack=True)\r\n\r\n M = 1023 # code period\r\n N = 23 # code bits per line\r\n Sca = 'ca' + str(Nca)\r\n f = open(fname_out, 'wt')\r\n f.write('//define a CA code\\n\\n')\r\n f.write('#include \\n\\n')\r\n f.write('#ifndef N_CA\\n')\r\n f.write('#define N_CA %d\\n' % M)\r\n f.write('#endif\\n')\r\n f.write('/*******************************************************************/\\n');\r\n f.write('/* 1023 Bit CA Gold Code %2d */\\n' \\\r\n % Nca);\r\n f.write('int8_t ca%d[N_CA] = {' % Nca)\r\n kk = 0;\r\n for k in range(M):\r\n # k_mod = k % M\r\n if (kk < N - 1) and (k < M - 1):\r\n f.write('%d,' % ca[k])\r\n kk += 1\r\n elif (kk == N - 1) & (k < M - 1):\r\n f.write('%d,\\n' % ca[k])\r\n if k < M:\r\n if Nca < 10:\r\n f.write(' ')\r\n else:\r\n f.write(' ')\r\n kk = 0\r\n else:\r\n f.write('%d' % ca[k])\r\n f.write('};\\n')\r\n f.write('/*******************************************************************/\\n')\r\n f.close()"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\neye pattern plot of a baseband digital communications waveform. The signal must be real, but can be multivalued in terms of the underlying modulation scheme. Used for BPSK eye plots in the Case Study article. Parameters ---------- x : ndarray of the real input data vector/array L : display length in samples (usually two symbols) S : start index Returns ------- None : A plot window opens containing the eye plot Notes ----- Increase S to eliminate filter transients. Examples -------- 1000 bits at 10 samples per bit with 'rc' shaping. >>> import matplotlib.pyplot as plt >>> from sk_dsp_comm import digitalcom as dc >>> x,b, data = dc.NRZ_bits(1000,10,'rc') >>> dc.eye_plot(x,20,60) >>> plt.show()", "response": "def eye_plot(x,L,S=0):\n \"\"\"\n Eye pattern plot of a baseband digital communications waveform.\n\n The signal must be real, but can be multivalued in terms of the underlying\n modulation scheme. Used for BPSK eye plots in the Case Study article.\n\n Parameters\n ----------\n x : ndarray of the real input data vector/array\n L : display length in samples (usually two symbols)\n S : start index\n\n Returns\n -------\n None : A plot window opens containing the eye plot\n \n Notes\n -----\n Increase S to eliminate filter transients.\n \n Examples\n --------\n 1000 bits at 10 samples per bit with 'rc' shaping.\n\n >>> import matplotlib.pyplot as plt\n >>> from sk_dsp_comm import digitalcom as dc\n >>> x,b, data = dc.NRZ_bits(1000,10,'rc')\n >>> dc.eye_plot(x,20,60)\n >>> plt.show()\n \"\"\"\n plt.figure(figsize=(6,4))\n idx = np.arange(0,L+1)\n plt.plot(idx,x[S:S+L+1],'b')\n k_max = int((len(x) - S)/L)-1\n for k in range(1,k_max):\n plt.plot(idx,x[S+k*L:S+L+1+k*L],'b')\n plt.grid()\n plt.xlabel('Time Index - n')\n plt.ylabel('Amplitude')\n plt.title('Eye Plot')\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef scatter(x,Ns,start):\n xI = np.real(x[start::Ns])\n xQ = np.imag(x[start::Ns])\n return xI, xQ", "response": "Scatter the signal x with NRZ bits and return the real and imaginary part of x."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef strips(x,Nx,fig_size=(6,4)):\n plt.figure(figsize=fig_size)\n #ax = fig.add_subplot(111)\n N = len(x)\n Mx = int(np.ceil(N/float(Nx)))\n x_max = np.max(np.abs(x))\n for kk in range(Mx):\n plt.plot(np.array([0,Nx]),-kk*Nx*np.array([1,1]),'r-.')\n plt.plot(x[kk*Nx:(kk+1)*Nx]/x_max*0.4*Nx-kk*Nx,'b')\n plt.axis([0,Nx,-Nx*(Mx-0.5),Nx*0.5])\n plt.yticks(np.arange(0,-Nx*Mx,-Nx),np.arange(0,Nx*Mx,Nx))\n plt.xlabel('Index')\n plt.ylabel('Strip Amplitude and Starting Index')\n return 0", "response": "Plots the contents of real ndarray x as a vertical stacking of\n strips each of length Nx."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ncounting bit errors between a transmitted and received BPSK signal.", "response": "def bit_errors(tx_data,rx_data,Ncorr = 1024,Ntransient = 0):\n \"\"\"\n Count bit errors between a transmitted and received BPSK signal.\n Time delay between streams is detected as well as ambiquity resolution\n due to carrier phase lock offsets of :math:`k*\\\\pi`, k=0,1.\n The ndarray tx_data is Tx 0/1 bits as real numbers I.\n The ndarray rx_data is Rx 0/1 bits as real numbers I.\n Note: Ncorr needs to be even\n \"\"\"\n \n # Remove Ntransient symbols and level shift to {-1,+1}\n tx_data = 2*tx_data[Ntransient:]-1\n rx_data = 2*rx_data[Ntransient:]-1\n # Correlate the first Ncorr symbols at four possible phase rotations\n R0 = np.fft.ifft(np.fft.fft(rx_data,Ncorr)*\n np.conj(np.fft.fft(tx_data,Ncorr)))\n R1 = np.fft.ifft(np.fft.fft(-1*rx_data,Ncorr)*\n np.conj(np.fft.fft(tx_data,Ncorr)))\n #Place the zero lag value in the center of the array\n R0 = np.fft.fftshift(R0)\n R1 = np.fft.fftshift(R1)\n R0max = np.max(R0.real)\n R1max = np.max(R1.real)\n R = np.array([R0max,R1max])\n Rmax = np.max(R)\n kphase_max = np.where(R == Rmax)[0]\n kmax = kphase_max[0]\n # Correlation lag value is zero at the center of the array\n if kmax == 0:\n lagmax = np.where(R0.real == Rmax)[0] - Ncorr/2\n elif kmax == 1:\n lagmax = np.where(R1.real == Rmax)[0] - Ncorr/2\n taumax = lagmax[0]\n print('kmax = %d, taumax = %d' % (kmax, taumax))\n\n # Count bit and symbol errors over the entire input ndarrays\n # Begin by making tx and rx length equal and apply phase rotation to rx\n if taumax < 0:\n tx_data = tx_data[int(-taumax):]\n tx_data = tx_data[:min(len(tx_data),len(rx_data))]\n rx_data = (-1)**kmax*rx_data[:len(tx_data)]\n else:\n rx_data = (-1)**kmax * rx_data[int(taumax):]\n rx_data = rx_data[:min(len(tx_data),len(rx_data))]\n tx_data = tx_data[:len(rx_data)]\n # Convert to 0's and 1's\n Bit_count = len(tx_data)\n tx_I = np.int16((tx_data.real + 1)/2)\n rx_I = np.int16((rx_data.real + 1)/2)\n Bit_errors = tx_I ^ rx_I\n return Bit_count,np.sum(Bit_errors)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef QAM_bb(N_symb,Ns,mod_type='16qam',pulse='rect',alpha=0.35):\n # Filter the impulse train waveform with a square root raised\n # cosine pulse shape designed as follows:\n\n # Design the filter to be of duration 12 symbols and\n # fix the excess bandwidth factor at alpha = 0.35\n # If SRC = 0 use a simple rectangle pulse shape\n if pulse.lower() == 'src':\n b = sqrt_rc_imp(Ns,alpha,6)\n elif pulse.lower() == 'rc':\n b = rc_imp(Ns,alpha,6) \n elif pulse.lower() == 'rect':\n b = np.ones(int(Ns)) #alt. rect. pulse shape\n else:\n raise ValueError('pulse shape must be src, rc, or rect')\n \n if mod_type.lower() == 'qpsk':\n M = 2 # bits per symbol\n elif mod_type.lower() == '16qam':\n M = 4\n elif mod_type.lower() == '64qam':\n M = 8\n elif mod_type.lower() == '256qam':\n M = 16\n else:\n raise ValueError('Unknown mod_type')\n\n # Create random symbols for the I & Q channels\n xI = np.random.randint(0,M,N_symb)\n xI = 2*xI - (M-1)\n xQ = np.random.randint(0,M,N_symb)\n xQ = 2*xQ - (M-1)\n # Employ differential encoding to counter phase ambiquities\n # Create a zero padded (interpolated by Ns) symbol sequence.\n # This prepares the symbol sequence for arbitrary pulse shaping.\n symbI = np.hstack((xI.reshape(N_symb,1),np.zeros((N_symb,int(Ns)-1))))\n symbI = symbI.flatten()\n symbQ = np.hstack((xQ.reshape(N_symb,1),np.zeros((N_symb,int(Ns)-1))))\n symbQ = symbQ.flatten()\n symb = symbI + 1j*symbQ\n if M > 2:\n symb /= (M-1)\n \n # The impulse train waveform contains one pulse per Ns (or Ts) samples\n # imp_train = [ones(K,1) zeros(K,Ns-1)]';\n # imp_train = reshape(imp_train,Ns*K,1);\n\n # Filter the impulse train signal\n x = signal.lfilter(b,1,symb)\n x = x.flatten() # out is a 1D vector\n # Scale shaping filter to have unity DC gain\n b = b/sum(b)\n return x, b, xI+1j*xQ", "response": "A simple baseband transmitter for a set of samples."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef MPSK_bb(N_symb,Ns,M,pulse='rect',alpha = 0.25,MM=6):\n data = np.random.randint(0,M,N_symb) \n xs = np.exp(1j*2*np.pi/M*data)\n x = np.hstack((xs.reshape(N_symb,1),np.zeros((N_symb,int(Ns)-1))))\n x =x.flatten()\n if pulse.lower() == 'rect':\n b = np.ones(int(Ns))\n elif pulse.lower() == 'rc':\n b = rc_imp(Ns,alpha,MM)\n elif pulse.lower() == 'src':\n b = sqrt_rc_imp(Ns,alpha,MM)\n else:\n raise ValueError('pulse type must be rec, rc, or src')\n x = signal.lfilter(b,1,x)\n if M == 4:\n x = x*np.exp(1j*np.pi/4); # For QPSK points in quadrants\n return x,b/float(Ns),data", "response": "Generate a complex baseband MPSK signal with pulse shaping."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngenerates a BIPS - K transmitter with adjacent channel interference.", "response": "def BPSK_tx(N_bits,Ns,ach_fc=2.0,ach_lvl_dB=-100,pulse='rect',alpha = 0.25,M=6):\n \"\"\"\n Generates biphase shift keyed (BPSK) transmitter with adjacent channel interference.\n\n Generates three BPSK signals with rectangular or square root raised cosine (SRC) \n pulse shaping of duration N_bits and Ns samples per bit. The desired signal is\n centered on f = 0, which the adjacent channel signals to the left and right\n are also generated at dB level relative to the desired signal. Used in the \n digital communications Case Study supplement.\n\n Parameters\n ----------\n N_bits : the number of bits to simulate\n Ns : the number of samples per bit\n ach_fc : the frequency offset of the adjacent channel signals (default 2.0)\n ach_lvl_dB : the level of the adjacent channel signals in dB (default -100)\n pulse : the pulse shape 'rect' or 'src'\n alpha : square root raised cosine pulse shape factor (default = 0.25)\n M : square root raised cosine pulse truncation factor (default = 6)\n\n Returns\n -------\n x : ndarray of the composite signal x0 + ach_lvl*(x1p + x1m)\n b : the transmit pulse shape \n data0 : the data bits used to form the desired signal; used for error checking\n\n Notes\n -----\n\n Examples\n --------\n >>> x,b,data0 = BPSK_tx(1000,10,'src')\n \"\"\"\n x0,b,data0 = NRZ_bits(N_bits,Ns,pulse,alpha,M)\n x1p,b,data1p = NRZ_bits(N_bits,Ns,pulse,alpha,M)\n x1m,b,data1m = NRZ_bits(N_bits,Ns,pulse,alpha,M)\n n = np.arange(len(x0))\n x1p = x1p*np.exp(1j*2*np.pi*ach_fc/float(Ns)*n)\n x1m = x1m*np.exp(-1j*2*np.pi*ach_fc/float(Ns)*n)\n ach_lvl = 10**(ach_lvl_dB/20.)\n return x0 + ach_lvl*(x1p + x1m), b, data0"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef RZ_bits(N_bits,Ns,pulse='rect',alpha = 0.25,M=6):\n data = np.random.randint(0,2,N_bits) \n x = np.hstack((data.reshape(N_bits,1),np.zeros((N_bits,int(Ns)-1))))\n x =x.flatten()\n if pulse.lower() == 'rect':\n b = np.ones(int(Ns))\n elif pulse.lower() == 'rc':\n b = rc_imp(Ns,alpha,M)\n elif pulse.lower() == 'src':\n b = sqrt_rc_imp(Ns,alpha,M)\n else:\n print('pulse type must be rec, rc, or src')\n x = signal.lfilter(b,1,x)\n return x,b/float(Ns),data", "response": "Generate return - to - zero RZ data bits with pulse shaping."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef xcorr(x1,x2,Nlags):\n K = 2*(int(np.floor(len(x1)/2)))\n X1 = fft.fft(x1[:K])\n X2 = fft.fft(x2[:K])\n E1 = sum(abs(x1[:K])**2)\n E2 = sum(abs(x2[:K])**2)\n r12 = np.fft.ifft(X1*np.conj(X2))/np.sqrt(E1*E2)\n k = np.arange(K) - int(np.floor(K/2))\n r12 = np.fft.fftshift(r12)\n idx = np.nonzero(np.ravel(abs(k) <= Nlags))\n return r12[idx], k[idx]", "response": "Compute the energy normalized cross correlation between two sequences x1 and x2."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nencoding signal samples to a serial bit stream of 0 or 1 bits.", "response": "def PCM_encode(x,N_bits):\n \"\"\"\n Parameters\n ----------\n x : signal samples to be PCM encoded\n N_bits ; bit precision of PCM samples\n\n Returns\n -------\n x_bits = encoded serial bit stream of 0/1 values. MSB first.\n\n Mark Wickert, Mark 2015\n \"\"\"\n xq = np.int16(np.rint(x*2**(N_bits-1)))\n x_bits = np.zeros((N_bits,len(xq)))\n for k, xk in enumerate(xq):\n x_bits[:,k] = to_bin(xk,N_bits)\n # Reshape into a serial bit stream\n x_bits = np.reshape(x_bits,(1,len(x)*N_bits),'F')\n return np.int16(x_bits.flatten())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef to_bin(data, width):\n data_str = bin(data & (2**width-1))[2:].zfill(width)\n return [int(x) for x in tuple(data_str)]", "response": "Convert an unsigned integer to a numpy binary array with the first element of the MSB and the last element of the LSB."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_bin(bin_array):\n width = len(bin_array)\n bin_wgts = 2**np.arange(width-1,-1,-1)\n return int(np.dot(bin_array,bin_wgts))", "response": "Convert binary array back a nonnegative integer."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef PCM_decode(x_bits,N_bits):\n N_samples = len(x_bits)//N_bits\n # Convert serial bit stream into parallel words with each \n # column holdingthe N_bits binary sample value\n xrs_bits = x_bits.copy()\n xrs_bits = np.reshape(xrs_bits,(N_bits,N_samples),'F')\n # Convert N_bits binary words into signed integer values\n xq = np.zeros(N_samples)\n w = 2**np.arange(N_bits-1,-1,-1) # binary weights for bin \n # to dec conversion\n for k in range(N_samples):\n xq[k] = np.dot(xrs_bits[:,k],w) - xrs_bits[0,k]*2**N_bits\n return xq/2**(N_bits-1)", "response": "Decodes a PCM signal into a sequence of signed integer values."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef NDA_symb_sync(z,Ns,L,BnTs,zeta=0.707,I_ord=3):\n # Loop filter parameters\n K0 = -1.0 # The modulo 1 counter counts down so a sign change in loop\n Kp = 1.0\n K1 = 4*zeta/(zeta + 1/(4*zeta))*BnTs/Ns/Kp/K0\n K2 = 4/(zeta + 1/(4*zeta))**2*(BnTs/Ns)**2/Kp/K0\n zz = np.zeros(len(z),dtype=np.complex128)\n #zz = np.zeros(int(np.floor(len(z)/float(Ns))),dtype=np.complex128)\n e_tau = np.zeros(len(z))\n #e_tau = np.zeros(int(np.floor(len(z)/float(Ns))))\n #z_TED_buff = np.zeros(Ns)\n c1_buff = np.zeros(2*L+1)\n\n vi = 0\n CNT_next = 0\n mu_next = 0\n underflow = 0\n epsilon = 0\n mm = 1\n z = np.hstack(([0], z))\n for nn in range(1,Ns*int(np.floor(len(z)/float(Ns)-(Ns-1)))):\n # Define variables used in linear interpolator control\n CNT = CNT_next\n mu = mu_next\n if underflow == 1:\n if I_ord == 1:\n # Decimated interpolator output (piecewise linear)\n z_interp = mu*z[nn] + (1 - mu)*z[nn-1]\n elif I_ord == 2:\n # Decimated interpolator output (piecewise parabolic)\n # in Farrow form with alpha = 1/2\n v2 = 1/2.*np.sum(z[nn+2:nn-1-1:-1]*[1, -1, -1, 1])\n v1 = 1/2.*np.sum(z[nn+2:nn-1-1:-1]*[-1, 3, -1, -1])\n v0 = z[nn]\n z_interp = (mu*v2 + v1)*mu + v0\n elif I_ord == 3:\n # Decimated interpolator output (piecewise cubic)\n # in Farrow form\n v3 = np.sum(z[nn+2:nn-1-1:-1]*[1/6., -1/2., 1/2., -1/6.])\n v2 = np.sum(z[nn+2:nn-1-1:-1]*[0, 1/2., -1, 1/2.])\n v1 = np.sum(z[nn+2:nn-1-1:-1]*[-1/6., 1, -1/2., -1/3.])\n v0 = z[nn]\n z_interp = ((mu*v3 + v2)*mu + v1)*mu + v0\n else:\n print('Error: I_ord must 1, 2, or 3')\n # Form TED output that is smoothed using 2*L+1 samples\n # We need Ns interpolants for this TED: 0:Ns-1\n c1 = 0\n for kk in range(Ns):\n if I_ord == 1:\n # piecewise linear interp over Ns samples for TED\n z_TED_interp = mu*z[nn+kk] + (1 - mu)*z[nn-1+kk]\n elif I_ord == 2:\n # piecewise parabolic in Farrow form with alpha = 1/2\n v2 = 1/2.*np.sum(z[nn+kk+2:nn+kk-1-1:-1]*[1, -1, -1, 1])\n v1 = 1/2.*np.sum(z[nn+kk+2:nn+kk-1-1:-1]*[-1, 3, -1, -1])\n v0 = z[nn+kk]\n z_TED_interp = (mu*v2 + v1)*mu + v0\n elif I_ord == 3:\n # piecewise cubic in Farrow form\n v3 = np.sum(z[nn+kk+2:nn+kk-1-1:-1]*[1/6., -1/2., 1/2., -1/6.])\n v2 = np.sum(z[nn+kk+2:nn+kk-1-1:-1]*[0, 1/2., -1, 1/2.])\n v1 = np.sum(z[nn+kk+2:nn+kk-1-1:-1]*[-1/6., 1, -1/2., -1/3.])\n v0 = z[nn+kk]\n z_TED_interp = ((mu*v3 + v2)*mu + v1)*mu + v0\n else:\n print('Error: I_ord must 1, 2, or 3')\n c1 = c1 + np.abs(z_TED_interp)**2 * np.exp(-1j*2*np.pi/Ns*kk)\n c1 = c1/Ns\n # Update 2*L+1 length buffer for TED output smoothing\n c1_buff = np.hstack(([c1], c1_buff[:-1]))\n # Form the smoothed TED output\n epsilon = -1/(2*np.pi)*np.angle(np.sum(c1_buff)/(2*L+1))\n # Save symbol spaced (decimated to symbol rate) interpolants in zz\n zz[mm] = z_interp\n e_tau[mm] = epsilon # log the error to the output vector e\n mm += 1\n else:\n # Simple zezo-order hold interpolation between symbol samples\n # we just coast using the old value\n #epsilon = 0\n pass\n vp = K1*epsilon # proportional component of loop filter\n vi = vi + K2*epsilon # integrator component of loop filter\n v = vp + vi # loop filter output\n W = 1/float(Ns) + v # counter control word\n \n # update registers\n CNT_next = CNT - W # Update counter value for next cycle\n if CNT_next < 0: # Test to see if underflow has occured\n CNT_next = 1 + CNT_next # Reduce counter value modulo-1 if underflow\n underflow = 1 # Set the underflow flag\n mu_next = CNT/W # update mu\n else:\n underflow = 0\n mu_next = mu\n # Remove zero samples at end\n zz = zz[:-(len(zz)-mm+1)]\n # Normalize so symbol values have a unity magnitude\n zz /=np.std(zz)\n e_tau = e_tau[:-(len(e_tau)-mm+1)]\n return zz, e_tau", "response": "This function is used to generate a NDA filter for a given baseband signal at nominal number of samples and time period."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef time_step(z,Ns,t_step,Nstep):\n z_step = np.hstack((z[:Ns*Nstep], z[(Ns*Nstep+t_step):], np.zeros(t_step)))\n return z_step", "response": "Create a one sample per symbol signal containing a phase rotation and step Nsymb into the waveform."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef phase_step(z,Ns,p_step,Nstep):\n nn = np.arange(0,len(z[::Ns]))\n theta = np.zeros(len(nn))\n idx = np.where(nn >= Nstep)\n theta[idx] = p_step*np.ones(len(idx))\n z_rot = z[::Ns]*np.exp(1j*theta)\n return z_rot", "response": "Create a one sample per symbol signal containing a phase rotation\n step Nsymb into the waveform."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef PLL1(theta,fs,loop_type,Kv,fn,zeta,non_lin):\n T = 1/float(fs)\n Kv = 2*np.pi*Kv # convert Kv in Hz/v to rad/s/v\n\n if loop_type == 1:\n # First-order loop parameters\n # Note Bn = K/4 Hz but K has units of rad/s\n #fn = 4*Bn/(2*pi);\n K = 2*np.pi*fn # loop natural frequency in rad/s\n elif loop_type == 2:\n # Second-order loop parameters\n #fn = 1/(2*pi) * 2*Bn/(zeta + 1/(4*zeta));\n K = 4 *np.pi*zeta*fn # loop natural frequency in rad/s\n tau2 = zeta/(np.pi*fn)\n elif loop_type == 3:\n # Second-order loop parameters for one-pole lowpass with\n # phase lead correction.\n #fn = 1/(2*pi) * 2*Bn/(zeta + 1/(4*zeta));\n K = Kv # Essentially the VCO gain sets the single-sided\n # hold-in range in Hz, as it is assumed that Kp = 1\n # and KLF = 1.\n tau1 = K/((2*np.pi*fn)**2)\n tau2 = 2*zeta/(2*np.pi*fn)*(1 - 2*np.pi*fn/K*1/(2*zeta))\n else:\n print('Loop type must be 1, 2, or 3')\n\n # Initialize integration approximation filters\n filt_in_last = 0; filt_out_last = 0;\n vco_in_last = 0; vco_out = 0; vco_out_last = 0;\n\n # Initialize working and final output vectors\n n = np.arange(len(theta))\n theta_hat = np.zeros_like(theta)\n ev = np.zeros_like(theta)\n phi = np.zeros_like(theta)\n\n # Begin the simulation loop\n for k in range(len(n)):\n phi[k] = theta[k] - vco_out\n if non_lin == 1:\n # sinusoidal phase detector\n pd_out = np.sin(phi[k])\n else:\n # Linear phase detector\n pd_out = phi[k]\n # Loop gain\n gain_out = K/Kv*pd_out # apply VCO gain at VCO\n # Loop filter\n if loop_type == 2:\n filt_in = (1/tau2)*gain_out\n filt_out = filt_out_last + T/2*(filt_in + filt_in_last)\n filt_in_last = filt_in\n filt_out_last = filt_out\n filt_out = filt_out + gain_out\n elif loop_type == 3:\n filt_in = (tau2/tau1)*gain_out - (1/tau1)*filt_out_last\n u3 = filt_in + (1/tau2)*filt_out_last\n filt_out = filt_out_last + T/2*(filt_in + filt_in_last)\n filt_in_last = filt_in\n filt_out_last = filt_out\n else:\n filt_out = gain_out;\n # VCO\n vco_in = filt_out\n if loop_type == 3:\n vco_in = u3\n vco_out = vco_out_last + T/2*(vco_in + vco_in_last)\n vco_in_last = vco_in\n vco_out_last = vco_out\n vco_out = Kv*vco_out # apply Kv\n # Measured loop signals\n ev[k] = vco_in\n theta_hat[k] = vco_out\n return theta_hat, ev, phi", "response": "This function is used to generate the baseband analog PLL model for the first loop in the baseband."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a new alphabet with upper bound for the code rate R and Ck.", "response": "def conv_Pb_bound(R,dfree,Ck,SNRdB,hard_soft,M=2):\r\n \"\"\"\r\n Coded bit error probabilty\r\n\r\n Convolution coding bit error probability upper bound\r\n according to Ziemer & Peterson 7-16, p. 507\r\n \r\n Mark Wickert November 2014\r\n\r\n Parameters\r\n ----------\r\n R: Code rate\r\n dfree: Free distance of the code\r\n Ck: Weight coefficient\r\n SNRdB: Signal to noise ratio in dB\r\n hard_soft: 0 hard, 1 soft, 2 uncoded\r\n M: M-ary\r\n\r\n Examples\r\n --------\r\n >>> import numpy as np\r\n >>> from sk_dsp_comm import fec_conv as fec\r\n >>> import matplotlib.pyplot as plt\r\n >>> SNRdB = np.arange(2,12,.1)\r\n >>> Pb = fec.conv_Pb_bound(1./2,10,[36, 0, 211, 0, 1404, 0, 11633],SNRdB,2)\r\n >>> Pb_1_2 = fec.conv_Pb_bound(1./2,10,[36, 0, 211, 0, 1404, 0, 11633],SNRdB,1)\r\n >>> Pb_3_4 = fec.conv_Pb_bound(3./4,4,[164, 0, 5200, 0, 151211, 0, 3988108],SNRdB,1)\r\n >>> plt.semilogy(SNRdB,Pb)\r\n >>> plt.semilogy(SNRdB,Pb_1_2)\r\n >>> plt.semilogy(SNRdB,Pb_3_4)\r\n >>> plt.axis([2,12,1e-7,1e0])\r\n >>> plt.xlabel(r'$E_b/N_0$ (dB)')\r\n >>> plt.ylabel(r'Symbol Error Probability')\r\n >>> plt.legend(('Uncoded BPSK','R=1/2, K=7, Soft','R=3/4 (punc), K=7, Soft'),loc='best')\r\n >>> plt.grid();\r\n >>> plt.show()\r\n\r\n Notes\r\n -----\r\n The code rate R is given by :math:`R_{s} = \\\\frac{k}{n}`.\r\n Mark Wickert and Andrew Smit 2018\r\n \"\"\"\r\n Pb = np.zeros_like(SNRdB)\r\n SNR = 10.**(SNRdB/10.)\r\n\r\n for n,SNRn in enumerate(SNR):\r\n for k in range(dfree,len(Ck)+dfree):\r\n if hard_soft == 0: # Evaluate hard decision bound\r\n Pb[n] += Ck[k-dfree]*hard_Pk(k,R,SNRn,M)\r\n elif hard_soft == 1: # Evaluate soft decision bound\r\n Pb[n] += Ck[k-dfree]*soft_Pk(k,R,SNRn,M)\r\n else: # Compute Uncoded Pe\r\n if M == 2:\r\n Pb[n] = Q_fctn(np.sqrt(2.*SNRn))\r\n else:\r\n Pb[n] = 4./np.log2(M)*(1 - 1/np.sqrt(M))*\\\r\n np.gaussQ(np.sqrt(3*np.log2(M)/(M-1)*SNRn));\r\n return Pb"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef hard_Pk(k,R,SNR,M=2):\r\n\r\n k = int(k)\r\n\r\n if M == 2:\r\n p = Q_fctn(np.sqrt(2.*R*SNR))\r\n else:\r\n p = 4./np.log2(M)*(1 - 1./np.sqrt(M))*\\\r\n Q_fctn(np.sqrt(3*R*np.log2(M)/float(M-1)*SNR))\r\n Pk = 0\r\n #if 2*k//2 == k:\r\n if np.mod(k,2) == 0:\r\n for e in range(int(k/2+1),int(k+1)):\r\n Pk += float(factorial(k))/(factorial(e)*factorial(k-e))*p**e*(1-p)**(k-e);\r\n # Pk += 1./2*float(factorial(k))/(factorial(int(k/2))*factorial(int(k-k/2)))*\\\r\n # p**(k/2)*(1-p)**(k//2);\r\n Pk += 1./2*float(factorial(k))/(factorial(int(k/2))*factorial(int(k-k/2)))*\\\r\n p**(k/2)*(1-p)**(k/2);\r\n elif np.mod(k,2) == 1:\r\n for e in range(int((k+1)//2),int(k+1)):\r\n Pk += factorial(k)/(factorial(e)*factorial(k-e))*p**e*(1-p)**(k-e);\r\n return Pk", "response": "Calculates the hard version of the image."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ncalculate Pk for a given time series and time series.", "response": "def soft_Pk(k,R,SNR,M=2):\r\n \"\"\"\r\n Pk = soft_Pk(k,R,SNR)\r\n \r\n Calculates Pk as found in Ziemer & Peterson eq. 7-13, p.505\r\n \r\n Mark Wickert November 2014\r\n \"\"\"\r\n if M == 2:\r\n Pk = Q_fctn(np.sqrt(2.*k*R*SNR))\r\n else:\r\n Pk = 4./np.log2(M)*(1 - 1./np.sqrt(M))*\\\r\n Q_fctn(np.sqrt(3*k*R*np.log2(M)/float(M-1)*SNR))\r\n \r\n return Pk"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef viterbi_decoder(self,x,metric_type='soft',quant_level=3):\r\n if metric_type == 'hard':\r\n # If hard decision must have 0/1 integers for input else float\r\n if np.issubdtype(x.dtype, np.integer):\r\n if x.max() > 1 or x.min() < 0:\r\n raise ValueError('Integer bit values must be 0 or 1')\r\n else:\r\n raise ValueError('Decoder inputs must be integers on [0,1] for hard decisions')\r\n # Initialize cumulative metrics array\r\n cm_present = np.zeros((self.Nstates,1))\r\n\r\n NS = len(x) # number of channel symbols to process; \r\n # must be even for rate 1/2\r\n # must be a multiple of 3 for rate 1/3\r\n y = np.zeros(NS-self.decision_depth) # Decoded bit sequence\r\n k = 0\r\n symbolL = self.rate.denominator\r\n\r\n # Calculate branch metrics and update traceback states and traceback bits\r\n for n in range(0,NS,symbolL):\r\n cm_past = self.paths.cumulative_metric[:,0]\r\n tb_states_temp = self.paths.traceback_states[:,:-1].copy()\r\n tb_bits_temp = self.paths.traceback_bits[:,:-1].copy()\r\n for m in range(self.Nstates):\r\n d1 = self.bm_calc(self.branches.bits1[m],\r\n x[n:n+symbolL],metric_type,\r\n quant_level)\r\n d1 = d1 + cm_past[self.branches.states1[m]]\r\n d2 = self.bm_calc(self.branches.bits2[m],\r\n x[n:n+symbolL],metric_type,\r\n quant_level)\r\n d2 = d2 + cm_past[self.branches.states2[m]]\r\n if d1 <= d2: # Find the survivor assuming minimum distance wins\r\n cm_present[m] = d1\r\n self.paths.traceback_states[m,:] = np.hstack((self.branches.states1[m],\r\n tb_states_temp[int(self.branches.states1[m]),:]))\r\n self.paths.traceback_bits[m,:] = np.hstack((self.branches.input1[m],\r\n tb_bits_temp[int(self.branches.states1[m]),:]))\r\n else:\r\n cm_present[m] = d2\r\n self.paths.traceback_states[m,:] = np.hstack((self.branches.states2[m],\r\n tb_states_temp[int(self.branches.states2[m]),:]))\r\n self.paths.traceback_bits[m,:] = np.hstack((self.branches.input2[m],\r\n tb_bits_temp[int(self.branches.states2[m]),:]))\r\n # Update cumulative metric history\r\n self.paths.cumulative_metric = np.hstack((cm_present, \r\n self.paths.cumulative_metric[:,:-1]))\r\n \r\n # Obtain estimate of input bit sequence from the oldest bit in \r\n # the traceback having the smallest (most likely) cumulative metric\r\n min_metric = min(self.paths.cumulative_metric[:,0])\r\n min_idx = np.where(self.paths.cumulative_metric[:,0] == min_metric)\r\n if n >= symbolL*self.decision_depth-symbolL: # 2 since Rate = 1/2\r\n y[k] = self.paths.traceback_bits[min_idx[0][0],-1]\r\n k += 1\r\n y = y[:k] # trim final length\r\n return y", "response": "A method which performs Viterbi decoding of the given noisy bit stream and returns a new set of noisy bit values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef bm_calc(self,ref_code_bits, rec_code_bits, metric_type, quant_level):\r\n distance = 0\r\n if metric_type == 'soft': # squared distance metric\r\n bits = binary(int(ref_code_bits),self.rate.denominator)\r\n for k in range(len(bits)):\r\n ref_bit = (2**quant_level-1)*int(bits[k],2)\r\n distance += (int(rec_code_bits[k]) - ref_bit)**2\r\n elif metric_type == 'hard': # hard decisions\r\n bits = binary(int(ref_code_bits),self.rate.denominator)\r\n for k in range(len(rec_code_bits)):\r\n distance += abs(rec_code_bits[k] - int(bits[k]))\r\n elif metric_type == 'unquant': # unquantized\r\n bits = binary(int(ref_code_bits),self.rate.denominator)\r\n for k in range(len(bits)):\r\n distance += (float(rec_code_bits[k])-float(bits[k]))**2\r\n else:\r\n print('Invalid metric type specified')\r\n raise ValueError('Invalid metric type specified. Use soft, hard, or unquant')\r\n return distance", "response": "Calculates the distance between two records in the branch and a given metric type and quant level."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\noutputs state = conv_encoder(input,state) We get the 1/2 or 1/3 rate from self.rate Polys G1 and G2 are entered as binary strings, e.g, G1 = '111' and G2 = '101' for K = 3 G1 = '1011011' and G2 = '1111001' for K = 7 G3 is also included for rate 1/3 Input state as a binary string of length K-1, e.g., '00' or '0000000' e.g., state = '00' for K = 3 e.g., state = '000000' for K = 7 Mark Wickert and Andrew Smit 2018", "response": "def conv_encoder(self,input,state):\r\n \"\"\"\r\n output, state = conv_encoder(input,state)\r\n We get the 1/2 or 1/3 rate from self.rate\r\n Polys G1 and G2 are entered as binary strings, e.g,\r\n G1 = '111' and G2 = '101' for K = 3\r\n G1 = '1011011' and G2 = '1111001' for K = 7\r\n G3 is also included for rate 1/3\r\n Input state as a binary string of length K-1, e.g., '00' or '0000000' \r\n e.g., state = '00' for K = 3\r\n e.g., state = '000000' for K = 7\r\n Mark Wickert and Andrew Smit 2018\r\n \"\"\"\r\n\r\n output = []\r\n\r\n if(self.rate == Fraction(1,2)):\r\n for n in range(len(input)):\r\n u1 = int(input[n])\r\n u2 = int(input[n])\r\n for m in range(1,self.constraint_length):\r\n if int(self.G_polys[0][m]) == 1: # XOR if we have a connection\r\n u1 = u1 ^ int(state[m-1])\r\n if int(self.G_polys[1][m]) == 1: # XOR if we have a connection\r\n u2 = u2 ^ int(state[m-1])\r\n # G1 placed first, G2 placed second\r\n output = np.hstack((output, [u1, u2]))\r\n state = bin(int(input[n]))[-1] + state[:-1]\r\n elif(self.rate == Fraction(1,3)):\r\n for n in range(len(input)):\r\n if(int(self.G_polys[0][0]) == 1):\r\n u1 = int(input[n])\r\n else:\r\n u1 = 0\r\n if(int(self.G_polys[1][0]) == 1):\r\n u2 = int(input[n])\r\n else:\r\n u2 = 0\r\n if(int(self.G_polys[2][0]) == 1):\r\n u3 = int(input[n])\r\n else:\r\n u3 = 0\r\n for m in range(1,self.constraint_length):\r\n if int(self.G_polys[0][m]) == 1: # XOR if we have a connection\r\n u1 = u1 ^ int(state[m-1])\r\n if int(self.G_polys[1][m]) == 1: # XOR if we have a connection\r\n u2 = u2 ^ int(state[m-1])\r\n if int(self.G_polys[2][m]) == 1: # XOR if we have a connection\r\n u3 = u3 ^ int(state[m-1])\r\n # G1 placed first, G2 placed second, G3 placed third\r\n output = np.hstack((output, [u1, u2, u3]))\r\n \r\n state = bin(int(input[n]))[-1] + state[:-1]\r\n\r\n return output, state"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\napply puncture to the serial bits produced by convolutionally encoding.", "response": "def puncture(self,code_bits,puncture_pattern = ('110','101')):\r\n \"\"\"\r\n Apply puncturing to the serial bits produced by convolutionally\r\n encoding.\r\n\r\n :param code_bits:\r\n :param puncture_pattern:\r\n :return:\r\n\r\n Examples\r\n --------\r\n This example uses the following puncture matrix:\r\n\r\n .. math::\r\n\r\n \\\\begin{align*}\r\n \\\\mathbf{A} = \\\\begin{bmatrix}\r\n 1 & 1 & 0 \\\\\\\\\r\n 1 & 0 & 1\r\n \\\\end{bmatrix}\r\n \\\\end{align*}\r\n\r\n The upper row operates on the outputs for the :math:`G_{1}` polynomial and the lower row operates on the outputs of\r\n the :math:`G_{2}` polynomial.\r\n\r\n >>> import numpy as np\r\n >>> from sk_dsp_comm.fec_conv import fec_conv\r\n >>> cc = fec_conv(('101','111'))\r\n >>> x = np.array([0, 0, 1, 1, 1, 0, 0, 0, 0, 0])\r\n >>> state = '00'\r\n >>> y, state = cc.conv_encoder(x, state)\r\n >>> cc.puncture(y, ('110','101'))\r\n array([ 0., 0., 0., 1., 1., 0., 0., 0., 1., 1., 0., 0.])\r\n \"\"\"\r\n # Check to see that the length of code_bits is consistent with a rate\r\n # 1/2 code.\r\n L_pp = len(puncture_pattern[0])\r\n N_codewords = int(np.floor(len(code_bits)/float(2)))\r\n if 2*N_codewords != len(code_bits):\r\n warnings.warn('Number of code bits must be even!')\r\n warnings.warn('Truncating bits to be compatible.')\r\n code_bits = code_bits[:2*N_codewords]\r\n # Extract the G1 and G2 encoded bits from the serial stream.\r\n # Assume the stream is of the form [G1 G2 G1 G2 ... ]\r\n x_G1 = code_bits.reshape(N_codewords,2).take([0],\r\n axis=1).reshape(1,N_codewords).flatten()\r\n x_G2 = code_bits.reshape(N_codewords,2).take([1],\r\n axis=1).reshape(1,N_codewords).flatten()\r\n # Check to see that the length of x_G1 and x_G2 is consistent with the\r\n # length of the puncture pattern\r\n N_punct_periods = int(np.floor(N_codewords/float(L_pp)))\r\n if L_pp*N_punct_periods != N_codewords:\r\n warnings.warn('Code bit length is not a multiple pp = %d!' % L_pp)\r\n warnings.warn('Truncating bits to be compatible.')\r\n x_G1 = x_G1[:L_pp*N_punct_periods]\r\n x_G2 = x_G2[:L_pp*N_punct_periods]\r\n #Puncture x_G1 and x_G1\r\n g1_pp1 = [k for k,g1 in enumerate(puncture_pattern[0]) if g1 == '1']\r\n g2_pp1 = [k for k,g2 in enumerate(puncture_pattern[1]) if g2 == '1']\r\n N_pp = len(g1_pp1)\r\n y_G1 = x_G1.reshape(N_punct_periods,L_pp).take(g1_pp1,\r\n axis=1).reshape(N_pp*N_punct_periods,1)\r\n y_G2 = x_G2.reshape(N_punct_periods,L_pp).take(g2_pp1,\r\n axis=1).reshape(N_pp*N_punct_periods,1)\r\n # Interleave y_G1 and y_G2 for modulation via a serial bit stream\r\n y = np.hstack((y_G1,y_G2)).reshape(1,2*N_pp*N_punct_periods).flatten()\r\n return y"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the values of the given soft bits in the channel that can be used to decode the next set of soft bits.", "response": "def depuncture(self,soft_bits,puncture_pattern = ('110','101'),\r\n erase_value = 3.5):\r\n \"\"\"\r\n Apply de-puncturing to the soft bits coming from the channel. Erasure bits\r\n are inserted to return the soft bit values back to a form that can be\r\n Viterbi decoded.\r\n\r\n :param soft_bits:\r\n :param puncture_pattern:\r\n :param erase_value:\r\n :return:\r\n\r\n Examples\r\n --------\r\n This example uses the following puncture matrix:\r\n\r\n .. math::\r\n\r\n \\\\begin{align*}\r\n \\\\mathbf{A} = \\\\begin{bmatrix}\r\n 1 & 1 & 0 \\\\\\\\\r\n 1 & 0 & 1\r\n \\\\end{bmatrix}\r\n \\\\end{align*}\r\n\r\n The upper row operates on the outputs for the :math:`G_{1}` polynomial and the lower row operates on the outputs of\r\n the :math:`G_{2}` polynomial.\r\n\r\n >>> import numpy as np\r\n >>> from sk_dsp_comm.fec_conv import fec_conv\r\n >>> cc = fec_conv(('101','111'))\r\n >>> x = np.array([0, 0, 1, 1, 1, 0, 0, 0, 0, 0])\r\n >>> state = '00'\r\n >>> y, state = cc.conv_encoder(x, state)\r\n >>> yp = cc.puncture(y, ('110','101'))\r\n >>> cc.depuncture(yp, ('110', '101'), 1)\r\n array([ 0., 0., 0., 1., 1., 1., 1., 0., 0., 1., 1., 0., 1., 1., 0., 1., 1., 0.]\r\n \"\"\"\r\n # Check to see that the length of soft_bits is consistent with a rate\r\n # 1/2 code.\r\n L_pp = len(puncture_pattern[0])\r\n L_pp1 = len([g1 for g1 in puncture_pattern[0] if g1 == '1'])\r\n L_pp0 = len([g1 for g1 in puncture_pattern[0] if g1 == '0'])\r\n #L_pp0 = len([g1 for g1 in pp1 if g1 == '0'])\r\n N_softwords = int(np.floor(len(soft_bits)/float(2)))\r\n if 2*N_softwords != len(soft_bits):\r\n warnings.warn('Number of soft bits must be even!')\r\n warnings.warn('Truncating bits to be compatible.')\r\n soft_bits = soft_bits[:2*N_softwords]\r\n # Extract the G1p and G2p encoded bits from the serial stream.\r\n # Assume the stream is of the form [G1p G2p G1p G2p ... ],\r\n # which for QPSK may be of the form [Ip Qp Ip Qp Ip Qp ... ]\r\n x_G1 = soft_bits.reshape(N_softwords,2).take([0],\r\n axis=1).reshape(1,N_softwords).flatten()\r\n x_G2 = soft_bits.reshape(N_softwords,2).take([1],\r\n axis=1).reshape(1,N_softwords).flatten()\r\n # Check to see that the length of x_G1 and x_G2 is consistent with the\r\n # puncture length period of the soft bits\r\n N_punct_periods = int(np.floor(N_softwords/float(L_pp1)))\r\n if L_pp1*N_punct_periods != N_softwords:\r\n warnings.warn('Number of soft bits per puncture period is %d' % L_pp1)\r\n warnings.warn('The number of soft bits is not a multiple')\r\n warnings.warn('Truncating soft bits to be compatible.')\r\n x_G1 = x_G1[:L_pp1*N_punct_periods]\r\n x_G2 = x_G2[:L_pp1*N_punct_periods]\r\n x_G1 = x_G1.reshape(N_punct_periods,L_pp1)\r\n x_G2 = x_G2.reshape(N_punct_periods,L_pp1)\r\n #Depuncture x_G1 and x_G1\r\n g1_pp1 = [k for k,g1 in enumerate(puncture_pattern[0]) if g1 == '1']\r\n g1_pp0 = [k for k,g1 in enumerate(puncture_pattern[0]) if g1 == '0']\r\n g2_pp1 = [k for k,g2 in enumerate(puncture_pattern[1]) if g2 == '1']\r\n g2_pp0 = [k for k,g2 in enumerate(puncture_pattern[1]) if g2 == '0']\r\n x_E = erase_value*np.ones((N_punct_periods,L_pp0))\r\n y_G1 = np.hstack((x_G1,x_E))\r\n y_G2 = np.hstack((x_G2,x_E))\r\n [g1_pp1.append(val) for idx,val in enumerate(g1_pp0)]\r\n g1_comp = list(zip(g1_pp1,list(range(L_pp))))\r\n g1_comp.sort()\r\n G1_col_permute = [g1_comp[idx][1] for idx in range(L_pp)]\r\n [g2_pp1.append(val) for idx,val in enumerate(g2_pp0)]\r\n g2_comp = list(zip(g2_pp1,list(range(L_pp))))\r\n g2_comp.sort()\r\n G2_col_permute = [g2_comp[idx][1] for idx in range(L_pp)]\r\n #permute columns to place erasure bits in the correct position\r\n y = np.hstack((y_G1[:,G1_col_permute].reshape(L_pp*N_punct_periods,1),\r\n y_G2[:,G2_col_permute].reshape(L_pp*N_punct_periods,\r\n 1))).reshape(1,2*L_pp*N_punct_periods).flatten()\r\n return y"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns a trellis diagram of the possible state transitions.", "response": "def trellis_plot(self,fsize=(6,4)):\r\n \"\"\"\r\n Plots a trellis diagram of the possible state transitions.\r\n\r\n Parameters\r\n ----------\r\n fsize : Plot size for matplotlib.\r\n\r\n Examples\r\n --------\r\n >>> import matplotlib.pyplot as plt\r\n >>> from sk_dsp_comm.fec_conv import fec_conv\r\n >>> cc = fec_conv()\r\n >>> cc.trellis_plot()\r\n >>> plt.show()\r\n \"\"\"\r\n\r\n branches_from = self.branches\r\n plt.figure(figsize=fsize)\r\n\r\n plt.plot(0,0,'.')\r\n plt.axis([-0.01, 1.01, -(self.Nstates-1)-0.05, 0.05])\r\n for m in range(self.Nstates):\r\n if branches_from.input1[m] == 0:\r\n plt.plot([0, 1],[-branches_from.states1[m], -m],'b')\r\n plt.plot([0, 1],[-branches_from.states1[m], -m],'r.')\r\n if branches_from.input2[m] == 0:\r\n plt.plot([0, 1],[-branches_from.states2[m], -m],'b')\r\n plt.plot([0, 1],[-branches_from.states2[m], -m],'r.')\r\n if branches_from.input1[m] == 1:\r\n plt.plot([0, 1],[-branches_from.states1[m], -m],'g')\r\n plt.plot([0, 1],[-branches_from.states1[m], -m],'r.')\r\n if branches_from.input2[m] == 1:\r\n plt.plot([0, 1],[-branches_from.states2[m], -m],'g')\r\n plt.plot([0, 1],[-branches_from.states2[m], -m],'r.')\r\n #plt.grid()\r\n plt.xlabel('One Symbol Transition')\r\n plt.ylabel('-State Index')\r\n msg = 'Rate %s, K = %d Trellis' %(self.rate, int(np.ceil(np.log2(self.Nstates)+1)))\r\n plt.title(msg)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef traceback_plot(self,fsize=(6,4)):\r\n traceback_states = self.paths.traceback_states\r\n plt.figure(figsize=fsize)\r\n plt.axis([-self.decision_depth+1, 0, \r\n -(self.Nstates-1)-0.5, 0.5])\r\n M,N = traceback_states.shape\r\n traceback_states = -traceback_states[:,::-1]\r\n\r\n plt.plot(range(-(N-1),0+1),traceback_states.T)\r\n plt.xlabel('Traceback Symbol Periods')\r\n plt.ylabel('State Index $0$ to -$2^{(K-1)}$')\r\n plt.title('Survivor Paths Traced Back From All %d States' % self.Nstates)\r\n plt.grid()", "response": "Plot the traceback of the possible last 4 states."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef up(self,x):\n y = self.M*ssd.upsample(x,self.M)\n y = signal.lfilter(self.b,self.a,y)\n return y", "response": "Upsample and filter the signal"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dn(self,x):\n y = signal.lfilter(self.b,self.a,x)\n y = ssd.downsample(y,self.M)\n return y", "response": "Downsample and filter the signal"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef filter(self,x):\n y = signal.lfilter(self.b,[1],x)\n return y", "response": "Filter the signal by the given value"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nplot the poles and zeros of the FIR filter in the z - plane", "response": "def zplane(self,auto_scale=True,size=2,detect_mult=True,tol=0.001):\n \"\"\"\n Plot the poles and zeros of the FIR filter in the z-plane\n \"\"\"\n ssd.zplane(self.b,[1],auto_scale,size,tol)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef filter(self,x):\n y = signal.sosfilt(self.sos,x)\n return y", "response": "Filter the signal using second - order sections"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dn(self,x,M_change = 12):\n y = signal.sosfilt(self.sos,x)\n y = ssd.downsample(y,M_change)\n return y", "response": "Downsample and filter the signal"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nplot the frequency response of the current species", "response": "def freq_resp(self, mode= 'dB', fs = 8000, ylim = [-100,2]):\n \"\"\"\n Frequency response plot\n \"\"\"\n iir_d.freqz_resp_cas_list([self.sos],mode,fs=fs)\n pylab.grid()\n pylab.ylim(ylim)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nplots the poles and zeros of the FIR filter in the z - plane", "response": "def zplane(self,auto_scale=True,size=2,detect_mult=True,tol=0.001):\n \"\"\"\n Plot the poles and zeros of the FIR filter in the z-plane\n \"\"\"\n iir_d.sos_zplane(self.sos,auto_scale,size,tol)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ser2ber(q,n,d,t,ps):\n lnps = len(ps) # len of error vector\n ber = np.zeros(lnps) # inialize output vector\n for k in range(0,lnps): # iterate error vector\n ser = ps[k] # channel symbol error rate\n sum1 = 0 # initialize sums\n sum2 = 0\n for i in range(t+1,d+1):\n term = special.comb(n,i)*(ser**i)*((1-ser))**(n-i)\n sum1 = sum1 + term\n for i in range(d+1,n+1):\n term = (i)*special.comb(n,i)*(ser**i)*((1-ser)**(n-i))\n sum2 = sum2+term\n ber[k] = (q/(2*(q-1)))*((d/n)*sum1+(1/n)*sum2)\n \n return ber", "response": "This function converts a symbol error rate to bit error rate."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds the bit error probability bounds according to Ziemer and Tranter page 656. parameters: ----------- j: number of parity bits used in single error correction block code SNRdB: Eb/N0 values in dB coded: Select single error correction code (True) or uncoded (False) M: modulation order returns: -------- Pb: bit error probability bound", "response": "def block_single_error_Pb_bound(j,SNRdB,coded=True,M=2):\n \"\"\"\n Finds the bit error probability bounds according to Ziemer and Tranter \n page 656.\n \n parameters:\n -----------\n j: number of parity bits used in single error correction block code\n SNRdB: Eb/N0 values in dB\n coded: Select single error correction code (True) or uncoded (False)\n M: modulation order\n \n returns:\n --------\n Pb: bit error probability bound\n \n \"\"\"\n Pb = np.zeros_like(SNRdB)\n Ps = np.zeros_like(SNRdB)\n SNR = 10.**(SNRdB/10.)\n n = 2**j-1\n k = n-j\n \n for i,SNRn in enumerate(SNR):\n if coded: # compute Hamming code Ps\n if M == 2:\n Ps[i] = Q_fctn(np.sqrt(k*2.*SNRn/n))\n else:\n Ps[i] = 4./np.log2(M)*(1 - 1/np.sqrt(M))*\\\n np.gaussQ(np.sqrt(3*np.log2(M)/(M-1)*SNRn))/k\n else: # Compute Uncoded Pb\n if M == 2:\n Pb[i] = Q_fctn(np.sqrt(2.*SNRn))\n else:\n Pb[i] = 4./np.log2(M)*(1 - 1/np.sqrt(M))*\\\n np.gaussQ(np.sqrt(3*np.log2(M)/(M-1)*SNRn))\n \n # Convert symbol error probability to bit error probability\n if coded:\n Pb = ser2ber(M,n,3,1,Ps)\n return Pb"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef hamm_gen(self,j):\n if(j < 3):\n raise ValueError('j must be > 2')\n\n # calculate codeword length\n n = 2**j-1\n \n # calculate source bit length\n k = n-j\n \n # Allocate memory for Matrices\n G = np.zeros((k,n),dtype=int)\n H = np.zeros((j,n),dtype=int)\n P = np.zeros((j,k),dtype=int)\n R = np.zeros((k,n),dtype=int)\n \n # Encode parity-check matrix columns with binary 1-n\n for i in range(1,n+1):\n b = list(binary(i,j))\n for m in range(0,len(b)):\n b[m] = int(b[m])\n H[:,i-1] = np.array(b)\n\n # Reformat H to be systematic\n H1 = np.zeros((1,j),dtype=int)\n H2 = np.zeros((1,j),dtype=int)\n for i in range(0,j):\n idx1 = 2**i-1\n idx2 = n-i-1\n H1[0,:] = H[:,idx1]\n H2[0,:] = H[:,idx2]\n H[:,idx1] = H2\n H[:,idx2] = H1\n \n # Get parity matrix from H\n P = H[:,:k]\n \n # Use P to calcuate generator matrix P\n G[:,:k] = np.diag(np.ones(k))\n G[:,k:] = P.T\n \n # Get k x k identity matrix\n R[:,:k] = np.diag(np.ones(k))\n\n return G, H, R, n, k", "response": "Generate parity check matrix G H and generator matrix H and R."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nencode input bit array x using hamming block code. parameters ---------- x: array of source bits to be encoded by block encoder. returns ------- codewords: array of code words generated by generator matrix G and input x. Andrew Smit November 2018", "response": "def hamm_encoder(self,x):\n \"\"\"\n Encodes input bit array x using hamming block code.\n \n parameters\n ----------\n x: array of source bits to be encoded by block encoder.\n \n returns\n -------\n codewords: array of code words generated by generator\n matrix G and input x.\n \n Andrew Smit November 2018\n \"\"\"\n if(np.dtype(x[0]) != int):\n raise ValueError('Error: Invalid data type. Input must be a vector of ints')\n\n if(len(x) % self.k or len(x) < self.k):\n raise ValueError('Error: Invalid input vector length. Length must be a multiple of %d' %self.k)\n\n N_symbols = int(len(x)/self.k)\n codewords = np.zeros(N_symbols*self.n)\n x = np.reshape(x,(1,len(x)))\n\n for i in range(0,N_symbols):\n codewords[i*self.n:(i+1)*self.n] = np.matmul(x[:,i*self.k:(i+1)*self.k],self.G)%2\n return codewords"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ndecodes hamming encoded codewords. Make sure code words are of the appropriate length for the object. parameters --------- codewords: bit array of codewords returns ------- decoded_bits: bit array of decoded source bits Andrew Smit November 2018", "response": "def hamm_decoder(self,codewords):\n \"\"\"\n Decode hamming encoded codewords. Make sure code words are of\n the appropriate length for the object.\n \n parameters\n ---------\n codewords: bit array of codewords \n \n returns\n -------\n decoded_bits: bit array of decoded source bits\n \n Andrew Smit November 2018\n \"\"\"\n if(np.dtype(codewords[0]) != int):\n raise ValueError('Error: Invalid data type. Input must be a vector of ints')\n\n if(len(codewords) % self.n or len(codewords) < self.n):\n raise ValueError('Error: Invalid input vector length. Length must be a multiple of %d' %self.n)\n\n # Calculate the number of symbols (codewords) in the input array\n N_symbols = int(len(codewords)/self.n)\n \n # Allocate memory for decoded sourcebits\n decoded_bits = np.zeros(N_symbols*self.k)\n \n # Loop through codewords to decode one block at a time\n codewords = np.reshape(codewords,(1,len(codewords)))\n for i in range(0,N_symbols):\n \n # find the syndrome of each codeword\n S = np.matmul(self.H,codewords[:,i*self.n:(i+1)*self.n].T) % 2\n\n # convert binary syndrome to an integer\n bits = ''\n for m in range(0,len(S)):\n bit = str(int(S[m,:]))\n bits = bits + bit\n error_pos = int(bits,2)\n h_pos = self.H[:,error_pos-1]\n \n # Use the syndrome to find the position of an error within the block\n bits = ''\n for m in range(0,len(S)):\n bit = str(int(h_pos[m]))\n bits = bits + bit\n decoded_pos = int(bits,2)-1\n\n # correct error if present\n if(error_pos):\n codewords[:,i*self.n+decoded_pos] = (codewords[:,i*self.n+decoded_pos] + 1) % 2\n \n # Decode the corrected codeword\n decoded_bits[i*self.k:(i+1)*self.k] = np.matmul(self.R,codewords[:,i*self.n:(i+1)*self.n].T).T % 2\n return decoded_bits.astype(int)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nencodes input bit array x using cyclic block code. parameters ---------- x: vector of source bits to be encoded by block encoder. Numpy array of integers expected. returns ------- codewords: vector of code words generated from input vector Andrew Smit November 2018", "response": "def cyclic_encoder(self,x,G='1011'):\n \"\"\"\n Encodes input bit array x using cyclic block code.\n \n parameters\n ----------\n x: vector of source bits to be encoded by block encoder. Numpy array\n of integers expected.\n \n returns\n -------\n codewords: vector of code words generated from input vector\n \n Andrew Smit November 2018\n \"\"\"\n \n # Check block length\n if(len(x) % self.k or len(x) < self.k):\n raise ValueError('Error: Incomplete block in input array. Make sure input array length is a multiple of %d' %self.k)\n \n # Check data type of input vector\n if(np.dtype(x[0]) != int):\n raise ValueError('Error: Input array should be int data type')\n \n # Calculate number of blocks\n Num_blocks = int(len(x) / self.k)\n \n codewords = np.zeros((Num_blocks,self.n),dtype=int)\n x = np.reshape(x,(Num_blocks,self.k))\n \n #print(x)\n \n for p in range(Num_blocks):\n S = np.zeros(len(self.G))\n codeword = np.zeros(self.n)\n current_block = x[p,:]\n #print(current_block)\n for i in range(0,self.n):\n if(i < self.k):\n S[0] = current_block[i]\n S0temp = 0\n for m in range(0,len(self.G)):\n if(self.G[m] == '1'):\n S0temp = S0temp + S[m]\n #print(j,S0temp,S[j])\n S0temp = S0temp % 2\n S = np.roll(S,1)\n codeword[i] = current_block[i]\n S[1] = S0temp\n else:\n out = 0\n for m in range(1,len(self.G)):\n if(self.G[m] == '1'):\n out = out + S[m]\n codeword[i] = out % 2\n S = np.roll(S,1)\n S[1] = 0\n codewords[p,:] = codeword\n #print(codeword)\n \n codewords = np.reshape(codewords,np.size(codewords))\n \n return codewords.astype(int)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndecoding a vector of cyclic coded codewords. parameters ---------- codewords: vector of codewords to be decoded. Numpy array of integers expected. returns ------- decoded_blocks: vector of decoded bits Andrew Smit November 2018", "response": "def cyclic_decoder(self,codewords):\n \"\"\"\n Decodes a vector of cyclic coded codewords.\n \n parameters\n ----------\n codewords: vector of codewords to be decoded. Numpy array of integers expected.\n \n returns\n -------\n decoded_blocks: vector of decoded bits\n \n Andrew Smit November 2018\n \"\"\"\n \n # Check block length\n if(len(codewords) % self.n or len(codewords) < self.n):\n raise ValueError('Error: Incomplete coded block in input array. Make sure coded input array length is a multiple of %d' %self.n)\n \n # Check input data type\n if(np.dtype(codewords[0]) != int):\n raise ValueError('Error: Input array should be int data type')\n \n # Calculate number of blocks\n Num_blocks = int(len(codewords) / self.n)\n \n decoded_blocks = np.zeros((Num_blocks,self.k),dtype=int)\n codewords = np.reshape(codewords,(Num_blocks,self.n))\n\n for p in range(Num_blocks):\n codeword = codewords[p,:]\n Ureg = np.zeros(self.n)\n S = np.zeros(len(self.G))\n decoded_bits = np.zeros(self.k)\n output = np.zeros(self.n)\n for i in range(0,self.n): # Switch A closed B open\n Ureg = np.roll(Ureg,1)\n Ureg[0] = codeword[i]\n S0temp = 0\n S[0] = codeword[i]\n for m in range(len(self.G)):\n if(self.G[m] == '1'):\n S0temp = S0temp + S[m]\n S0 = S\n S = np.roll(S,1)\n S[1] = S0temp % 2\n\n for i in range(0,self.n): # Switch B closed A open\n Stemp = 0\n for m in range(1,len(self.G)):\n if(self.G[m] == '1'):\n Stemp = Stemp + S[m]\n S = np.roll(S,1)\n S[1] = Stemp % 2\n and_out = 1\n for m in range(1,len(self.G)):\n if(m > 1):\n and_out = and_out and ((S[m]+1) % 2)\n else:\n and_out = and_out and S[m]\n output[i] = (and_out + Ureg[len(Ureg)-1]) % 2\n Ureg = np.roll(Ureg,1)\n Ureg[0] = 0\n decoded_bits = output[0:self.k].astype(int)\n decoded_blocks[p,:] = decoded_bits\n \n return np.reshape(decoded_blocks,np.size(decoded_blocks)).astype(int)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nselects the proper LockManager based on the current Celery backend.", "response": "def _select_manager(backend_name):\n \"\"\"Select the proper LockManager based on the current backend used by Celery.\n\n :raise NotImplementedError: If Celery is using an unsupported backend.\n\n :param str backend_name: Class name of the current Celery backend. Usually value of\n current_app.extensions['celery'].celery.backend.__class__.__name__.\n\n :return: Class definition object (not instance). One of the _LockManager* classes.\n \"\"\"\n if backend_name == 'RedisBackend':\n lock_manager = _LockManagerRedis\n elif backend_name == 'DatabaseBackend':\n lock_manager = _LockManagerDB\n else:\n raise NotImplementedError\n return lock_manager"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef single_instance(func=None, lock_timeout=None, include_args=False):\n if func is None:\n return partial(single_instance, lock_timeout=lock_timeout, include_args=include_args)\n\n @wraps(func)\n def wrapped(celery_self, *args, **kwargs):\n \"\"\"Wrapped Celery task, for single_instance().\"\"\"\n # Select the manager and get timeout.\n timeout = (\n lock_timeout or celery_self.soft_time_limit or celery_self.time_limit\n or celery_self.app.conf.get('CELERYD_TASK_SOFT_TIME_LIMIT')\n or celery_self.app.conf.get('CELERYD_TASK_TIME_LIMIT')\n or (60 * 5)\n )\n manager_class = _select_manager(celery_self.backend.__class__.__name__)\n lock_manager = manager_class(celery_self, timeout, include_args, args, kwargs)\n\n # Lock and execute.\n with lock_manager:\n ret_value = func(*args, **kwargs)\n return ret_value\n return wrapped", "response": "Decorator for celery tasks that allows a task to only have one running instance at a time."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef task_identifier(self):\n task_id = self.celery_self.name\n if self.include_args:\n merged_args = str(self.args) + str([(k, self.kwargs[k]) for k in sorted(self.kwargs)])\n task_id += '.args.{0}'.format(hashlib.md5(merged_args.encode('utf-8')).hexdigest())\n return task_id", "response": "Return the unique identifier of a task instance."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns True if lock exists and has not timed out.", "response": "def is_already_running(self):\n \"\"\"Return True if lock exists and has not timed out.\"\"\"\n redis_key = self.CELERY_LOCK.format(task_id=self.task_identifier)\n return self.celery_self.backend.client.exists(redis_key)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef reset_lock(self):\n redis_key = self.CELERY_LOCK.format(task_id=self.task_identifier)\n self.celery_self.backend.client.delete(redis_key)", "response": "Removes the lock regardless of timeout."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn True if lock exists and has not timed out.", "response": "def is_already_running(self):\n \"\"\"Return True if lock exists and has not timed out.\"\"\"\n date_done = (self.restore_group(self.task_identifier) or dict()).get('date_done')\n if not date_done:\n return False\n difference = datetime.utcnow() - date_done\n return difference < timedelta(seconds=self.timeout)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef init_app(self, app):\n _state._register_app = self.original_register_app # Restore Celery app registration function.\n if not hasattr(app, 'extensions'):\n app.extensions = dict()\n if 'celery' in app.extensions:\n raise ValueError('Already registered extension CELERY.')\n app.extensions['celery'] = _CeleryState(self, app)\n\n # Instantiate celery and read config.\n super(Celery, self).__init__(app.import_name, broker=app.config['CELERY_BROKER_URL'])\n\n # Set result backend default.\n if 'CELERY_RESULT_BACKEND' in app.config:\n self._preconf['CELERY_RESULT_BACKEND'] = app.config['CELERY_RESULT_BACKEND']\n\n self.conf.update(app.config)\n task_base = self.Task\n\n # Add Flask app context to celery instance.\n class ContextTask(task_base):\n def __call__(self, *_args, **_kwargs):\n with app.app_context():\n return task_base.__call__(self, *_args, **_kwargs)\n setattr(ContextTask, 'abstract', True)\n setattr(self, 'Task', ContextTask)", "response": "Initializes the Celery instance."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef iter_chunksize(num_samples, chunksize):\n last_chunksize = int(np.mod(num_samples, chunksize))\n chunksize = int(chunksize)\n for _ in range(int(num_samples) // chunksize):\n yield chunksize\n if last_chunksize > 0:\n yield last_chunksize", "response": "Iterator used to iterate over a array of size num_samples."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef iter_chunk_slice(num_samples, chunksize):\n i = 0\n for c_size in iter_chunksize(num_samples, chunksize):\n yield slice(i, i + c_size)\n i += c_size", "response": "Iterator used to iterate over chunks over an array of size num_samples."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef iter_chunk_index(num_samples, chunksize):\n i = 0\n for c_size in iter_chunksize(num_samples, chunksize):\n yield i, i + c_size\n i += c_size", "response": "Iterator used to iterate over a chunk of size num_samples."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreduce with func chunk by chunk.", "response": "def reduce_chunk(func, array):\n \"\"\"Reduce with `func`, chunk by chunk, the passed pytable `array`.\n \"\"\"\n res = []\n for slice in iter_chunk_slice(array.shape[-1], array.chunkshape[-1]):\n res.append(func(array[..., slice]))\n return func(res)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nmaps with func chunk by chunk", "response": "def map_chunk(func, array, out_array):\n \"\"\"Map with `func`, chunk by chunk, the input pytable `array`.\n The result is stored in the output pytable array `out_array`.\n \"\"\"\n for slice in iter_chunk_slice(array.shape[-1], array.chunkshape[-1]):\n out_array.append(func(array[..., slice]))\n return out_array"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngenerating timestamps from a set of remote simulations in dview.", "response": "def parallel_gen_timestamps(dview, max_em_rate, bg_rate):\n \"\"\"Generate timestamps from a set of remote simulations in `dview`.\n Assumes that all the engines have an `S` object already containing\n an emission trace (`S.em`). The \"photons\" timestamps are generated\n from these emission traces and merged into a single array of timestamps.\n `max_em_rate` and `bg_rate` are passed to `S.sim_timetrace()`.\n \"\"\"\n dview.execute('S.sim_timestamps_em_store(max_rate=%d, bg_rate=%d, '\n 'seed=S.EID, overwrite=True)' % (max_em_rate, bg_rate))\n dview.execute('times = S.timestamps[:]')\n dview.execute('times_par = S.timestamps_par[:]')\n Times = dview['times']\n Times_par = dview['times_par']\n # Assuming all t_max equal, just take the first\n t_max = dview['S.t_max'][0]\n t_tot = np.sum(dview['S.t_max'])\n dview.execute(\"sim_name = S.compact_name_core(t_max=False, hashdigit=0)\")\n # Core names contains no ID or t_max\n sim_name = dview['sim_name'][0]\n times_all, times_par_all = merge_ph_times(Times, Times_par,\n time_block=t_max)\n return times_all, times_par_all, t_tot, sim_name"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef merge_ph_times(times_list, times_par_list, time_block):\n offsets = np.arange(len(times_list)) * time_block\n cum_sizes = np.cumsum([ts.size for ts in times_list])\n times = np.zeros(cum_sizes[-1])\n times_par = np.zeros(cum_sizes[-1], dtype='uint8')\n i1 = 0\n for i2, ts, ts_par, offset in zip(cum_sizes, times_list, times_par_list,\n offsets):\n times[i1:i2] = ts + offset\n times_par[i1:i2] = ts_par\n i1 = i2\n return times, times_par", "response": "Build an array of timestamps joining the arrays in ph_times_list."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning a merged timestamp array for Donor + Accept. and bool mask for A.", "response": "def merge_DA_ph_times(ph_times_d, ph_times_a):\n \"\"\"Returns a merged timestamp array for Donor+Accept. and bool mask for A.\n \"\"\"\n ph_times = np.hstack([ph_times_d, ph_times_a])\n a_em = np.hstack([np.zeros(ph_times_d.size, dtype=np.bool),\n np.ones(ph_times_a.size, dtype=np.bool)])\n index_sort = ph_times.argsort()\n return ph_times[index_sort], a_em[index_sort]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn a simulation object summing the emissions and particles in SS.", "response": "def merge_particle_emission(SS):\n \"\"\"Returns a sim object summing the emissions and particles in SS (list).\n \"\"\"\n # Merge all the particles\n P = reduce(lambda x, y: x + y, [Si.particles for Si in SS])\n s = SS[0]\n S = ParticlesSimulation(t_step=s.t_step, t_max=s.t_max,\n particles=P, box=s.box, psf=s.psf)\n S.em = np.zeros(s.em.shape, dtype=np.float64)\n for Si in SS:\n S.em += Si.em\n return S"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load_PSFLab_file(fname):\n if os.path.exists(fname) or os.path.exists(fname + '.mat'):\n return loadmat(fname)['data']\n else:\n raise IOError(\"Can't find PSF file '%s'\" % fname)", "response": "Load the array data in the. mat file fname."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef convert_PSFLab_xz(data, x_step=0.5, z_step=0.5, normalize=False):\n z_len, x_len = data.shape\n hdata = data[:, (x_len - 1) // 2:]\n x = np.arange(hdata.shape[1]) * x_step\n z = np.arange(-(z_len - 1) / 2, (z_len - 1) / 2 + 1) * z_step\n if normalize:\n hdata /= hdata.max() # normalize to 1 at peak\n return x, z, hdata, hdata[:, 0].argmax()", "response": "Convert a 2D array containing a x - z PSF slice into a 2D array containing a x - z PSF slice."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef eval(self, x, y, z):\n xc, yc, zc = self.rc\n sx, sy, sz = self.s\n\n ## Method1: direct evaluation\n #return exp(-(((x-xc)**2)/(2*sx**2) + ((y-yc)**2)/(2*sy**2) +\\\n # ((z-zc)**2)/(2*sz**2)))\n\n ## Method2: evaluation using numexpr\n def arg(s):\n return \"((%s-%sc)**2)/(2*s%s**2)\" % (s, s, s)\n return NE.evaluate(\"exp(-(%s + %s + %s))\" %\n (arg(\"x\"), arg(\"y\"), arg(\"z\")))", "response": "Evaluate the function in ( x y z."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef eval(self, x, y, z):\n ro = np.sqrt(x**2 + y**2)\n zs, xs = ro.shape\n v = self.eval_xz(ro.ravel(), z.ravel())\n return v.reshape(zs, xs)", "response": "Evaluate the function in x y z."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nstores the PSF data in file_handle.", "response": "def to_hdf5(self, file_handle, parent_node='/'):\n \"\"\"Store the PSF data in `file_handle` (pytables) in `parent_node`.\n\n The raw PSF array name is stored with same name as the original fname.\n Also, the following attribues are set: fname, dir_, x_step, z_step.\n \"\"\"\n tarray = file_handle.create_array(parent_node, name=self.fname,\n obj=self.psflab_psf_raw,\n title='PSF x-z slice (PSFLab array)')\n for name in ['fname', 'dir_', 'x_step', 'z_step']:\n file_handle.set_node_attr(tarray, name, getattr(self, name))\n return tarray"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef hash(self):\n hash_list = []\n for key, value in sorted(self.__dict__.items()):\n if not callable(value):\n if isinstance(value, np.ndarray):\n hash_list.append(value.tostring())\n else:\n hash_list.append(str(value))\n return hashlib.md5(repr(hash_list).encode()).hexdigest()", "response": "Return an md5 hash string computed on the PSF data."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nchecking whether the git executable is found.", "response": "def git_path_valid(git_path=None):\n \"\"\"\n Check whether the git executable is found.\n \"\"\"\n if git_path is None and GIT_PATH is None:\n return False\n if git_path is None: git_path = GIT_PATH\n try:\n call([git_path, '--version'])\n return True\n except OSError:\n return False"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_git_version(git_path=None):\n if git_path is None: git_path = GIT_PATH\n git_version = check_output([git_path, \"--version\"]).split()[2]\n return git_version", "response": "Get the Git version."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef check_clean_status(git_path=None):\n output = get_status(git_path)\n is_unmodified = (len(output.strip()) == 0)\n return is_unmodified", "response": "Returns whether there are uncommitted changes in the working dir."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_last_commit_line(git_path=None):\n if git_path is None: git_path = GIT_PATH\n output = check_output([git_path, \"log\", \"--pretty=format:'%ad %h %s'\",\n \"--date=short\", \"-n1\"])\n return output.strip()[1:-1]", "response": "Get one - line description of HEAD commit for current dir."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nget the last commit SHA1 of the current git repository.", "response": "def get_last_commit(git_path=None):\n \"\"\"\n Get the HEAD commit SHA1 of repository in current dir.\n \"\"\"\n if git_path is None: git_path = GIT_PATH\n line = get_last_commit_line(git_path)\n revision_id = line.split()[1]\n return revision_id"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nprint the summary of the current version of the current revision.", "response": "def print_summary(string='Repository', git_path=None):\n \"\"\"\n Print the last commit line and eventual uncommitted changes.\n \"\"\"\n if git_path is None: git_path = GIT_PATH\n\n # If git is available, check fretbursts version\n if not git_path_valid():\n print('\\n%s revision unknown (git not found).' % string)\n else:\n last_commit = get_last_commit_line()\n print('\\n{} revision:\\n {}\\n'.format(string, last_commit))\n if not check_clean_status():\n print('\\nWARNING -> Uncommitted changes:')\n print(get_status())"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ngetting the filenames for donor and acceptor timestamps for the given parameters", "response": "def get_bromo_fnames_da(d_em_kHz, d_bg_kHz, a_em_kHz, a_bg_kHz,\n ID='1+2+3+4+5+6', t_tot='480', num_p='30', pM='64',\n t_step=0.5e-6, D=1.2e-11, dir_=''):\n \"\"\"Get filenames for donor and acceptor timestamps for the given parameters\n \"\"\"\n\n clk_p = t_step/32. # with t_step=0.5us -> 156.25 ns\n E_sim = 1.*a_em_kHz/(a_em_kHz + d_em_kHz)\n\n FRET_val = 100.*E_sim\n print(\"Simulated FRET value: %.1f%%\" % FRET_val)\n\n d_em_kHz_str = \"%04d\" % d_em_kHz\n a_em_kHz_str = \"%04d\" % a_em_kHz\n d_bg_kHz_str = \"%04.1f\" % d_bg_kHz\n a_bg_kHz_str = \"%04.1f\" % a_bg_kHz\n\n print(\"D: EM %s BG %s \" % (d_em_kHz_str, d_bg_kHz_str))\n print(\"A: EM %s BG %s \" % (a_em_kHz_str, a_bg_kHz_str))\n\n fname_d = ('ph_times_{t_tot}s_D{D}_{np}P_{pM}pM_'\n 'step{ts_us}us_ID{ID}_EM{em}kHz_BG{bg}kHz.npy').format(\n em=d_em_kHz_str, bg=d_bg_kHz_str, t_tot=t_tot, pM=pM,\n np=num_p, ID=ID, ts_us=t_step*1e6, D=D)\n\n fname_a = ('ph_times_{t_tot}s_D{D}_{np}P_{pM}pM_'\n 'step{ts_us}us_ID{ID}_EM{em}kHz_BG{bg}kHz.npy').format(\n em=a_em_kHz_str, bg=a_bg_kHz_str, t_tot=t_tot, pM=pM,\n np=num_p, ID=ID, ts_us=t_step*1e6, D=D)\n print(fname_d)\n print(fname_a)\n\n name = ('BroSim_E{:.1f}_dBG{:.1f}k_aBG{:.1f}k_'\n 'dEM{:.0f}k').format(FRET_val, d_bg_kHz, a_bg_kHz, d_em_kHz)\n\n return dir_+fname_d, dir_+fname_a, name, clk_p, E_sim"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nstoring parameters in params in ParticlesSimulation", "response": "def set_sim_params(self, nparams, attr_params):\n \"\"\"Store parameters in `params` in `h5file.root.parameters`.\n\n `nparams` (dict)\n A dict as returned by `get_params()` in `ParticlesSimulation()`\n The format is:\n keys:\n used as parameter name\n values: (2-elements tuple)\n first element is the parameter value\n second element is a string used as \"title\" (description)\n `attr_params` (dict)\n A dict whole items are stored as attributes in '/parameters'\n \"\"\"\n for name, value in nparams.items():\n val = value[0] if value[0] is not None else 'none'\n self.h5file.create_array('/parameters', name, obj=val,\n title=value[1])\n for name, value in attr_params.items():\n self.h5file.set_node_attr('/parameters', name, value)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef numeric_params(self):\n nparams = dict()\n for p in self.h5file.root.parameters:\n nparams[p.name] = p.read()\n return nparams", "response": "Return a dict containing all numeric parameters stored in '/parameters'"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef numeric_params_meta(self):\n nparams = dict()\n for p in self.h5file.root.parameters:\n nparams[p.name] = (p.read(), p.title)\n return nparams", "response": "Return a dict with all parameters and metadata in '/parameters'."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds a trajectory to the HDF5 file.", "response": "def add_trajectory(self, name, overwrite=False, shape=(0,), title='',\n chunksize=2**19, comp_filter=default_compression,\n atom=tables.Float64Atom(), params=dict(),\n chunkslice='bytes'):\n \"\"\"Add an trajectory array in '/trajectories'.\n \"\"\"\n group = self.h5file.root.trajectories\n if name in group:\n print(\"%s already exists ...\" % name, end='')\n if overwrite:\n self.h5file.remove_node(group, name)\n print(\" deleted.\")\n else:\n print(\" old returned.\")\n return group.get_node(name)\n\n nparams = self.numeric_params\n num_t_steps = nparams['t_max'] / nparams['t_step']\n\n chunkshape = self.calc_chunkshape(chunksize, shape, kind=chunkslice)\n store_array = self.h5file.create_earray(\n group, name, atom=atom,\n shape = shape,\n chunkshape = chunkshape,\n expectedrows = num_t_steps,\n filters = comp_filter,\n title = title)\n\n # Set the array parameters/attributes\n for key, value in params.items():\n store_array.set_attr(key, value)\n store_array.set_attr('PyBroMo', __version__)\n store_array.set_attr('creation_time', current_time())\n return store_array"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_emission_tot(self, chunksize=2**19, comp_filter=default_compression,\n overwrite=False, params=dict(),\n chunkslice='bytes'):\n \"\"\"Add the `emission_tot` array in '/trajectories'.\n \"\"\"\n kwargs = dict(overwrite=overwrite, chunksize=chunksize, params=params,\n comp_filter=comp_filter, atom=tables.Float32Atom(),\n title='Summed emission trace of all the particles')\n return self.add_trajectory('emission_tot', **kwargs)", "response": "Add the emission_tot array in '/trajectories."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_emission(self, chunksize=2**19, comp_filter=default_compression,\n overwrite=False, params=dict(), chunkslice='bytes'):\n \"\"\"Add the `emission` array in '/trajectories'.\n \"\"\"\n nparams = self.numeric_params\n num_particles = nparams['np']\n\n return self.add_trajectory('emission', shape=(num_particles, 0),\n overwrite=overwrite, chunksize=chunksize,\n comp_filter=comp_filter,\n atom=tables.Float32Atom(),\n title='Emission trace of each particle',\n params=params)", "response": "Add the emission array in '/trajectories."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef add_position(self, radial=False, chunksize=2**19, chunkslice='bytes',\n comp_filter=default_compression, overwrite=False,\n params=dict()):\n \"\"\"Add the `position` array in '/trajectories'.\n \"\"\"\n nparams = self.numeric_params\n num_particles = nparams['np']\n\n name, ncoords, prefix = 'position', 3, 'X-Y-Z'\n if radial:\n name, ncoords, prefix = 'position_rz', 2, 'R-Z'\n title = '%s position trace of each particle' % prefix\n return self.add_trajectory(name, shape=(num_particles, ncoords, 0),\n overwrite=overwrite, chunksize=chunksize,\n comp_filter=comp_filter,\n atom=tables.Float32Atom(),\n title=title,\n params=params)", "response": "Add the position array in '/trajectories."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfolds all the values of a outside a1.. a2 inside that interval.", "response": "def wrap_periodic(a, a1, a2):\n \"\"\"Folds all the values of `a` outside [a1..a2] inside that interval.\n This function is used to apply periodic boundary conditions.\n \"\"\"\n a -= a1\n wrapped = np.mod(a, a2 - a1) + a1\n return wrapped"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef wrap_mirror(a, a1, a2):\n a[a > a2] = a2 - (a[a > a2] - a2)\n a[a < a1] = a1 + (a1 - a[a < a1])\n return a", "response": "Folds all the values of a outside a1.. a2 inside that interval."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sim_timetrace(emission, max_rate, t_step):\n emission_rates = emission * max_rate * t_step\n return np.random.poisson(lam=emission_rates).astype(np.uint8)", "response": "Draw random emitted photons from Poisson"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sim_timetrace_bg(emission, max_rate, bg_rate, t_step, rs=None):\n if rs is None:\n rs = np.random.RandomState()\n em = np.atleast_2d(emission).astype('float64', copy=False)\n counts_nrows = em.shape[0]\n if bg_rate is not None:\n counts_nrows += 1 # add a row for poisson background\n counts = np.zeros((counts_nrows, em.shape[1]), dtype='u1')\n # In-place computation\n # NOTE: the caller will see the modification\n em *= (max_rate * t_step)\n # Use automatic type conversion int64 (counts_par) -> uint8 (counts)\n counts_par = rs.poisson(lam=em)\n if bg_rate is None:\n counts[:] = counts_par\n else:\n counts[:-1] = counts_par\n counts[-1] = rs.poisson(lam=bg_rate * t_step, size=em.shape[1])\n return counts", "response": "Draw random emissions from r. v. ~ Poisson background."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndrawing random emitted photons from r. v. ~ Poisson ( emission_rates ).", "response": "def sim_timetrace_bg2(emission, max_rate, bg_rate, t_step, rs=None):\n \"\"\"Draw random emitted photons from r.v. ~ Poisson(emission_rates).\n\n This is an alternative implementation of :func:`sim_timetrace_bg`.\n \"\"\"\n if rs is None:\n rs = np.random.RandomState()\n emiss_bin_rate = np.zeros((emission.shape[0] + 1, emission.shape[1]),\n dtype='float64')\n emiss_bin_rate[:-1] = emission * max_rate * t_step\n if bg_rate is not None:\n emiss_bin_rate[-1] = bg_rate * t_step\n counts = rs.poisson(lam=emiss_bin_rate).astype('uint8')\n else:\n counts = rs.poisson(lam=emiss_bin_rate[:-1]).astype('uint8')\n return counts"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef volume(self):\n return (self.x2 - self.x1) * (self.y2 - self.y1) * (self.z2 - self.z1)", "response": "Box volume in m^3."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate a list of Particles.", "response": "def _generate(num_particles, D, box, rs):\n \"\"\"Generate a list of `Particle` objects.\"\"\"\n X0 = rs.rand(num_particles) * (box.x2 - box.x1) + box.x1\n Y0 = rs.rand(num_particles) * (box.y2 - box.y1) + box.y1\n Z0 = rs.rand(num_particles) * (box.z2 - box.z1) + box.z1\n return [Particle(D=D, x0=x0, y0=y0, z0=z0)\n for x0, y0, z0 in zip(X0, Y0, Z0)]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nadding particles with diffusion coefficient D at random positions.", "response": "def add(self, num_particles, D):\n \"\"\"Add particles with diffusion coefficient `D` at random positions.\n \"\"\"\n self._plist += self._generate(num_particles, D, box=self.box,\n rs=self.rs)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef positions(self):\n return np.vstack([p.r0 for p in self]).reshape(len(self), 3, 1)", "response": "Initial position for each particle. Shape N 3 1."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef diffusion_coeff_counts(self):\n return [(key, len(list(group)))\n for key, group in itertools.groupby(self.diffusion_coeff)]", "response": "List of tuples of diffusion coefficient counts pairs."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef datafile_from_hash(hash_, prefix, path):\n pattern = '%s_%s*.h*' % (prefix, hash_)\n datafiles = list(path.glob(pattern))\n if len(datafiles) == 0:\n raise NoMatchError('No matches for \"%s\"' % pattern)\n if len(datafiles) > 1:\n raise MultipleMatchesError('More than 1 match for \"%s\"' % pattern)\n return datafiles[0]", "response": "Return pathlib. Path for a data - file with given hash and prefix."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef from_datafile(hash_, path='./', ignore_timestamps=False, mode='r'):\n path = Path(path)\n assert path.exists()\n\n file_traj = ParticlesSimulation.datafile_from_hash(\n hash_, prefix=ParticlesSimulation._PREFIX_TRAJ, path=path)\n store = TrajectoryStore(file_traj, mode='r')\n\n psf_pytables = store.h5file.get_node('/psf/default_psf')\n psf = NumericPSF(psf_pytables=psf_pytables)\n box = store.h5file.get_node_attr('/parameters', 'box')\n P = store.h5file.get_node_attr('/parameters', 'particles')\n\n names = ['t_step', 't_max', 'EID', 'ID']\n kwargs = {name: store.numeric_params[name] for name in names}\n S = ParticlesSimulation(particles=Particles.from_json(P), box=box,\n psf=psf, **kwargs)\n\n # Emulate S.open_store_traj()\n S.store = store\n S.psf_pytables = psf_pytables\n S.traj_group = S.store.h5file.root.trajectories\n S.emission = S.traj_group.emission\n S.emission_tot = S.traj_group.emission_tot\n if 'position' in S.traj_group:\n S.position = S.traj_group.position\n elif 'position_rz' in S.traj_group:\n S.position = S.traj_group.position_rz\n S.chunksize = S.store.h5file.get_node('/parameters', 'chunksize')\n if not ignore_timestamps:\n try:\n file_ts = ParticlesSimulation.datafile_from_hash(\n hash_, prefix=ParticlesSimulation._PREFIX_TS, path=path)\n except NoMatchError:\n # There are no timestamps saved.\n pass\n else:\n # Load the timestamps\n S.ts_store = TimestampStore(file_ts, mode=mode)\n S.ts_group = S.ts_store.h5file.root.timestamps\n print(' - Found matching timestamps.')\n return S", "response": "Load simulation from disk trajectories and timestamps."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _get_group_randomstate(rs, seed, group):\n if rs is None:\n rs = np.random.RandomState(seed=seed)\n # Try to set the random state from the last session to preserve\n # a single random stream when simulating timestamps multiple times\n if 'last_random_state' in group._v_attrs:\n rs.set_state(group._v_attrs['last_random_state'])\n print(\"INFO: Random state set to last saved state in '%s'.\" %\n group._v_name)\n else:\n print(\"INFO: Random state initialized from seed (%d).\" % seed)\n return rs", "response": "Return a RandomState equal to the input unless rs is None."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns an md5 hash for the simulation parameters", "response": "def hash(self):\n \"\"\"Return an hash for the simulation parameters (excluding ID and EID)\n This can be used to generate unique file names for simulations\n that have the same parameters and just different ID or EID.\n \"\"\"\n hash_numeric = 't_step=%.3e, t_max=%.2f, np=%d, conc=%.2e' % \\\n (self.t_step, self.t_max, self.num_particles, self.concentration())\n hash_list = [hash_numeric, self.particles.short_repr(), repr(self.box),\n self.psf.hash()]\n return hashlib.md5(repr(hash_list).encode()).hexdigest()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncompacting representation of simulation params.", "response": "def compact_name_core(self, hashsize=6, t_max=False):\n \"\"\"Compact representation of simulation params (no ID, EID and t_max)\n \"\"\"\n Moles = self.concentration()\n name = \"%s_%dpM_step%.1fus\" % (\n self.particles.short_repr(), Moles * 1e12, self.t_step * 1e6)\n if hashsize > 0:\n name = self.hash()[:hashsize] + '_' + name\n if t_max:\n name += \"_t_max%.1fs\" % self.t_max\n return name"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncompacting representation of all simulation parameters", "response": "def compact_name(self, hashsize=6):\n \"\"\"Compact representation of all simulation parameters\n \"\"\"\n # this can be made more robust for ID > 9 (double digit)\n s = self.compact_name_core(hashsize, t_max=True)\n s += \"_ID%d-%d\" % (self.ID, self.EID)\n return s"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nprinting on - disk array sizes required for current set of parameters.", "response": "def print_sizes(self):\n \"\"\"Print on-disk array sizes required for current set of parameters.\"\"\"\n float_size = 4\n MB = 1024 * 1024\n size_ = self.n_samples * float_size\n em_size = size_ * self.num_particles / MB\n pos_size = 3 * size_ * self.num_particles / MB\n print(\" Number of particles:\", self.num_particles)\n print(\" Number of time steps:\", self.n_samples)\n print(\" Emission array - 1 particle (float32): %.1f MB\" % (size_ / MB))\n print(\" Emission array (float32): %.1f MB\" % em_size)\n print(\" Position array (float32): %.1f MB \" % pos_size)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the concentration of the particles in the box.", "response": "def concentration(self, pM=False):\n \"\"\"Return the concentration (in Moles) of the particles in the box.\n \"\"\"\n concentr = (self.num_particles / NA) / self.box.volume_L\n if pM:\n concentr *= 1e12\n return concentr"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef _sim_trajectories(self, time_size, start_pos, rs,\n total_emission=False, save_pos=False, radial=False,\n wrap_func=wrap_periodic):\n \"\"\"Simulate (in-memory) `time_size` steps of trajectories.\n\n Simulate Brownian motion diffusion and emission of all the particles.\n Uses the attributes: num_particles, sigma_1d, box, psf.\n\n Arguments:\n time_size (int): number of time steps to be simulated.\n start_pos (array): shape (num_particles, 3), particles start\n positions. This array is modified to store the end position\n after this method is called.\n rs (RandomState): a `numpy.random.RandomState` object used\n to generate the random numbers.\n total_emission (bool): if True, store only the total emission array\n containing the sum of emission of all the particles.\n save_pos (bool): if True, save the particles 3D trajectories\n wrap_func (function): the function used to apply the boundary\n condition (use :func:`wrap_periodic` or :func:`wrap_mirror`).\n\n Returns:\n POS (list): list of 3D trajectories arrays (3 x time_size)\n em (array): array of emission (total or per-particle)\n \"\"\"\n time_size = int(time_size)\n num_particles = self.num_particles\n if total_emission:\n em = np.zeros(time_size, dtype=np.float32)\n else:\n em = np.zeros((num_particles, time_size), dtype=np.float32)\n\n POS = []\n # pos_w = np.zeros((3, c_size))\n for i, sigma_1d in enumerate(self.sigma_1d):\n delta_pos = rs.normal(loc=0, scale=sigma_1d,\n size=3 * time_size)\n delta_pos = delta_pos.reshape(3, time_size)\n pos = np.cumsum(delta_pos, axis=-1, out=delta_pos)\n pos += start_pos[i]\n\n # Coordinates wrapping using the specified boundary conditions\n for coord in (0, 1, 2):\n pos[coord] = wrap_func(pos[coord], *self.box.b[coord])\n\n # Sample the PSF along i-th trajectory then square to account\n # for emission and detection PSF.\n Ro = sqrt(pos[0]**2 + pos[1]**2) # radial pos. on x-y plane\n Z = pos[2]\n current_em = self.psf.eval_xz(Ro, Z)**2\n if total_emission:\n # Add the current particle emission to the total emission\n em += current_em.astype(np.float32)\n else:\n # Store the individual emission of current particle\n em[i] = current_em.astype(np.float32)\n if save_pos:\n pos_save = np.vstack((Ro, Z)) if radial else pos\n POS.append(pos_save[np.newaxis, :, :])\n # Update start_pos in-place for current particle\n start_pos[i] = pos[:, -1:]\n return POS, em", "response": "Simulate the trajectories of all particles."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef simulate_diffusion(self, save_pos=False, total_emission=True,\n radial=False, rs=None, seed=1, path='./',\n wrap_func=wrap_periodic,\n chunksize=2**19, chunkslice='times', verbose=True):\n \"\"\"Simulate Brownian motion trajectories and emission rates.\n\n This method performs the Brownian motion simulation using the current\n set of parameters. Before running this method you can check the\n disk-space requirements using :method:`print_sizes`.\n\n Results are stored to disk in HDF5 format and are accessible in\n in `self.emission`, `self.emission_tot` and `self.position` as\n pytables arrays.\n\n Arguments:\n save_pos (bool): if True, save the particles 3D trajectories\n total_emission (bool): if True, store only the total emission array\n containing the sum of emission of all the particles.\n rs (RandomState object): random state object used as random number\n generator. If None, use a random state initialized from seed.\n seed (uint): when `rs` is None, `seed` is used to initialize the\n random state, otherwise is ignored.\n wrap_func (function): the function used to apply the boundary\n condition (use :func:`wrap_periodic` or :func:`wrap_mirror`).\n path (string): a folder where simulation data is saved.\n verbose (bool): if False, prints no output.\n \"\"\"\n if rs is None:\n rs = np.random.RandomState(seed=seed)\n self.open_store_traj(chunksize=chunksize, chunkslice=chunkslice,\n radial=radial, path=path)\n # Save current random state for reproducibility\n self.traj_group._v_attrs['init_random_state'] = rs.get_state()\n\n em_store = self.emission_tot if total_emission else self.emission\n\n print('- Start trajectories simulation - %s' % ctime(), flush=True)\n if verbose:\n print('[PID %d] Diffusion time:' % os.getpid(), end='')\n i_chunk = 0\n t_chunk_size = self.emission.chunkshape[1]\n chunk_duration = t_chunk_size * self.t_step\n\n par_start_pos = self.particles.positions\n prev_time = 0\n for time_size in iter_chunksize(self.n_samples, t_chunk_size):\n if verbose:\n curr_time = int(chunk_duration * (i_chunk + 1))\n if curr_time > prev_time:\n print(' %ds' % curr_time, end='', flush=True)\n prev_time = curr_time\n\n POS, em = self._sim_trajectories(time_size, par_start_pos, rs,\n total_emission=total_emission,\n save_pos=save_pos, radial=radial,\n wrap_func=wrap_func)\n\n ## Append em to the permanent storage\n # if total_emission, data is just a linear array\n # otherwise is a 2-D array (self.num_particles, c_size)\n em_store.append(em)\n if save_pos:\n self.position.append(np.vstack(POS).astype('float32'))\n i_chunk += 1\n self.store.h5file.flush()\n\n # Save current random state\n self.traj_group._v_attrs['last_random_state'] = rs.get_state()\n self.store.h5file.flush()\n print('\\n- End trajectories simulation - %s' % ctime(), flush=True)", "response": "This method simulates the diffusion of the current trajectory and the emission rates."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_timestamps_part(self, name):\n par_name = name + '_par'\n timestamps = self.ts_store.h5file.get_node('/timestamps', name)\n particles = self.ts_store.h5file.get_node('/timestamps', par_name)\n return timestamps, particles", "response": "Return matching timestamps and particles pytables arrays.\n "} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nsimulates timestamps from emission trajectories.", "response": "def _sim_timestamps(self, max_rate, bg_rate, emission, i_start, rs,\n ip_start=0, scale=10, sort=True):\n \"\"\"Simulate timestamps from emission trajectories.\n\n Uses attributes: `.t_step`.\n\n Returns:\n A tuple of two arrays: timestamps and particles.\n \"\"\"\n counts_chunk = sim_timetrace_bg(emission, max_rate, bg_rate,\n self.t_step, rs=rs)\n nrows = emission.shape[0]\n if bg_rate is not None:\n nrows += 1\n assert counts_chunk.shape == (nrows, emission.shape[1])\n max_counts = counts_chunk.max()\n if max_counts == 0:\n return np.array([], dtype=np.int64), np.array([], dtype=np.int64)\n\n time_start = i_start * scale\n time_stop = time_start + counts_chunk.shape[1] * scale\n ts_range = np.arange(time_start, time_stop, scale, dtype='int64')\n\n # Loop for each particle to compute timestamps\n times_chunk_p = []\n par_index_chunk_p = []\n for ip, counts_chunk_ip in enumerate(counts_chunk):\n # Compute timestamps for particle ip for all bins with counts\n times_c_ip = []\n for v in range(1, max_counts + 1):\n times_c_ip.append(ts_range[counts_chunk_ip >= v])\n\n # Stack the timestamps from different \"counts\"\n t = np.hstack(times_c_ip)\n # Append current particle\n times_chunk_p.append(t)\n par_index_chunk_p.append(np.full(t.size, ip + ip_start, dtype='u1'))\n\n # Merge the arrays of different particles\n times_chunk = np.hstack(times_chunk_p)\n par_index_chunk = np.hstack(par_index_chunk_p)\n\n if sort:\n # Sort timestamps inside the merged chunk\n index_sort = times_chunk.argsort(kind='mergesort')\n times_chunk = times_chunk[index_sort]\n par_index_chunk = par_index_chunk[index_sort]\n\n return times_chunk, par_index_chunk"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef simulate_timestamps_mix(self, max_rates, populations, bg_rate,\n rs=None, seed=1, chunksize=2**16,\n comp_filter=None, overwrite=False,\n skip_existing=False, scale=10,\n path=None, t_chunksize=None, timeslice=None):\n \"\"\"Compute one timestamps array for a mixture of N populations.\n\n Timestamp data are saved to disk and accessible as pytables arrays in\n `._timestamps` and `._tparticles`.\n The background generated timestamps are assigned a\n conventional particle number (last particle index + 1).\n\n Arguments:\n max_rates (list): list of the peak max emission rate for each\n population.\n populations (list of slices): slices to `self.particles`\n defining each population.\n bg_rate (float, cps): rate for a Poisson background process\n rs (RandomState object): random state object used as random number\n generator. If None, use a random state initialized from seed.\n seed (uint): when `rs` is None, `seed` is used to initialize the\n random state, otherwise is ignored.\n chunksize (int): chunk size used for the on-disk timestamp array\n comp_filter (tables.Filter or None): compression filter to use\n for the on-disk `timestamps` and `tparticles` arrays.\n If None use default compression.\n overwrite (bool): if True, overwrite any pre-existing timestamps\n array. If False, never overwrite. The outcome of simulating an\n existing array is controlled by `skip_existing` flag.\n skip_existing (bool): if True, skip simulation if the same\n timestamps array is already present.\n scale (int): `self.t_step` is multiplied by `scale` to obtain the\n timestamps units in seconds.\n path (string): folder where to save the data.\n timeslice (float or None): timestamps are simulated until\n `timeslice` seconds. If None, simulate until `self.t_max`.\n \"\"\"\n self.open_store_timestamp(chunksize=chunksize, path=path)\n rs = self._get_group_randomstate(rs, seed, self.ts_group)\n if t_chunksize is None:\n t_chunksize = self.emission.chunkshape[1]\n timeslice_size = self.n_samples\n if timeslice is not None:\n timeslice_size = timeslice // self.t_step\n\n name = self._get_ts_name_mix(max_rates, populations, bg_rate, rs=rs)\n kw = dict(name=name, clk_p=self.t_step / scale,\n max_rates=max_rates, bg_rate=bg_rate, populations=populations,\n num_particles=self.num_particles,\n bg_particle=self.num_particles,\n overwrite=overwrite, chunksize=chunksize)\n if comp_filter is not None:\n kw.update(comp_filter=comp_filter)\n try:\n self._timestamps, self._tparticles = (self.ts_store\n .add_timestamps(**kw))\n except ExistingArrayError as e:\n if skip_existing:\n print(' - Skipping already present timestamps array.')\n return\n else:\n raise e\n\n self.ts_group._v_attrs['init_random_state'] = rs.get_state()\n self._timestamps.attrs['init_random_state'] = rs.get_state()\n self._timestamps.attrs['PyBroMo'] = __version__\n\n ts_list, part_list = [], []\n # Load emission in chunks, and save only the final timestamps\n bg_rates = [None] * (len(max_rates) - 1) + [bg_rate]\n prev_time = 0\n for i_start, i_end in iter_chunk_index(timeslice_size, t_chunksize):\n\n curr_time = np.around(i_start * self.t_step, decimals=0)\n if curr_time > prev_time:\n print(' %.1fs' % curr_time, end='', flush=True)\n prev_time = curr_time\n\n em_chunk = self.emission[:, i_start:i_end]\n\n times_chunk_s, par_index_chunk_s = \\\n self._sim_timestamps_populations(\n em_chunk, max_rates, populations, bg_rates, i_start,\n rs, scale)\n\n # Save sorted timestamps (suffix '_s') and corresponding particles\n ts_list.append(times_chunk_s)\n part_list.append(par_index_chunk_s)\n\n for ts, part in zip(ts_list, part_list):\n self._timestamps.append(ts)\n self._tparticles.append(part)\n\n # Save current random state so it can be resumed in the next session\n self.ts_group._v_attrs['last_random_state'] = rs.get_state()\n self._timestamps.attrs['last_random_state'] = rs.get_state()\n self.ts_store.h5file.flush()", "response": "Simulate timestamps for a mixture of N populations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef simulate_timestamps_mix_da(self, max_rates_d, max_rates_a,\n populations, bg_rate_d, bg_rate_a,\n rs=None, seed=1, chunksize=2**16,\n comp_filter=None, overwrite=False,\n skip_existing=False, scale=10,\n path=None, t_chunksize=2**19,\n timeslice=None):\n\n \"\"\"Compute D and A timestamps arrays for a mixture of N populations.\n\n This method reads the emission from disk once, and generates a pair\n of timestamps arrays (e.g. donor and acceptor) from each chunk.\n\n Timestamp data are saved to disk and accessible as pytables arrays in\n `._timestamps_d/a` and `._tparticles_d/a`.\n The background generated timestamps are assigned a\n conventional particle number (last particle index + 1).\n\n Arguments:\n max_rates_d (list): list of the peak max emission rate in the\n donor channel for each population.\n max_rates_a (list): list of the peak max emission rate in the\n acceptor channel for each population.\n populations (list of slices): slices to `self.particles`\n defining each population.\n bg_rate_d (float, cps): rate for a Poisson background process\n in the donor channel.\n bg_rate_a (float, cps): rate for a Poisson background process\n in the acceptor channel.\n rs (RandomState object): random state object used as random number\n generator. If None, use a random state initialized from seed.\n seed (uint): when `rs` is None, `seed` is used to initialize the\n random state, otherwise is ignored.\n chunksize (int): chunk size used for the on-disk timestamp array\n comp_filter (tables.Filter or None): compression filter to use\n for the on-disk `timestamps` and `tparticles` arrays.\n If None use default compression.\n overwrite (bool): if True, overwrite any pre-existing timestamps\n array. If False, never overwrite. The outcome of simulating an\n existing array is controlled by `skip_existing` flag.\n skip_existing (bool): if True, skip simulation if the same\n timestamps array is already present.\n scale (int): `self.t_step` is multiplied by `scale` to obtain the\n timestamps units in seconds.\n path (string): folder where to save the data.\n timeslice (float or None): timestamps are simulated until\n `timeslice` seconds. If None, simulate until `self.t_max`.\n \"\"\"\n self.open_store_timestamp(chunksize=chunksize, path=path)\n rs = self._get_group_randomstate(rs, seed, self.ts_group)\n if t_chunksize is None:\n t_chunksize = self.emission.chunkshape[1]\n timeslice_size = self.n_samples\n if timeslice is not None:\n timeslice_size = timeslice // self.t_step\n\n name_d = self._get_ts_name_mix(max_rates_d, populations, bg_rate_d, rs)\n name_a = self._get_ts_name_mix(max_rates_a, populations, bg_rate_a, rs)\n\n kw = dict(clk_p=self.t_step / scale,\n populations=populations,\n num_particles=self.num_particles,\n bg_particle=self.num_particles,\n overwrite=overwrite, chunksize=chunksize)\n if comp_filter is not None:\n kw.update(comp_filter=comp_filter)\n\n kw.update(name=name_d, max_rates=max_rates_d, bg_rate=bg_rate_d)\n try:\n self._timestamps_d, self._tparticles_d = (self.ts_store\n .add_timestamps(**kw))\n except ExistingArrayError as e:\n if skip_existing:\n print(' - Skipping already present timestamps array.')\n return\n else:\n raise e\n\n kw.update(name=name_a, max_rates=max_rates_a, bg_rate=bg_rate_a)\n try:\n self._timestamps_a, self._tparticles_a = (self.ts_store\n .add_timestamps(**kw))\n except ExistingArrayError as e:\n if skip_existing:\n print(' - Skipping already present timestamps array.')\n return\n else:\n raise e\n\n self.ts_group._v_attrs['init_random_state'] = rs.get_state()\n self._timestamps_d.attrs['init_random_state'] = rs.get_state()\n self._timestamps_d.attrs['PyBroMo'] = __version__\n self._timestamps_a.attrs['init_random_state'] = rs.get_state()\n self._timestamps_a.attrs['PyBroMo'] = __version__\n\n # Load emission in chunks, and save only the final timestamps\n bg_rates_d = [None] * (len(max_rates_d) - 1) + [bg_rate_d]\n bg_rates_a = [None] * (len(max_rates_a) - 1) + [bg_rate_a]\n prev_time = 0\n for i_start, i_end in iter_chunk_index(timeslice_size, t_chunksize):\n\n curr_time = np.around(i_start * self.t_step, decimals=1)\n if curr_time > prev_time:\n print(' %.1fs' % curr_time, end='', flush=True)\n prev_time = curr_time\n\n em_chunk = self.emission[:, i_start:i_end]\n\n times_chunk_s_d, par_index_chunk_s_d = \\\n self._sim_timestamps_populations(\n em_chunk, max_rates_d, populations, bg_rates_d, i_start,\n rs, scale)\n\n times_chunk_s_a, par_index_chunk_s_a = \\\n self._sim_timestamps_populations(\n em_chunk, max_rates_a, populations, bg_rates_a, i_start,\n rs, scale)\n\n # Save sorted timestamps (suffix '_s') and corresponding particles\n self._timestamps_d.append(times_chunk_s_d)\n self._tparticles_d.append(par_index_chunk_s_d)\n self._timestamps_a.append(times_chunk_s_a)\n self._tparticles_a.append(par_index_chunk_s_a)\n\n # Save current random state so it can be resumed in the next session\n self.ts_group._v_attrs['last_random_state'] = rs.get_state()\n self._timestamps_d._v_attrs['last_random_state'] = rs.get_state()\n self.ts_store.h5file.flush()", "response": "This method reads the timestamps from disk and generates the timestamps arrays for a mixture of N populations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef merge_da(ts_d, ts_par_d, ts_a, ts_par_a):\n ts = np.hstack([ts_d, ts_a])\n ts_par = np.hstack([ts_par_d, ts_par_a])\n a_ch = np.hstack([np.zeros(ts_d.shape[0], dtype=bool),\n np.ones(ts_a.shape[0], dtype=bool)])\n index_sort = ts.argsort()\n return ts[index_sort], a_ch[index_sort], ts_par[index_sort]", "response": "Merge donor and acceptor timestamps and particle arrays."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncompute donor and acceptor emission rates from total emission rate and E.", "response": "def em_rates_from_E_DA(em_rate_tot, E_values):\n \"\"\"Donor and Acceptor emission rates from total emission rate and E (FRET).\n \"\"\"\n E_values = np.asarray(E_values)\n em_rates_a = E_values * em_rate_tot\n em_rates_d = em_rate_tot - em_rates_a\n return em_rates_d, em_rates_a"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef em_rates_from_E_unique(em_rate_tot, E_values):\n em_rates_d, em_rates_a = em_rates_from_E_DA(em_rate_tot, E_values)\n return np.unique(np.hstack([em_rates_d, em_rates_a]))", "response": "Array of unique emission rates for given total emission and E."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef em_rates_from_E_DA_mix(em_rates_tot, E_values):\n em_rates_d, em_rates_a = [], []\n for em_rate_tot, E_value in zip(em_rates_tot, E_values):\n em_rate_di, em_rate_ai = em_rates_from_E_DA(em_rate_tot, E_value)\n em_rates_d.append(em_rate_di)\n em_rates_a.append(em_rate_ai)\n return em_rates_d, em_rates_a", "response": "Returns D and A emission rates for two populations."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef populations_diff_coeff(particles, populations):\n D_counts = particles.diffusion_coeff_counts\n if len(D_counts) == 1:\n pop_sizes = [pop.stop - pop.start for pop in populations]\n assert D_counts[0][1] >= sum(pop_sizes)\n D_counts = [(D_counts[0][0], ps) for ps in pop_sizes]\n\n D_list = []\n D_pop_start = 0 # start index of diffusion-based populations\n for pop, (D, counts) in zip(populations, D_counts):\n D_list.append(D)\n assert pop.start >= D_pop_start\n assert pop.stop <= D_pop_start + counts\n D_pop_start += counts\n return D_list", "response": "Return the diffusion coefficients of the two specified populations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef populations_slices(particles, num_pop_list):\n slices = []\n i_prev = 0\n for num_pop in num_pop_list:\n slices.append(slice(i_prev, i_prev + num_pop))\n i_prev += num_pop\n return slices", "response": "2 - tuple of slices for selection of two populations."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _calc_hash_da(self, rs):\n self.hash_d = hash_(rs.get_state())[:6]\n self.hash_a = self.hash_d", "response": "Compute hash of D and A timestamps for single - step D + A case."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncomputing timestamps for current populations.", "response": "def run(self, rs, overwrite=True, skip_existing=False, path=None,\n chunksize=None):\n \"\"\"Compute timestamps for current populations.\"\"\"\n if path is None:\n path = str(self.S.store.filepath.parent)\n kwargs = dict(rs=rs, overwrite=overwrite, path=path,\n timeslice=self.timeslice, skip_existing=skip_existing)\n if chunksize is not None:\n kwargs['chunksize'] = chunksize\n header = ' - Mixture Simulation:'\n\n # Donor timestamps hash is from the input RandomState\n self.hash_d = hash_(rs.get_state())[:6] # needed by merge_da()\n print('%s Donor timestamps - %s' % (header, ctime()), flush=True)\n self.S.simulate_timestamps_mix(\n populations = self.populations,\n max_rates = self.em_rates_d,\n bg_rate = self.bg_rate_d,\n **kwargs)\n\n # Acceptor timestamps hash is from 'last_random_state' attribute\n # of the donor timestamps. This allows deterministic generation of\n # donor + acceptor timestamps given the input random state.\n ts_d, _ = self.S.get_timestamps_part(self.name_timestamps_d)\n rs.set_state(ts_d.attrs['last_random_state'])\n self.hash_a = hash_(rs.get_state())[:6] # needed by merge_da()\n print('\\n%s Acceptor timestamps - %s' % (header, ctime()), flush=True)\n self.S.simulate_timestamps_mix(\n populations = self.populations,\n max_rates = self.em_rates_a,\n bg_rate = self.bg_rate_a,\n **kwargs)\n print('\\n%s Completed. %s' % (header, ctime()), flush=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\ncompute timestamps for current populations.", "response": "def run_da(self, rs, overwrite=True, skip_existing=False, path=None,\n chunksize=None):\n \"\"\"Compute timestamps for current populations.\"\"\"\n if path is None:\n path = str(self.S.store.filepath.parent)\n kwargs = dict(rs=rs, overwrite=overwrite, path=path,\n timeslice=self.timeslice, skip_existing=skip_existing)\n if chunksize is not None:\n kwargs['chunksize'] = chunksize\n header = ' - Mixture Simulation:'\n\n # Donor timestamps hash is from the input RandomState\n self._calc_hash_da(rs)\n print('%s Donor + Acceptor timestamps - %s' %\n (header, ctime()), flush=True)\n self.S.simulate_timestamps_mix_da(\n max_rates_d = self.em_rates_d,\n max_rates_a = self.em_rates_a,\n populations = self.populations,\n bg_rate_d = self.bg_rate_d,\n bg_rate_a = self.bg_rate_a,\n **kwargs)\n print('\\n%s Completed. %s' % (header, ctime()), flush=True)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nmerging donor and acceptor timestamps computes ts a_ch part.", "response": "def merge_da(self):\n \"\"\"Merge donor and acceptor timestamps, computes `ts`, `a_ch`, `part`.\n \"\"\"\n print(' - Merging D and A timestamps', flush=True)\n ts_d, ts_par_d = self.S.get_timestamps_part(self.name_timestamps_d)\n ts_a, ts_par_a = self.S.get_timestamps_part(self.name_timestamps_a)\n ts, a_ch, part = merge_da(ts_d, ts_par_d, ts_a, ts_par_a)\n assert a_ch.sum() == ts_a.shape[0]\n assert (~a_ch).sum() == ts_d.shape[0]\n assert a_ch.size == ts_a.shape[0] + ts_d.shape[0]\n self.ts, self.a_ch, self.part = ts, a_ch, part\n self.clk_p = ts_d.attrs['clk_p']"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef save_photon_hdf5(self, identity=None, overwrite=True, path=None):\n filepath = self.filepath\n if path is not None:\n filepath = Path(path, filepath.name)\n self.merge_da()\n data = self._make_photon_hdf5(identity=identity)\n phc.hdf5.save_photon_hdf5(data, h5_fname=str(filepath),\n overwrite=overwrite)", "response": "Create a smFRET Photon - HDF5 file with current timestamps."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nprinting the HDF5 attributes for node_name.", "response": "def print_attrs(data_file, node_name='/', which='user', compress=False):\n \"\"\"Print the HDF5 attributes for `node_name`.\n\n Parameters:\n data_file (pytables HDF5 file object): the data file to print\n node_name (string): name of the path inside the file to be printed.\n Can be either a group or a leaf-node. Default: '/', the root node.\n which (string): Valid values are 'user' for user-defined attributes,\n 'sys' for pytables-specific attributes and 'all' to print both\n groups of attributes. Default 'user'.\n compress (bool): if True displays at most a line for each attribute.\n Default False.\n \"\"\"\n node = data_file.get_node(node_name)\n print ('List of attributes for:\\n %s\\n' % node)\n for attr in node._v_attrs._f_list():\n print ('\\t%s' % attr)\n attr_content = repr(node._v_attrs[attr])\n if compress:\n attr_content = attr_content.split('\\n')[0]\n print (\"\\t %s\" % attr_content)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nprinting all the sub - groups in group and leaf - nodes children of group.", "response": "def print_children(data_file, group='/'):\n \"\"\"Print all the sub-groups in `group` and leaf-nodes children of `group`.\n\n Parameters:\n data_file (pytables HDF5 file object): the data file to print\n group (string): path name of the group to be printed.\n Default: '/', the root node.\n \"\"\"\n base = data_file.get_node(group)\n print ('Groups in:\\n %s\\n' % base)\n\n for node in base._f_walk_groups():\n if node is not base:\n print (' %s' % node)\n\n print ('\\nLeaf-nodes in %s:' % group)\n for node in base._v_leaves.itervalues():\n info = node.shape\n if len(info) == 0:\n info = node.read()\n print ('\\t%s, %s' % (node.name, info))\n if len(node.title) > 0:\n print ('\\t %s' % node.title)"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ntrain model on given training examples and return the list of costs after each minibatch is processed.", "response": "def fit(self, trX, trY, batch_size=64, n_epochs=1, len_filter=LenFilter(), snapshot_freq=1, path=None):\n \"\"\"Train model on given training examples and return the list of costs after each minibatch is processed.\n\n Args:\n trX (list) -- Inputs\n trY (list) -- Outputs\n batch_size (int, optional) -- number of examples in a minibatch (default 64)\n n_epochs (int, optional) -- number of epochs to train for (default 1)\n len_filter (object, optional) -- object to filter training example by length (default LenFilter())\n snapshot_freq (int, optional) -- number of epochs between saving model snapshots (default 1)\n path (str, optional) -- prefix of path where model snapshots are saved.\n If None, no snapshots are saved (default None)\n\n Returns:\n list -- costs of model after processing each minibatch\n \"\"\"\n if len_filter is not None:\n trX, trY = len_filter.filter(trX, trY)\n trY = standardize_targets(trY, cost=self.cost)\n\n n = 0.\n t = time()\n costs = []\n for e in range(n_epochs):\n epoch_costs = []\n for xmb, ymb in self.iterator.iterXY(trX, trY):\n c = self._train(xmb, ymb)\n epoch_costs.append(c)\n n += len(ymb)\n if self.verbose >= 2:\n n_per_sec = n / (time() - t)\n n_left = len(trY) - n % len(trY)\n time_left = n_left/n_per_sec\n sys.stdout.write(\"\\rEpoch %d Seen %d samples Avg cost %0.4f Time left %d seconds\" % (e, n, np.mean(epoch_costs[-250:]), time_left))\n sys.stdout.flush()\n costs.extend(epoch_costs)\n\n status = \"Epoch %d Seen %d samples Avg cost %0.4f Time elapsed %d seconds\" % (e, n, np.mean(epoch_costs[-250:]), time() - t)\n if self.verbose >= 2:\n sys.stdout.write(\"\\r\"+status)\n sys.stdout.flush()\n sys.stdout.write(\"\\n\")\n elif self.verbose == 1:\n print(status)\n if path and e % snapshot_freq == 0:\n save(self, \"{0}.{1}\".format(path, e))\n return costs"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ngenerating a VAO that is a plane on the xz axis of a specific size and resolution.", "response": "def plane_xz(size=(10, 10), resolution=(10, 10)) -> VAO:\n \"\"\"\n Generates a plane on the xz axis of a specific size and resolution.\n Normals and texture coordinates are also included.\n\n Args:\n size: (x, y) tuple\n resolution: (x, y) tuple\n\n Returns:\n A :py:class:`demosys.opengl.vao.VAO` instance\n \"\"\"\n sx, sz = size\n rx, rz = resolution\n dx, dz = sx / rx, sz / rz # step\n ox, oz = -sx / 2, -sz / 2 # start offset\n\n def gen_pos():\n for z in range(rz):\n for x in range(rx):\n yield ox + x * dx\n yield 0\n yield oz + z * dz\n\n def gen_uv():\n for z in range(rz):\n for x in range(rx):\n yield x / (rx - 1)\n yield 1 - z / (rz - 1)\n\n def gen_normal():\n for _ in range(rx * rz):\n yield 0.0\n yield 1.0\n yield 0.0\n\n def gen_index():\n for z in range(rz - 1):\n for x in range(rx - 1):\n # quad poly left\n yield z * rz + x + 1\n yield z * rz + x\n yield z * rz + x + rx\n # quad poly right\n yield z * rz + x + 1\n yield z * rz + x + rx\n yield z * rz + x + rx + 1\n\n pos_data = numpy.fromiter(gen_pos(), dtype=numpy.float32)\n uv_data = numpy.fromiter(gen_uv(), dtype=numpy.float32)\n normal_data = numpy.fromiter(gen_normal(), dtype=numpy.float32)\n index_data = numpy.fromiter(gen_index(), dtype=numpy.uint32)\n\n vao = VAO(\"plane_xz\", mode=moderngl.TRIANGLES)\n\n vao.buffer(pos_data, '3f', ['in_position'])\n vao.buffer(uv_data, '2f', ['in_uv'])\n vao.buffer(normal_data, '3f', ['in_normal'])\n\n vao.index_buffer(index_data, index_element_size=4)\n\n return vao"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load(self):\n self.path = self.find_scene(self.meta.path)\n if not self.path:\n raise ValueError(\"Scene '{}' not found\".format(self.meta.path))\n\n self.scene = Scene(self.path)\n\n # Load gltf json file\n if self.path.suffix == '.gltf':\n self.load_gltf()\n\n # Load binary gltf file\n if self.path.suffix == '.glb':\n self.load_glb()\n\n self.meta.check_version()\n self.meta.check_extensions(self.supported_extensions)\n self.load_images()\n self.load_samplers()\n self.load_textures()\n self.load_materials()\n self.load_meshes()\n self.load_nodes()\n\n self.scene.calc_scene_bbox()\n self.scene.prepare()\n\n return self.scene", "response": "Load the object containing the current metadata and return it."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nload a gltf json file", "response": "def load_gltf(self):\n \"\"\"Loads a gltf json file\"\"\"\n with open(self.path) as fd:\n self.meta = GLTFMeta(self.path, json.load(fd))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_glb(self):\n with open(self.path, 'rb') as fd:\n # Check header\n magic = fd.read(4)\n if magic != GLTF_MAGIC_HEADER:\n raise ValueError(\"{} has incorrect header {} != {}\".format(self.path, magic, GLTF_MAGIC_HEADER))\n\n version = struct.unpack(' buffer_views -> buffers\n for acc in self.accessors:\n acc.bufferView = self.buffer_views[acc.bufferViewId]\n\n for buffer_view in self.buffer_views:\n buffer_view.buffer = self.buffers[buffer_view.bufferId]\n\n # Link accessors to mesh primitives\n for mesh in self.meshes:\n for primitive in mesh.primitives:\n if getattr(primitive, \"indices\", None) is not None:\n primitive.indices = self.accessors[primitive.indices]\n for name, value in primitive.attributes.items():\n primitive.attributes[name] = self.accessors[value]\n\n # Link buffer views to images\n for image in self.images:\n if image.bufferViewId is not None:\n image.bufferView = self.buffer_views[image.bufferViewId]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncheck if the extensions are supported.", "response": "def check_extensions(self, supported):\n \"\"\"\n \"extensionsRequired\": [\"KHR_draco_mesh_compression\"],\n \"extensionsUsed\": [\"KHR_draco_mesh_compression\"]\n \"\"\"\n if self.data.get('extensionsRequired'):\n for ext in self.data.get('extensionsRequired'):\n if ext not in supported:\n raise ValueError(\"Extension {} not supported\".format(ext))\n\n if self.data.get('extensionsUsed'):\n for ext in self.data.get('extensionsUsed'):\n if ext not in supported:\n raise ValueError(\"Extension {} not supported\".format(ext))"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if the bin files referenced in this object exist", "response": "def buffers_exist(self):\n \"\"\"Checks if the bin files referenced exist\"\"\"\n for buff in self.buffers:\n if not buff.is_separate_file:\n continue\n\n path = self.path.parent / buff.uri\n if not os.path.exists(path):\n raise FileNotFoundError(\"Buffer {} referenced in {} not found\".format(path, self.path))"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nloading the index buffer and polygon list for a primitive", "response": "def load_indices(self, primitive):\n \"\"\"Loads the index buffer / polygon list for a primitive\"\"\"\n if getattr(primitive, \"indices\") is None:\n return None, None\n\n _, component_type, buffer = primitive.indices.read()\n return component_type, buffer"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ngets the bounding box for the mesh", "response": "def get_bbox(self, primitive):\n \"\"\"Get the bounding box for the mesh\"\"\"\n accessor = primitive.attributes.get('POSITION')\n return accessor.min, accessor.max"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef interleaves(self, info):\n return info.byte_offset == self.component_type.size * self.components", "response": "Does the buffer interleave with this one?"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nread buffer data :return: component count, component type, data", "response": "def read(self):\n \"\"\"\n Reads buffer data\n :return: component count, component type, data\n \"\"\"\n # ComponentType helps us determine the datatype\n dtype = NP_COMPONENT_DTYPE[self.componentType.value]\n return ACCESSOR_TYPE[self.type], self.componentType, self.bufferView.read(\n byte_offset=self.byteOffset,\n dtype=dtype,\n count=self.count * ACCESSOR_TYPE[self.type],\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef info(self):\n buffer, byte_length, byte_offset = self.bufferView.info(byte_offset=self.byteOffset)\n return buffer, self.bufferView, \\\n byte_length, byte_offset, \\\n self.componentType, ACCESSOR_TYPE[self.type], self.count", "response": "Get underlying buffer info for this accessor\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ngetting the underlying buffer info", "response": "def info(self, byte_offset=0):\n \"\"\"\n Get the underlying buffer info\n :param byte_offset: byte offset from accessor\n :return: buffer, byte_length, byte_offset\n \"\"\"\n return self.buffer, self.byteLength, byte_offset + self.byteOffset"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nset the 3D position of the camera in the cache", "response": "def set_position(self, x, y, z):\n \"\"\"\n Set the 3D position of the camera\n\n :param x: float\n :param y: float\n :param z: float\n \"\"\"\n self.position = Vector3([x, y, z])"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef view_matrix(self):\n self._update_yaw_and_pitch()\n return self._gl_look_at(self.position, self.position + self.dir, self._up)", "response": "Returns the current view matrix for the camera."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nupdating the camera vectors based on the current yaw and pitch.", "response": "def _update_yaw_and_pitch(self):\n \"\"\"\n Updates the camera vectors based on the current yaw and pitch\n \"\"\"\n front = Vector3([0.0, 0.0, 0.0])\n front.x = cos(radians(self.yaw)) * cos(radians(self.pitch))\n front.y = sin(radians(self.pitch))\n front.z = sin(radians(self.yaw)) * cos(radians(self.pitch))\n\n self.dir = vector.normalise(front)\n self.right = vector.normalise(vector3.cross(self.dir, self._up))\n self.up = vector.normalise(vector3.cross(self.right, self.dir))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nlooking at a specific point in the camera.", "response": "def look_at(self, vec=None, pos=None):\n \"\"\"\n Look at a specific point\n\n :param vec: Vector3 position\n :param pos: python list [x, y, x]\n :return: Camera matrix\n \"\"\"\n if pos is None:\n vec = Vector3(pos)\n\n if vec is None:\n raise ValueError(\"vector or pos must be set\")\n\n return self._gl_look_at(self.position, vec, self._up)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nupdating the camera position move state", "response": "def move_state(self, direction, activate):\n \"\"\"\n Set the camera position move state\n\n :param direction: What direction to update\n :param activate: Start or stop moving in the direction\n \"\"\"\n if direction == RIGHT:\n self._xdir = POSITIVE if activate else STILL\n elif direction == LEFT:\n self._xdir = NEGATIVE if activate else STILL\n elif direction == FORWARD:\n self._zdir = NEGATIVE if activate else STILL\n elif direction == BACKWARD:\n self._zdir = POSITIVE if activate else STILL\n elif direction == UP:\n self._ydir = POSITIVE if activate else STILL\n elif direction == DOWN:\n self._ydir = NEGATIVE if activate else STILL"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef rot_state(self, x, y):\n if self.last_x is None:\n self.last_x = x\n if self.last_y is None:\n self.last_y = y\n\n x_offset = self.last_x - x\n y_offset = self.last_y - y\n\n self.last_x = x\n self.last_y = y\n\n x_offset *= self.mouse_sensitivity\n y_offset *= self.mouse_sensitivity\n\n self.yaw -= x_offset\n self.pitch += y_offset\n\n if self.pitch > 85.0:\n self.pitch = 85.0\n if self.pitch < -85.0:\n self.pitch = -85.0\n\n self._update_yaw_and_pitch()", "response": "Set the rotation state of the camera."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the current view matrix for the camera.", "response": "def view_matrix(self):\n \"\"\"\n :return: The current view matrix for the camera\n \"\"\"\n # Use separate time in camera so we can move it when the demo is paused\n now = time.time()\n # If the camera has been inactive for a while, a large time delta\n # can suddenly move the camera far away from the scene\n t = max(now - self._last_time, 0)\n self._last_time = now\n\n # X Movement\n if self._xdir == POSITIVE:\n self.position += self.right * self.velocity * t\n elif self._xdir == NEGATIVE:\n self.position -= self.right * self.velocity * t\n\n # Z Movement\n if self._zdir == NEGATIVE:\n self.position += self.dir * self.velocity * t\n elif self._zdir == POSITIVE:\n self.position -= self.dir * self.velocity * t\n\n # Y Movement\n if self._ydir == POSITIVE:\n self.position += self.up * self.velocity * t\n elif self._ydir == NEGATIVE:\n self.position -= self.up * self.velocity * t\n\n return self._gl_look_at(self.position, self.position + self.dir, self._up)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ntranslate string into character texture positions", "response": "def _translate_string(self, data, length):\r\n \"\"\"Translate string into character texture positions\"\"\"\r\n for index, char in enumerate(data):\r\n if index == length:\r\n break\r\n\r\n yield self._meta.characters - 1 - self._ct[char]"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\ngenerate character translation map", "response": "def _generate_character_map(self):\r\n \"\"\"Generate character translation map (latin1 pos to texture pos)\"\"\"\r\n self._ct = [-1] * 256\r\n index = 0\r\n for crange in self._meta.character_ranges:\r\n for cpos in range(crange['min'], crange['max'] + 1):\r\n self._ct[cpos] = index\r\n index += 1"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef buffer_format(frmt: str) -> BufferFormat:\n try:\n return BUFFER_FORMATS[frmt]\n except KeyError:\n raise ValueError(\"Buffer format '{}' unknown. Valid formats: {}\".format(\n frmt, BUFFER_FORMATS.keys()\n ))", "response": "Return the buffer format instance for the given format string."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef attribute_format(frmt: str) -> BufferFormat:\n try:\n return ATTRIBUTE_FORMATS[frmt]\n except KeyError:\n raise ValueError(\"Buffer format '{}' unknown. Valid formats: {}\".format(\n frmt, ATTRIBUTE_FORMATS.keys()\n ))", "response": "Return the buffer format instance for the given format."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef init(window=None, project=None, timeline=None):\n from demosys.effects.registry import Effect\n from demosys.scene import camera\n\n window.timeline = timeline\n\n # Inject attributes into the base Effect class\n setattr(Effect, '_window', window)\n setattr(Effect, '_ctx', window.ctx)\n setattr(Effect, '_project', project)\n\n # Set up the default system camera\n window.sys_camera = camera.SystemCamera(aspect=window.aspect_ratio, fov=60.0, near=1, far=1000)\n setattr(Effect, '_sys_camera', window.sys_camera)\n\n print(\"Loading started at\", time.time())\n project.load()\n\n # Initialize timer\n timer_cls = import_string(settings.TIMER)\n window.timer = timer_cls()\n window.timer.start()", "response": "Initialize load and run\n "} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef draw(self, projection_matrix=None, camera_matrix=None, time=0):\n projection_matrix = projection_matrix.astype('f4').tobytes()\n camera_matrix = camera_matrix.astype('f4').tobytes()\n\n for node in self.root_nodes:\n node.draw(\n projection_matrix=projection_matrix,\n camera_matrix=camera_matrix,\n time=time,\n )\n\n self.ctx.clear_samplers(0, 4)", "response": "Draw all the nodes in the scene"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef draw_bbox(self, projection_matrix=None, camera_matrix=None, all=True):\n projection_matrix = projection_matrix.astype('f4').tobytes()\n camera_matrix = camera_matrix.astype('f4').tobytes()\n\n # Scene bounding box\n self.bbox_program[\"m_proj\"].write(projection_matrix)\n self.bbox_program[\"m_view\"].write(self._view_matrix.astype('f4').tobytes())\n self.bbox_program[\"m_cam\"].write(camera_matrix)\n self.bbox_program[\"bb_min\"].write(self.bbox_min.astype('f4').tobytes())\n self.bbox_program[\"bb_max\"].write(self.bbox_max.astype('f4').tobytes())\n self.bbox_program[\"color\"].value = (1.0, 0.0, 0.0)\n self.bbox_vao.render(self.bbox_program)\n\n if not all:\n return\n\n # Draw bounding box for children\n for node in self.root_nodes:\n node.draw_bbox(projection_matrix, camera_matrix, self.bbox_program, self.bbox_vao)", "response": "Draw bounding boxes for the current scene and mesh."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef apply_mesh_programs(self, mesh_programs=None):\n if not mesh_programs:\n mesh_programs = [ColorProgram(), TextureProgram(), FallbackProgram()]\n\n for mesh in self.meshes:\n for mp in mesh_programs:\n instance = mp.apply(mesh)\n if instance is not None:\n if isinstance(instance, MeshProgram):\n mesh.mesh_program = mp\n break\n else:\n raise ValueError(\"apply() must return a MeshProgram instance, not {}\".format(type(instance)))\n\n if not mesh.mesh_program:\n print(\"WARING: No mesh program applied to '{}'\".format(mesh.name))", "response": "Applies the given list of mesh programs to the current instance of the current mesh."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncalculating scene bounding box", "response": "def calc_scene_bbox(self):\n \"\"\"Calculate scene bbox\"\"\"\n bbox_min, bbox_max = None, None\n for node in self.root_nodes:\n bbox_min, bbox_max = node.calc_global_bbox(\n matrix44.create_identity(),\n bbox_min,\n bbox_max\n )\n\n self.bbox_min = bbox_min\n self.bbox_max = bbox_max\n\n self.diagonal_size = vector3.length(self.bbox_max - self.bbox_min)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate random 3D points inside a confied box.", "response": "def points_random_3d(count, range_x=(-10.0, 10.0), range_y=(-10.0, 10.0), range_z=(-10.0, 10.0), seed=None) -> VAO:\n \"\"\"\n Generates random positions inside a confied box.\n\n Args:\n count (int): Number of points to generate\n\n Keyword Args:\n range_x (tuple): min-max range for x axis: Example (-10.0. 10.0)\n range_y (tuple): min-max range for y axis: Example (-10.0. 10.0)\n range_z (tuple): min-max range for z axis: Example (-10.0. 10.0)\n seed (int): The random seed\n\n Returns:\n A :py:class:`demosys.opengl.vao.VAO` instance\n \"\"\"\n random.seed(seed)\n\n def gen():\n for _ in range(count):\n yield random.uniform(*range_x)\n yield random.uniform(*range_y)\n yield random.uniform(*range_z)\n\n data = numpy.fromiter(gen(), count=count * 3, dtype=numpy.float32)\n\n vao = VAO(\"geometry:points_random_3d\", mode=moderngl.POINTS)\n vao.buffer(data, '3f', ['in_position'])\n\n return vao"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget the current time in seconds", "response": "def get_time(self) -> float:\n \"\"\"\n Get the current position in the music in seconds\n \"\"\"\n if self.paused:\n return self.pause_time\n\n return mixer.music.get_pos() / 1000.0"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef set_time(self, value: float):\n if value < 0:\n value = 0\n\n # mixer.music.play(start=value)\n mixer.music.set_pos(value)", "response": "Set the current time in the music in seconds causing the player to seek to this location."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef draw_buffers(self, near, far):\n self.ctx.disable(moderngl.DEPTH_TEST)\n\n helper.draw(self.gbuffer.color_attachments[0], pos=(0.0, 0.0), scale=(0.25, 0.25))\n helper.draw(self.gbuffer.color_attachments[1], pos=(0.5, 0.0), scale=(0.25, 0.25))\n helper.draw_depth(self.gbuffer.depth_attachment, near, far, pos=(1.0, 0.0), scale=(0.25, 0.25))\n helper.draw(self.lightbuffer.color_attachments[0], pos=(1.5, 0.0), scale=(0.25, 0.25))", "response": "Draw the framebuffers for debug purposes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef add_point_light(self, position, radius):\n self.point_lights.append(PointLight(position, radius))", "response": "Add point light to the set"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nrender the light volumes from the inside skeleton", "response": "def render_lights(self, camera_matrix, projection):\n \"\"\"Render light volumes\"\"\"\n # Draw light volumes from the inside\n self.ctx.front_face = 'cw'\n self.ctx.blend_func = moderngl.ONE, moderngl.ONE\n\n helper._depth_sampler.use(location=1)\n with self.lightbuffer_scope:\n for light in self.point_lights:\n # Calc light properties\n light_size = light.radius\n m_light = matrix44.multiply(light.matrix, camera_matrix)\n # Draw the light volume\n self.point_light_shader[\"m_proj\"].write(projection.tobytes())\n self.point_light_shader[\"m_light\"].write(m_light.astype('f4').tobytes())\n self.gbuffer.color_attachments[1].use(location=0)\n self.point_light_shader[\"g_normal\"].value = 0\n self.gbuffer.depth_attachment.use(location=1)\n self.point_light_shader[\"g_depth\"].value = 1\n self.point_light_shader[\"screensize\"].value = (self.width, self.height)\n self.point_light_shader[\"proj_const\"].value = projection.projection_constants\n self.point_light_shader[\"radius\"].value = light_size\n self.unit_cube.render(self.point_light_shader)\n\n helper._depth_sampler.clear(location=1)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nrender outlines of light volumes", "response": "def render_lights_debug(self, camera_matrix, projection):\n \"\"\"Render outlines of light volumes\"\"\"\n self.ctx.enable(moderngl.BLEND)\n self.ctx.blend_func = moderngl.SRC_ALPHA, moderngl.ONE_MINUS_SRC_ALPHA\n\n for light in self.point_lights:\n m_mv = matrix44.multiply(light.matrix, camera_matrix)\n light_size = light.radius\n self.debug_shader[\"m_proj\"].write(projection.tobytes())\n self.debug_shader[\"m_mv\"].write(m_mv.astype('f4').tobytes())\n self.debug_shader[\"size\"].value = light_size\n self.unit_cube.render(self.debug_shader, mode=moderngl.LINE_STRIP)\n\n self.ctx.disable(moderngl.BLEND)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncombining diffuse and light buffer", "response": "def combine(self):\n \"\"\"Combine diffuse and light buffer\"\"\"\n self.gbuffer.color_attachments[0].use(location=0)\n self.combine_shader[\"diffuse_buffer\"].value = 0\n self.lightbuffer.color_attachments[0].use(location=1)\n self.combine_shader[\"light_buffer\"].value = 1\n self.quad.render(self.combine_shader)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef load_shader(self, shader_type: str, path: str):\r\n if path:\r\n resolved_path = self.find_program(path)\r\n if not resolved_path:\r\n raise ValueError(\"Cannot find {} shader '{}'\".format(shader_type, path))\r\n\r\n print(\"Loading:\", path)\r\n\r\n with open(resolved_path, 'r') as fd:\r\n return fd.read()", "response": "Load a single shader"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nload a texture array", "response": "def load(self):\r\n \"\"\"Load a texture array\"\"\"\r\n self._open_image()\r\n\r\n width, height, depth = self.image.size[0], self.image.size[1] // self.layers, self.layers\r\n components, data = image_data(self.image)\r\n\r\n texture = self.ctx.texture_array(\r\n (width, height, depth),\r\n components,\r\n data,\r\n )\r\n texture.extra = {'meta': self.meta}\r\n\r\n if self.meta.mipmap:\r\n texture.build_mipmaps()\r\n\r\n self._close_image()\r\n\r\n return texture"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ndraws the mesh using the assigned program", "response": "def draw(self, projection_matrix=None, view_matrix=None, camera_matrix=None, time=0):\n \"\"\"\n Draw the mesh using the assigned mesh program\n\n :param projection_matrix: projection_matrix (bytes)\n :param view_matrix: view_matrix (bytes)\n :param camera_matrix: camera_matrix (bytes)\n \"\"\"\n if self.mesh_program:\n self.mesh_program.draw(\n self,\n projection_matrix=projection_matrix,\n view_matrix=view_matrix,\n camera_matrix=camera_matrix,\n time=time\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nadd metadata about the mesh", "response": "def add_attribute(self, attr_type, name, components):\n \"\"\"\n Add metadata about the mesh\n :param attr_type: POSITION, NORMAL etc\n :param name: The attribute name used in the program\n :param components: Number of floats\n \"\"\"\n self.attributes[attr_type] = {\"name\": name, \"components\": components}"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nset the current time jumping in the timeline.", "response": "def set_time(self, value: float):\n \"\"\"\n Set the current time jumping in the timeline.\n\n Args:\n value (float): The new time\n \"\"\"\n if value < 0:\n value = 0\n\n self.controller.row = self.rps * value"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndraw function called by the system every frame when the effect is active. This method raises ``NotImplementedError`` unless implemented. Args: time (float): The current time in seconds. frametime (float): The time the previous frame used to render in seconds. target (``moderngl.Framebuffer``): The target FBO for the effect.", "response": "def draw(self, time: float, frametime: float, target: moderngl.Framebuffer):\n \"\"\"\n Draw function called by the system every frame when the effect is active.\n This method raises ``NotImplementedError`` unless implemented.\n\n Args:\n time (float): The current time in seconds.\n frametime (float): The time the previous frame used to render in seconds.\n target (``moderngl.Framebuffer``): The target FBO for the effect.\n \"\"\"\n raise NotImplementedError(\"draw() is not implemented\")"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ngets a program by its label", "response": "def get_program(self, label: str) -> moderngl.Program:\n \"\"\"\n Get a program by its label\n\n Args:\n label (str): The label for the program\n\n Returns: py:class:`moderngl.Program` instance\n \"\"\"\n return self._project.get_program(label)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nget a texture by its label", "response": "def get_texture(self, label: str) -> Union[moderngl.Texture, moderngl.TextureArray,\n moderngl.Texture3D, moderngl.TextureCube]:\n \"\"\"\n Get a texture by its label\n\n Args:\n label (str): The Label for the texture\n\n Returns:\n The py:class:`moderngl.Texture` instance\n \"\"\"\n return self._project.get_texture(label)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_effect_class(self, effect_name: str, package_name: str = None) -> Type['Effect']:\n return self._project.get_effect_class(effect_name, package_name=package_name)", "response": "Returns the class object corresponding to the given effect name."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_projection(self, fov: float = 75.0, near: float = 1.0, far: float = 100.0, aspect_ratio: float = None):\n return matrix44.create_perspective_projection_matrix(\n fov,\n aspect_ratio or self.window.aspect_ratio,\n near,\n far,\n dtype='f4',\n )", "response": "Create a projection matrix for the specified field of view near and far."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a transformation matrix woth rotations and translation.", "response": "def create_transformation(self, rotation=None, translation=None):\n \"\"\"\n Creates a transformation matrix woth rotations and translation.\n\n Args:\n rotation: 3 component vector as a list, tuple, or :py:class:`pyrr.Vector3`\n translation: 3 component vector as a list, tuple, or :py:class:`pyrr.Vector3`\n\n Returns:\n A 4x4 matrix as a :py:class:`numpy.array`\n \"\"\"\n mat = None\n if rotation is not None:\n mat = Matrix44.from_eulers(Vector3(rotation))\n\n if translation is not None:\n trans = matrix44.create_from_translation(Vector3(translation))\n if mat is None:\n mat = trans\n else:\n mat = matrix44.multiply(mat, trans)\n\n return mat"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef create_normal_matrix(self, modelview):\n normal_m = Matrix33.from_matrix44(modelview)\n normal_m = normal_m.inverse\n normal_m = normal_m.transpose()\n return normal_m", "response": "Creates a normal matrix from a modelview matrix."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nscans for available templates in effect_templates", "response": "def available_templates(value):\n \"\"\"Scan for available templates in effect_templates\"\"\"\n templates = list_templates()\n\n if value not in templates:\n raise ArgumentTypeError(\"Effect template '{}' does not exist.\\n Available templates: {} \".format(\n value, \", \".join(templates)))\n\n return value"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget the absolute path to the root of the demosys package", "response": "def root_path():\n \"\"\"Get the absolute path to the root of the demosys package\"\"\"\n module_dir = os.path.dirname(globals()['__file__'])\n return os.path.dirname(os.path.dirname(module_dir))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef load(self):\r\n self.meta.resolved_path = self.find_data(self.meta.path)\r\n\r\n if not self.meta.resolved_path:\r\n raise ImproperlyConfigured(\"Data file '{}' not found\".format(self.meta.path))\r\n\r\n print(\"Loading:\", self.meta.path)\r\n\r\n with open(self.meta.resolved_path, 'r') as fd:\r\n return fd.read()", "response": "Load a file in text mode"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn an instance of the finder class from an import path.", "response": "def get_finder(import_path):\n \"\"\"\n Get a finder class from an import path.\n Raises ``demosys.core.exceptions.ImproperlyConfigured`` if the finder is not found.\n This function uses an lru cache.\n\n :param import_path: string representing an import path\n :return: An instance of the finder\n \"\"\"\n Finder = import_string(import_path)\n if not issubclass(Finder, BaseFileSystemFinder):\n raise ImproperlyConfigured('Finder {} is not a subclass of core.finders.FileSystemFinder'.format(import_path))\n return Finder()"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nfind a file in the path.", "response": "def find(self, path: Path):\n \"\"\"\n Find a file in the path. The file may exist in multiple\n paths. The last found file will be returned.\n\n :param path: The path to find\n :return: The absolute path to the file or None if not found\n \"\"\"\n # Update paths from settings to make them editable runtime\n # This is only possible for FileSystemFinders\n if getattr(self, 'settings_attr', None):\n self.paths = getattr(settings, self.settings_attr)\n\n path_found = None\n\n for entry in self.paths:\n abspath = entry / path\n if abspath.exists():\n path_found = abspath\n\n return path_found"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nupdating the internal projection matrix based on current values of the current key - value store.", "response": "def update(self, aspect_ratio=None, fov=None, near=None, far=None):\n \"\"\"\n Update the internal projection matrix based on current values\n or values passed in if specified.\n\n :param aspect_ratio: New aspect ratio\n :param fov: New field of view\n :param near: New near value\n :param far: New far value\n \"\"\"\n self.aspect_ratio = aspect_ratio or self.aspect_ratio\n self.fov = fov or self.fov\n self.near = near or self.near\n self.far = far or self.far\n\n self.matrix = Matrix44.perspective_projection(self.fov, self.aspect_ratio, self.near, self.far)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the x y tuple of the current projection constants for the current projection.", "response": "def projection_constants(self):\n \"\"\"\n Returns the (x, y) projection constants for the current projection.\n :return: x, y tuple projection constants\n \"\"\"\n return self.far / (self.far - self.near), (self.far * self.near) / (self.near - self.far)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ndraw node and children of this node and all children of this node.", "response": "def draw(self, projection_matrix=None, camera_matrix=None, time=0):\n \"\"\"\n Draw node and children\n\n :param projection_matrix: projection matrix (bytes)\n :param camera_matrix: camera_matrix (bytes)\n :param time: The current time\n \"\"\"\n if self.mesh:\n self.mesh.draw(\n projection_matrix=projection_matrix,\n view_matrix=self.matrix_global_bytes,\n camera_matrix=camera_matrix,\n time=time\n )\n\n for child in self.children:\n child.draw(\n projection_matrix=projection_matrix,\n camera_matrix=camera_matrix,\n time=time\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef swap_buffers(self):\n self.frames += 1\n glfw.swap_buffers(self.window)\n self.poll_events()", "response": "Swap buffers incement the framecounter and pull events."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef resize(self, width, height):\n self.width = width\n self.height = height\n self.buffer_width, self.buffer_height = glfw.get_framebuffer_size(self.window)\n self.set_default_viewport()", "response": "Sets the new size and buffer size internally"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef check_glfw_version(self):\n print(\"glfw version: {} (python wrapper version {})\".format(glfw.get_version(), glfw.__version__))\n if glfw.get_version() < self.min_glfw_version:\n raise ValueError(\"Please update glfw binaries to version {} or later\".format(self.min_glfw_version))", "response": "Checks that the glfw library version is compatible with the current version of glfw binaries."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nkeys event callback for glfw. Translates and forwards keyboard event to :py:func:`keyboard_event` :param window: Window event origin :param key: The key that was pressed or released. :param scancode: The system-specific scancode of the key. :param action: GLFW_PRESS, GLFW_RELEASE or GLFW_REPEAT :param mods: Bit field describing which modifier keys were held down.", "response": "def key_event_callback(self, window, key, scancode, action, mods):\n \"\"\"\n Key event callback for glfw.\n Translates and forwards keyboard event to :py:func:`keyboard_event`\n\n :param window: Window event origin\n :param key: The key that was pressed or released.\n :param scancode: The system-specific scancode of the key.\n :param action: GLFW_PRESS, GLFW_RELEASE or GLFW_REPEAT\n :param mods: Bit field describing which modifier keys were held down.\n \"\"\"\n self.keyboard_event(key, action, mods)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef quad_2d(width, height, xpos=0.0, ypos=0.0) -> VAO:\n pos = numpy.array([\n xpos - width / 2.0, ypos + height / 2.0, 0.0,\n xpos - width / 2.0, ypos - height / 2.0, 0.0,\n xpos + width / 2.0, ypos - height / 2.0, 0.0,\n xpos - width / 2.0, ypos + height / 2.0, 0.0,\n xpos + width / 2.0, ypos - height / 2.0, 0.0,\n xpos + width / 2.0, ypos + height / 2.0, 0.0,\n ], dtype=numpy.float32)\n\n normals = numpy.array([\n 0.0, 0.0, 1.0,\n 0.0, 0.0, 1.0,\n 0.0, 0.0, 1.0,\n 0.0, 0.0, 1.0,\n 0.0, 0.0, 1.0,\n 0.0, 0.0, 1.0,\n ], dtype=numpy.float32)\n\n uvs = numpy.array([\n 0.0, 1.0,\n 0.0, 0.0,\n 1.0, 0.0,\n 0.0, 1.0,\n 1.0, 0.0,\n 1.0, 1.0,\n ], dtype=numpy.float32)\n\n vao = VAO(\"geometry:quad\", mode=moderngl.TRIANGLES)\n vao.buffer(pos, '3f', [\"in_position\"])\n vao.buffer(normals, '3f', [\"in_normal\"])\n vao.buffer(uvs, '2f', [\"in_uv\"])\n\n return vao", "response": "Create a 2D quad VAO with 2 triangles with normals and texture coordinates."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef translate_buffer_format(vertex_format):\n buffer_format = []\n attributes = []\n mesh_attributes = []\n\n if \"T2F\" in vertex_format:\n buffer_format.append(\"2f\")\n attributes.append(\"in_uv\")\n mesh_attributes.append((\"TEXCOORD_0\", \"in_uv\", 2))\n\n if \"C3F\" in vertex_format:\n buffer_format.append(\"3f\")\n attributes.append(\"in_color\")\n mesh_attributes.append((\"NORMAL\", \"in_color\", 3))\n\n if \"N3F\" in vertex_format:\n buffer_format.append(\"3f\")\n attributes.append(\"in_normal\")\n mesh_attributes.append((\"NORMAL\", \"in_normal\", 3))\n\n buffer_format.append(\"3f\")\n attributes.append(\"in_position\")\n mesh_attributes.append((\"POSITION\", \"in_position\", 3))\n\n return \" \".join(buffer_format), attributes, mesh_attributes", "response": "Translate the buffer format into a single string."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef load(self):\n path = self.find_scene(self.meta.path)\n\n if not path:\n raise ValueError(\"Scene '{}' not found\".format(self.meta.path))\n\n if path.suffix == '.bin':\n path = path.parent / path.stem\n\n data = pywavefront.Wavefront(str(path), create_materials=True, cache=True)\n scene = Scene(self.meta.resolved_path)\n texture_cache = {}\n\n for _, mat in data.materials.items():\n mesh = Mesh(mat.name)\n\n # Traditional loader\n if mat.vertices:\n buffer_format, attributes, mesh_attributes = translate_buffer_format(mat.vertex_format)\n vbo = numpy.array(mat.vertices, dtype='f4')\n\n vao = VAO(mat.name, mode=moderngl.TRIANGLES)\n vao.buffer(vbo, buffer_format, attributes)\n mesh.vao = vao\n\n for attrs in mesh_attributes:\n mesh.add_attribute(*attrs)\n\n # Binary cache loader\n elif hasattr(mat, 'vao'):\n mesh = Mesh(mat.name)\n mesh.vao = mat.vao\n for attrs in mat.mesh_attributes:\n mesh.add_attribute(*attrs)\n else:\n # Empty\n continue\n\n scene.meshes.append(mesh)\n\n mesh.material = Material(mat.name)\n scene.materials.append(mesh.material)\n mesh.material.color = mat.diffuse\n\n if mat.texture:\n # A texture can be referenced multiple times, so we need to cache loaded ones\n texture = texture_cache.get(mat.texture.path)\n if not texture:\n print(\"Loading:\", mat.texture.path)\n texture = textures.load(TextureDescription(\n label=mat.texture.path,\n path=mat.texture.path,\n mipmap=True,\n ))\n texture_cache[mat.texture.path] = texture\n\n mesh.material.mat_texture = MaterialTexture(\n texture=texture,\n sampler=None,\n )\n\n node = Node(mesh=mesh)\n scene.root_nodes.append(node)\n\n # Not supported yet for obj\n # self.calc_scene_bbox()\n scene.prepare()\n\n return scene", "response": "Load the current object into the internal state."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef start(self):\n if self.start_time is None:\n self.start_time = time.time()\n # Play after pause\n else:\n # Add the duration of the paused interval to the total offset\n pause_duration = time.time() - self.pause_time\n self.offset += pause_duration\n # print(\"pause duration\", pause_duration, \"offset\", self.offset)\n # Exit the paused state\n self.pause_time = None", "response": "Start the paused state."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nstop the timer and return the time the timer was stopped.", "response": "def stop(self) -> float:\n \"\"\"\n Stop the timer\n\n Returns:\n The time the timer was stopped\n \"\"\"\n self.stop_time = time.time()\n return self.stop_time - self.start_time - self.offset"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ngets the current time in seconds", "response": "def get_time(self) -> float:\n \"\"\"\n Get the current time in seconds\n\n Returns:\n The current time in seconds\n \"\"\"\n if self.pause_time is not None:\n curr_time = self.pause_time - self.offset - self.start_time\n return curr_time\n\n curr_time = time.time()\n return curr_time - self.start_time - self.offset"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nsetting the current time in the calendar.", "response": "def set_time(self, value: float):\n \"\"\"\n Set the current time. This can be used to jump in the timeline.\n\n Args:\n value (float): The new time\n \"\"\"\n if value < 0:\n value = 0\n\n self.offset += self.get_time() - value"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nresolve the loader class based on file extension", "response": "def resolve_loader(self, meta: SceneDescription):\n \"\"\"\n Resolve scene loader based on file extension\n \"\"\"\n for loader_cls in self._loaders:\n if loader_cls.supports_file(meta):\n meta.loader_cls = loader_cls\n break\n else:\n raise ImproperlyConfigured(\n \"Scene {} has no loader class registered. Check settings.SCENE_LOADERS\".format(meta.path))"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef on_key_press(self, symbol, modifiers):\r\n self.keyboard_event(symbol, self.keys.ACTION_PRESS, modifiers)", "response": "Handles key press events."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nhandling key release events.", "response": "def on_key_release(self, symbol, modifiers):\r\n \"\"\"\r\n Pyglet specific key release callback.\r\n Forwards and translates the events to :py:func:`keyboard_event`\r\n \"\"\"\r\n self.keyboard_event(symbol, self.keys.ACTION_RELEASE, modifiers)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef on_mouse_motion(self, x, y, dx, dy):\r\n # screen coordinates relative to the lower-left corner\r\n self.cursor_event(x, self.buffer_height - y, dx, dy)", "response": "Handles mouse motion events."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef swap_buffers(self):\r\n if not self.window.context:\r\n return\r\n\r\n self.frames += 1\r\n self.window.flip()\r\n self.window.dispatch_events()", "response": "Swap buffers in the current window and dispatch events."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\ncreate a new instance of the Knockout sphere.", "response": "def sphere(radius=0.5, sectors=32, rings=16) -> VAO:\n \"\"\"\n Creates a sphere.\n\n Keyword Args:\n radius (float): Radius or the sphere\n rings (int): number or horizontal rings\n sectors (int): number of vertical segments\n\n Returns:\n A :py:class:`demosys.opengl.vao.VAO` instance\n \"\"\"\n R = 1.0 / (rings - 1)\n S = 1.0 / (sectors - 1)\n\n vertices = [0] * (rings * sectors * 3)\n normals = [0] * (rings * sectors * 3)\n uvs = [0] * (rings * sectors * 2)\n\n v, n, t = 0, 0, 0\n for r in range(rings):\n for s in range(sectors):\n y = math.sin(-math.pi / 2 + math.pi * r * R)\n x = math.cos(2 * math.pi * s * S) * math.sin(math.pi * r * R)\n z = math.sin(2 * math.pi * s * S) * math.sin(math.pi * r * R)\n\n uvs[t] = s * S\n uvs[t + 1] = r * R\n\n vertices[v] = x * radius\n vertices[v + 1] = y * radius\n vertices[v + 2] = z * radius\n\n normals[n] = x\n normals[n + 1] = y\n normals[n + 2] = z\n\n t += 2\n v += 3\n n += 3\n\n indices = [0] * rings * sectors * 6\n i = 0\n for r in range(rings - 1):\n for s in range(sectors - 1):\n indices[i] = r * sectors + s\n indices[i + 1] = (r + 1) * sectors + (s + 1)\n indices[i + 2] = r * sectors + (s + 1)\n\n indices[i + 3] = r * sectors + s\n indices[i + 4] = (r + 1) * sectors + s\n indices[i + 5] = (r + 1) * sectors + (s + 1)\n i += 6\n\n vbo_vertices = numpy.array(vertices, dtype=numpy.float32)\n vbo_normals = numpy.array(normals, dtype=numpy.float32)\n vbo_uvs = numpy.array(uvs, dtype=numpy.float32)\n vbo_elements = numpy.array(indices, dtype=numpy.uint32)\n\n vao = VAO(\"sphere\", mode=mlg.TRIANGLES)\n # VBOs\n vao.buffer(vbo_vertices, '3f', ['in_position'])\n vao.buffer(vbo_normals, '3f', ['in_normal'])\n vao.buffer(vbo_uvs, '2f', ['in_uv'])\n vao.index_buffer(vbo_elements, index_element_size=4)\n\n return vao"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\ncalling the superclass draw method and closes the session if necessary.", "response": "def draw(self, current_time, frame_time):\n \"\"\"\n Calls the superclass ``draw()`` methods and checks ``HEADLESS_FRAMES``/``HEADLESS_DURATION``\n \"\"\"\n super().draw(current_time, frame_time)\n\n if self.headless_duration and current_time >= self.headless_duration:\n self.close()"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nswaps the buffers of the current instance.", "response": "def swap_buffers(self):\n \"\"\"\n Headless window currently don't support double buffering.\n We only increment the frame counter here.\n \"\"\"\n self.frames += 1\n\n if self.headless_frames and self.frames >= self.headless_frames:\n self.close()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nload a resource or returns existing one.", "response": "def load(self, meta: ResourceDescription) -> Any:\n \"\"\"\n Loads a resource or return existing one\n\n :param meta: The resource description\n \"\"\"\n self._check_meta(meta)\n self.resolve_loader(meta)\n return meta.loader_cls(meta).load()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nadds a resource to this pool.", "response": "def add(self, meta):\n \"\"\"\n Add a resource to this pool.\n The resource is loaded and returned when ``load_pool()`` is called.\n\n :param meta: The resource description\n \"\"\"\n self._check_meta(meta)\n self.resolve_loader(meta)\n self._resources.append(meta)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef load_pool(self):\n for meta in self._resources:\n resource = self.load(meta)\n yield meta, resource\n\n self._resources = []", "response": "Load all the data files into memory and return a generator that yields each resource."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nresolving a loader class to a resource description.", "response": "def resolve_loader(self, meta: ResourceDescription):\n \"\"\"\n Attempts to assign a loader class to a resource description\n\n :param meta: The resource description instance\n \"\"\"\n meta.loader_cls = self.get_loader(meta, raise_on_error=True)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_loader(self, meta: ResourceDescription, raise_on_error=False) -> BaseLoader:\n for loader in self._loaders:\n if loader.name == meta.loader:\n return loader\n\n if raise_on_error:\n raise ImproperlyConfigured(\n \"Resource has invalid loader '{}': {}\\nAvailiable loaders: {}\".format(\n meta.loader, meta, [loader.name for loader in self._loaders]))", "response": "Attempts to get a loader from the resource description instance."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef keyPressEvent(self, event):\n self.keyboard_event(event.key(), self.keys.ACTION_PRESS, 0)", "response": "This callback function handles key press events."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef keyReleaseEvent(self, event):\n self.keyboard_event(event.key(), self.keys.ACTION_RELEASE, 0)", "response": "Handles a key release event."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ndraw a 2D texture using a fullscreen quad.", "response": "def draw(self, texture, pos=(0.0, 0.0), scale=(1.0, 1.0)):\r\n \"\"\"\r\n Draw texture using a fullscreen quad.\r\n By default this will conver the entire screen.\r\n\r\n :param pos: (tuple) offset x, y\r\n :param scale: (tuple) scale x, y\r\n \"\"\"\r\n if not self.initialized:\r\n self.init()\r\n\r\n self._texture2d_shader[\"offset\"].value = (pos[0] - 1.0, pos[1] - 1.0)\r\n self._texture2d_shader[\"scale\"].value = (scale[0], scale[1])\r\n texture.use(location=0)\r\n self._texture2d_sampler.use(location=0)\r\n self._texture2d_shader[\"texture0\"].value = 0\r\n self._quad.render(self._texture2d_shader)\r\n self._texture2d_sampler.clear(location=0)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndrawing the depth buffer linearized.", "response": "def draw_depth(self, texture, near, far, pos=(0.0, 0.0), scale=(1.0, 1.0)):\r\n \"\"\"\r\n Draw depth buffer linearized.\r\n By default this will draw the texture as a full screen quad.\r\n A sampler will be used to ensure the right conditions to draw the depth buffer.\r\n\r\n :param near: Near plane in projection\r\n :param far: Far plane in projection\r\n :param pos: (tuple) offset x, y\r\n :param scale: (tuple) scale x, y\r\n \"\"\"\r\n if not self.initialized:\r\n self.init()\r\n\r\n self._depth_shader[\"offset\"].value = (pos[0] - 1.0, pos[1] - 1.0)\r\n self._depth_shader[\"scale\"].value = (scale[0], scale[1])\r\n self._depth_shader[\"near\"].value = near\r\n self._depth_shader[\"far\"].value = far\r\n self._depth_sampler.use(location=0)\r\n texture.use(location=0)\r\n self._depth_shader[\"texture0\"].value = 0\r\n self._quad.render(self._depth_shader)\r\n self._depth_sampler.clear(location=0)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ninitializes geometry and shader for drawing FBO layers", "response": "def _init_texture2d_draw(self):\r\n \"\"\"Initialize geometry and shader for drawing FBO layers\"\"\"\r\n if not TextureHelper._quad:\r\n TextureHelper._quad = geometry.quad_fs()\r\n\r\n # Shader for drawing color layers\r\n TextureHelper._texture2d_shader = context.ctx().program(\r\n vertex_shader=\"\"\"\r\n #version 330\r\n\r\n in vec3 in_position;\r\n in vec2 in_uv;\r\n out vec2 uv;\r\n uniform vec2 offset;\r\n uniform vec2 scale;\r\n\r\n void main() {\r\n uv = in_uv;\r\n gl_Position = vec4((in_position.xy + vec2(1.0, 1.0)) * scale + offset, 0.0, 1.0);\r\n }\r\n \"\"\",\r\n fragment_shader=\"\"\"\r\n #version 330\r\n\r\n out vec4 out_color;\r\n in vec2 uv;\r\n uniform sampler2D texture0;\r\n\r\n void main() {\r\n out_color = texture(texture0, uv);\r\n }\r\n \"\"\"\r\n )\r\n\r\n TextureHelper._texture2d_sampler = self.ctx.sampler(\r\n filter=(moderngl.LINEAR, moderngl.LINEAR),\r\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _init_depth_texture_draw(self):\r\n from demosys import geometry\r\n\r\n if not TextureHelper._quad:\r\n TextureHelper._quad = geometry.quad_fs()\r\n\r\n # Shader for drawing depth layers\r\n TextureHelper._depth_shader = context.ctx().program(\r\n vertex_shader=\"\"\"\r\n #version 330\r\n\r\n in vec3 in_position;\r\n in vec2 in_uv;\r\n out vec2 uv;\r\n uniform vec2 offset;\r\n uniform vec2 scale;\r\n\r\n void main() {\r\n uv = in_uv;\r\n gl_Position = vec4((in_position.xy + vec2(1.0, 1.0)) * scale + offset, 0.0, 1.0);\r\n }\r\n \"\"\",\r\n fragment_shader=\"\"\"\r\n #version 330\r\n\r\n out vec4 out_color;\r\n in vec2 uv;\r\n uniform sampler2D texture0;\r\n uniform float near;\r\n uniform float far;\r\n\r\n void main() {\r\n float z = texture(texture0, uv).r;\r\n float d = (2.0 * near) / (far + near - z * (far - near));\r\n out_color = vec4(d);\r\n }\r\n \"\"\"\r\n )\r\n\r\n TextureHelper._depth_sampler = self.ctx.sampler(\r\n filter=(moderngl.LINEAR, moderngl.LINEAR),\r\n compare_func='',\r\n )", "response": "Initialize geometry and shader for drawing depth layers"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\ndrawing a frame. Internally it calls the configured timeline's draw method. Args: current_time (float): The current time (preferrably always from the configured timer class) frame_time (float): The duration of the previous frame in seconds", "response": "def draw(self, current_time, frame_time):\n \"\"\"\n Draws a frame. Internally it calls the\n configured timeline's draw method.\n\n Args:\n current_time (float): The current time (preferrably always from the configured timer class)\n frame_time (float): The duration of the previous frame in seconds\n \"\"\"\n self.set_default_viewport()\n self.timeline.draw(current_time, frame_time, self.fbo)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nclear the window buffer", "response": "def clear(self):\n \"\"\"\n Clear the window buffer\n \"\"\"\n self.ctx.fbo.clear(\n red=self.clear_color[0],\n green=self.clear_color[1],\n blue=self.clear_color[2],\n alpha=self.clear_color[3],\n depth=self.clear_depth,\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the clear color and depth values for the current locale.", "response": "def clear_values(self, red=0.0, green=0.0, blue=0.0, alpha=0.0, depth=1.0):\n \"\"\"\n Sets the clear values for the window buffer.\n\n Args:\n red (float): red compoent\n green (float): green compoent\n blue (float): blue compoent\n alpha (float): alpha compoent\n depth (float): depth value\n \"\"\"\n self.clear_color = (red, green, blue, alpha)\n self.clear_depth = depth"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nhandle the keyboard events such as camera movement and camera movement.", "response": "def keyboard_event(self, key, action, modifier):\n \"\"\"\n Handles the standard keyboard events such as camera movements,\n taking a screenshot, closing the window etc.\n\n Can be overriden add new keyboard events. Ensure this method\n is also called if you want to keep the standard features.\n\n Arguments:\n key: The key that was pressed or released\n action: The key action. Can be `ACTION_PRESS` or `ACTION_RELEASE`\n modifier: Modifiers such as holding shift or ctrl\n \"\"\"\n # The well-known standard key for quick exit\n if key == self.keys.ESCAPE:\n self.close()\n return\n\n # Toggle pause time\n if key == self.keys.SPACE and action == self.keys.ACTION_PRESS:\n self.timer.toggle_pause()\n\n # Camera movement\n # Right\n if key == self.keys.D:\n if action == self.keys.ACTION_PRESS:\n self.sys_camera.move_right(True)\n elif action == self.keys.ACTION_RELEASE:\n self.sys_camera.move_right(False)\n # Left\n elif key == self.keys.A:\n if action == self.keys.ACTION_PRESS:\n self.sys_camera.move_left(True)\n elif action == self.keys.ACTION_RELEASE:\n self.sys_camera.move_left(False)\n # Forward\n elif key == self.keys.W:\n if action == self.keys.ACTION_PRESS:\n self.sys_camera.move_forward(True)\n if action == self.keys.ACTION_RELEASE:\n self.sys_camera.move_forward(False)\n # Backwards\n elif key == self.keys.S:\n if action == self.keys.ACTION_PRESS:\n self.sys_camera.move_backward(True)\n if action == self.keys.ACTION_RELEASE:\n self.sys_camera.move_backward(False)\n\n # UP\n elif key == self.keys.Q:\n if action == self.keys.ACTION_PRESS:\n self.sys_camera.move_down(True)\n if action == self.keys.ACTION_RELEASE:\n self.sys_camera.move_down(False)\n\n # Down\n elif key == self.keys.E:\n if action == self.keys.ACTION_PRESS:\n self.sys_camera.move_up(True)\n if action == self.keys.ACTION_RELEASE:\n self.sys_camera.move_up(False)\n\n # Screenshots\n if key == self.keys.X and action == self.keys.ACTION_PRESS:\n screenshot.create()\n\n if key == self.keys.R and action == self.keys.ACTION_PRESS:\n project.instance.reload_programs()\n\n if key == self.keys.RIGHT and action == self.keys.ACTION_PRESS:\n self.timer.set_time(self.timer.get_time() + 10.0)\n\n if key == self.keys.LEFT and action == self.keys.ACTION_PRESS:\n self.timer.set_time(self.timer.get_time() - 10.0)\n\n # Forward the event to the timeline\n self.timeline.key_event(key, action, modifier)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef cursor_event(self, x, y, dx, dy):\n self.sys_camera.rot_state(x, y)", "response": "Handles the standard mouse movement event."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef set_default_viewport(self):\n # The expected height with the current viewport width\n expected_height = int(self.buffer_width / self.aspect_ratio)\n\n # How much positive or negative y padding\n blank_space = self.buffer_height - expected_height\n self.fbo.viewport = (0, blank_space // 2, self.buffer_width, expected_height)", "response": "Sets the default viewport for the current page."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\ncheck if the loader supports a file extension", "response": "def supports_file(cls, meta):\n \"\"\"Check if the loader has a supported file extension\"\"\"\n path = Path(meta.path)\n\n for ext in cls.file_extensions:\n if path.suffixes[:len(ext)] == ext:\n return True\n\n return False"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef get(self, name) -> Track:\n name = name.lower()\n track = self.track_map.get(name)\n if not track:\n track = Track(name)\n self.tacks.append(track)\n self.track_map[name] = track\n return track", "response": "Get or create a Track object."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nfinding all commands in the folder command_dir", "response": "def find_commands(command_dir: str) -> List[str]:\n \"\"\"\n Get all command names in the a folder\n\n :return: List of commands names\n \"\"\"\n if not command_dir:\n return []\n\n return [name for _, name, is_pkg in pkgutil.iter_modules([command_dir])\n if not is_pkg and not name.startswith('_')]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nexecuting a command from the command line.", "response": "def execute_from_command_line(argv=None):\n \"\"\"\n Currently the only entrypoint (manage.py, demosys-admin)\n \"\"\"\n if not argv:\n argv = sys.argv\n\n # prog_name = argv[0]\n system_commands = find_commands(system_command_dir())\n project_commands = find_commands(project_command_dir())\n\n project_package = project_package_name()\n\n command = argv[1] if len(argv) > 1 else None\n\n # Are we running a core command?\n if command in system_commands:\n cmd = load_command_class('demosys', command)\n cmd.run_from_argv(argv)\n elif command in project_commands:\n cmd = load_command_class(project_package, command)\n cmd.run_from_argv(argv)\n else:\n print(\"Available commands:\")\n for name in system_commands:\n print(\" - {}\".format(name))\n for name in project_commands:\n print(\" - {}\".format(name))"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nhack in program directory", "response": "def add_program_dir(self, directory):\n \"\"\"Hack in program directory\"\"\"\n dirs = list(self.PROGRAM_DIRS)\n dirs.append(directory)\n self.PROGRAM_DIRS = dirs"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef add_texture_dir(self, directory):\n dirs = list(self.TEXTURE_DIRS)\n dirs.append(directory)\n self.TEXTURE_DIRS = dirs", "response": "Hack in texture directory"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nhacks in a data directory", "response": "def add_data_dir(self, directory):\n \"\"\"Hack in a data directory\"\"\"\n dirs = list(self.DATA_DIRS)\n dirs.append(directory)\n self.DATA_DIRS = dirs"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nbuild the content tuple for the buffer", "response": "def content(self, attributes: List[str]):\n \"\"\"Build content tuple for the buffer\"\"\"\n formats = []\n attrs = []\n for attrib_format, attrib in zip(self.attrib_formats, self.attributes):\n\n if attrib not in attributes:\n formats.append(attrib_format.pad_str())\n continue\n\n formats.append(attrib_format.format)\n attrs.append(attrib)\n\n attributes.remove(attrib)\n\n if not attrs:\n return None\n\n return (\n self.buffer,\n \"{}{}\".format(\" \".join(formats), '/i' if self.per_instance else ''),\n *attrs\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef render(self, program: moderngl.Program, mode=None, vertices=-1, first=0, instances=1):\n vao = self.instance(program)\n\n if mode is None:\n mode = self.mode\n\n vao.render(mode, vertices=vertices, first=first, instances=instances)", "response": "Render the VAO.\n\n Args:\n program: The ``moderngl.Program``\n\n Keyword Args:\n mode: Override the draw mode (``TRIANGLES`` etc)\n vertices (int): The number of vertices to transform\n first (int): The index of the first vertex to start with\n instances (int): The number of instances"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nrendering indirect draw commands for the specified moderngl. Program.", "response": "def render_indirect(self, program: moderngl.Program, buffer, mode=None, count=-1, *, first=0):\n \"\"\"\n The render primitive (mode) must be the same as the input primitive of the GeometryShader.\n The draw commands are 5 integers: (count, instanceCount, firstIndex, baseVertex, baseInstance).\n\n Args:\n program: The ``moderngl.Program``\n buffer: The ``moderngl.Buffer`` containing indirect draw commands\n\n Keyword Args:\n mode (int): By default :py:data:`TRIANGLES` will be used.\n count (int): The number of draws.\n first (int): The index of the first indirect draw command.\n \"\"\"\n vao = self.instance(program)\n\n if mode is None:\n mode = self.mode\n\n vao.render_indirect(buffer, mode=mode, count=count, first=first)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ntransforms vertices. Stores the output in a single buffer.", "response": "def transform(self, program: moderngl.Program, buffer: moderngl.Buffer,\n mode=None, vertices=-1, first=0, instances=1):\n \"\"\"\n Transform vertices. Stores the output in a single buffer.\n\n Args:\n program: The ``moderngl.Program``\n buffer: The ``moderngl.buffer`` to store the output\n\n Keyword Args:\n mode: Draw mode (for example ``moderngl.POINTS``)\n vertices (int): The number of vertices to transform\n first (int): The index of the first vertex to start with\n instances (int): The number of instances\n \"\"\"\n vao = self.instance(program)\n\n if mode is None:\n mode = self.mode\n\n vao.transform(buffer, mode=mode, vertices=vertices, first=first, instances=instances)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef buffer(self, buffer, buffer_format: str, attribute_names, per_instance=False):\n if not isinstance(attribute_names, list):\n attribute_names = [attribute_names, ]\n\n if not type(buffer) in [moderngl.Buffer, numpy.ndarray, bytes]:\n raise VAOError(\n (\n \"buffer parameter must be a moderngl.Buffer, numpy.ndarray or bytes instance\"\n \"(not {})\".format(type(buffer))\n )\n )\n\n if isinstance(buffer, numpy.ndarray):\n buffer = self.ctx.buffer(buffer.tobytes())\n\n if isinstance(buffer, bytes):\n buffer = self.ctx.buffer(data=buffer)\n\n formats = buffer_format.split()\n if len(formats) != len(attribute_names):\n raise VAOError(\"Format '{}' does not describe attributes {}\".format(buffer_format, attribute_names))\n\n self.buffers.append(BufferInfo(buffer, buffer_format, attribute_names, per_instance=per_instance))\n self.vertex_count = self.buffers[-1].vertices\n\n return buffer", "response": "Register a buffer for the current VAO."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nsets the index buffer for this VAO.", "response": "def index_buffer(self, buffer, index_element_size=4):\n \"\"\"\n Set the index buffer for this VAO\n\n Args:\n buffer: ``moderngl.Buffer``, ``numpy.array`` or ``bytes``\n\n Keyword Args:\n index_element_size (int): Byte size of each element. 1, 2 or 4\n \"\"\"\n if not type(buffer) in [moderngl.Buffer, numpy.ndarray, bytes]:\n raise VAOError(\"buffer parameter must be a moderngl.Buffer, numpy.ndarray or bytes instance\")\n\n if isinstance(buffer, numpy.ndarray):\n buffer = self.ctx.buffer(buffer.tobytes())\n\n if isinstance(buffer, bytes):\n buffer = self.ctx.buffer(data=buffer)\n\n self._index_buffer = buffer\n self._index_element_size = index_element_size"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the instance of the given program.", "response": "def instance(self, program: moderngl.Program) -> moderngl.VertexArray:\n \"\"\"\n Obtain the ``moderngl.VertexArray`` instance for the program.\n The instance is only created once and cached internally.\n\n Returns: ``moderngl.VertexArray`` instance\n \"\"\"\n vao = self.vaos.get(program.glo)\n if vao:\n return vao\n\n program_attributes = [name for name, attr in program._members.items() if isinstance(attr, moderngl.Attribute)]\n\n # Make sure all attributes are covered\n for attrib_name in program_attributes:\n # Ignore built in attributes for now\n if attrib_name.startswith('gl_'):\n continue\n\n # Do we have a buffer mapping to this attribute?\n if not sum(buffer.has_attribute(attrib_name) for buffer in self.buffers):\n raise VAOError(\"VAO {} doesn't have attribute {} for program {}\".format(\n self.name, attrib_name, program.name))\n\n vao_content = []\n\n # Pick out the attributes we can actually map\n for buffer in self.buffers:\n content = buffer.content(program_attributes)\n if content:\n vao_content.append(content)\n\n # Any attribute left is not accounted for\n if program_attributes:\n for attrib_name in program_attributes:\n if attrib_name.startswith('gl_'):\n continue\n\n raise VAOError(\"Did not find a buffer mapping for {}\".format([n for n in program_attributes]))\n\n # Create the vao\n if self._index_buffer:\n vao = context.ctx().vertex_array(program, vao_content,\n self._index_buffer, self._index_element_size)\n else:\n vao = context.ctx().vertex_array(program, vao_content)\n\n self.vaos[program.glo] = vao\n return vao"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ndestroying the vao object", "response": "def release(self, buffer=True):\n \"\"\"\n Destroy the vao object\n\n Keyword Args:\n buffers (bool): also release buffers\n \"\"\"\n for key, vao in self.vaos:\n vao.release()\n\n if buffer:\n for buff in self.buffers:\n buff.buffer.release()\n\n if self._index_buffer:\n self._index_buffer.release()"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn a new cube VAO with the given width height and depth centered at center.", "response": "def cube(width, height, depth, center=(0.0, 0.0, 0.0), normals=True, uvs=True) -> VAO:\n \"\"\"\n Creates a cube VAO with normals and texture coordinates\n\n Args:\n width (float): Width of the cube\n height (float): Height of the cube\n depth (float): Depth of the cube\n\n Keyword Args:\n center: center of the cube as a 3-component tuple\n normals: (bool) Include normals\n uvs: (bool) include uv coordinates\n\n Returns:\n A :py:class:`demosys.opengl.vao.VAO` instance\n \"\"\"\n width, height, depth = width / 2.0, height / 2.0, depth / 2.0\n\n pos = numpy.array([\n center[0] + width, center[1] - height, center[2] + depth,\n center[0] + width, center[1] + height, center[2] + depth,\n center[0] - width, center[1] - height, center[2] + depth,\n center[0] + width, center[1] + height, center[2] + depth,\n center[0] - width, center[1] + height, center[2] + depth,\n center[0] - width, center[1] - height, center[2] + depth,\n center[0] + width, center[1] - height, center[2] - depth,\n center[0] + width, center[1] + height, center[2] - depth,\n center[0] + width, center[1] - height, center[2] + depth,\n center[0] + width, center[1] + height, center[2] - depth,\n center[0] + width, center[1] + height, center[2] + depth,\n center[0] + width, center[1] - height, center[2] + depth,\n center[0] + width, center[1] - height, center[2] - depth,\n center[0] + width, center[1] - height, center[2] + depth,\n center[0] - width, center[1] - height, center[2] + depth,\n center[0] + width, center[1] - height, center[2] - depth,\n center[0] - width, center[1] - height, center[2] + depth,\n center[0] - width, center[1] - height, center[2] - depth,\n center[0] - width, center[1] - height, center[2] + depth,\n center[0] - width, center[1] + height, center[2] + depth,\n center[0] - width, center[1] + height, center[2] - depth,\n center[0] - width, center[1] - height, center[2] + depth,\n center[0] - width, center[1] + height, center[2] - depth,\n center[0] - width, center[1] - height, center[2] - depth,\n center[0] + width, center[1] + height, center[2] - depth,\n center[0] + width, center[1] - height, center[2] - depth,\n center[0] - width, center[1] - height, center[2] - depth,\n center[0] + width, center[1] + height, center[2] - depth,\n center[0] - width, center[1] - height, center[2] - depth,\n center[0] - width, center[1] + height, center[2] - depth,\n center[0] + width, center[1] + height, center[2] - depth,\n center[0] - width, center[1] + height, center[2] - depth,\n center[0] + width, center[1] + height, center[2] + depth,\n center[0] - width, center[1] + height, center[2] - depth,\n center[0] - width, center[1] + height, center[2] + depth,\n center[0] + width, center[1] + height, center[2] + depth,\n ], dtype=numpy.float32)\n\n if normals:\n normal_data = numpy.array([\n -0, 0, 1,\n -0, 0, 1,\n -0, 0, 1,\n 0, 0, 1,\n 0, 0, 1,\n 0, 0, 1,\n 1, 0, 0,\n 1, 0, 0,\n 1, 0, 0,\n 1, 0, 0,\n 1, 0, 0,\n 1, 0, 0,\n 0, -1, 0,\n 0, -1, 0,\n 0, -1, 0,\n 0, -1, 0,\n 0, -1, 0,\n 0, -1, 0,\n -1, -0, 0,\n -1, -0, 0,\n -1, -0, 0,\n -1, -0, 0,\n -1, -0, 0,\n -1, -0, 0,\n 0, 0, -1,\n 0, 0, -1,\n 0, 0, -1,\n 0, 0, -1,\n 0, 0, -1,\n 0, 0, -1,\n 0, 1, 0,\n 0, 1, 0,\n 0, 1, 0,\n 0, 1, 0,\n 0, 1, 0,\n 0, 1, 0,\n ], dtype=numpy.float32)\n\n if uvs:\n uvs_data = numpy.array([\n 1, 0,\n 1, 1,\n 0, 0,\n 1, 1,\n 0, 1,\n 0, 0,\n 1, 0,\n 1, 1,\n 0, 0,\n 1, 1,\n 0, 1,\n 0, 0,\n 1, 1,\n 0, 1,\n 0, 0,\n 1, 1,\n 0, 0,\n 1, 0,\n 0, 1,\n 0, 0,\n 1, 0,\n 0, 1,\n 1, 0,\n 1, 1,\n 1, 0,\n 1, 1,\n 0, 1,\n 1, 0,\n 0, 1,\n 0, 0,\n 1, 1,\n 0, 1,\n 1, 0,\n 0, 1,\n 0, 0,\n 1, 0\n ], dtype=numpy.float32)\n\n vao = VAO(\"geometry:cube\")\n\n # Add buffers\n vao.buffer(pos, '3f', ['in_position'])\n if normals:\n vao.buffer(normal_data, '3f', ['in_normal'])\n if uvs:\n vao.buffer(uvs_data, '2f', ['in_uv'])\n\n return vao"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef draw(self, mesh, projection_matrix=None, view_matrix=None, camera_matrix=None, time=0):\n self.program[\"m_proj\"].write(projection_matrix)\n self.program[\"m_mv\"].write(view_matrix)\n mesh.vao.render(self.program)", "response": "Draw the current region of the current region."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef get_time(self) -> float:\n if self.paused:\n return self.pause_time\n\n return self.player.get_time() / 1000.0", "response": "Get the current time in seconds"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nparse the effect package string.", "response": "def parse_package_string(path):\n \"\"\"\n Parse the effect package string.\n Can contain the package python path or path to effect class in an effect package.\n\n Examples::\n\n # Path to effect pacakge\n examples.cubes\n\n # Path to effect class\n examples.cubes.Cubes\n\n Args:\n path: python path to effect package. May also include effect class name.\n\n Returns:\n tuple: (package_path, effect_class)\n \"\"\"\n parts = path.split('.')\n\n # Is the last entry in the path capitalized?\n if parts[-1][0].isupper():\n return \".\".join(parts[:-1]), parts[-1]\n\n return path, \"\""} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nget all directories that are available for the current application.", "response": "def get_dirs(self) -> List[str]:\n \"\"\"\n Get all effect directories for registered effects.\n \"\"\"\n for package in self.packages:\n yield os.path.join(package.path, 'resources')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef get_effect_resources(self) -> List[Any]:\n resources = []\n for package in self.packages:\n resources.extend(package.resources)\n\n return resources", "response": "Get all resources registed in effect packages."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd a single effect package to the effect package list.", "response": "def add_package(self, name):\n \"\"\"\n Registers a single package\n\n :param name: (str) The effect package to add\n \"\"\"\n name, cls_name = parse_package_string(name)\n\n if name in self.package_map:\n return\n\n package = EffectPackage(name)\n package.load()\n\n self.packages.append(package)\n self.package_map[package.name] = package\n\n # Load effect package dependencies\n self.polulate(package.effect_packages)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef get_package(self, name) -> 'EffectPackage':\n name, cls_name = parse_package_string(name)\n\n try:\n return self.package_map[name]\n except KeyError:\n raise EffectError(\"No package '{}' registered\".format(name))", "response": "Get a package by python path. Can also contain path to an effect."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds an effect class by class name or full python path to class", "response": "def find_effect_class(self, path) -> Type[Effect]:\n \"\"\"\n Find an effect class by class name or full python path to class\n\n Args:\n path (str): effect class name or full python path to effect class\n\n Returns:\n Effect class\n\n Raises:\n EffectError if no class is found\n \"\"\"\n package_name, class_name = parse_package_string(path)\n\n if package_name:\n package = self.get_package(package_name)\n return package.find_effect_class(class_name, raise_for_error=True)\n\n for package in self.packages:\n effect_cls = package.find_effect_class(class_name)\n if effect_cls:\n return effect_cls\n\n raise EffectError(\"No effect class '{}' found in any packages\".format(class_name))"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef runnable_effects(self) -> List[Type[Effect]]:\n return [cls for cls in self.effect_classes if cls.runnable]", "response": "Returns the runnable effects in the package"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nfinds the effect package", "response": "def load_package(self):\n \"\"\"FInd the effect package\"\"\"\n try:\n self.package = importlib.import_module(self.name)\n except ModuleNotFoundError:\n raise ModuleNotFoundError(\"Effect package '{}' not found.\".format(self.name))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\niterating the module attributes picking out effects", "response": "def load_effects_classes(self):\n \"\"\"Iterate the module attributes picking out effects\"\"\"\n self.effect_classes = []\n\n for _, cls in inspect.getmembers(self.effect_module):\n if inspect.isclass(cls):\n if cls == Effect:\n continue\n\n if issubclass(cls, Effect):\n self.effect_classes.append(cls)\n self.effect_class_map[cls.__name__] = cls\n cls._name = \"{}.{}\".format(self.effect_module_name, cls.__name__)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef load_resource_module(self):\n # Attempt to load the dependencies module\n try:\n name = '{}.{}'.format(self.name, 'dependencies')\n self.dependencies_module = importlib.import_module(name)\n except ModuleNotFoundError as err:\n raise EffectError(\n (\n \"Effect package '{}' has no 'dependencies' module or the module has errors. \"\n \"Forwarded error from importlib: {}\"\n ).format(self.name, err))\n\n # Fetch the resource descriptions\n try:\n self.resources = getattr(self.dependencies_module, 'resources')\n except AttributeError:\n raise EffectError(\"Effect dependencies module '{}' has no 'resources' attribute\".format(name))\n\n if not isinstance(self.resources, list):\n raise EffectError(\n \"Effect dependencies module '{}': 'resources' is of type {} instead of a list\".format(\n name, type(self.resources)))\n\n # Fetch the effect class list\n try:\n self.effect_packages = getattr(self.dependencies_module, 'effect_packages')\n except AttributeError:\n raise EffectError(\"Effect dependencies module '{}' has 'effect_packages' attribute\".format(name))\n\n if not isinstance(self.effect_packages, list):\n raise EffectError(\n \"Effect dependencies module '{}': 'effect_packages' is of type {} instead of a list\".format(\n name, type(self.effects)))", "response": "Load the resource module and the resource list"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncreates a screenshot of the current page.", "response": "def create(file_format='png', name=None):\n \"\"\"\n Create a screenshot\n :param file_format: formats supported by PIL (png, jpeg etc)\n \"\"\"\n dest = \"\"\n if settings.SCREENSHOT_PATH:\n if not os.path.exists(settings.SCREENSHOT_PATH):\n print(\"SCREENSHOT_PATH does not exist. creating: {}\".format(settings.SCREENSHOT_PATH))\n os.makedirs(settings.SCREENSHOT_PATH)\n dest = settings.SCREENSHOT_PATH\n else:\n print(\"SCREENSHOT_PATH not defined in settings. Using cwd as fallback.\")\n\n if not Config.target:\n Config.target = context.window().fbo\n\n image = Image.frombytes(\n \"RGB\",\n (Config.target.viewport[2], Config.target.viewport[3]),\n Config.target.read(viewport=Config.target.viewport, alignment=Config.alignment),\n )\n image = image.transpose(Image.FLIP_TOP_BOTTOM)\n\n if not name:\n name = \"{}.{}\".format(datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S-%f\"), file_format)\n\n dest = os.path.join(dest, name)\n print(\"Creating screenshot:\", dest)\n image.save(dest, format=file_format)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef draw(self, time, frametime, target):\r\n for effect in self.effects:\r\n value = effect.rocket_timeline_track.time_value(time)\r\n if value > 0.5:\r\n effect.draw(time, frametime, target)", "response": "Draw the current time frame and target."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nload a 2d texture", "response": "def load(self):\r\n \"\"\"Load a 2d texture\"\"\"\r\n self._open_image()\r\n\r\n components, data = image_data(self.image)\r\n\r\n texture = self.ctx.texture(\r\n self.image.size,\r\n components,\r\n data,\r\n )\r\n texture.extra = {'meta': self.meta}\r\n\r\n if self.meta.mipmap:\r\n texture.build_mipmaps()\r\n\r\n self._close_image()\r\n\r\n return texture"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef from_single(cls, meta: ProgramDescription, source: str):\r\n instance = cls(meta)\r\n instance.vertex_source = ShaderSource(\r\n VERTEX_SHADER,\r\n meta.path or meta.vertex_shader,\r\n source\r\n )\r\n\r\n if GEOMETRY_SHADER in source:\r\n instance.geometry_source = ShaderSource(\r\n GEOMETRY_SHADER,\r\n meta.path or meta.geometry_shader,\r\n source,\r\n )\r\n\r\n if FRAGMENT_SHADER in source:\r\n instance.fragment_source = ShaderSource(\r\n FRAGMENT_SHADER,\r\n meta.path or meta.fragment_shader,\r\n source,\r\n )\r\n\r\n if TESS_CONTROL_SHADER in source:\r\n instance.tess_control_source = ShaderSource(\r\n TESS_CONTROL_SHADER,\r\n meta.path or meta.tess_control_shader,\r\n source,\r\n )\r\n\r\n if TESS_EVALUATION_SHADER in source:\r\n instance.tess_evaluation_source = ShaderSource(\r\n TESS_EVALUATION_SHADER,\r\n meta.path or meta.tess_evaluation_shader,\r\n source,\r\n )\r\n\r\n return instance", "response": "Initialize a single glsl string containing all shaders"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ninitialize multiple shader strings", "response": "def from_separate(cls, meta: ProgramDescription, vertex_source, geometry_source=None, fragment_source=None,\r\n tess_control_source=None, tess_evaluation_source=None):\r\n \"\"\"Initialize multiple shader strings\"\"\"\r\n instance = cls(meta)\r\n instance.vertex_source = ShaderSource(\r\n VERTEX_SHADER,\r\n meta.path or meta.vertex_shader,\r\n vertex_source,\r\n )\r\n\r\n if geometry_source:\r\n instance.geometry_source = ShaderSource(\r\n GEOMETRY_SHADER,\r\n meta.path or meta.geometry_shader,\r\n geometry_source,\r\n )\r\n\r\n if fragment_source:\r\n instance.fragment_source = ShaderSource(\r\n FRAGMENT_SHADER,\r\n meta.path or meta.fragment_shader,\r\n fragment_source,\r\n )\r\n\r\n if tess_control_source:\r\n instance.tess_control_source = ShaderSource(\r\n TESS_CONTROL_SHADER,\r\n meta.path or meta.tess_control_shader,\r\n tess_control_source,\r\n )\r\n\r\n if tess_evaluation_source:\r\n instance.tess_evaluation_source = ShaderSource(\r\n TESS_EVALUATION_SHADER,\r\n meta.path or meta.tess_control_shader,\r\n tess_evaluation_source,\r\n )\r\n\r\n return instance"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\ncreates a new program instance with the same attributes as the current one.", "response": "def create(self):\r\n \"\"\"\r\n Creates a shader program.\r\n\r\n Returns:\r\n ModernGL Program instance\r\n \"\"\"\r\n # Get out varyings\r\n out_attribs = []\r\n\r\n # If no fragment shader is present we are doing transform feedback\r\n if not self.fragment_source:\r\n # Out attributes is present in geometry shader if present\r\n if self.geometry_source:\r\n out_attribs = self.geometry_source.find_out_attribs()\r\n # Otherwise they are specified in vertex shader\r\n else:\r\n out_attribs = self.vertex_source.find_out_attribs()\r\n\r\n program = self.ctx.program(\r\n vertex_shader=self.vertex_source.source,\r\n geometry_shader=self.geometry_source.source if self.geometry_source else None,\r\n fragment_shader=self.fragment_source.source if self.fragment_source else None,\r\n tess_control_shader=self.tess_control_source.source if self.tess_control_source else None,\r\n tess_evaluation_shader=self.tess_evaluation_source.source if self.tess_evaluation_source else None,\r\n varyings=out_attribs,\r\n )\r\n program.extra = {'meta': self.meta}\r\n return program"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef find_out_attribs(self):\r\n names = []\r\n for line in self.lines:\r\n if line.strip().startswith(\"out \"):\r\n names.append(line.split()[2].replace(';', ''))\r\n return names", "response": "Find all out attributes in the shader source."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nprints the shader lines", "response": "def print(self):\r\n \"\"\"Print the shader lines\"\"\"\r\n print(\"---[ START {} ]---\".format(self.name))\r\n\r\n for i, line in enumerate(self.lines):\r\n print(\"{}: {}\".format(str(i).zfill(3), line))\r\n\r\n print(\"---[ END {} ]---\".format(self.name))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\ncreating an effect instance adding it to the internal effects dictionary using the label as key.", "response": "def create_effect(self, label: str, name: str, *args, **kwargs) -> Effect:\r\n \"\"\"\r\n Create an effect instance adding it to the internal effects dictionary using the label as key.\r\n\r\n Args:\r\n label (str): The unique label for the effect instance\r\n name (str): Name or full python path to the effect class we want to instantiate\r\n args: Positional arguments to the effect initializer\r\n kwargs: Keyword arguments to the effect initializer\r\n\r\n Returns:\r\n The newly created Effect instance\r\n \"\"\"\r\n effect_cls = effects.find_effect_class(name)\r\n effect = effect_cls(*args, **kwargs)\r\n effect._label = label\r\n\r\n if label in self._effects:\r\n raise ValueError(\"An effect with label '{}' already exists\".format(label))\r\n\r\n self._effects[label] = effect\r\n\r\n return effect"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef load(self):\r\n self.create_effect_classes()\r\n\r\n self._add_resource_descriptions_to_pools(self.create_external_resources())\r\n self._add_resource_descriptions_to_pools(self.create_resources())\r\n\r\n for meta, resource in resources.textures.load_pool():\r\n self._textures[meta.label] = resource\r\n\r\n for meta, resource in resources.programs.load_pool():\r\n self._programs[meta.label] = resource\r\n\r\n for meta, resource in resources.scenes.load_pool():\r\n self._scenes[meta.label] = resource\r\n\r\n for meta, resource in resources.data.load_pool():\r\n self._data[meta.label] = resource\r\n\r\n self.create_effect_instances()\r\n self.post_load()", "response": "Loads all resources and effects from the pool."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _add_resource_descriptions_to_pools(self, meta_list):\r\n if not meta_list:\r\n return\r\n\r\n for meta in meta_list:\r\n getattr(resources, meta.resource_type).add(meta)", "response": "Adds the resource descriptions to the resource pools."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef reload_programs(self):\r\n print(\"Reloading programs:\")\r\n for name, program in self._programs.items():\r\n if getattr(program, 'program', None):\r\n print(\" - {}\".format(program.meta.label))\r\n program.program = resources.programs.load(program.meta)", "response": "Reload all shader programs with the reloadable flag set set\r\n "} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the latest instance of the effect with the given label", "response": "def get_effect(self, label: str) -> Effect:\r\n \"\"\"\r\n Get an effect instance by label\r\n\r\n Args:\r\n label (str): The label for the effect instance\r\n\r\n Returns:\r\n Effect class instance\r\n \"\"\"\r\n return self._get_resource(label, self._effects, \"effect\")"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns an effect class from the effect registry.", "response": "def get_effect_class(self, class_name, package_name=None) -> Type[Effect]:\r\n \"\"\"\r\n Get an effect class from the effect registry.\r\n\r\n Args:\r\n class_name (str): The exact class name of the effect\r\n\r\n Keyword Args:\r\n package_name (str): The python path to the effect package the effect name is located.\r\n This is optional and can be used to avoid issue with class name collisions.\r\n\r\n Returns:\r\n Effect class\r\n \"\"\"\r\n if package_name:\r\n return effects.find_effect_class(\"{}.{}\".format(package_name, class_name))\r\n\r\n return effects.find_effect_class(class_name)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_scene(self, label: str) -> Scene:\r\n return self._get_resource(label, self._scenes, \"scene\")", "response": "Returns a new instance of the Scene with the specified label."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef get_texture(self, label: str) -> Union[moderngl.Texture, moderngl.TextureArray,\r\n moderngl.Texture3D, moderngl.TextureCube]:\r\n \"\"\"\r\n Get a texture by label\r\n\r\n Args:\r\n label (str): The label for the texture to fetch\r\n\r\n Returns:\r\n Texture instance\r\n \"\"\"\r\n return self._get_resource(label, self._textures, \"texture\")", "response": "Returns the first texture in the list of available textures."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the data object with the specified label", "response": "def get_data(self, label: str) -> Any:\r\n \"\"\"\r\n Get a data resource by label\r\n\r\n Args:\r\n label (str): The labvel for the data resource to fetch\r\n\r\n Returns:\r\n The requeted data object\r\n \"\"\"\r\n return self._get_resource(label, self._data, \"data\")"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef _get_resource(self, label: str, source: dict, resource_type: str):\r\n try:\r\n return source[label]\r\n except KeyError:\r\n raise ValueError(\"Cannot find {0} with label '{1}'.\\nExisting {0} labels: {2}\".format(\r\n resource_type, label, list(source.keys())))", "response": "Generic method to fetch the latest resource from the dictionary"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef get_runnable_effects(self) -> List[Effect]:\r\n return [effect for name, effect in self._effects.items() if effect.runnable]", "response": "Returns all runnable effects in the project."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef image_data(image):\r\n # NOTE: We might want to check the actual image.mode\r\n # and convert to an acceptable format.\r\n # At the moment we load the data as is.\r\n data = image.tobytes()\r\n components = len(data) // (image.size[0] * image.size[1])\r\n return components, data", "response": "Get components and bytes for an image"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef run_from_argv(self, argv):\r\n parser = self.create_parser(argv[0], argv[1])\r\n options = parser.parse_args(argv[2:])\r\n cmd_options = vars(options)\r\n args = cmd_options.pop('args', ())\r\n self.handle(*args, **cmd_options)", "response": "Called by the system when executing the command from the command line."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef create_parser(self, prog_name, subcommand):\r\n parser = argparse.ArgumentParser(prog_name, subcommand)\r\n # Add generic arguments here\r\n self.add_arguments(parser)\r\n return parser", "response": "Create argument parser and deal with add_arguments."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nvalidating that the name is valid.", "response": "def validate_name(self, name):\r\n \"\"\"\r\n Can the name be used as a python module or package?\r\n Raises ``ValueError`` if the name is invalid.\r\n\r\n :param name: the name to check\r\n \"\"\"\r\n if not name:\r\n raise ValueError(\"Name cannot be empty\")\r\n\r\n # Can the name be used as an identifier in python (module or package name)\r\n if not name.isidentifier():\r\n raise ValueError(\"{} is not a valid identifier\".format(name))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ngenerate a bounding box for the current entry.", "response": "def bbox(width=1.0, height=1.0, depth=1.0):\n \"\"\"\n Generates a bounding box with (0.0, 0.0, 0.0) as the center.\n This is simply a box with ``LINE_STRIP`` as draw mode.\n\n Keyword Args:\n width (float): Width of the box\n height (float): Height of the box\n depth (float): Depth of the box\n\n Returns:\n A :py:class:`demosys.opengl.vao.VAO` instance\n \"\"\"\n width, height, depth = width / 2.0, height / 2.0, depth / 2.0\n pos = numpy.array([\n width, -height, depth,\n width, height, depth,\n -width, -height, depth,\n width, height, depth,\n -width, height, depth,\n -width, -height, depth,\n width, -height, -depth,\n width, height, -depth,\n width, -height, depth,\n width, height, -depth,\n width, height, depth,\n width, -height, depth,\n width, -height, -depth,\n width, -height, depth,\n -width, -height, depth,\n width, -height, -depth,\n -width, -height, depth,\n -width, -height, -depth,\n -width, -height, depth,\n -width, height, depth,\n -width, height, -depth,\n -width, -height, depth,\n -width, height, -depth,\n -width, -height, -depth,\n width, height, -depth,\n width, -height, -depth,\n -width, -height, -depth,\n width, height, -depth,\n -width, -height, -depth,\n -width, height, -depth,\n width, height, -depth,\n -width, height, -depth,\n width, height, depth,\n -width, height, -depth,\n -width, height, depth,\n width, height, depth,\n ], dtype=numpy.float32)\n\n vao = VAO(\"geometry:cube\", mode=moderngl.LINE_STRIP)\n vao.buffer(pos, '3f', [\"in_position\"])\n\n return vao"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nfinds the last occurance of the file in finders", "response": "def _find_last_of(self, path, finders):\r\n \"\"\"Find the last occurance of the file in finders\"\"\"\r\n found_path = None\r\n for finder in finders:\r\n result = finder.find(path)\r\n if result:\r\n found_path = result\r\n\r\n return found_path"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef initial_sanity_check(self):\r\n # Check for python module collision\r\n self.try_import(self.project_name)\r\n\r\n # Is the name a valid identifier?\r\n self.validate_name(self.project_name)\r\n\r\n # Make sure we don't mess with existing directories\r\n if os.path.exists(self.project_name):\r\n print(\"Directory {} already exist. Aborting.\".format(self.project_name))\r\n return False\r\n\r\n if os.path.exists('manage.py'):\r\n print(\"A manage.py file already exist in the current directory. Aborting.\")\r\n return False\r\n\r\n return True", "response": "Checks if we can create the project"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef create_entrypoint(self):\r\n with open(os.path.join(self.template_dir, 'manage.py'), 'r') as fd:\r\n data = fd.read().format(project_name=self.project_name)\r\n\r\n with open('manage.py', 'w') as fd:\r\n fd.write(data)\r\n\r\n os.chmod('manage.py', 0o777)", "response": "Create the manage. py file in the current directory"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef get_template_dir(self):\r\n directory = os.path.dirname(os.path.abspath(__file__))\r\n directory = os.path.dirname(os.path.dirname(directory))\r\n directory = os.path.join(directory, 'project_template')\r\n return directory", "response": "Returns the absolute path to template directory"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef davidson(lname, fname='.', omit_fname=False):\n return Davidson().encode(lname, fname, omit_fname)", "response": "Return a Davidson s Consonant Code."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns a string that encodes the given last name and optional first name.", "response": "def encode(self, lname, fname='.', omit_fname=False):\n \"\"\"Return Davidson's Consonant Code.\n\n Parameters\n ----------\n lname : str\n Last name (or word) to be encoded\n fname : str\n First name (optional), of which the first character is included in\n the code.\n omit_fname : bool\n Set to True to completely omit the first character of the first\n name\n\n Returns\n -------\n str\n Davidson's Consonant Code\n\n Example\n -------\n >>> pe = Davidson()\n >>> pe.encode('Gough')\n 'G .'\n >>> pe.encode('pneuma')\n 'PNM .'\n >>> pe.encode('knight')\n 'KNGT.'\n >>> pe.encode('trice')\n 'TRC .'\n >>> pe.encode('judge')\n 'JDG .'\n >>> pe.encode('Smith', 'James')\n 'SMT J'\n >>> pe.encode('Wasserman', 'Tabitha')\n 'WSRMT'\n\n \"\"\"\n lname = text_type(lname.upper())\n code = self._delete_consecutive_repeats(\n lname[:1] + lname[1:].translate(self._trans)\n )\n code = code[:4] + (4 - len(code)) * ' '\n\n if not omit_fname:\n code += fname[:1].upper()\n\n return code"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nencode a text using arithmetic coding with the provided probabilities.", "response": "def ac_encode(text, probs):\n \"\"\"Encode a text using arithmetic coding with the provided probabilities.\n\n This is a wrapper for :py:meth:`Arithmetic.encode`.\n\n Parameters\n ----------\n text : str\n A string to encode\n probs : dict\n A probability statistics dictionary generated by\n :py:meth:`Arithmetic.train`\n\n Returns\n -------\n tuple\n The arithmetically coded text\n\n Example\n -------\n >>> pr = ac_train('the quick brown fox jumped over the lazy dog')\n >>> ac_encode('align', pr)\n (16720586181, 34)\n\n \"\"\"\n coder = Arithmetic()\n coder.set_probs(probs)\n return coder.encode(text)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef ac_decode(longval, nbits, probs):\n coder = Arithmetic()\n coder.set_probs(probs)\n return coder.decode(longval, nbits)", "response": "Decode the number to a string using the given statistics."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef train(self, text):\n text = text_type(text)\n if '\\x00' in text:\n text = text.replace('\\x00', ' ')\n counts = Counter(text)\n counts['\\x00'] = 1\n tot_letters = sum(counts.values())\n\n tot = 0\n self._probs = {}\n prev = Fraction(0)\n for char, count in sorted(\n counts.items(), key=lambda x: (x[1], x[0]), reverse=True\n ):\n follow = Fraction(tot + count, tot_letters)\n self._probs[char] = (prev, follow)\n prev = follow\n tot = tot + count", "response": "r Generate a probability dict from the provided text."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encode(self, text):\n text = text_type(text)\n if '\\x00' in text:\n text = text.replace('\\x00', ' ')\n minval = Fraction(0)\n maxval = Fraction(1)\n\n for char in text + '\\x00':\n prob_range = self._probs[char]\n delta = maxval - minval\n maxval = minval + prob_range[1] * delta\n minval = minval + prob_range[0] * delta\n\n # I tried without the /2 just to check. Doesn't work.\n # Keep scaling up until the error range is >= 1. That\n # gives me the minimum number of bits needed to resolve\n # down to the end-of-data character.\n delta = (maxval - minval) / 2\n nbits = long(0)\n while delta < 1:\n nbits += 1\n delta *= 2\n # The below condition shouldn't ever be false\n if nbits == 0: # pragma: no cover\n return 0, 0\n # using -1 instead of /2\n avg = (maxval + minval) * 2 ** (nbits - 1)\n # Could return a rational instead ...\n # the division truncation is deliberate\n return avg.numerator // avg.denominator, nbits", "response": "Encode a string using arithmetic coding."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef decode(self, longval, nbits):\n val = Fraction(longval, long(1) << nbits)\n letters = []\n\n probs_items = [\n (char, minval, maxval)\n for (char, (minval, maxval)) in self._probs.items()\n ]\n\n char = '\\x00'\n while True:\n for (char, minval, maxval) in probs_items: # noqa: B007\n if minval <= val < maxval:\n break\n\n if char == '\\x00':\n break\n letters.append(char)\n delta = maxval - minval\n val = (val - minval) / delta\n return ''.join(letters)", "response": "Decode the number to a string using the given statistics."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef fuzzy_soundex(word, max_length=5, zero_pad=True):\n return FuzzySoundex().encode(word, max_length, zero_pad)", "response": "Returns the Fuzzy Soundex code for a word."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nencoding a string into a Fuzzy Soundex code.", "response": "def encode(self, word, max_length=5, zero_pad=True):\n \"\"\"Return the Fuzzy Soundex code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to 4)\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length\n string\n\n Returns\n -------\n str\n The Fuzzy Soundex value\n\n Examples\n --------\n >>> pe = FuzzySoundex()\n >>> pe.encode('Christopher')\n 'K6931'\n >>> pe.encode('Niall')\n 'N4000'\n >>> pe.encode('Smith')\n 'S5300'\n >>> pe.encode('Smith')\n 'S5300'\n\n \"\"\"\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n\n # Clamp max_length to [4, 64]\n if max_length != -1:\n max_length = min(max(4, max_length), 64)\n else:\n max_length = 64\n\n if not word:\n if zero_pad:\n return '0' * max_length\n return '0'\n\n if word[:2] in {'CS', 'CZ', 'TS', 'TZ'}:\n word = 'SS' + word[2:]\n elif word[:2] == 'GN':\n word = 'NN' + word[2:]\n elif word[:2] in {'HR', 'WR'}:\n word = 'RR' + word[2:]\n elif word[:2] == 'HW':\n word = 'WW' + word[2:]\n elif word[:2] in {'KN', 'NG'}:\n word = 'NN' + word[2:]\n\n if word[-2:] == 'CH':\n word = word[:-2] + 'KK'\n elif word[-2:] == 'NT':\n word = word[:-2] + 'TT'\n elif word[-2:] == 'RT':\n word = word[:-2] + 'RR'\n elif word[-3:] == 'RDT':\n word = word[:-3] + 'RR'\n\n word = word.replace('CA', 'KA')\n word = word.replace('CC', 'KK')\n word = word.replace('CK', 'KK')\n word = word.replace('CE', 'SE')\n word = word.replace('CHL', 'KL')\n word = word.replace('CL', 'KL')\n word = word.replace('CHR', 'KR')\n word = word.replace('CR', 'KR')\n word = word.replace('CI', 'SI')\n word = word.replace('CO', 'KO')\n word = word.replace('CU', 'KU')\n word = word.replace('CY', 'SY')\n word = word.replace('DG', 'GG')\n word = word.replace('GH', 'HH')\n word = word.replace('MAC', 'MK')\n word = word.replace('MC', 'MK')\n word = word.replace('NST', 'NSS')\n word = word.replace('PF', 'FF')\n word = word.replace('PH', 'FF')\n word = word.replace('SCH', 'SSS')\n word = word.replace('TIO', 'SIO')\n word = word.replace('TIA', 'SIO')\n word = word.replace('TCH', 'CHH')\n\n sdx = word.translate(self._trans)\n sdx = sdx.replace('-', '')\n\n # remove repeating characters\n sdx = self._delete_consecutive_repeats(sdx)\n\n if word[0] in {'H', 'W', 'Y'}:\n sdx = word[0] + sdx\n else:\n sdx = word[0] + sdx[1:]\n\n sdx = sdx.replace('0', '')\n\n if zero_pad:\n sdx += '0' * max_length\n\n return sdx[:max_length]"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef corpus_importer(self, corpus, n_val=1, bos='_START_', eos='_END_'):\n if not corpus or not isinstance(corpus, Corpus):\n raise TypeError('Corpus argument of the Corpus class required.')\n\n sentences = corpus.sents()\n\n for sent in sentences:\n ngs = Counter(sent)\n for key in ngs.keys():\n self._add_to_ngcorpus(self.ngcorpus, [key], ngs[key])\n\n if n_val > 1:\n if bos and bos != '':\n sent = [bos] + sent\n if eos and eos != '':\n sent += [eos]\n for i in range(2, n_val + 1):\n for j in range(len(sent) - i + 1):\n self._add_to_ngcorpus(\n self.ngcorpus, sent[j : j + i], 1\n )", "response": "This function creates a new NGramCorpus instance from a Corpus object."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nadd a new entry to a corpus.", "response": "def _add_to_ngcorpus(self, corpus, words, count):\n \"\"\"Build up a corpus entry recursively.\n\n Parameters\n ----------\n corpus : Corpus\n The corpus\n words : [str]\n Words to add to the corpus\n count : int\n Count of words\n\n \"\"\"\n if words[0] not in corpus:\n corpus[words[0]] = Counter()\n\n if len(words) == 1:\n corpus[words[0]][None] += count\n else:\n self._add_to_ngcorpus(corpus[words[0]], words[1:], count)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef gng_importer(self, corpus_file):\n with c_open(corpus_file, 'r', encoding='utf-8') as gng:\n for line in gng:\n line = line.rstrip().split('\\t')\n words = line[0].split()\n\n self._add_to_ngcorpus(self.ngcorpus, words, int(line[2]))", "response": "Fill in self. ngcorpus from a Google NGram corpus file."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef tf(self, term):\n if ' ' in term:\n raise ValueError(\n 'tf can only calculate the term frequency of individual words'\n )\n tcount = self.get_count(term)\n if tcount == 0:\n return 0.0\n return 1 + log10(tcount)", "response": "r Return the term frequency for a given term."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef encode(self, word):\n\n def _raise_word_ex():\n \"\"\"Raise an AttributeError.\n\n Raises\n ------\n AttributeError\n Word attribute must be a string with a space or period dividing\n the first and last names or a tuple/list consisting of the\n first and last names\n\n \"\"\"\n raise AttributeError(\n 'Word attribute must be a string with a space or period '\n + 'dividing the first and last names or a tuple/list '\n + 'consisting of the first and last names'\n )\n\n if not word:\n return ''\n\n names = []\n if isinstance(word, (str, text_type)):\n names = word.split('.', 1)\n if len(names) != 2:\n names = word.split(' ', 1)\n if len(names) != 2:\n _raise_word_ex()\n elif hasattr(word, '__iter__'):\n if len(word) != 2:\n _raise_word_ex()\n names = word\n else:\n _raise_word_ex()\n\n names = [\n unicode_normalize(\n 'NFKD', text_type(_.strip().replace('\u00df', 'SS').upper())\n )\n for _ in names\n ]\n code = ''\n\n def _steps_one_to_three(name):\n \"\"\"Perform the first three steps of SPFC.\n\n Parameters\n ----------\n name : str\n Name to transform\n\n Returns\n -------\n str\n Transformed name\n\n \"\"\"\n # filter out non A-Z\n name = ''.join(_ for _ in name if _ in self._uc_set)\n\n # 1. In the field, convert DK to K, DT to T, SC to S, KN to N,\n # and MN to N\n for subst in self._substitutions:\n name = name.replace(subst[0], subst[1])\n\n # 2. In the name field, replace multiple letters with a single\n # letter\n name = self._delete_consecutive_repeats(name)\n\n # 3. Remove vowels, W, H, and Y, but keep the first letter in the\n # name field.\n if name:\n name = name[0] + ''.join(\n _\n for _ in name[1:]\n if _ not in {'A', 'E', 'H', 'I', 'O', 'U', 'W', 'Y'}\n )\n return name\n\n names = [_steps_one_to_three(_) for _ in names]\n\n # 4. The first digit of the code is obtained using PF1 and the first\n # letter of the name field. Remove this letter after coding.\n if names[1]:\n code += names[1][0].translate(self._pf1)\n names[1] = names[1][1:]\n\n # 5. Using the last letters of the name, use Table PF3 to obtain the\n # second digit of the code. Use as many letters as possible and remove\n # after coding.\n if names[1]:\n if names[1][-3:] == 'STN' or names[1][-3:] == 'PRS':\n code += '8'\n names[1] = names[1][:-3]\n elif names[1][-2:] == 'SN':\n code += '8'\n names[1] = names[1][:-2]\n elif names[1][-3:] == 'STR':\n code += '9'\n names[1] = names[1][:-3]\n elif names[1][-2:] in {'SR', 'TN', 'TD'}:\n code += '9'\n names[1] = names[1][:-2]\n elif names[1][-3:] == 'DRS':\n code += '7'\n names[1] = names[1][:-3]\n elif names[1][-2:] in {'TR', 'MN'}:\n code += '7'\n names[1] = names[1][:-2]\n else:\n code += names[1][-1].translate(self._pf3)\n names[1] = names[1][:-1]\n\n # 6. The third digit is found using Table PF2 and the first character\n # of the first name. Remove after coding.\n if names[0]:\n code += names[0][0].translate(self._pf2)\n names[0] = names[0][1:]\n\n # 7. The fourth digit is found using Table PF2 and the first character\n # of the name field. If no letters remain use zero. After coding remove\n # the letter.\n # 8. The fifth digit is found in the same manner as the fourth using\n # the remaining characters of the name field if any.\n for _ in range(2):\n if names[1]:\n code += names[1][0].translate(self._pf2)\n names[1] = names[1][1:]\n else:\n code += '0'\n\n return code", "response": "Encodes a string into a standardized Phonetic Frequency Code."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef encode(self, word, terminator='\\0'):\n if word:\n if terminator in word:\n raise ValueError(\n 'Specified terminator, {}, already in word.'.format(\n terminator if terminator != '\\0' else '\\\\0'\n )\n )\n else:\n word += terminator\n wordlist = sorted(\n word[i:] + word[:i] for i in range(len(word))\n )\n return ''.join([w[-1] for w in wordlist])\n else:\n return terminator", "response": "r Return the Burrows - Wheeler transformed form of a word."} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the indel distance between two strings.", "response": "def dist_abs(self, src, tar):\n \"\"\"Return the indel distance between two strings.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n int\n Indel distance\n\n Examples\n --------\n >>> cmp = Indel()\n >>> cmp.dist_abs('cat', 'hat')\n 2\n >>> cmp.dist_abs('Niall', 'Neil')\n 3\n >>> cmp.dist_abs('Colin', 'Cuilen')\n 5\n >>> cmp.dist_abs('ATCG', 'TAGC')\n 4\n\n \"\"\"\n return self._lev.dist_abs(\n src, tar, mode='lev', cost=(1, 1, 9999, 9999)\n )"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the normalized Levenshtein distance between two strings.", "response": "def dist(self, src, tar):\n \"\"\"Return the normalized indel distance between two strings.\n\n This is equivalent to normalized Levenshtein distance, when only\n inserts and deletes are possible.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n float\n Normalized indel distance\n\n Examples\n --------\n >>> cmp = Indel()\n >>> round(cmp.dist('cat', 'hat'), 12)\n 0.333333333333\n >>> round(cmp.dist('Niall', 'Neil'), 12)\n 0.333333333333\n >>> round(cmp.dist('Colin', 'Cuilen'), 12)\n 0.454545454545\n >>> cmp.dist('ATCG', 'TAGC')\n 0.5\n\n \"\"\"\n if src == tar:\n return 0.0\n return self.dist_abs(src, tar) / (len(src) + len(tar))"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nencoding a string into a Haase Phonetik value.", "response": "def encode(self, word, primary_only=False):\n \"\"\"Return the Haase Phonetik (numeric output) code for a word.\n\n While the output code is numeric, it is nevertheless a str.\n\n Parameters\n ----------\n word : str\n The word to transform\n primary_only : bool\n If True, only the primary code is returned\n\n Returns\n -------\n tuple\n The Haase Phonetik value as a numeric string\n\n Examples\n --------\n >>> pe = Haase()\n >>> pe.encode('Joachim')\n ('9496',)\n >>> pe.encode('Christoph')\n ('4798293', '8798293')\n >>> pe.encode('J\u00f6rg')\n ('974',)\n >>> pe.encode('Smith')\n ('8692',)\n >>> pe.encode('Schmidt')\n ('8692', '4692')\n\n \"\"\"\n\n def _after(word, pos, letters):\n \"\"\"Return True if word[pos] follows one of the supplied letters.\n\n Parameters\n ----------\n word : str\n Word to modify\n pos : int\n Position to examine\n letters : set\n Letters to check for\n\n Returns\n -------\n bool\n True if word[pos] follows one of letters\n\n \"\"\"\n if pos > 0 and word[pos - 1] in letters:\n return True\n return False\n\n def _before(word, pos, letters):\n \"\"\"Return True if word[pos] precedes one of the supplied letters.\n\n Parameters\n ----------\n word : str\n Word to modify\n pos : int\n Position to examine\n letters : set\n Letters to check for\n\n Returns\n -------\n bool\n True if word[pos] precedes one of letters\n\n \"\"\"\n if pos + 1 < len(word) and word[pos + 1] in letters:\n return True\n return False\n\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n\n word = word.replace('\u00c4', 'AE')\n word = word.replace('\u00d6', 'OE')\n word = word.replace('\u00dc', 'UE')\n word = ''.join(c for c in word if c in self._uc_set)\n\n variants = []\n if primary_only:\n variants = [word]\n else:\n pos = 0\n if word[:2] == 'CH':\n variants.append(('CH', 'SCH'))\n pos += 2\n len_3_vars = {\n 'OWN': 'AUN',\n 'WSK': 'RSK',\n 'SCH': 'CH',\n 'GLI': 'LI',\n 'AUX': 'O',\n 'EUX': 'O',\n }\n while pos < len(word):\n if word[pos : pos + 4] == 'ILLE':\n variants.append(('ILLE', 'I'))\n pos += 4\n elif word[pos : pos + 3] in len_3_vars:\n variants.append(\n (word[pos : pos + 3], len_3_vars[word[pos : pos + 3]])\n )\n pos += 3\n elif word[pos : pos + 2] == 'RB':\n variants.append(('RB', 'RW'))\n pos += 2\n elif len(word[pos:]) == 3 and word[pos:] == 'EAU':\n variants.append(('EAU', 'O'))\n pos += 3\n elif len(word[pos:]) == 1 and word[pos:] in {'A', 'O'}:\n if word[pos:] == 'O':\n variants.append(('O', 'OW'))\n else:\n variants.append(('A', 'AR'))\n pos += 1\n else:\n variants.append((word[pos],))\n pos += 1\n\n variants = [''.join(letters) for letters in product(*variants)]\n\n def _haase_code(word):\n sdx = ''\n for i in range(len(word)):\n if word[i] in self._uc_v_set:\n sdx += '9'\n elif word[i] == 'B':\n sdx += '1'\n elif word[i] == 'P':\n if _before(word, i, {'H'}):\n sdx += '3'\n else:\n sdx += '1'\n elif word[i] in {'D', 'T'}:\n if _before(word, i, {'C', 'S', 'Z'}):\n sdx += '8'\n else:\n sdx += '2'\n elif word[i] in {'F', 'V', 'W'}:\n sdx += '3'\n elif word[i] in {'G', 'K', 'Q'}:\n sdx += '4'\n elif word[i] == 'C':\n if _after(word, i, {'S', 'Z'}):\n sdx += '8'\n elif i == 0:\n if _before(\n word,\n i,\n {'A', 'H', 'K', 'L', 'O', 'Q', 'R', 'U', 'X'},\n ):\n sdx += '4'\n else:\n sdx += '8'\n elif _before(word, i, {'A', 'H', 'K', 'O', 'Q', 'U', 'X'}):\n sdx += '4'\n else:\n sdx += '8'\n elif word[i] == 'X':\n if _after(word, i, {'C', 'K', 'Q'}):\n sdx += '8'\n else:\n sdx += '48'\n elif word[i] == 'L':\n sdx += '5'\n elif word[i] in {'M', 'N'}:\n sdx += '6'\n elif word[i] == 'R':\n sdx += '7'\n elif word[i] in {'S', 'Z'}:\n sdx += '8'\n\n sdx = self._delete_consecutive_repeats(sdx)\n\n return sdx\n\n encoded = tuple(_haase_code(word) for word in variants)\n if len(encoded) > 1:\n encoded_set = set()\n encoded_single = []\n for code in encoded:\n if code not in encoded_set:\n encoded_set.add(code)\n encoded_single.append(code)\n return tuple(encoded_single)\n\n return encoded"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sim(self, src, tar, *args, **kwargs):\n return 1.0 - self.dist(src, tar, *args, **kwargs)", "response": "Return similarity between two strings src and tar."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dist_abs(self, src, tar, *args, **kwargs):\n return self.dist(src, tar, *args, **kwargs)", "response": "Return absolute distance.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n *args\n Variable length argument list.\n **kwargs\n Arbitrary keyword arguments.\n\n Returns\n -------\n int\n Absolute distance"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the occurrence fingerprint.", "response": "def occurrence_fingerprint(\n word, n_bits=16, most_common=MOST_COMMON_LETTERS_CG\n):\n \"\"\"Return the occurrence fingerprint.\n\n This is a wrapper for :py:meth:`Occurrence.fingerprint`.\n\n Parameters\n ----------\n word : str\n The word to fingerprint\n n_bits : int\n Number of bits in the fingerprint returned\n most_common : list\n The most common tokens in the target language, ordered by frequency\n\n Returns\n -------\n int\n The occurrence fingerprint\n\n Examples\n --------\n >>> bin(occurrence_fingerprint('hat'))\n '0b110000100000000'\n >>> bin(occurrence_fingerprint('niall'))\n '0b10110000100000'\n >>> bin(occurrence_fingerprint('colin'))\n '0b1110000110000'\n >>> bin(occurrence_fingerprint('atcg'))\n '0b110000000010000'\n >>> bin(occurrence_fingerprint('entreatment'))\n '0b1110010010000100'\n\n \"\"\"\n return Occurrence().fingerprint(word, n_bits, most_common)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the occurrence fingerprint for the given word.", "response": "def fingerprint(self, word, n_bits=16, most_common=MOST_COMMON_LETTERS_CG):\n \"\"\"Return the occurrence fingerprint.\n\n Parameters\n ----------\n word : str\n The word to fingerprint\n n_bits : int\n Number of bits in the fingerprint returned\n most_common : list\n The most common tokens in the target language, ordered by frequency\n\n Returns\n -------\n int\n The occurrence fingerprint\n\n Examples\n --------\n >>> of = Occurrence()\n >>> bin(of.fingerprint('hat'))\n '0b110000100000000'\n >>> bin(of.fingerprint('niall'))\n '0b10110000100000'\n >>> bin(of.fingerprint('colin'))\n '0b1110000110000'\n >>> bin(of.fingerprint('atcg'))\n '0b110000000010000'\n >>> bin(of.fingerprint('entreatment'))\n '0b1110010010000100'\n\n \"\"\"\n word = set(word)\n fingerprint = 0\n\n for letter in most_common:\n if letter in word:\n fingerprint += 1\n n_bits -= 1\n if n_bits:\n fingerprint <<= 1\n else:\n break\n\n n_bits -= 1\n if n_bits > 0:\n fingerprint <<= n_bits\n\n return fingerprint"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sim_baystat(src, tar, min_ss_len=None, left_ext=None, right_ext=None):\n return Baystat().sim(src, tar, min_ss_len, left_ext, right_ext)", "response": "Return the Baystat similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the Baystat distance.", "response": "def dist_baystat(src, tar, min_ss_len=None, left_ext=None, right_ext=None):\n \"\"\"Return the Baystat distance.\n\n This is a wrapper for :py:meth:`Baystat.dist`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n min_ss_len : int\n Minimum substring length to be considered\n left_ext : int\n Left-side extension length\n right_ext : int\n Right-side extension length\n\n Returns\n -------\n float\n The Baystat distance\n\n Examples\n --------\n >>> round(dist_baystat('cat', 'hat'), 12)\n 0.333333333333\n >>> dist_baystat('Niall', 'Neil')\n 0.6\n >>> round(dist_baystat('Colin', 'Cuilen'), 12)\n 0.833333333333\n >>> dist_baystat('ATCG', 'TAGC')\n 1.0\n\n \"\"\"\n return Baystat().dist(src, tar, min_ss_len, left_ext, right_ext)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sim(self, src, tar, min_ss_len=None, left_ext=None, right_ext=None):\n if src == tar:\n return 1.0\n if not src or not tar:\n return 0.0\n\n max_len = max(len(src), len(tar))\n\n if not (min_ss_len and left_ext and right_ext):\n # These can be set via arguments to the function. Otherwise they\n # are set automatically based on values from the article.\n if max_len >= 7:\n min_ss_len = 2\n left_ext = 2\n right_ext = 2\n else:\n # The paper suggests that for short names, (exclusively) one or\n # the other of left_ext and right_ext can be 1, with good\n # results. I use 0 & 0 as the default in this case.\n min_ss_len = 1\n left_ext = 0\n right_ext = 0\n\n pos = 0\n match_len = 0\n\n while True:\n if pos + min_ss_len > len(src):\n return match_len / max_len\n\n hit_len = 0\n ix = 1\n\n substring = src[pos : pos + min_ss_len]\n search_begin = pos - left_ext\n\n if search_begin < 0:\n search_begin = 0\n left_ext_len = pos\n else:\n left_ext_len = left_ext\n\n if pos + min_ss_len + right_ext >= len(tar):\n right_ext_len = len(tar) - pos - min_ss_len\n else:\n right_ext_len = right_ext\n\n if (\n search_begin + left_ext_len + min_ss_len + right_ext_len\n > search_begin\n ):\n search_val = tar[\n search_begin : (\n search_begin\n + left_ext_len\n + min_ss_len\n + right_ext_len\n )\n ]\n else:\n search_val = ''\n\n flagged_tar = ''\n while substring in search_val and pos + ix <= len(src):\n hit_len = len(substring)\n flagged_tar = tar.replace(substring, '#' * hit_len)\n\n if pos + min_ss_len + ix <= len(src):\n substring = src[pos : pos + min_ss_len + ix]\n\n if pos + min_ss_len + right_ext_len + 1 <= len(tar):\n right_ext_len += 1\n\n # The following is unnecessary, I think\n # if (search_begin + left_ext_len + min_ss_len + right_ext_len\n # <= len(tar)):\n search_val = tar[\n search_begin : (\n search_begin\n + left_ext_len\n + min_ss_len\n + right_ext_len\n )\n ]\n\n ix += 1\n\n if hit_len > 0:\n tar = flagged_tar\n\n match_len += hit_len\n pos += ix", "response": "Return the Baystat similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the Tversky index of two strings.", "response": "def sim_tversky(src, tar, qval=2, alpha=1, beta=1, bias=None):\n \"\"\"Return the Tversky index of two strings.\n\n This is a wrapper for :py:meth:`Tversky.sim`.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n alpha : float\n Tversky index parameter as described above\n beta : float\n Tversky index parameter as described above\n bias : float\n The symmetric Tversky index bias parameter\n\n Returns\n -------\n float\n Tversky similarity\n\n Examples\n --------\n >>> sim_tversky('cat', 'hat')\n 0.3333333333333333\n >>> sim_tversky('Niall', 'Neil')\n 0.2222222222222222\n >>> sim_tversky('aluminum', 'Catalan')\n 0.0625\n >>> sim_tversky('ATCG', 'TAGC')\n 0.0\n\n \"\"\"\n return Tversky().sim(src, tar, qval, alpha, beta, bias)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dist_tversky(src, tar, qval=2, alpha=1, beta=1, bias=None):\n return Tversky().dist(src, tar, qval, alpha, beta, bias)", "response": "Return the Tversky distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sim(self, src, tar, qval=2, alpha=1, beta=1, bias=None):\n if alpha < 0 or beta < 0:\n raise ValueError(\n 'Unsupported weight assignment; alpha and beta '\n + 'must be greater than or equal to 0.'\n )\n\n if src == tar:\n return 1.0\n elif not src or not tar:\n return 0.0\n\n q_src, q_tar = self._get_qgrams(src, tar, qval)\n q_src_mag = sum(q_src.values())\n q_tar_mag = sum(q_tar.values())\n q_intersection_mag = sum((q_src & q_tar).values())\n\n if not q_src or not q_tar:\n return 0.0\n\n if bias is None:\n return q_intersection_mag / (\n q_intersection_mag\n + alpha * (q_src_mag - q_intersection_mag)\n + beta * (q_tar_mag - q_intersection_mag)\n )\n\n a_val = min(\n q_src_mag - q_intersection_mag, q_tar_mag - q_intersection_mag\n )\n b_val = max(\n q_src_mag - q_intersection_mag, q_tar_mag - q_intersection_mag\n )\n c_val = q_intersection_mag + bias\n return c_val / (beta * (alpha * a_val + (1 - alpha) * b_val) + c_val)", "response": "Return the Tversky index of two strings."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the longest common subsequence of two strings.", "response": "def lcsseq(self, src, tar):\n \"\"\"Return the longest common subsequence of two strings.\n\n Based on the dynamic programming algorithm from\n http://rosettacode.org/wiki/Longest_common_subsequence\n :cite:`rosettacode:2018b`. This is licensed GFDL 1.2.\n\n Modifications include:\n conversion to a numpy array in place of a list of lists\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n str\n The longest common subsequence\n\n Examples\n --------\n >>> sseq = LCSseq()\n >>> sseq.lcsseq('cat', 'hat')\n 'at'\n >>> sseq.lcsseq('Niall', 'Neil')\n 'Nil'\n >>> sseq.lcsseq('aluminum', 'Catalan')\n 'aln'\n >>> sseq.lcsseq('ATCG', 'TAGC')\n 'AC'\n\n \"\"\"\n lengths = np_zeros((len(src) + 1, len(tar) + 1), dtype=np_int)\n\n # row 0 and column 0 are initialized to 0 already\n for i, src_char in enumerate(src):\n for j, tar_char in enumerate(tar):\n if src_char == tar_char:\n lengths[i + 1, j + 1] = lengths[i, j] + 1\n else:\n lengths[i + 1, j + 1] = max(\n lengths[i + 1, j], lengths[i, j + 1]\n )\n\n # read the substring out from the matrix\n result = ''\n i, j = len(src), len(tar)\n while i != 0 and j != 0:\n if lengths[i, j] == lengths[i - 1, j]:\n i -= 1\n elif lengths[i, j] == lengths[i, j - 1]:\n j -= 1\n else:\n result = src[i - 1] + result\n i -= 1\n j -= 1\n return result"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sim(self, src, tar):\n if src == tar:\n return 1.0\n elif not src or not tar:\n return 0.0\n return len(self.lcsseq(src, tar)) / max(len(src), len(tar))", "response": "Return the longest common subsequence similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef sim(self, src, tar):\n if src == tar:\n return 1.0\n if not src or not tar:\n return 0.0\n min_word, max_word = (src, tar) if len(src) < len(tar) else (tar, src)\n min_len = len(min_word)\n for i in range(min_len, 0, -1):\n if min_word[:i] == max_word[:i]:\n return i / min_len\n return 0.0", "response": "Return the similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef count_fingerprint(word, n_bits=16, most_common=MOST_COMMON_LETTERS_CG):\n return Count().fingerprint(word, n_bits, most_common)", "response": "Return the count fingerprint."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef fingerprint(self, word, n_bits=16, most_common=MOST_COMMON_LETTERS_CG):\n if n_bits % 2:\n n_bits += 1\n\n word = Counter(word)\n fingerprint = 0\n\n for letter in most_common:\n if n_bits:\n fingerprint <<= 2\n fingerprint += word[letter] & 3\n n_bits -= 2\n else:\n break\n\n if n_bits:\n fingerprint <<= n_bits\n\n return fingerprint", "response": "Return the count fingerprint of the given word."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the phonetic fingerprint of a phrase.", "response": "def phonetic_fingerprint(\n phrase, phonetic_algorithm=double_metaphone, joiner=' ', *args, **kwargs\n):\n \"\"\"Return the phonetic fingerprint of a phrase.\n\n This is a wrapper for :py:meth:`Phonetic.fingerprint`.\n\n Parameters\n ----------\n phrase : str\n The string from which to calculate the phonetic fingerprint\n phonetic_algorithm : function\n A phonetic algorithm that takes a string and returns a string\n (presumably a phonetic representation of the original string). By\n default, this function uses :py:func:`.double_metaphone`.\n joiner : str\n The string that will be placed between each word\n *args\n Variable length argument list\n **kwargs\n Arbitrary keyword arguments\n\n Returns\n -------\n str\n The phonetic fingerprint of the phrase\n\n Examples\n --------\n >>> phonetic_fingerprint('The quick brown fox jumped over the lazy dog.')\n '0 afr fks jmpt kk ls prn tk'\n >>> from abydos.phonetic import soundex\n >>> phonetic_fingerprint('The quick brown fox jumped over the lazy dog.',\n ... phonetic_algorithm=soundex)\n 'b650 d200 f200 j513 l200 o160 q200 t000'\n\n \"\"\"\n return Phonetic().fingerprint(\n phrase, phonetic_algorithm, joiner, *args, **kwargs\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the phonetic fingerprint of a phrase.", "response": "def fingerprint(\n self,\n phrase,\n phonetic_algorithm=double_metaphone,\n joiner=' ',\n *args,\n **kwargs\n ):\n \"\"\"Return the phonetic fingerprint of a phrase.\n\n Parameters\n ----------\n phrase : str\n The string from which to calculate the phonetic fingerprint\n phonetic_algorithm : function\n A phonetic algorithm that takes a string and returns a string\n (presumably a phonetic representation of the original string). By\n default, this function uses :py:func:`.double_metaphone`.\n joiner : str\n The string that will be placed between each word\n *args\n Variable length argument list\n **kwargs\n Arbitrary keyword arguments\n\n Returns\n -------\n str\n The phonetic fingerprint of the phrase\n\n Examples\n --------\n >>> pf = Phonetic()\n >>> pf.fingerprint('The quick brown fox jumped over the lazy dog.')\n '0 afr fks jmpt kk ls prn tk'\n >>> from abydos.phonetic import soundex\n >>> pf.fingerprint('The quick brown fox jumped over the lazy dog.',\n ... phonetic_algorithm=soundex)\n 'b650 d200 f200 j513 l200 o160 q200 t000'\n\n \"\"\"\n phonetic = ''\n for word in phrase.split():\n word = phonetic_algorithm(word, *args, **kwargs)\n if not isinstance(word, text_type) and hasattr(word, '__iter__'):\n word = word[0]\n phonetic += word + joiner\n phonetic = phonetic[: -len(joiner)]\n return super(self.__class__, self).fingerprint(phonetic)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the docs in the corpus with sentences flattened.", "response": "def docs_of_words(self):\n r\"\"\"Return the docs in the corpus, with sentences flattened.\n\n Each list within the corpus represents all the words of that document.\n Thus the sentence level of lists has been flattened.\n\n Returns\n -------\n [[str]]\n The docs in the corpus as a list of list of strs\n\n Example\n -------\n >>> tqbf = 'The quick brown fox jumped over the lazy dog.\\n'\n >>> tqbf += 'And then it slept.\\n And the dog ran off.'\n >>> corp = Corpus(tqbf)\n >>> corp.docs_of_words()\n [['The', 'quick', 'brown', 'fox', 'jumped', 'over', 'the', 'lazy',\n 'dog.', 'And', 'then', 'it', 'slept.', 'And', 'the', 'dog', 'ran',\n 'off.']]\n >>> len(corp.docs_of_words())\n 1\n\n \"\"\"\n return [\n [words for sents in doc for words in sents] for doc in self.corpus\n ]"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the raw corpus. This is reconstructed by joining sub - components with the corpus split characters Returns ------- str The raw corpus.", "response": "def raw(self):\n r\"\"\"Return the raw corpus.\n\n This is reconstructed by joining sub-components with the corpus' split\n characters\n\n Returns\n -------\n str\n The raw corpus\n\n Example\n -------\n >>> tqbf = 'The quick brown fox jumped over the lazy dog.\\n'\n >>> tqbf += 'And then it slept.\\n And the dog ran off.'\n >>> corp = Corpus(tqbf)\n >>> print(corp.raw())\n The quick brown fox jumped over the lazy dog.\n And then it slept.\n And the dog ran off.\n >>> len(corp.raw())\n 85\n\n \"\"\"\n doc_list = []\n for doc in self.corpus:\n sent_list = []\n for sent in doc:\n sent_list.append(' '.join(sent))\n doc_list.append(self.sent_split.join(sent_list))\n del sent_list\n return self.doc_split.join(doc_list)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stem(self, word):\n terminate = False\n intact = True\n while not terminate:\n for n in range(6, 0, -1):\n if word[-n:] in self._rule_table[n]:\n accept = False\n if len(self._rule_table[n][word[-n:]]) < 4:\n for rule in self._rule_table[n][word[-n:]]:\n (\n word,\n accept,\n intact,\n terminate,\n ) = self._apply_rule(word, rule, intact, terminate)\n if accept:\n break\n else:\n rule = self._rule_table[n][word[-n:]]\n (word, accept, intact, terminate) = self._apply_rule(\n word, rule, intact, terminate\n )\n\n if accept:\n break\n else:\n break\n\n return word", "response": "Returns the stemmed version of the given word."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the Reth - Schek Phonetik code for a word.", "response": "def encode(self, word):\n \"\"\"Return Reth-Schek Phonetik code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n\n Returns\n -------\n str\n The Reth-Schek Phonetik code\n\n Examples\n --------\n >>> reth_schek_phonetik('Joachim')\n 'JOAGHIM'\n >>> reth_schek_phonetik('Christoph')\n 'GHRISDOF'\n >>> reth_schek_phonetik('J\u00f6rg')\n 'JOERG'\n >>> reth_schek_phonetik('Smith')\n 'SMID'\n >>> reth_schek_phonetik('Schmidt')\n 'SCHMID'\n\n \"\"\"\n # Uppercase\n word = word.upper()\n\n # Replace umlauts/eszett\n word = word.replace('\u00c4', 'AE')\n word = word.replace('\u00d6', 'OE')\n word = word.replace('\u00dc', 'UE')\n word = word.replace('\u00df', 'SS')\n\n # Main loop, using above replacements table\n pos = 0\n while pos < len(word):\n for num in range(3, 0, -1):\n if word[pos : pos + num] in self._replacements[num]:\n word = (\n word[:pos]\n + self._replacements[num][word[pos : pos + num]]\n + word[pos + num :]\n )\n pos += 1\n break\n else:\n pos += 1 # Advance if nothing is recognized\n\n # Change 'CH' back(?) to 'SCH'\n word = word.replace('CH', 'SCH')\n\n # Replace final sequences\n if word[-2:] == 'ER':\n word = word[:-2] + 'R'\n elif word[-2:] == 'EL':\n word = word[:-2] + 'L'\n elif word[-1:] == 'H':\n word = word[:-1]\n\n return word"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nencodes a string into a SfinxBis code.", "response": "def encode(self, word, max_length=-1):\n \"\"\"Return the SfinxBis code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to unlimited)\n\n Returns\n -------\n tuple\n The SfinxBis value\n\n Examples\n --------\n >>> pe = SfinxBis()\n >>> pe.encode('Christopher')\n ('K68376',)\n >>> pe.encode('Niall')\n ('N4',)\n >>> pe.encode('Smith')\n ('S53',)\n >>> pe.encode('Schmidt')\n ('S53',)\n\n >>> pe.encode('Johansson')\n ('J585',)\n >>> pe.encode('Sj\u00f6berg')\n ('#162',)\n\n \"\"\"\n\n def _foersvensker(lokal_ordet):\n \"\"\"Return the Swedish-ized form of the word.\n\n Parameters\n ----------\n lokal_ordet : str\n Word to transform\n\n Returns\n -------\n str\n Transformed word\n\n \"\"\"\n lokal_ordet = lokal_ordet.replace('STIERN', 'STJ\u00c4RN')\n lokal_ordet = lokal_ordet.replace('HIE', 'HJ')\n lokal_ordet = lokal_ordet.replace('SI\u00d6', 'SJ\u00d6')\n lokal_ordet = lokal_ordet.replace('SCH', 'SH')\n lokal_ordet = lokal_ordet.replace('QU', 'KV')\n lokal_ordet = lokal_ordet.replace('IO', 'JO')\n lokal_ordet = lokal_ordet.replace('PH', 'F')\n\n for i in self._harde_vokaler:\n lokal_ordet = lokal_ordet.replace(i + '\u00dc', i + 'J')\n lokal_ordet = lokal_ordet.replace(i + 'Y', i + 'J')\n lokal_ordet = lokal_ordet.replace(i + 'I', i + 'J')\n for i in self._mjuka_vokaler:\n lokal_ordet = lokal_ordet.replace(i + '\u00dc', i + 'J')\n lokal_ordet = lokal_ordet.replace(i + 'Y', i + 'J')\n lokal_ordet = lokal_ordet.replace(i + 'I', i + 'J')\n\n if 'H' in lokal_ordet:\n for i in self._uc_c_set:\n lokal_ordet = lokal_ordet.replace('H' + i, i)\n\n lokal_ordet = lokal_ordet.translate(self._substitutions)\n\n lokal_ordet = lokal_ordet.replace('\u00d0', 'ETH')\n lokal_ordet = lokal_ordet.replace('\u00de', 'TH')\n lokal_ordet = lokal_ordet.replace('\u00df', 'SS')\n\n return lokal_ordet\n\n def _koda_foersta_ljudet(lokal_ordet):\n \"\"\"Return the word with the first sound coded.\n\n Parameters\n ----------\n lokal_ordet : str\n Word to transform\n\n Returns\n -------\n str\n Transformed word\n\n \"\"\"\n if (\n lokal_ordet[0:1] in self._mjuka_vokaler\n or lokal_ordet[0:1] in self._harde_vokaler\n ):\n lokal_ordet = '$' + lokal_ordet[1:]\n elif lokal_ordet[0:2] in ('DJ', 'GJ', 'HJ', 'LJ'):\n lokal_ordet = 'J' + lokal_ordet[2:]\n elif (\n lokal_ordet[0:1] == 'G'\n and lokal_ordet[1:2] in self._mjuka_vokaler\n ):\n lokal_ordet = 'J' + lokal_ordet[1:]\n elif lokal_ordet[0:1] == 'Q':\n lokal_ordet = 'K' + lokal_ordet[1:]\n elif lokal_ordet[0:2] == 'CH' and lokal_ordet[2:3] in frozenset(\n self._mjuka_vokaler | self._harde_vokaler\n ):\n lokal_ordet = '#' + lokal_ordet[2:]\n elif (\n lokal_ordet[0:1] == 'C'\n and lokal_ordet[1:2] in self._harde_vokaler\n ):\n lokal_ordet = 'K' + lokal_ordet[1:]\n elif (\n lokal_ordet[0:1] == 'C' and lokal_ordet[1:2] in self._uc_c_set\n ):\n lokal_ordet = 'K' + lokal_ordet[1:]\n elif lokal_ordet[0:1] == 'X':\n lokal_ordet = 'S' + lokal_ordet[1:]\n elif (\n lokal_ordet[0:1] == 'C'\n and lokal_ordet[1:2] in self._mjuka_vokaler\n ):\n lokal_ordet = 'S' + lokal_ordet[1:]\n elif lokal_ordet[0:3] in ('SKJ', 'STJ', 'SCH'):\n lokal_ordet = '#' + lokal_ordet[3:]\n elif lokal_ordet[0:2] in ('SH', 'KJ', 'TJ', 'SJ'):\n lokal_ordet = '#' + lokal_ordet[2:]\n elif (\n lokal_ordet[0:2] == 'SK'\n and lokal_ordet[2:3] in self._mjuka_vokaler\n ):\n lokal_ordet = '#' + lokal_ordet[2:]\n elif (\n lokal_ordet[0:1] == 'K'\n and lokal_ordet[1:2] in self._mjuka_vokaler\n ):\n lokal_ordet = '#' + lokal_ordet[1:]\n return lokal_ordet\n\n # Steg 1, Versaler\n word = unicode_normalize('NFC', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = word.replace('-', ' ')\n\n # Steg 2, Ta bort adelsprefix\n for adelstitel in self._adelstitler:\n while adelstitel in word:\n word = word.replace(adelstitel, ' ')\n if word.startswith(adelstitel[1:]):\n word = word[len(adelstitel) - 1 :]\n\n # Split word into tokens\n ordlista = word.split()\n\n # Steg 3, Ta bort dubbelteckning i b\u00f6rjan p\u00e5 namnet\n ordlista = [\n self._delete_consecutive_repeats(ordet) for ordet in ordlista\n ]\n if not ordlista:\n # noinspection PyRedundantParentheses\n return ('',)\n\n # Steg 4, F\u00f6rsvenskning\n ordlista = [_foersvensker(ordet) for ordet in ordlista]\n\n # Steg 5, Ta bort alla tecken som inte \u00e4r A-\u00d6 (65-90,196,197,214)\n ordlista = [\n ''.join(c for c in ordet if c in self._uc_set)\n for ordet in ordlista\n ]\n\n # Steg 6, Koda f\u00f6rsta ljudet\n ordlista = [_koda_foersta_ljudet(ordet) for ordet in ordlista]\n\n # Steg 7, Dela upp namnet i tv\u00e5 delar\n rest = [ordet[1:] for ordet in ordlista]\n\n # Steg 8, Utf\u00f6r fonetisk transformation i resten\n rest = [ordet.replace('DT', 'T') for ordet in rest]\n rest = [ordet.replace('X', 'KS') for ordet in rest]\n\n # Steg 9, Koda resten till en sifferkod\n for vokal in self._mjuka_vokaler:\n rest = [ordet.replace('C' + vokal, '8' + vokal) for ordet in rest]\n rest = [ordet.translate(self._trans) for ordet in rest]\n\n # Steg 10, Ta bort intilliggande dubbletter\n rest = [self._delete_consecutive_repeats(ordet) for ordet in rest]\n\n # Steg 11, Ta bort alla \"9\"\n rest = [ordet.replace('9', '') for ordet in rest]\n\n # Steg 12, S\u00e4tt ihop delarna igen\n ordlista = [\n ''.join(ordet) for ordet in zip((_[0:1] for _ in ordlista), rest)\n ]\n\n # truncate, if max_length is set\n if max_length > 0:\n ordlista = [ordet[:max_length] for ordet in ordlista]\n\n return tuple(ordlista)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef bmpm(\n word,\n language_arg=0,\n name_mode='gen',\n match_mode='approx',\n concat=False,\n filter_langs=False,\n):\n \"\"\"Return the Beider-Morse Phonetic Matching encoding(s) of a term.\n\n This is a wrapper for :py:meth:`BeiderMorse.encode`.\n\n Parameters\n ----------\n word : str\n The word to transform\n language_arg : str\n The language of the term; supported values include:\n\n - ``any``\n - ``arabic``\n - ``cyrillic``\n - ``czech``\n - ``dutch``\n - ``english``\n - ``french``\n - ``german``\n - ``greek``\n - ``greeklatin``\n - ``hebrew``\n - ``hungarian``\n - ``italian``\n - ``latvian``\n - ``polish``\n - ``portuguese``\n - ``romanian``\n - ``russian``\n - ``spanish``\n - ``turkish``\n\n name_mode : str\n The name mode of the algorithm:\n\n - ``gen`` -- general (default)\n - ``ash`` -- Ashkenazi\n - ``sep`` -- Sephardic\n\n match_mode : str\n Matching mode: ``approx`` or ``exact``\n concat : bool\n Concatenation mode\n filter_langs : bool\n Filter out incompatible languages\n\n Returns\n -------\n tuple\n The Beider-Morse phonetic value(s)\n\n Examples\n --------\n >>> bmpm('Christopher')\n 'xrQstopir xrQstYpir xristopir xristYpir xrQstofir xrQstYfir xristofir\n xristYfir xristopi xritopir xritopi xristofi xritofir xritofi\n tzristopir tzristofir zristopir zristopi zritopir zritopi zristofir\n zristofi zritofir zritofi'\n >>> bmpm('Niall')\n 'nial niol'\n >>> bmpm('Smith')\n 'zmit'\n >>> bmpm('Schmidt')\n 'zmit stzmit'\n\n >>> bmpm('Christopher', language_arg='German')\n 'xrQstopir xrQstYpir xristopir xristYpir xrQstofir xrQstYfir xristofir\n xristYfir'\n >>> bmpm('Christopher', language_arg='English')\n 'tzristofir tzrQstofir tzristafir tzrQstafir xristofir xrQstofir\n xristafir xrQstafir'\n >>> bmpm('Christopher', language_arg='German', name_mode='ash')\n 'xrQstopir xrQstYpir xristopir xristYpir xrQstofir xrQstYfir xristofir\n xristYfir'\n\n >>> bmpm('Christopher', language_arg='German', match_mode='exact')\n 'xriStopher xriStofer xristopher xristofer'\n\n \"\"\"\n return BeiderMorse().encode(\n word, language_arg, name_mode, match_mode, concat, filter_langs\n )", "response": "Return the Beider - Morse phonetic encoding of a word."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _language(self, name, name_mode):\n name = name.strip().lower()\n rules = BMDATA[name_mode]['language_rules']\n all_langs = (\n sum(_LANG_DICT[_] for _ in BMDATA[name_mode]['languages']) - 1\n )\n choices_remaining = all_langs\n for rule in rules:\n letters, languages, accept = rule\n if search(letters, name) is not None:\n if accept:\n choices_remaining &= languages\n else:\n choices_remaining &= (~languages) % (all_langs + 1)\n if choices_remaining == L_NONE:\n choices_remaining = L_ANY\n return choices_remaining", "response": "Return the best guess language ID for the word and language choices."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _redo_language(\n self, term, name_mode, rules, final_rules1, final_rules2, concat\n ):\n \"\"\"Reassess the language of the terms and call the phonetic encoder.\n\n Uses a split multi-word term.\n\n Parameters\n ----------\n term : str\n The term to encode via Beider-Morse\n name_mode : str\n The name mode of the algorithm: ``gen`` (default),\n ``ash`` (Ashkenazi), or ``sep`` (Sephardic)\n rules : tuple\n The set of initial phonetic transform regexps\n final_rules1 : tuple\n The common set of final phonetic transform regexps\n final_rules2 : tuple\n The specific set of final phonetic transform regexps\n concat : bool\n A flag to indicate concatenation\n\n Returns\n -------\n str\n A Beider-Morse phonetic code\n\n \"\"\"\n language_arg = self._language(term, name_mode)\n return self._phonetic(\n term,\n name_mode,\n rules,\n final_rules1,\n final_rules2,\n language_arg,\n concat,\n )", "response": "Reassess the language of the terms and call the phonetic encoder."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _phonetic(\n self,\n term,\n name_mode,\n rules,\n final_rules1,\n final_rules2,\n language_arg=0,\n concat=False,\n ):\n \"\"\"Return the Beider-Morse encoding(s) of a term.\n\n Parameters\n ----------\n term : str\n The term to encode via Beider-Morse\n name_mode : str\n The name mode of the algorithm: ``gen`` (default),\n ``ash`` (Ashkenazi), or ``sep`` (Sephardic)\n rules : tuple\n The set of initial phonetic transform regexps\n final_rules1 : tuple\n The common set of final phonetic transform regexps\n final_rules2 : tuple\n The specific set of final phonetic transform regexps\n language_arg : int\n The language of the term\n concat : bool\n A flag to indicate concatenation\n\n Returns\n -------\n str\n A Beider-Morse phonetic code\n\n \"\"\"\n term = term.replace('-', ' ').strip()\n\n if name_mode == 'gen': # generic case\n # discard and concatenate certain words if at the start of the name\n for pfx in BMDATA['gen']['discards']:\n if term.startswith(pfx):\n remainder = term[len(pfx) :]\n combined = pfx[:-1] + remainder\n result = (\n self._redo_language(\n remainder,\n name_mode,\n rules,\n final_rules1,\n final_rules2,\n concat,\n )\n + '-'\n + self._redo_language(\n combined,\n name_mode,\n rules,\n final_rules1,\n final_rules2,\n concat,\n )\n )\n return result\n\n words = (\n term.split()\n ) # create array of the individual words in the name\n words2 = []\n\n if name_mode == 'sep': # Sephardic case\n # for each word in the name, delete portions of word preceding\n # apostrophe\n # ex: d'avila d'aguilar --> avila aguilar\n # also discard certain words in the name\n\n # note that we can never get a match on \"de la\" because we are\n # checking single words below\n # this is a bug, but I won't try to fix it now\n\n for word in words:\n word = word[word.rfind('\\'') + 1 :]\n if word not in BMDATA['sep']['discards']:\n words2.append(word)\n\n elif name_mode == 'ash': # Ashkenazic case\n # discard certain words if at the start of the name\n if len(words) > 1 and words[0] in BMDATA['ash']['discards']:\n words2 = words[1:]\n else:\n words2 = list(words)\n else:\n words2 = list(words)\n\n if concat:\n # concatenate the separate words of a multi-word name\n # (normally used for exact matches)\n term = ' '.join(words2)\n elif len(words2) == 1: # not a multi-word name\n term = words2[0]\n else:\n # encode each word in a multi-word name separately\n # (normally used for approx matches)\n result = '-'.join(\n [\n self._redo_language(\n w, name_mode, rules, final_rules1, final_rules2, concat\n )\n for w in words2\n ]\n )\n return result\n\n term_length = len(term)\n\n # apply language rules to map to phonetic alphabet\n phonetic = ''\n skip = 0\n for i in range(term_length):\n if skip:\n skip -= 1\n continue\n found = False\n for rule in rules:\n pattern = rule[_PATTERN_POS]\n pattern_length = len(pattern)\n lcontext = rule[_LCONTEXT_POS]\n rcontext = rule[_RCONTEXT_POS]\n\n # check to see if next sequence in input matches the string in\n # the rule\n if (pattern_length > term_length - i) or (\n term[i : i + pattern_length] != pattern\n ): # no match\n continue\n\n right = '^' + rcontext\n left = lcontext + '$'\n\n # check that right context is satisfied\n if rcontext != '':\n if not search(right, term[i + pattern_length :]):\n continue\n\n # check that left context is satisfied\n if lcontext != '':\n if not search(left, term[:i]):\n continue\n\n # check for incompatible attributes\n candidate = self._apply_rule_if_compat(\n phonetic, rule[_PHONETIC_POS], language_arg\n )\n # The below condition shouldn't ever be false\n if candidate is not None: # pragma: no branch\n phonetic = candidate\n found = True\n break\n\n if (\n not found\n ): # character in name that is not in table -- e.g., space\n pattern_length = 1\n skip = pattern_length - 1\n\n # apply final rules on phonetic-alphabet,\n # doing a substitution of certain characters\n phonetic = self._apply_final_rules(\n phonetic, final_rules1, language_arg, False\n ) # apply common rules\n # final_rules1 are the common approx rules,\n # final_rules2 are approx rules for specific language\n phonetic = self._apply_final_rules(\n phonetic, final_rules2, language_arg, True\n ) # apply lang specific rules\n\n return phonetic", "response": "This function returns the Beider - Morse encoding of a term."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef _apply_final_rules(self, phonetic, final_rules, language_arg, strip):\n # optimization to save time\n if not final_rules:\n return phonetic\n\n # expand the result\n phonetic = self._expand_alternates(phonetic)\n phonetic_array = phonetic.split('|')\n\n for k in range(len(phonetic_array)):\n phonetic = phonetic_array[k]\n phonetic2 = ''\n phoneticx = self._normalize_lang_attrs(phonetic, True)\n\n i = 0\n while i < len(phonetic):\n found = False\n\n if phonetic[i] == '[': # skip over language attribute\n attrib_start = i\n i += 1\n while True:\n if phonetic[i] == ']':\n i += 1\n phonetic2 += phonetic[attrib_start:i]\n break\n i += 1\n continue\n\n for rule in final_rules:\n pattern = rule[_PATTERN_POS]\n pattern_length = len(pattern)\n lcontext = rule[_LCONTEXT_POS]\n rcontext = rule[_RCONTEXT_POS]\n\n right = '^' + rcontext\n left = lcontext + '$'\n\n # check to see if next sequence in phonetic matches the\n # string in the rule\n if (pattern_length > len(phoneticx) - i) or phoneticx[\n i : i + pattern_length\n ] != pattern:\n continue\n\n # check that right context is satisfied\n if rcontext != '':\n if not search(right, phoneticx[i + pattern_length :]):\n continue\n\n # check that left context is satisfied\n if lcontext != '':\n if not search(left, phoneticx[:i]):\n continue\n\n # check for incompatible attributes\n candidate = self._apply_rule_if_compat(\n phonetic2, rule[_PHONETIC_POS], language_arg\n )\n # The below condition shouldn't ever be false\n if candidate is not None: # pragma: no branch\n phonetic2 = candidate\n found = True\n break\n\n if not found:\n # character in name for which there is no substitution in\n # the table\n phonetic2 += phonetic[i]\n pattern_length = 1\n\n i += pattern_length\n\n phonetic_array[k] = self._expand_alternates(phonetic2)\n\n phonetic = '|'.join(phonetic_array)\n if strip:\n phonetic = self._normalize_lang_attrs(phonetic, True)\n\n if '|' in phonetic:\n phonetic = '(' + self._remove_dupes(phonetic) + ')'\n\n return phonetic", "response": "This function applies the final rules to the phonetic encoding."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nexpand phonetic alternates separated by |s.", "response": "def _expand_alternates(self, phonetic):\n \"\"\"Expand phonetic alternates separated by |s.\n\n Parameters\n ----------\n phonetic : str\n A Beider-Morse phonetic encoding\n\n Returns\n -------\n str\n A Beider-Morse phonetic code\n\n \"\"\"\n alt_start = phonetic.find('(')\n if alt_start == -1:\n return self._normalize_lang_attrs(phonetic, False)\n\n prefix = phonetic[:alt_start]\n alt_start += 1 # get past the (\n alt_end = phonetic.find(')', alt_start)\n alt_string = phonetic[alt_start:alt_end]\n alt_end += 1 # get past the )\n suffix = phonetic[alt_end:]\n alt_array = alt_string.split('|')\n result = ''\n\n for i in range(len(alt_array)):\n alt = alt_array[i]\n alternate = self._expand_alternates(prefix + alt + suffix)\n if alternate != '' and alternate != '[0]':\n if result != '':\n result += '|'\n result += alternate\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\njoining the phonetic numbers and suffixes in cases of alternate phonetic values.", "response": "def _pnums_with_leading_space(self, phonetic):\n \"\"\"Join prefixes & suffixes in cases of alternate phonetic values.\n\n Parameters\n ----------\n phonetic : str\n A Beider-Morse phonetic encoding\n\n Returns\n -------\n str\n A Beider-Morse phonetic code\n\n \"\"\"\n alt_start = phonetic.find('(')\n if alt_start == -1:\n return ' ' + self._phonetic_number(phonetic)\n\n prefix = phonetic[:alt_start]\n alt_start += 1 # get past the (\n alt_end = phonetic.find(')', alt_start)\n alt_string = phonetic[alt_start:alt_end]\n alt_end += 1 # get past the )\n suffix = phonetic[alt_end:]\n alt_array = alt_string.split('|')\n result = ''\n for alt in alt_array:\n result += self._pnums_with_leading_space(prefix + alt + suffix)\n\n return result"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nprepare & join phonetic numbers.", "response": "def _phonetic_numbers(self, phonetic):\n \"\"\"Prepare & join phonetic numbers.\n\n Split phonetic value on '-', run through _pnums_with_leading_space,\n and join with ' '\n\n Parameters\n ----------\n phonetic : str\n A Beider-Morse phonetic encoding\n\n Returns\n -------\n str\n A Beider-Morse phonetic code\n\n \"\"\"\n phonetic_array = phonetic.split('-') # for names with spaces in them\n result = ' '.join(\n [self._pnums_with_leading_space(i)[1:] for i in phonetic_array]\n )\n return result"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nremoving duplicates from a phonetic encoding list.", "response": "def _remove_dupes(self, phonetic):\n \"\"\"Remove duplicates from a phonetic encoding list.\n\n Parameters\n ----------\n phonetic : str\n A Beider-Morse phonetic encoding\n\n Returns\n -------\n str\n A Beider-Morse phonetic code\n\n \"\"\"\n alt_string = phonetic\n alt_array = alt_string.split('|')\n\n result = '|'\n for i in range(len(alt_array)):\n alt = alt_array[i]\n if alt and '|' + alt + '|' not in result:\n result += alt + '|'\n\n return result[1:-1]"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _normalize_lang_attrs(self, text, strip):\n uninitialized = -1 # all 1's\n attrib = uninitialized\n while '[' in text:\n bracket_start = text.find('[')\n bracket_end = text.find(']', bracket_start)\n if bracket_end == -1:\n raise ValueError(\n 'No closing square bracket: text=('\n + text\n + ') strip=('\n + text_type(strip)\n + ')'\n )\n attrib &= int(text[bracket_start + 1 : bracket_end])\n text = text[:bracket_start] + text[bracket_end + 1 :]\n\n if attrib == uninitialized or strip:\n return text\n elif attrib == 0:\n # means that the attributes were incompatible and there is no\n # alternative here\n return '[0]'\n return text + '[' + str(attrib) + ']'", "response": "Remove embedded bracketed attributes.\n\n This (potentially) bitwise-ands bracketed attributes together and adds\n to the end.\n This is applied to a single alternative at a time -- not to a\n parenthesized list.\n It removes all embedded bracketed attributes, logically-ands them\n together, and places them at the end.\n However if strip is true, this can indeed remove embedded bracketed\n attributes from a parenthesized list.\n\n Parameters\n ----------\n text : str\n A Beider-Morse phonetic encoding (in progress)\n strip : bool\n Remove the bracketed attributes (and throw away)\n\n Returns\n -------\n str\n A Beider-Morse phonetic code\n\n Raises\n ------\n ValueError\n No closing square bracket"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef _apply_rule_if_compat(self, phonetic, target, language_arg):\n candidate = phonetic + target\n if '[' not in candidate: # no attributes so we need test no further\n return candidate\n\n # expand the result, converting incompatible attributes to [0]\n candidate = self._expand_alternates(candidate)\n candidate_array = candidate.split('|')\n\n # drop each alternative that has incompatible attributes\n candidate = ''\n found = False\n\n for i in range(len(candidate_array)):\n this_candidate = candidate_array[i]\n if language_arg != 1:\n this_candidate = self._normalize_lang_attrs(\n this_candidate + '[' + str(language_arg) + ']', False\n )\n if this_candidate != '[0]':\n found = True\n if candidate:\n candidate += '|'\n candidate += this_candidate\n\n # return false if no compatible alternatives remain\n if not found:\n return None\n\n # return the result of applying the rule\n if '|' in candidate:\n candidate = '(' + candidate + ')'\n return candidate", "response": "This function will apply a phonetic regex if compatible. This function will return the result of applying the rule if the phonetic regex is not compatible."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the index value for a language code.", "response": "def _language_index_from_code(self, code, name_mode):\n \"\"\"Return the index value for a language code.\n\n This returns l_any if more than one code is specified or the code is\n out of bounds.\n\n Parameters\n ----------\n code : int\n The language code to interpret\n name_mode : str\n The name mode of the algorithm: ``gen`` (default),\n ``ash`` (Ashkenazi), or ``sep`` (Sephardic)\n\n Returns\n -------\n int\n Language code index\n\n \"\"\"\n if code < 1 or code > sum(\n _LANG_DICT[_] for _ in BMDATA[name_mode]['languages']\n ): # code out of range\n return L_ANY\n if (\n code & (code - 1)\n ) != 0: # choice was more than one language; use any\n return L_ANY\n return code"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef encode(\n self,\n word,\n language_arg=0,\n name_mode='gen',\n match_mode='approx',\n concat=False,\n filter_langs=False,\n ):\n \"\"\"Return the Beider-Morse Phonetic Matching encoding(s) of a term.\n\n Parameters\n ----------\n word : str\n The word to transform\n language_arg : int\n The language of the term; supported values include:\n\n - ``any``\n - ``arabic``\n - ``cyrillic``\n - ``czech``\n - ``dutch``\n - ``english``\n - ``french``\n - ``german``\n - ``greek``\n - ``greeklatin``\n - ``hebrew``\n - ``hungarian``\n - ``italian``\n - ``latvian``\n - ``polish``\n - ``portuguese``\n - ``romanian``\n - ``russian``\n - ``spanish``\n - ``turkish``\n\n name_mode : str\n The name mode of the algorithm:\n\n - ``gen`` -- general (default)\n - ``ash`` -- Ashkenazi\n - ``sep`` -- Sephardic\n\n match_mode : str\n Matching mode: ``approx`` or ``exact``\n concat : bool\n Concatenation mode\n filter_langs : bool\n Filter out incompatible languages\n\n Returns\n -------\n tuple\n The Beider-Morse phonetic value(s)\n\n Raises\n ------\n ValueError\n Unknown language\n\n Examples\n --------\n >>> pe = BeiderMorse()\n >>> pe.encode('Christopher')\n 'xrQstopir xrQstYpir xristopir xristYpir xrQstofir xrQstYfir\n xristofir xristYfir xristopi xritopir xritopi xristofi xritofir\n xritofi tzristopir tzristofir zristopir zristopi zritopir zritopi\n zristofir zristofi zritofir zritofi'\n >>> pe.encode('Niall')\n 'nial niol'\n >>> pe.encode('Smith')\n 'zmit'\n >>> pe.encode('Schmidt')\n 'zmit stzmit'\n\n >>> pe.encode('Christopher', language_arg='German')\n 'xrQstopir xrQstYpir xristopir xristYpir xrQstofir xrQstYfir\n xristofir xristYfir'\n >>> pe.encode('Christopher', language_arg='English')\n 'tzristofir tzrQstofir tzristafir tzrQstafir xristofir xrQstofir\n xristafir xrQstafir'\n >>> pe.encode('Christopher', language_arg='German', name_mode='ash')\n 'xrQstopir xrQstYpir xristopir xristYpir xrQstofir xrQstYfir\n xristofir xristYfir'\n\n >>> pe.encode('Christopher', language_arg='German', match_mode='exact')\n 'xriStopher xriStofer xristopher xristofer'\n\n \"\"\"\n word = normalize('NFC', text_type(word.strip().lower()))\n\n name_mode = name_mode.strip().lower()[:3]\n if name_mode not in {'ash', 'sep', 'gen'}:\n name_mode = 'gen'\n\n if match_mode != 'exact':\n match_mode = 'approx'\n\n # Translate the supplied language_arg value into an integer\n # representing a set of languages\n all_langs = (\n sum(_LANG_DICT[_] for _ in BMDATA[name_mode]['languages']) - 1\n )\n lang_choices = 0\n if isinstance(language_arg, (int, float, long)):\n lang_choices = int(language_arg)\n elif language_arg != '' and isinstance(language_arg, (text_type, str)):\n for lang in text_type(language_arg).lower().split(','):\n if lang in _LANG_DICT and (_LANG_DICT[lang] & all_langs):\n lang_choices += _LANG_DICT[lang]\n elif not filter_langs:\n raise ValueError(\n 'Unknown \\''\n + name_mode\n + '\\' language: \\''\n + lang\n + '\\''\n )\n\n # Language choices are either all incompatible with the name mode or\n # no choices were given, so try to autodetect\n if lang_choices == 0:\n language_arg = self._language(word, name_mode)\n else:\n language_arg = lang_choices\n language_arg2 = self._language_index_from_code(language_arg, name_mode)\n\n rules = BMDATA[name_mode]['rules'][language_arg2]\n final_rules1 = BMDATA[name_mode][match_mode]['common']\n final_rules2 = BMDATA[name_mode][match_mode][language_arg2]\n\n result = self._phonetic(\n word,\n name_mode,\n rules,\n final_rules1,\n final_rules2,\n language_arg,\n concat,\n )\n result = self._phonetic_numbers(result)\n\n return result", "response": "Return the Beider - Morse Phonetic Matching encoding of a word."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sim_strcmp95(src, tar, long_strings=False):\n return Strcmp95().sim(src, tar, long_strings)", "response": "Return the strcmp95 similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the strcmp95 distance between two strings.", "response": "def dist_strcmp95(src, tar, long_strings=False):\n \"\"\"Return the strcmp95 distance between two strings.\n\n This is a wrapper for :py:meth:`Strcmp95.dist`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n long_strings : bool\n Set to True to increase the probability of a match when the number of\n matched characters is large. This option allows for a little more\n tolerance when the strings are large. It is not an appropriate test\n when comparing fixed length fields such as phone and social security\n numbers.\n\n Returns\n -------\n float\n Strcmp95 distance\n\n Examples\n --------\n >>> round(dist_strcmp95('cat', 'hat'), 12)\n 0.222222222222\n >>> round(dist_strcmp95('Niall', 'Neil'), 12)\n 0.1545\n >>> round(dist_strcmp95('aluminum', 'Catalan'), 12)\n 0.345238095238\n >>> round(dist_strcmp95('ATCG', 'TAGC'), 12)\n 0.166666666667\n\n \"\"\"\n return Strcmp95().dist(src, tar, long_strings)"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the strcmp95 similarity of two strings.", "response": "def sim(self, src, tar, long_strings=False):\n \"\"\"Return the strcmp95 similarity of two strings.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n long_strings : bool\n Set to True to increase the probability of a match when the number\n of matched characters is large. This option allows for a little\n more tolerance when the strings are large. It is not an appropriate\n test when comparing fixed length fields such as phone and social\n security numbers.\n\n Returns\n -------\n float\n Strcmp95 similarity\n\n Examples\n --------\n >>> cmp = Strcmp95()\n >>> cmp.sim('cat', 'hat')\n 0.7777777777777777\n >>> cmp.sim('Niall', 'Neil')\n 0.8454999999999999\n >>> cmp.sim('aluminum', 'Catalan')\n 0.6547619047619048\n >>> cmp.sim('ATCG', 'TAGC')\n 0.8333333333333334\n\n \"\"\"\n\n def _in_range(char):\n \"\"\"Return True if char is in the range (0, 91).\n\n Parameters\n ----------\n char : str\n The character to check\n\n Returns\n -------\n bool\n True if char is in the range (0, 91)\n\n \"\"\"\n return 91 > ord(char) > 0\n\n ying = src.strip().upper()\n yang = tar.strip().upper()\n\n if ying == yang:\n return 1.0\n # If either string is blank - return - added in Version 2\n if not ying or not yang:\n return 0.0\n\n adjwt = defaultdict(int)\n\n # Initialize the adjwt array on the first call to the function only.\n # The adjwt array is used to give partial credit for characters that\n # may be errors due to known phonetic or character recognition errors.\n # A typical example is to match the letter \"O\" with the number \"0\"\n for i in self._sp_mx:\n adjwt[(i[0], i[1])] = 3\n adjwt[(i[1], i[0])] = 3\n\n if len(ying) > len(yang):\n search_range = len(ying)\n minv = len(yang)\n else:\n search_range = len(yang)\n minv = len(ying)\n\n # Blank out the flags\n ying_flag = [0] * search_range\n yang_flag = [0] * search_range\n search_range = max(0, search_range // 2 - 1)\n\n # Looking only within the search range,\n # count and flag the matched pairs.\n num_com = 0\n yl1 = len(yang) - 1\n for i in range(len(ying)):\n low_lim = (i - search_range) if (i >= search_range) else 0\n hi_lim = (i + search_range) if ((i + search_range) <= yl1) else yl1\n for j in range(low_lim, hi_lim + 1):\n if (yang_flag[j] == 0) and (yang[j] == ying[i]):\n yang_flag[j] = 1\n ying_flag[i] = 1\n num_com += 1\n break\n\n # If no characters in common - return\n if num_com == 0:\n return 0.0\n\n # Count the number of transpositions\n k = n_trans = 0\n for i in range(len(ying)):\n if ying_flag[i] != 0:\n j = 0\n for j in range(k, len(yang)): # pragma: no branch\n if yang_flag[j] != 0:\n k = j + 1\n break\n if ying[i] != yang[j]:\n n_trans += 1\n n_trans //= 2\n\n # Adjust for similarities in unmatched characters\n n_simi = 0\n if minv > num_com:\n for i in range(len(ying)):\n if ying_flag[i] == 0 and _in_range(ying[i]):\n for j in range(len(yang)):\n if yang_flag[j] == 0 and _in_range(yang[j]):\n if (ying[i], yang[j]) in adjwt:\n n_simi += adjwt[(ying[i], yang[j])]\n yang_flag[j] = 2\n break\n num_sim = n_simi / 10.0 + num_com\n\n # Main weight computation\n weight = (\n num_sim / len(ying)\n + num_sim / len(yang)\n + (num_com - n_trans) / num_com\n )\n weight /= 3.0\n\n # Continue to boost the weight if the strings are similar\n if weight > 0.7:\n\n # Adjust for having up to the first 4 characters in common\n j = 4 if (minv >= 4) else minv\n i = 0\n while (i < j) and (ying[i] == yang[i]) and (not ying[i].isdigit()):\n i += 1\n if i:\n weight += i * 0.1 * (1.0 - weight)\n\n # Optionally adjust for long strings.\n\n # After agreeing beginning chars, at least two more must agree and\n # the agreeing characters must be > .5 of remaining characters.\n if (\n long_strings\n and (minv > 4)\n and (num_com > i + 1)\n and (2 * num_com >= minv + i)\n ):\n if not ying[0].isdigit():\n weight += (1.0 - weight) * (\n (num_com - i - 1) / (len(ying) + len(yang) - i * 2 + 2)\n )\n\n return weight"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the Naval Research Laboratory phonetic encoding of a word.", "response": "def encode(self, word):\n \"\"\"Return the Naval Research Laboratory phonetic encoding of a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n\n Returns\n -------\n str\n The NRL phonetic encoding\n\n Examples\n --------\n >>> pe = NRL()\n >>> pe.encode('the')\n 'DHAX'\n >>> pe.encode('round')\n 'rAWnd'\n >>> pe.encode('quick')\n 'kwIHk'\n >>> pe.encode('eaten')\n 'IYtEHn'\n >>> pe.encode('Smith')\n 'smIHTH'\n >>> pe.encode('Larsen')\n 'lAArsEHn'\n\n \"\"\"\n\n def _to_regex(pattern, left_match=True):\n new_pattern = ''\n replacements = {\n '#': '[AEIOU]+',\n ':': '[BCDFGHJKLMNPQRSTVWXYZ]*',\n '^': '[BCDFGHJKLMNPQRSTVWXYZ]',\n '.': '[BDVGJLMNTWZ]',\n '%': '(ER|E|ES|ED|ING|ELY)',\n '+': '[EIY]',\n ' ': '^',\n }\n for char in pattern:\n new_pattern += (\n replacements[char] if char in replacements else char\n )\n\n if left_match:\n new_pattern += '$'\n if '^' not in pattern:\n new_pattern = '^.*' + new_pattern\n else:\n new_pattern = '^' + new_pattern.replace('^', '$')\n if '$' not in new_pattern:\n new_pattern += '.*$'\n\n return new_pattern\n\n word = word.upper()\n\n pron = ''\n pos = 0\n while pos < len(word):\n left_orig = word[:pos]\n right_orig = word[pos:]\n first = word[pos] if word[pos] in self._rules else ' '\n for rule in self._rules[first]:\n left, match, right, out = rule\n if right_orig.startswith(match):\n if left:\n l_pattern = _to_regex(left, left_match=True)\n if right:\n r_pattern = _to_regex(right, left_match=False)\n if (not left or re_match(l_pattern, left_orig)) and (\n not right\n or re_match(r_pattern, right_orig[len(match) :])\n ):\n pron += out\n pos += len(match)\n break\n else:\n pron += word[pos]\n pos += 1\n\n return pron"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef lcsstr(self, src, tar):\n lengths = np_zeros((len(src) + 1, len(tar) + 1), dtype=np_int)\n longest, i_longest = 0, 0\n for i in range(1, len(src) + 1):\n for j in range(1, len(tar) + 1):\n if src[i - 1] == tar[j - 1]:\n lengths[i, j] = lengths[i - 1, j - 1] + 1\n if lengths[i, j] > longest:\n longest = lengths[i, j]\n i_longest = i\n else:\n lengths[i, j] = 0\n return src[i_longest - longest : i_longest]", "response": "Return the longest common substring of two strings."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef needleman_wunsch(src, tar, gap_cost=1, sim_func=sim_ident):\n return NeedlemanWunsch().dist_abs(src, tar, gap_cost, sim_func)", "response": "Return the Needleman - Wunsch score of two strings."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sim_matrix(\n src,\n tar,\n mat=None,\n mismatch_cost=0,\n match_cost=1,\n symmetric=True,\n alphabet=None,\n ):\n \"\"\"Return the matrix similarity of two strings.\n\n With the default parameters, this is identical to sim_ident.\n It is possible for sim_matrix to return values outside of the range\n :math:`[0, 1]`, if values outside that range are present in mat,\n mismatch_cost, or match_cost.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n mat : dict\n A dict mapping tuples to costs; the tuples are (src, tar) pairs of\n symbols from the alphabet parameter\n mismatch_cost : float\n The value returned if (src, tar) is absent from mat when src does\n not equal tar\n match_cost : float\n The value returned if (src, tar) is absent from mat when src equals\n tar\n symmetric : bool\n True if the cost of src not matching tar is identical to the cost\n of tar not matching src; in this case, the values in mat need only\n contain (src, tar) or (tar, src), not both\n alphabet : str\n A collection of tokens from which src and tar are drawn; if this is\n defined a ValueError is raised if either tar or src is not found in\n alphabet\n\n Returns\n -------\n float\n Matrix similarity\n\n Raises\n ------\n ValueError\n src value not in alphabet\n ValueError\n tar value not in alphabet\n\n Examples\n --------\n >>> NeedlemanWunsch.sim_matrix('cat', 'hat')\n 0\n >>> NeedlemanWunsch.sim_matrix('hat', 'hat')\n 1\n\n \"\"\"\n if alphabet:\n alphabet = tuple(alphabet)\n for i in src:\n if i not in alphabet:\n raise ValueError('src value not in alphabet')\n for i in tar:\n if i not in alphabet:\n raise ValueError('tar value not in alphabet')\n\n if src == tar:\n if mat and (src, src) in mat:\n return mat[(src, src)]\n return match_cost\n if mat and (src, tar) in mat:\n return mat[(src, tar)]\n elif symmetric and mat and (tar, src) in mat:\n return mat[(tar, src)]\n return mismatch_cost", "response": "Return the similarity matrix of two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef encode(self, word, max_length=14):\n alpha = ['']\n pos = 0\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n\n # Clamp max_length to [4, 64]\n if max_length != -1:\n max_length = min(max(4, max_length), 64)\n else:\n max_length = 64\n\n # Do special processing for initial substrings\n for k in self._alpha_sis_initials_order:\n if word.startswith(k):\n alpha[0] += self._alpha_sis_initials[k]\n pos += len(k)\n break\n\n # Add a '0' if alpha is still empty\n if not alpha[0]:\n alpha[0] += '0'\n\n # Whether or not any special initial codes were encoded, iterate\n # through the length of the word in the main encoding loop\n while pos < len(word):\n orig_pos = pos\n for k in self._alpha_sis_basic_order:\n if word[pos:].startswith(k):\n if isinstance(self._alpha_sis_basic[k], tuple):\n newalpha = []\n for i in range(len(self._alpha_sis_basic[k])):\n newalpha += [\n _ + self._alpha_sis_basic[k][i] for _ in alpha\n ]\n alpha = newalpha\n else:\n alpha = [_ + self._alpha_sis_basic[k] for _ in alpha]\n pos += len(k)\n break\n if pos == orig_pos:\n alpha = [_ + '_' for _ in alpha]\n pos += 1\n\n # Trim doublets and placeholders\n for i in range(len(alpha)):\n pos = 1\n while pos < len(alpha[i]):\n if alpha[i][pos] == alpha[i][pos - 1]:\n alpha[i] = alpha[i][:pos] + alpha[i][pos + 1 :]\n pos += 1\n alpha = (_.replace('_', '') for _ in alpha)\n\n # Trim codes and return tuple\n alpha = ((_ + ('0' * max_length))[:max_length] for _ in alpha)\n return tuple(alpha)", "response": "Return the IBM Alpha Search Inquiry System code for a word."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef encode(self, word, max_length=-1):\n # uppercase, normalize, and decompose, filter to A-Z minus vowels & W\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = ''.join(c for c in word if c in self._uc_set)\n\n # merge repeated Ls & Rs\n word = word.replace('LL', 'L')\n word = word.replace('R', 'R')\n\n # apply the Soundex algorithm\n sdx = word.translate(self._trans)\n\n if max_length > 0:\n sdx = (sdx + ('0' * max_length))[:max_length]\n\n return sdx", "response": "Return the PhoneticSpanish coding of word."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the q - gram fingerprint of a phrase.", "response": "def qgram_fingerprint(phrase, qval=2, start_stop='', joiner=''):\n \"\"\"Return Q-Gram fingerprint.\n\n This is a wrapper for :py:meth:`QGram.fingerprint`.\n\n Parameters\n ----------\n phrase : str\n The string from which to calculate the q-gram fingerprint\n qval : int\n The length of each q-gram (by default 2)\n start_stop : str\n The start & stop symbol(s) to concatenate on either end of the phrase,\n as defined in :py:class:`tokenizer.QGrams`\n joiner : str\n The string that will be placed between each word\n\n Returns\n -------\n str\n The q-gram fingerprint of the phrase\n\n Examples\n --------\n >>> qgram_fingerprint('The quick brown fox jumped over the lazy dog.')\n 'azbrckdoedeleqerfoheicjukblampnfogovowoxpequrortthuiumvewnxjydzy'\n >>> qgram_fingerprint('Christopher')\n 'cherhehrisopphristto'\n >>> qgram_fingerprint('Niall')\n 'aliallni'\n\n \"\"\"\n return QGram().fingerprint(phrase, qval, start_stop, joiner)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fingerprint(self, phrase, qval=2, start_stop='', joiner=''):\n phrase = unicode_normalize('NFKD', text_type(phrase.strip().lower()))\n phrase = ''.join(c for c in phrase if c.isalnum())\n phrase = QGrams(phrase, qval, start_stop)\n phrase = joiner.join(sorted(phrase))\n return phrase", "response": "Return the q - gram fingerprint of the phrase."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the NCD between two strings using BWT plus RLE.", "response": "def dist(self, src, tar):\n \"\"\"Return the NCD between two strings using BWT plus RLE.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n float\n Compression distance\n\n Examples\n --------\n >>> cmp = NCDbwtrle()\n >>> cmp.dist('cat', 'hat')\n 0.75\n >>> cmp.dist('Niall', 'Neil')\n 0.8333333333333334\n >>> cmp.dist('aluminum', 'Catalan')\n 1.0\n >>> cmp.dist('ATCG', 'TAGC')\n 0.8\n\n \"\"\"\n if src == tar:\n return 0.0\n\n src_comp = self._rle.encode(self._bwt.encode(src))\n tar_comp = self._rle.encode(self._bwt.encode(tar))\n concat_comp = self._rle.encode(self._bwt.encode(src + tar))\n concat_comp2 = self._rle.encode(self._bwt.encode(tar + src))\n\n return (\n min(len(concat_comp), len(concat_comp2))\n - min(len(src_comp), len(tar_comp))\n ) / max(len(src_comp), len(tar_comp))"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dm_soundex(word, max_length=6, zero_pad=True):\n return DaitchMokotoff().encode(word, max_length, zero_pad)", "response": "Return the Daitch - Mokotoff Soundex code for a word."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the Daitch - Mokotoff Soundex code for a word.", "response": "def encode(self, word, max_length=6, zero_pad=True):\n \"\"\"Return the Daitch-Mokotoff Soundex code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to 6; must be between 6\n and 64)\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length\n string\n\n Returns\n -------\n str\n The Daitch-Mokotoff Soundex value\n\n Examples\n --------\n >>> pe = DaitchMokotoff()\n >>> sorted(pe.encode('Christopher'))\n ['494379', '594379']\n >>> pe.encode('Niall')\n {'680000'}\n >>> pe.encode('Smith')\n {'463000'}\n >>> pe.encode('Schmidt')\n {'463000'}\n\n >>> sorted(pe.encode('The quick brown fox', max_length=20,\n ... zero_pad=False))\n ['35457976754', '3557976754']\n\n \"\"\"\n dms = [''] # initialize empty code list\n\n # Require a max_length of at least 6 and not more than 64\n if max_length != -1:\n max_length = min(max(6, max_length), 64)\n else:\n max_length = 64\n\n # uppercase, normalize, decompose, and filter non-A-Z\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n\n # Nothing to convert, return base case\n if not word:\n if zero_pad:\n return {'0' * max_length}\n return {'0'}\n\n pos = 0\n while pos < len(word):\n # Iterate through _dms_order, which specifies the possible\n # substrings for which codes exist in the Daitch-Mokotoff coding\n for sstr in self._dms_order[word[pos]]: # pragma: no branch\n if word[pos:].startswith(sstr):\n # Having determined a valid substring start, retrieve the\n # code\n dm_val = self._dms_table[sstr]\n\n # Having retried the code (triple), determine the correct\n # positional variant (first, pre-vocalic, elsewhere)\n if pos == 0:\n dm_val = dm_val[0]\n elif (\n pos + len(sstr) < len(word)\n and word[pos + len(sstr)] in self._uc_v_set\n ):\n dm_val = dm_val[1]\n else:\n dm_val = dm_val[2]\n\n # Build the code strings\n if isinstance(dm_val, tuple):\n dms = [_ + text_type(dm_val[0]) for _ in dms] + [\n _ + text_type(dm_val[1]) for _ in dms\n ]\n else:\n dms = [_ + text_type(dm_val) for _ in dms]\n pos += len(sstr)\n break\n\n # Filter out double letters and _ placeholders\n dms = (\n ''.join(c for c in self._delete_consecutive_repeats(_) if c != '_')\n for _ in dms\n )\n\n # Trim codes and return set\n if zero_pad:\n dms = ((_ + ('0' * max_length))[:max_length] for _ in dms)\n else:\n dms = (_[:max_length] for _ in dms)\n return set(dms)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encode(self, word):\n word = word.upper()\n\n code = ''\n skip = 0\n\n if word[0:2] == 'AA':\n code = '\u00c5'\n skip = 2\n elif word[0:2] == 'GI':\n code = 'J'\n skip = 2\n elif word[0:3] == 'SKY':\n code = 'X'\n skip = 3\n elif word[0:2] == 'EI':\n code = '\u00c6'\n skip = 2\n elif word[0:2] == 'KY':\n code = 'X'\n skip = 2\n elif word[:1] == 'C':\n code = 'K'\n skip = 1\n elif word[:1] == '\u00c4':\n code = '\u00c6'\n skip = 1\n elif word[:1] == '\u00d6':\n code = '\u00d8'\n skip = 1\n\n if word[-2:] == 'DT':\n word = word[:-2] + 'T'\n # Though the rules indicate this rule applies in all positions, the\n # reference implementation indicates it applies only in final position.\n elif word[-2:-1] in self._uc_v_set and word[-1:] == 'D':\n word = word[:-2]\n\n for pos, char in enumerate(word):\n if skip:\n skip -= 1\n else:\n for length in sorted(self._replacements, reverse=True):\n if word[pos : pos + length] in self._replacements[length]:\n code += self._replacements[length][\n word[pos : pos + length]\n ]\n skip = length - 1\n break\n else:\n if not pos or char not in self._uc_v_set:\n code += char\n\n code = self._delete_consecutive_repeats(code)\n\n return code", "response": "Encodes a string into a Norphone code."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\ncasts to tuple. Returns ------- tuple The confusion table as a 4-tuple (tp, tn, fp, fn) Example ------- >>> ct = ConfusionTable(120, 60, 20, 30) >>> ct.to_tuple() (120, 60, 20, 30)", "response": "def to_tuple(self):\n \"\"\"Cast to tuple.\n\n Returns\n -------\n tuple\n The confusion table as a 4-tuple (tp, tn, fp, fn)\n\n Example\n -------\n >>> ct = ConfusionTable(120, 60, 20, 30)\n >>> ct.to_tuple()\n (120, 60, 20, 30)\n\n \"\"\"\n return self._tp, self._tn, self._fp, self._fn"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\ncasts to dict. Returns dict", "response": "def to_dict(self):\n \"\"\"Cast to dict.\n\n Returns\n -------\n dict\n The confusion table as a dict\n\n Example\n -------\n >>> ct = ConfusionTable(120, 60, 20, 30)\n >>> import pprint\n >>> pprint.pprint(ct.to_dict())\n {'fn': 30, 'fp': 20, 'tn': 60, 'tp': 120}\n\n \"\"\"\n return {'tp': self._tp, 'tn': self._tn, 'fp': self._fp, 'fn': self._fn}"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef population(self):\n return self._tp + self._tn + self._fp + self._fn", "response": "Return the population of the confusion table."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef precision(self):\n if self._tp + self._fp == 0:\n return float('NaN')\n return self._tp / (self._tp + self._fp)", "response": "Return precision of the confusion table."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef precision_gain(self):\n if self.population() == 0:\n return float('NaN')\n random_precision = self.cond_pos_pop() / self.population()\n return self.precision() / random_precision", "response": "Return gain in precision."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns recall of the confusion table entry.", "response": "def recall(self):\n r\"\"\"Return recall.\n\n Recall is defined as :math:`\\frac{tp}{tp + fn}`\n\n AKA sensitivity\n\n AKA true positive rate (TPR)\n\n Cf. https://en.wikipedia.org/wiki/Precision_and_recall\n\n Cf. https://en.wikipedia.org/wiki/Sensitivity_(test)\n\n Cf. https://en.wikipedia.org/wiki/Information_retrieval#Recall\n\n Returns\n -------\n float\n The recall of the confusion table\n\n Example\n -------\n >>> ct = ConfusionTable(120, 60, 20, 30)\n >>> ct.recall()\n 0.8\n\n \"\"\"\n if self._tp + self._fn == 0:\n return float('NaN')\n return self._tp / (self._tp + self._fn)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef specificity(self):\n if self._tn + self._fp == 0:\n return float('NaN')\n return self._tn / (self._tn + self._fp)", "response": "Return specificity of the confusion table."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns negative predictive value of the confusion table.", "response": "def npv(self):\n r\"\"\"Return negative predictive value (NPV).\n\n NPV is defined as :math:`\\frac{tn}{tn + fn}`\n\n Cf. https://en.wikipedia.org/wiki/Negative_predictive_value\n\n Returns\n -------\n float\n The negative predictive value of the confusion table\n\n Example\n -------\n >>> ct = ConfusionTable(120, 60, 20, 30)\n >>> ct.npv()\n 0.6666666666666666\n\n \"\"\"\n if self._tn + self._fn == 0:\n return float('NaN')\n return self._tn / (self._tn + self._fn)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the fall - out of the confusion table.", "response": "def fallout(self):\n r\"\"\"Return fall-out.\n\n Fall-out is defined as :math:`\\frac{fp}{fp + tn}`\n\n AKA false positive rate (FPR)\n\n Cf. https://en.wikipedia.org/wiki/Information_retrieval#Fall-out\n\n Returns\n -------\n float\n The fall-out of the confusion table\n\n Example\n -------\n >>> ct = ConfusionTable(120, 60, 20, 30)\n >>> ct.fallout()\n 0.25\n\n \"\"\"\n if self._fp + self._tn == 0:\n return float('NaN')\n return self._fp / (self._fp + self._tn)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef fdr(self):\n if self._fp + self._tp == 0:\n return float('NaN')\n return self._fp / (self._fp + self._tp)", "response": "Return the false discovery rate of the confusion table."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the accuracy of the confusion table.", "response": "def accuracy(self):\n r\"\"\"Return accuracy.\n\n Accuracy is defined as :math:`\\frac{tp + tn}{population}`\n\n Cf. https://en.wikipedia.org/wiki/Accuracy\n\n Returns\n -------\n float\n The accuracy of the confusion table\n\n Example\n -------\n >>> ct = ConfusionTable(120, 60, 20, 30)\n >>> ct.accuracy()\n 0.782608695652174\n\n \"\"\"\n if self.population() == 0:\n return float('NaN')\n return (self._tp + self._tn) / self.population()"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef accuracy_gain(self):\n if self.population() == 0:\n return float('NaN')\n random_accuracy = (self.cond_pos_pop() / self.population()) ** 2 + (\n self.cond_neg_pop() / self.population()\n ) ** 2\n return self.accuracy() / random_accuracy", "response": "Return gain in accuracy."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef pr_lmean(self):\n precision = self.precision()\n recall = self.recall()\n if not precision or not recall:\n return 0.0\n elif precision == recall:\n return precision\n return (precision - recall) / (math.log(precision) - math.log(recall))", "response": "Return logarithmic mean of precision & recall."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mcc(self):\n if (\n (\n (self._tp + self._fp)\n * (self._tp + self._fn)\n * (self._tn + self._fp)\n * (self._tn + self._fn)\n )\n ) == 0:\n return float('NaN')\n return ((self._tp * self._tn) - (self._fp * self._fn)) / math.sqrt(\n (self._tp + self._fp)\n * (self._tp + self._fn)\n * (self._tn + self._fp)\n * (self._tn + self._fn)\n )", "response": "Return Matthews correlation coefficient of the confusion table entry."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef significance(self):\n if (\n (\n (self._tp + self._fp)\n * (self._tp + self._fn)\n * (self._tn + self._fp)\n * (self._tn + self._fn)\n )\n ) == 0:\n return float('NaN')\n return (\n (self._tp * self._tn - self._fp * self._fn) ** 2\n * (self._tp + self._tn + self._fp + self._fn)\n ) / (\n (self._tp + self._fp)\n * (self._tp + self._fn)\n * (self._tn + self._fp)\n * (self._tn + self._fn)\n )", "response": "Return the significance of the confusion table."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn \u03ba statistic of the confusion table.", "response": "def kappa_statistic(self):\n r\"\"\"Return \u03ba statistic.\n\n The \u03ba statistic is defined as:\n :math:`\\kappa = \\frac{accuracy - random~ accuracy}\n {1 - random~ accuracy}`\n\n The \u03ba statistic compares the performance of the classifier relative to\n the performance of a random classifier. :math:`\\kappa` = 0 indicates\n performance identical to random. :math:`\\kappa` = 1 indicates perfect\n predictive success. :math:`\\kappa` = -1 indicates perfect predictive\n failure.\n\n Returns\n -------\n float\n The \u03ba statistic of the confusion table\n\n Example\n -------\n >>> ct = ConfusionTable(120, 60, 20, 30)\n >>> ct.kappa_statistic()\n 0.5344129554655871\n\n \"\"\"\n if self.population() == 0:\n return float('NaN')\n random_accuracy = (\n (self._tn + self._fp) * (self._tn + self._fn)\n + (self._fn + self._tp) * (self._fp + self._tp)\n ) / self.population() ** 2\n return (self.accuracy() - random_accuracy) / (1 - random_accuracy)"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef encode(self, word, max_length=-1):\n # Require a max_length of at least 4\n if max_length != -1:\n max_length = max(4, max_length)\n\n primary = ''\n secondary = ''\n\n def _slavo_germanic():\n \"\"\"Return True if the word appears to be Slavic or Germanic.\n\n Returns\n -------\n bool\n True if the word appears to be Slavic or Germanic\n\n \"\"\"\n if 'W' in word or 'K' in word or 'CZ' in word:\n return True\n return False\n\n def _metaph_add(pri, sec=''):\n \"\"\"Return a new metaphone tuple with the supplied elements.\n\n Parameters\n ----------\n pri : str\n The primary element\n sec : str\n The secondary element\n\n Returns\n -------\n tuple\n A new metaphone tuple with the supplied elements\n\n \"\"\"\n newpri = primary\n newsec = secondary\n if pri:\n newpri += pri\n if sec:\n if sec != ' ':\n newsec += sec\n else:\n newsec += pri\n return newpri, newsec\n\n def _is_vowel(pos):\n \"\"\"Return True if the character at word[pos] is a vowel.\n\n Parameters\n ----------\n pos : int\n Position in the word\n\n Returns\n -------\n bool\n True if the character is a vowel\n\n \"\"\"\n if pos >= 0 and word[pos] in {'A', 'E', 'I', 'O', 'U', 'Y'}:\n return True\n return False\n\n def _get_at(pos):\n \"\"\"Return the character at word[pos].\n\n Parameters\n ----------\n pos : int\n Position in the word\n\n Returns\n -------\n str\n Character at word[pos]\n\n \"\"\"\n return word[pos]\n\n def _string_at(pos, slen, substrings):\n \"\"\"Return True if word[pos:pos+slen] is in substrings.\n\n Parameters\n ----------\n pos : int\n Position in the word\n slen : int\n Substring length\n substrings : set\n Substrings to search\n\n Returns\n -------\n bool\n True if word[pos:pos+slen] is in substrings\n\n \"\"\"\n if pos < 0:\n return False\n return word[pos : pos + slen] in substrings\n\n current = 0\n length = len(word)\n if length < 1:\n return '', ''\n last = length - 1\n\n word = word.upper()\n word = word.replace('\u00df', 'SS')\n\n # Pad the original string so that we can index beyond the edge of the\n # world\n word += ' '\n\n # Skip these when at start of word\n if word[0:2] in {'GN', 'KN', 'PN', 'WR', 'PS'}:\n current += 1\n\n # Initial 'X' is pronounced 'Z' e.g. 'Xavier'\n if _get_at(0) == 'X':\n primary, secondary = _metaph_add('S') # 'Z' maps to 'S'\n current += 1\n\n # Main loop\n while True:\n if current >= length:\n break\n\n if _get_at(current) in {'A', 'E', 'I', 'O', 'U', 'Y'}:\n if current == 0:\n # All init vowels now map to 'A'\n primary, secondary = _metaph_add('A')\n current += 1\n continue\n\n elif _get_at(current) == 'B':\n # \"-mb\", e.g\", \"dumb\", already skipped over...\n primary, secondary = _metaph_add('P')\n if _get_at(current + 1) == 'B':\n current += 2\n else:\n current += 1\n continue\n\n elif _get_at(current) == '\u00c7':\n primary, secondary = _metaph_add('S')\n current += 1\n continue\n\n elif _get_at(current) == 'C':\n # Various Germanic\n if (\n current > 1\n and not _is_vowel(current - 2)\n and _string_at((current - 1), 3, {'ACH'})\n and (\n (_get_at(current + 2) != 'I')\n and (\n (_get_at(current + 2) != 'E')\n or _string_at(\n (current - 2), 6, {'BACHER', 'MACHER'}\n )\n )\n )\n ):\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n # Special case 'caesar'\n elif current == 0 and _string_at(current, 6, {'CAESAR'}):\n primary, secondary = _metaph_add('S')\n current += 2\n continue\n\n # Italian 'chianti'\n elif _string_at(current, 4, {'CHIA'}):\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n elif _string_at(current, 2, {'CH'}):\n # Find 'Michael'\n if current > 0 and _string_at(current, 4, {'CHAE'}):\n primary, secondary = _metaph_add('K', 'X')\n current += 2\n continue\n\n # Greek roots e.g. 'chemistry', 'chorus'\n elif (\n current == 0\n and (\n _string_at((current + 1), 5, {'HARAC', 'HARIS'})\n or _string_at(\n (current + 1), 3, {'HOR', 'HYM', 'HIA', 'HEM'}\n )\n )\n and not _string_at(0, 5, {'CHORE'})\n ):\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n # Germanic, Greek, or otherwise 'ch' for 'kh' sound\n elif (\n (\n _string_at(0, 4, {'VAN ', 'VON '})\n or _string_at(0, 3, {'SCH'})\n )\n or\n # 'architect but not 'arch', 'orchestra', 'orchid'\n _string_at(\n (current - 2), 6, {'ORCHES', 'ARCHIT', 'ORCHID'}\n )\n or _string_at((current + 2), 1, {'T', 'S'})\n or (\n (\n _string_at(\n (current - 1), 1, {'A', 'O', 'U', 'E'}\n )\n or (current == 0)\n )\n and\n # e.g., 'wachtler', 'wechsler', but not 'tichner'\n _string_at(\n (current + 2),\n 1,\n {\n 'L',\n 'R',\n 'N',\n 'M',\n 'B',\n 'H',\n 'F',\n 'V',\n 'W',\n ' ',\n },\n )\n )\n ):\n primary, secondary = _metaph_add('K')\n\n else:\n if current > 0:\n if _string_at(0, 2, {'MC'}):\n # e.g., \"McHugh\"\n primary, secondary = _metaph_add('K')\n else:\n primary, secondary = _metaph_add('X', 'K')\n else:\n primary, secondary = _metaph_add('X')\n\n current += 2\n continue\n\n # e.g, 'czerny'\n elif _string_at(current, 2, {'CZ'}) and not _string_at(\n (current - 2), 4, {'WICZ'}\n ):\n primary, secondary = _metaph_add('S', 'X')\n current += 2\n continue\n\n # e.g., 'focaccia'\n elif _string_at((current + 1), 3, {'CIA'}):\n primary, secondary = _metaph_add('X')\n current += 3\n\n # double 'C', but not if e.g. 'McClellan'\n elif _string_at(current, 2, {'CC'}) and not (\n (current == 1) and (_get_at(0) == 'M')\n ):\n # 'bellocchio' but not 'bacchus'\n if _string_at(\n (current + 2), 1, {'I', 'E', 'H'}\n ) and not _string_at((current + 2), 2, {'HU'}):\n # 'accident', 'accede' 'succeed'\n if (\n (current == 1) and _get_at(current - 1) == 'A'\n ) or _string_at((current - 1), 5, {'UCCEE', 'UCCES'}):\n primary, secondary = _metaph_add('KS')\n # 'bacci', 'bertucci', other italian\n else:\n primary, secondary = _metaph_add('X')\n current += 3\n continue\n else: # Pierce's rule\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n elif _string_at(current, 2, {'CK', 'CG', 'CQ'}):\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n elif _string_at(current, 2, {'CI', 'CE', 'CY'}):\n # Italian vs. English\n if _string_at(current, 3, {'CIO', 'CIE', 'CIA'}):\n primary, secondary = _metaph_add('S', 'X')\n else:\n primary, secondary = _metaph_add('S')\n current += 2\n continue\n\n # else\n else:\n primary, secondary = _metaph_add('K')\n\n # name sent in 'mac caffrey', 'mac gregor\n if _string_at((current + 1), 2, {' C', ' Q', ' G'}):\n current += 3\n elif _string_at(\n (current + 1), 1, {'C', 'K', 'Q'}\n ) and not _string_at((current + 1), 2, {'CE', 'CI'}):\n current += 2\n else:\n current += 1\n continue\n\n elif _get_at(current) == 'D':\n if _string_at(current, 2, {'DG'}):\n if _string_at((current + 2), 1, {'I', 'E', 'Y'}):\n # e.g. 'edge'\n primary, secondary = _metaph_add('J')\n current += 3\n continue\n else:\n # e.g. 'edgar'\n primary, secondary = _metaph_add('TK')\n current += 2\n continue\n\n elif _string_at(current, 2, {'DT', 'DD'}):\n primary, secondary = _metaph_add('T')\n current += 2\n continue\n\n # else\n else:\n primary, secondary = _metaph_add('T')\n current += 1\n continue\n\n elif _get_at(current) == 'F':\n if _get_at(current + 1) == 'F':\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('F')\n continue\n\n elif _get_at(current) == 'G':\n if _get_at(current + 1) == 'H':\n if (current > 0) and not _is_vowel(current - 1):\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n # 'ghislane', ghiradelli\n elif current == 0:\n if _get_at(current + 2) == 'I':\n primary, secondary = _metaph_add('J')\n else:\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n # Parker's rule (with some further refinements) -\n # e.g., 'hugh'\n elif (\n (\n (current > 1)\n and _string_at((current - 2), 1, {'B', 'H', 'D'})\n )\n or\n # e.g., 'bough'\n (\n (current > 2)\n and _string_at((current - 3), 1, {'B', 'H', 'D'})\n )\n or\n # e.g., 'broughton'\n (\n (current > 3)\n and _string_at((current - 4), 1, {'B', 'H'})\n )\n ):\n current += 2\n continue\n else:\n # e.g. 'laugh', 'McLaughlin', 'cough',\n # 'gough', 'rough', 'tough'\n if (\n (current > 2)\n and (_get_at(current - 1) == 'U')\n and (\n _string_at(\n (current - 3), 1, {'C', 'G', 'L', 'R', 'T'}\n )\n )\n ):\n primary, secondary = _metaph_add('F')\n elif (current > 0) and _get_at(current - 1) != 'I':\n primary, secondary = _metaph_add('K')\n current += 2\n continue\n\n elif _get_at(current + 1) == 'N':\n if (\n (current == 1)\n and _is_vowel(0)\n and not _slavo_germanic()\n ):\n primary, secondary = _metaph_add('KN', 'N')\n # not e.g. 'cagney'\n elif (\n not _string_at((current + 2), 2, {'EY'})\n and (_get_at(current + 1) != 'Y')\n and not _slavo_germanic()\n ):\n primary, secondary = _metaph_add('N', 'KN')\n else:\n primary, secondary = _metaph_add('KN')\n current += 2\n continue\n\n # 'tagliaro'\n elif (\n _string_at((current + 1), 2, {'LI'})\n and not _slavo_germanic()\n ):\n primary, secondary = _metaph_add('KL', 'L')\n current += 2\n continue\n\n # -ges-, -gep-, -gel-, -gie- at beginning\n elif (current == 0) and (\n (_get_at(current + 1) == 'Y')\n or _string_at(\n (current + 1),\n 2,\n {\n 'ES',\n 'EP',\n 'EB',\n 'EL',\n 'EY',\n 'IB',\n 'IL',\n 'IN',\n 'IE',\n 'EI',\n 'ER',\n },\n )\n ):\n primary, secondary = _metaph_add('K', 'J')\n current += 2\n continue\n\n # -ger-, -gy-\n elif (\n (\n _string_at((current + 1), 2, {'ER'})\n or (_get_at(current + 1) == 'Y')\n )\n and not _string_at(0, 6, {'DANGER', 'RANGER', 'MANGER'})\n and not _string_at((current - 1), 1, {'E', 'I'})\n and not _string_at((current - 1), 3, {'RGY', 'OGY'})\n ):\n primary, secondary = _metaph_add('K', 'J')\n current += 2\n continue\n\n # italian e.g, 'biaggi'\n elif _string_at(\n (current + 1), 1, {'E', 'I', 'Y'}\n ) or _string_at((current - 1), 4, {'AGGI', 'OGGI'}):\n # obvious germanic\n if (\n _string_at(0, 4, {'VAN ', 'VON '})\n or _string_at(0, 3, {'SCH'})\n ) or _string_at((current + 1), 2, {'ET'}):\n primary, secondary = _metaph_add('K')\n elif _string_at((current + 1), 4, {'IER '}):\n primary, secondary = _metaph_add('J')\n else:\n primary, secondary = _metaph_add('J', 'K')\n current += 2\n continue\n\n else:\n if _get_at(current + 1) == 'G':\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('K')\n continue\n\n elif _get_at(current) == 'H':\n # only keep if first & before vowel or btw. 2 vowels\n if ((current == 0) or _is_vowel(current - 1)) and _is_vowel(\n current + 1\n ):\n primary, secondary = _metaph_add('H')\n current += 2\n else: # also takes care of 'HH'\n current += 1\n continue\n\n elif _get_at(current) == 'J':\n # obvious spanish, 'jose', 'san jacinto'\n if _string_at(current, 4, {'JOSE'}) or _string_at(\n 0, 4, {'SAN '}\n ):\n if (\n (current == 0) and (_get_at(current + 4) == ' ')\n ) or _string_at(0, 4, {'SAN '}):\n primary, secondary = _metaph_add('H')\n else:\n primary, secondary = _metaph_add('J', 'H')\n current += 1\n continue\n\n elif (current == 0) and not _string_at(current, 4, {'JOSE'}):\n # Yankelovich/Jankelowicz\n primary, secondary = _metaph_add('J', 'A')\n # Spanish pron. of e.g. 'bajador'\n elif (\n _is_vowel(current - 1)\n and not _slavo_germanic()\n and (\n (_get_at(current + 1) == 'A')\n or (_get_at(current + 1) == 'O')\n )\n ):\n primary, secondary = _metaph_add('J', 'H')\n elif current == last:\n primary, secondary = _metaph_add('J', ' ')\n elif not _string_at(\n (current + 1), 1, {'L', 'T', 'K', 'S', 'N', 'M', 'B', 'Z'}\n ) and not _string_at((current - 1), 1, {'S', 'K', 'L'}):\n primary, secondary = _metaph_add('J')\n\n if _get_at(current + 1) == 'J': # it could happen!\n current += 2\n else:\n current += 1\n continue\n\n elif _get_at(current) == 'K':\n if _get_at(current + 1) == 'K':\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('K')\n continue\n\n elif _get_at(current) == 'L':\n if _get_at(current + 1) == 'L':\n # Spanish e.g. 'cabrillo', 'gallegos'\n if (\n (current == (length - 3))\n and _string_at(\n (current - 1), 4, {'ILLO', 'ILLA', 'ALLE'}\n )\n ) or (\n (\n _string_at((last - 1), 2, {'AS', 'OS'})\n or _string_at(last, 1, {'A', 'O'})\n )\n and _string_at((current - 1), 4, {'ALLE'})\n ):\n primary, secondary = _metaph_add('L', ' ')\n current += 2\n continue\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('L')\n continue\n\n elif _get_at(current) == 'M':\n if (\n (\n _string_at((current - 1), 3, {'UMB'})\n and (\n ((current + 1) == last)\n or _string_at((current + 2), 2, {'ER'})\n )\n )\n or\n # 'dumb', 'thumb'\n (_get_at(current + 1) == 'M')\n ):\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('M')\n continue\n\n elif _get_at(current) == 'N':\n if _get_at(current + 1) == 'N':\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('N')\n continue\n\n elif _get_at(current) == '\u00d1':\n current += 1\n primary, secondary = _metaph_add('N')\n continue\n\n elif _get_at(current) == 'P':\n if _get_at(current + 1) == 'H':\n primary, secondary = _metaph_add('F')\n current += 2\n continue\n\n # also account for \"campbell\", \"raspberry\"\n elif _string_at((current + 1), 1, {'P', 'B'}):\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('P')\n continue\n\n elif _get_at(current) == 'Q':\n if _get_at(current + 1) == 'Q':\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('K')\n continue\n\n elif _get_at(current) == 'R':\n # french e.g. 'rogier', but exclude 'hochmeier'\n if (\n (current == last)\n and not _slavo_germanic()\n and _string_at((current - 2), 2, {'IE'})\n and not _string_at((current - 4), 2, {'ME', 'MA'})\n ):\n primary, secondary = _metaph_add('', 'R')\n else:\n primary, secondary = _metaph_add('R')\n\n if _get_at(current + 1) == 'R':\n current += 2\n else:\n current += 1\n continue\n\n elif _get_at(current) == 'S':\n # special cases 'island', 'isle', 'carlisle', 'carlysle'\n if _string_at((current - 1), 3, {'ISL', 'YSL'}):\n current += 1\n continue\n\n # special case 'sugar-'\n elif (current == 0) and _string_at(current, 5, {'SUGAR'}):\n primary, secondary = _metaph_add('X', 'S')\n current += 1\n continue\n\n elif _string_at(current, 2, {'SH'}):\n # Germanic\n if _string_at(\n (current + 1), 4, {'HEIM', 'HOEK', 'HOLM', 'HOLZ'}\n ):\n primary, secondary = _metaph_add('S')\n else:\n primary, secondary = _metaph_add('X')\n current += 2\n continue\n\n # Italian & Armenian\n elif _string_at(current, 3, {'SIO', 'SIA'}) or _string_at(\n current, 4, {'SIAN'}\n ):\n if not _slavo_germanic():\n primary, secondary = _metaph_add('S', 'X')\n else:\n primary, secondary = _metaph_add('S')\n current += 3\n continue\n\n # German & anglicisations, e.g. 'smith' match 'schmidt',\n # 'snider' match 'schneider'\n # also, -sz- in Slavic language although in Hungarian it is\n # pronounced 's'\n elif (\n (current == 0)\n and _string_at((current + 1), 1, {'M', 'N', 'L', 'W'})\n ) or _string_at((current + 1), 1, {'Z'}):\n primary, secondary = _metaph_add('S', 'X')\n if _string_at((current + 1), 1, {'Z'}):\n current += 2\n else:\n current += 1\n continue\n\n elif _string_at(current, 2, {'SC'}):\n # Schlesinger's rule\n if _get_at(current + 2) == 'H':\n # dutch origin, e.g. 'school', 'schooner'\n if _string_at(\n (current + 3),\n 2,\n {'OO', 'ER', 'EN', 'UY', 'ED', 'EM'},\n ):\n # 'schermerhorn', 'schenker'\n if _string_at((current + 3), 2, {'ER', 'EN'}):\n primary, secondary = _metaph_add('X', 'SK')\n else:\n primary, secondary = _metaph_add('SK')\n current += 3\n continue\n else:\n if (\n (current == 0)\n and not _is_vowel(3)\n and (_get_at(3) != 'W')\n ):\n primary, secondary = _metaph_add('X', 'S')\n else:\n primary, secondary = _metaph_add('X')\n current += 3\n continue\n\n elif _string_at((current + 2), 1, {'I', 'E', 'Y'}):\n primary, secondary = _metaph_add('S')\n current += 3\n continue\n\n # else\n else:\n primary, secondary = _metaph_add('SK')\n current += 3\n continue\n\n else:\n # french e.g. 'resnais', 'artois'\n if (current == last) and _string_at(\n (current - 2), 2, {'AI', 'OI'}\n ):\n primary, secondary = _metaph_add('', 'S')\n else:\n primary, secondary = _metaph_add('S')\n\n if _string_at((current + 1), 1, {'S', 'Z'}):\n current += 2\n else:\n current += 1\n continue\n\n elif _get_at(current) == 'T':\n if _string_at(current, 4, {'TION'}):\n primary, secondary = _metaph_add('X')\n current += 3\n continue\n\n elif _string_at(current, 3, {'TIA', 'TCH'}):\n primary, secondary = _metaph_add('X')\n current += 3\n continue\n\n elif _string_at(current, 2, {'TH'}) or _string_at(\n current, 3, {'TTH'}\n ):\n # special case 'thomas', 'thames' or germanic\n if (\n _string_at((current + 2), 2, {'OM', 'AM'})\n or _string_at(0, 4, {'VAN ', 'VON '})\n or _string_at(0, 3, {'SCH'})\n ):\n primary, secondary = _metaph_add('T')\n else:\n primary, secondary = _metaph_add('0', 'T')\n current += 2\n continue\n\n elif _string_at((current + 1), 1, {'T', 'D'}):\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('T')\n continue\n\n elif _get_at(current) == 'V':\n if _get_at(current + 1) == 'V':\n current += 2\n else:\n current += 1\n primary, secondary = _metaph_add('F')\n continue\n\n elif _get_at(current) == 'W':\n # can also be in middle of word\n if _string_at(current, 2, {'WR'}):\n primary, secondary = _metaph_add('R')\n current += 2\n continue\n elif (current == 0) and (\n _is_vowel(current + 1) or _string_at(current, 2, {'WH'})\n ):\n # Wasserman should match Vasserman\n if _is_vowel(current + 1):\n primary, secondary = _metaph_add('A', 'F')\n else:\n # need Uomo to match Womo\n primary, secondary = _metaph_add('A')\n\n # Arnow should match Arnoff\n if (\n ((current == last) and _is_vowel(current - 1))\n or _string_at(\n (current - 1), 5, {'EWSKI', 'EWSKY', 'OWSKI', 'OWSKY'}\n )\n or _string_at(0, 3, {'SCH'})\n ):\n primary, secondary = _metaph_add('', 'F')\n current += 1\n continue\n # Polish e.g. 'filipowicz'\n elif _string_at(current, 4, {'WICZ', 'WITZ'}):\n primary, secondary = _metaph_add('TS', 'FX')\n current += 4\n continue\n # else skip it\n else:\n current += 1\n continue\n\n elif _get_at(current) == 'X':\n # French e.g. breaux\n if not (\n (current == last)\n and (\n _string_at((current - 3), 3, {'IAU', 'EAU'})\n or _string_at((current - 2), 2, {'AU', 'OU'})\n )\n ):\n primary, secondary = _metaph_add('KS')\n\n if _string_at((current + 1), 1, {'C', 'X'}):\n current += 2\n else:\n current += 1\n continue\n\n elif _get_at(current) == 'Z':\n # Chinese Pinyin e.g. 'zhao'\n if _get_at(current + 1) == 'H':\n primary, secondary = _metaph_add('J')\n current += 2\n continue\n elif _string_at((current + 1), 2, {'ZO', 'ZI', 'ZA'}) or (\n _slavo_germanic()\n and ((current > 0) and _get_at(current - 1) != 'T')\n ):\n primary, secondary = _metaph_add('S', 'TS')\n else:\n primary, secondary = _metaph_add('S')\n\n if _get_at(current + 1) == 'Z':\n current += 2\n else:\n current += 1\n continue\n\n else:\n current += 1\n\n if max_length > 0:\n primary = primary[:max_length]\n secondary = secondary[:max_length]\n if primary == secondary:\n secondary = ''\n\n return primary, secondary", "response": "Encodes a word into a Double Metaphone code."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the stemmed version of the given word.", "response": "def stem(self, word):\n \"\"\"Return CLEF German stem.\n\n Parameters\n ----------\n word : str\n The word to stem\n\n Returns\n -------\n str\n Word stem\n\n Examples\n --------\n >>> stmr = CLEFGerman()\n >>> stmr.stem('lesen')\n 'lese'\n >>> stmr.stem('graues')\n 'grau'\n >>> stmr.stem('buchstabieren')\n 'buchstabier'\n\n \"\"\"\n # lowercase, normalize, and compose\n word = normalize('NFC', text_type(word.lower()))\n\n # remove umlauts\n word = word.translate(self._umlauts)\n\n # remove plurals\n wlen = len(word) - 1\n\n if wlen > 3:\n if wlen > 5:\n if word[-3:] == 'nen':\n return word[:-3]\n if wlen > 4:\n if word[-2:] in {'en', 'se', 'es', 'er'}:\n return word[:-2]\n if word[-1] in {'e', 'n', 'r', 's'}:\n return word[:-1]\n return word"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encode(self, word, mode=1, lang='de'):\n phonet_hash = Counter()\n alpha_pos = Counter()\n\n phonet_hash_1 = Counter()\n phonet_hash_2 = Counter()\n\n def _initialize_phonet(lang):\n \"\"\"Initialize phonet variables.\n\n Parameters\n ----------\n lang : str\n Language to use for rules\n\n \"\"\"\n if lang == 'none':\n _phonet_rules = self._rules_no_lang\n else:\n _phonet_rules = self._rules_german\n\n phonet_hash[''] = -1\n\n # German and international umlauts\n for j in {\n '\u00c0',\n '\u00c1',\n '\u00c2',\n '\u00c3',\n '\u00c4',\n '\u00c5',\n '\u00c6',\n '\u00c7',\n '\u00c8',\n '\u00c9',\n '\u00ca',\n '\u00cb',\n '\u00cc',\n '\u00cd',\n '\u00ce',\n '\u00cf',\n '\u00d0',\n '\u00d1',\n '\u00d2',\n '\u00d3',\n '\u00d4',\n '\u00d5',\n '\u00d6',\n '\u00d8',\n '\u00d9',\n '\u00da',\n '\u00db',\n '\u00dc',\n '\u00dd',\n '\u00de',\n '\u00df',\n '\u0152',\n '\u0160',\n '\u0178',\n }:\n alpha_pos[j] = 1\n phonet_hash[j] = -1\n\n # \"normal\" letters ('A'-'Z')\n for i, j in enumerate('ABCDEFGHIJKLMNOPQRSTUVWXYZ'):\n alpha_pos[j] = i + 2\n phonet_hash[j] = -1\n\n for i in range(26):\n for j in range(28):\n phonet_hash_1[i, j] = -1\n phonet_hash_2[i, j] = -1\n\n # for each phonetc rule\n for i in range(len(_phonet_rules)):\n rule = _phonet_rules[i]\n\n if rule and i % 3 == 0:\n # calculate first hash value\n k = _phonet_rules[i][0]\n\n if phonet_hash[k] < 0 and (\n _phonet_rules[i + 1] or _phonet_rules[i + 2]\n ):\n phonet_hash[k] = i\n\n # calculate second hash values\n if k and alpha_pos[k] >= 2:\n k = alpha_pos[k]\n\n j = k - 2\n rule = rule[1:]\n\n if not rule:\n rule = ' '\n elif rule[0] == '(':\n rule = rule[1:]\n else:\n rule = rule[0]\n\n while rule and (rule[0] != ')'):\n k = alpha_pos[rule[0]]\n\n if k > 0:\n # add hash value for this letter\n if phonet_hash_1[j, k] < 0:\n phonet_hash_1[j, k] = i\n phonet_hash_2[j, k] = i\n\n if phonet_hash_2[j, k] >= (i - 30):\n phonet_hash_2[j, k] = i\n else:\n k = -1\n\n if k <= 0:\n # add hash value for all letters\n if phonet_hash_1[j, 0] < 0:\n phonet_hash_1[j, 0] = i\n\n phonet_hash_2[j, 0] = i\n\n rule = rule[1:]\n\n def _phonet(term, mode, lang):\n \"\"\"Return the phonet coded form of a term.\n\n Parameters\n ----------\n term : str\n Term to transform\n mode : int\n The ponet variant to employ (1 or 2)\n lang : str\n ``de`` (default) for German, ``none`` for no language\n\n Returns\n -------\n str\n The phonet value\n\n \"\"\"\n if lang == 'none':\n _phonet_rules = self._rules_no_lang\n else:\n _phonet_rules = self._rules_german\n\n char0 = ''\n dest = term\n\n if not term:\n return ''\n\n term_length = len(term)\n\n # convert input string to upper-case\n src = term.translate(self._upper_trans)\n\n # check \"src\"\n i = 0\n j = 0\n zeta = 0\n\n while i < len(src):\n char = src[i]\n\n pos = alpha_pos[char]\n\n if pos >= 2:\n xpos = pos - 2\n\n if i + 1 == len(src):\n pos = alpha_pos['']\n else:\n pos = alpha_pos[src[i + 1]]\n\n start1 = phonet_hash_1[xpos, pos]\n start2 = phonet_hash_1[xpos, 0]\n end1 = phonet_hash_2[xpos, pos]\n end2 = phonet_hash_2[xpos, 0]\n\n # preserve rule priorities\n if (start2 >= 0) and ((start1 < 0) or (start2 < start1)):\n pos = start1\n start1 = start2\n start2 = pos\n pos = end1\n end1 = end2\n end2 = pos\n\n if (end1 >= start2) and (start2 >= 0):\n if end2 > end1:\n end1 = end2\n\n start2 = -1\n end2 = -1\n else:\n pos = phonet_hash[char]\n start1 = pos\n end1 = 10000\n start2 = -1\n end2 = -1\n\n pos = start1\n zeta0 = 0\n\n if pos >= 0:\n # check rules for this char\n while (_phonet_rules[pos] is None) or (\n _phonet_rules[pos][0] == char\n ):\n if pos > end1:\n if start2 > 0:\n pos = start2\n start1 = start2\n start2 = -1\n end1 = end2\n end2 = -1\n continue\n\n break\n\n if (_phonet_rules[pos] is None) or (\n _phonet_rules[pos + mode] is None\n ):\n # no conversion rule available\n pos += 3\n continue\n\n # check whole string\n matches = 1 # number of matching letters\n priority = 5 # default priority\n rule = _phonet_rules[pos]\n rule = rule[1:]\n\n while (\n rule\n and (len(src) > (i + matches))\n and (src[i + matches] == rule[0])\n and not rule[0].isdigit()\n and (rule not in '(-<^$')\n ):\n matches += 1\n rule = rule[1:]\n\n if rule and (rule[0] == '('):\n # check an array of letters\n if (\n (len(src) > (i + matches))\n and src[i + matches].isalpha()\n and (src[i + matches] in rule[1:])\n ):\n matches += 1\n\n while rule and rule[0] != ')':\n rule = rule[1:]\n\n # if rule[0] == ')':\n rule = rule[1:]\n\n if rule:\n priority0 = ord(rule[0])\n else:\n priority0 = 0\n\n matches0 = matches\n\n while rule and rule[0] == '-' and matches > 1:\n matches -= 1\n rule = rule[1:]\n\n if rule and rule[0] == '<':\n rule = rule[1:]\n\n if rule and rule[0].isdigit():\n # read priority\n priority = int(rule[0])\n rule = rule[1:]\n\n if rule and rule[0:2] == '^^':\n rule = rule[1:]\n\n if (\n not rule\n or (\n (rule[0] == '^')\n and ((i == 0) or not src[i - 1].isalpha())\n and (\n (rule[1:2] != '$')\n or (\n not (\n src[\n i + matches0 : i + matches0 + 1\n ].isalpha()\n )\n and (\n src[\n i + matches0 : i + matches0 + 1\n ]\n != '.'\n )\n )\n )\n )\n or (\n (rule[0] == '$')\n and (i > 0)\n and src[i - 1].isalpha()\n and (\n (\n not src[\n i + matches0 : i + matches0 + 1\n ].isalpha()\n )\n and (\n src[i + matches0 : i + matches0 + 1]\n != '.'\n )\n )\n )\n ):\n # look for continuation, if:\n # matches > 1 und NO '-' in first string */\n pos0 = -1\n\n start3 = 0\n start4 = 0\n end3 = 0\n end4 = 0\n\n if (\n (matches > 1)\n and src[i + matches : i + matches + 1]\n and (priority0 != ord('-'))\n ):\n char0 = src[i + matches - 1]\n pos0 = alpha_pos[char0]\n\n if pos0 >= 2 and src[i + matches]:\n xpos = pos0 - 2\n pos0 = alpha_pos[src[i + matches]]\n start3 = phonet_hash_1[xpos, pos0]\n start4 = phonet_hash_1[xpos, 0]\n end3 = phonet_hash_2[xpos, pos0]\n end4 = phonet_hash_2[xpos, 0]\n\n # preserve rule priorities\n if (start4 >= 0) and (\n (start3 < 0) or (start4 < start3)\n ):\n pos0 = start3\n start3 = start4\n start4 = pos0\n pos0 = end3\n end3 = end4\n end4 = pos0\n\n if (end3 >= start4) and (start4 >= 0):\n if end4 > end3:\n end3 = end4\n\n start4 = -1\n end4 = -1\n else:\n pos0 = phonet_hash[char0]\n start3 = pos0\n end3 = 10000\n start4 = -1\n end4 = -1\n\n pos0 = start3\n\n # check continuation rules for src[i+matches]\n if pos0 >= 0:\n while (_phonet_rules[pos0] is None) or (\n _phonet_rules[pos0][0] == char0\n ):\n if pos0 > end3:\n if start4 > 0:\n pos0 = start4\n start3 = start4\n start4 = -1\n end3 = end4\n end4 = -1\n continue\n\n priority0 = -1\n\n # important\n break\n\n if (_phonet_rules[pos0] is None) or (\n _phonet_rules[pos0 + mode] is None\n ):\n # no conversion rule available\n pos0 += 3\n continue\n\n # check whole string\n matches0 = matches\n priority0 = 5\n rule = _phonet_rules[pos0]\n rule = rule[1:]\n\n while (\n rule\n and (\n src[\n i + matches0 : i + matches0 + 1\n ]\n == rule[0]\n )\n and (\n not rule[0].isdigit()\n or (rule in '(-<^$')\n )\n ):\n matches0 += 1\n rule = rule[1:]\n\n if rule and rule[0] == '(':\n # check an array of letters\n if src[\n i + matches0 : i + matches0 + 1\n ].isalpha() and (\n src[i + matches0] in rule[1:]\n ):\n matches0 += 1\n\n while rule and rule[0] != ')':\n rule = rule[1:]\n\n # if rule[0] == ')':\n rule = rule[1:]\n\n while rule and rule[0] == '-':\n # \"matches0\" is NOT decremented\n # because of\n # \"if (matches0 == matches)\"\n rule = rule[1:]\n\n if rule and rule[0] == '<':\n rule = rule[1:]\n\n if rule and rule[0].isdigit():\n priority0 = int(rule[0])\n rule = rule[1:]\n\n if (\n not rule\n or\n # rule == '^' is not possible here\n (\n (rule[0] == '$')\n and not src[\n i + matches0 : i + matches0 + 1\n ].isalpha()\n and (\n src[\n i\n + matches0 : i\n + matches0\n + 1\n ]\n != '.'\n )\n )\n ):\n if matches0 == matches:\n # this is only a partial string\n pos0 += 3\n continue\n\n if priority0 < priority:\n # priority is too low\n pos0 += 3\n continue\n\n # continuation rule found\n break\n\n pos0 += 3\n\n # end of \"while\"\n if (priority0 >= priority) and (\n (_phonet_rules[pos0] is not None)\n and (_phonet_rules[pos0][0] == char0)\n ):\n\n pos += 3\n continue\n\n # replace string\n if _phonet_rules[pos] and (\n '<' in _phonet_rules[pos][1:]\n ):\n priority0 = 1\n else:\n priority0 = 0\n\n rule = _phonet_rules[pos + mode]\n\n if (priority0 == 1) and (zeta == 0):\n # rule with '<' is applied\n if (\n (j > 0)\n and rule\n and (\n (dest[j - 1] == char)\n or (dest[j - 1] == rule[0])\n )\n ):\n j -= 1\n\n zeta0 = 1\n zeta += 1\n matches0 = 0\n\n while rule and src[i + matches0]:\n src = (\n src[0 : i + matches0]\n + rule[0]\n + src[i + matches0 + 1 :]\n )\n matches0 += 1\n rule = rule[1:]\n\n if matches0 < matches:\n src = (\n src[0 : i + matches0]\n + src[i + matches :]\n )\n\n char = src[i]\n else:\n i = i + matches - 1\n zeta = 0\n\n while len(rule) > 1:\n if (j == 0) or (dest[j - 1] != rule[0]):\n dest = (\n dest[0:j]\n + rule[0]\n + dest[min(len(dest), j + 1) :]\n )\n j += 1\n\n rule = rule[1:]\n\n # new \"current char\"\n if not rule:\n rule = ''\n char = ''\n else:\n char = rule[0]\n\n if (\n _phonet_rules[pos]\n and '^^' in _phonet_rules[pos][1:]\n ):\n if char:\n dest = (\n dest[0:j]\n + char\n + dest[min(len(dest), j + 1) :]\n )\n j += 1\n\n src = src[i + 1 :]\n i = 0\n zeta0 = 1\n\n break\n\n pos += 3\n\n if pos > end1 and start2 > 0:\n pos = start2\n start1 = start2\n end1 = end2\n start2 = -1\n end2 = -1\n\n if zeta0 == 0:\n if char and ((j == 0) or (dest[j - 1] != char)):\n # delete multiple letters only\n dest = (\n dest[0:j] + char + dest[min(j + 1, term_length) :]\n )\n j += 1\n\n i += 1\n zeta = 0\n\n dest = dest[0:j]\n\n return dest\n\n _initialize_phonet(lang)\n\n word = unicode_normalize('NFKC', text_type(word))\n return _phonet(word, mode, lang)", "response": "Encodes a word into a set of phonet code."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef stem(self, word):\n # lowercase, normalize, and compose\n word = normalize('NFC', text_type(word.lower()))\n\n r1_start = min(max(3, self._sb_r1(word)), len(word))\n\n # Step 1\n _r1 = word[r1_start:]\n if _r1[-7:] == 'erendes':\n word = word[:-7]\n elif _r1[-6:] in {'erende', 'hedens'}:\n word = word[:-6]\n elif _r1[-5:] in {\n 'ethed',\n 'erede',\n 'heden',\n 'heder',\n 'endes',\n 'ernes',\n 'erens',\n 'erets',\n }:\n word = word[:-5]\n elif _r1[-4:] in {\n 'ered',\n 'ende',\n 'erne',\n 'eren',\n 'erer',\n 'heds',\n 'enes',\n 'eres',\n 'eret',\n }:\n word = word[:-4]\n elif _r1[-3:] in {'hed', 'ene', 'ere', 'ens', 'ers', 'ets'}:\n word = word[:-3]\n elif _r1[-2:] in {'en', 'er', 'es', 'et'}:\n word = word[:-2]\n elif _r1[-1:] == 'e':\n word = word[:-1]\n elif _r1[-1:] == 's':\n if len(word) > 1 and word[-2] in self._s_endings:\n word = word[:-1]\n\n # Step 2\n if word[r1_start:][-2:] in {'gd', 'dt', 'gt', 'kt'}:\n word = word[:-1]\n\n # Step 3\n if word[-4:] == 'igst':\n word = word[:-2]\n\n _r1 = word[r1_start:]\n repeat_step2 = False\n if _r1[-4:] == 'elig':\n word = word[:-4]\n repeat_step2 = True\n elif _r1[-4:] == 'l\u00f8st':\n word = word[:-1]\n elif _r1[-3:] in {'lig', 'els'}:\n word = word[:-3]\n repeat_step2 = True\n elif _r1[-2:] == 'ig':\n word = word[:-2]\n repeat_step2 = True\n\n if repeat_step2:\n if word[r1_start:][-2:] in {'gd', 'dt', 'gt', 'kt'}:\n word = word[:-1]\n\n # Step 4\n if (\n len(word[r1_start:]) >= 1\n and len(word) >= 2\n and word[-1] == word[-2]\n and word[-1] not in self._vowels\n ):\n word = word[:-1]\n\n return word", "response": "Return Snowball Danish stem."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the Snowball German stem.", "response": "def stem(self, word, alternate_vowels=False):\n \"\"\"Return Snowball German stem.\n\n Parameters\n ----------\n word : str\n The word to stem\n alternate_vowels : bool\n Composes ae as \u00e4, oe as \u00f6, and ue as \u00fc before running the algorithm\n\n Returns\n -------\n str\n Word stem\n\n Examples\n --------\n >>> stmr = SnowballGerman()\n >>> stmr.stem('lesen')\n 'les'\n >>> stmr.stem('graues')\n 'grau'\n >>> stmr.stem('buchstabieren')\n 'buchstabi'\n\n \"\"\"\n # lowercase, normalize, and compose\n word = normalize('NFC', word.lower())\n word = word.replace('\u00df', 'ss')\n\n if len(word) > 2:\n for i in range(2, len(word)):\n if word[i] in self._vowels and word[i - 2] in self._vowels:\n if word[i - 1] == 'u':\n word = word[: i - 1] + 'U' + word[i:]\n elif word[i - 1] == 'y':\n word = word[: i - 1] + 'Y' + word[i:]\n\n if alternate_vowels:\n word = word.replace('ae', '\u00e4')\n word = word.replace('oe', '\u00f6')\n word = word.replace('que', 'Q')\n word = word.replace('ue', '\u00fc')\n word = word.replace('Q', 'que')\n\n r1_start = max(3, self._sb_r1(word))\n r2_start = self._sb_r2(word)\n\n # Step 1\n niss_flag = False\n if word[-3:] == 'ern':\n if len(word[r1_start:]) >= 3:\n word = word[:-3]\n elif word[-2:] == 'em':\n if len(word[r1_start:]) >= 2:\n word = word[:-2]\n elif word[-2:] == 'er':\n if len(word[r1_start:]) >= 2:\n word = word[:-2]\n elif word[-2:] == 'en':\n if len(word[r1_start:]) >= 2:\n word = word[:-2]\n niss_flag = True\n elif word[-2:] == 'es':\n if len(word[r1_start:]) >= 2:\n word = word[:-2]\n niss_flag = True\n elif word[-1:] == 'e':\n if len(word[r1_start:]) >= 1:\n word = word[:-1]\n niss_flag = True\n elif word[-1:] == 's':\n if (\n len(word[r1_start:]) >= 1\n and len(word) >= 2\n and word[-2] in self._s_endings\n ):\n word = word[:-1]\n\n if niss_flag and word[-4:] == 'niss':\n word = word[:-1]\n\n # Step 2\n if word[-3:] == 'est':\n if len(word[r1_start:]) >= 3:\n word = word[:-3]\n elif word[-2:] == 'en':\n if len(word[r1_start:]) >= 2:\n word = word[:-2]\n elif word[-2:] == 'er':\n if len(word[r1_start:]) >= 2:\n word = word[:-2]\n elif word[-2:] == 'st':\n if (\n len(word[r1_start:]) >= 2\n and len(word) >= 6\n and word[-3] in self._st_endings\n ):\n word = word[:-2]\n\n # Step 3\n if word[-4:] == 'isch':\n if len(word[r2_start:]) >= 4 and word[-5] != 'e':\n word = word[:-4]\n elif word[-4:] in {'lich', 'heit'}:\n if len(word[r2_start:]) >= 4:\n word = word[:-4]\n if word[-2:] in {'er', 'en'} and len(word[r1_start:]) >= 2:\n word = word[:-2]\n elif word[-4:] == 'keit':\n if len(word[r2_start:]) >= 4:\n word = word[:-4]\n if word[-4:] == 'lich' and len(word[r2_start:]) >= 4:\n word = word[:-4]\n elif word[-2:] == 'ig' and len(word[r2_start:]) >= 2:\n word = word[:-2]\n elif word[-3:] in {'end', 'ung'}:\n if len(word[r2_start:]) >= 3:\n word = word[:-3]\n if (\n word[-2:] == 'ig'\n and len(word[r2_start:]) >= 2\n and word[-3] != 'e'\n ):\n word = word[:-2]\n elif word[-2:] in {'ig', 'ik'}:\n if len(word[r2_start:]) >= 2 and word[-3] != 'e':\n word = word[:-2]\n\n # Change 'Y' and 'U' back to lowercase if survived stemming\n for i in range(0, len(word)):\n if word[i] == 'Y':\n word = word[:i] + 'y' + word[i + 1 :]\n elif word[i] == 'U':\n word = word[:i] + 'u' + word[i + 1 :]\n\n # Remove umlauts\n _umlauts = dict(zip((ord(_) for _ in '\u00e4\u00f6\u00fc'), 'aou'))\n word = word.translate(_umlauts)\n\n return word"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dist_abs(self, src, tar, max_offset=5):\n if not src:\n return len(tar)\n\n if not tar:\n return len(src)\n\n src_len = len(src)\n tar_len = len(tar)\n\n src_cur = 0\n tar_cur = 0\n lcss = 0\n local_cs = 0\n\n while (src_cur < src_len) and (tar_cur < tar_len):\n if src[src_cur] == tar[tar_cur]:\n local_cs += 1\n else:\n lcss += local_cs\n local_cs = 0\n if src_cur != tar_cur:\n src_cur = tar_cur = max(src_cur, tar_cur)\n for i in range(max_offset):\n if not (\n (src_cur + i < src_len) or (tar_cur + i < tar_len)\n ):\n break\n if (src_cur + i < src_len) and (\n src[src_cur + i] == tar[tar_cur]\n ):\n src_cur += i\n local_cs += 1\n break\n if (tar_cur + i < tar_len) and (\n src[src_cur] == tar[tar_cur + i]\n ):\n tar_cur += i\n local_cs += 1\n break\n\n src_cur += 1\n tar_cur += 1\n\n lcss += local_cs\n return round(max(src_len, tar_len) - lcss)", "response": "Return the simplest Sift4 distance between two strings src and tar."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef typo(src, tar, metric='euclidean', cost=(1, 1, 0.5, 0.5), layout='QWERTY'):\n return Typo().dist_abs(src, tar, metric, cost, layout)", "response": "Return the typo distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the normalized typo distance between two strings.", "response": "def dist_typo(\n src, tar, metric='euclidean', cost=(1, 1, 0.5, 0.5), layout='QWERTY'\n):\n \"\"\"Return the normalized typo distance between two strings.\n\n This is a wrapper for :py:meth:`Typo.dist`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n metric : str\n Supported values include: ``euclidean``, ``manhattan``,\n ``log-euclidean``, and ``log-manhattan``\n cost : tuple\n A 4-tuple representing the cost of the four possible edits: inserts,\n deletes, substitutions, and shift, respectively (by default:\n (1, 1, 0.5, 0.5)) The substitution & shift costs should be\n significantly less than the cost of an insertion & deletion unless a\n log metric is used.\n layout : str\n Name of the keyboard layout to use (Currently supported:\n ``QWERTY``, ``Dvorak``, ``AZERTY``, ``QWERTZ``)\n\n Returns\n -------\n float\n Normalized typo distance\n\n Examples\n --------\n >>> round(dist_typo('cat', 'hat'), 12)\n 0.527046283086\n >>> round(dist_typo('Niall', 'Neil'), 12)\n 0.565028142929\n >>> round(dist_typo('Colin', 'Cuilen'), 12)\n 0.569035609563\n >>> dist_typo('ATCG', 'TAGC')\n 0.625\n\n \"\"\"\n return Typo().dist(src, tar, metric, cost, layout)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef sim_typo(\n src, tar, metric='euclidean', cost=(1, 1, 0.5, 0.5), layout='QWERTY'\n):\n \"\"\"Return the normalized typo similarity between two strings.\n\n This is a wrapper for :py:meth:`Typo.sim`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n metric : str\n Supported values include: ``euclidean``, ``manhattan``,\n ``log-euclidean``, and ``log-manhattan``\n cost : tuple\n A 4-tuple representing the cost of the four possible edits: inserts,\n deletes, substitutions, and shift, respectively (by default:\n (1, 1, 0.5, 0.5)) The substitution & shift costs should be\n significantly less than the cost of an insertion & deletion unless a\n log metric is used.\n layout : str\n Name of the keyboard layout to use (Currently supported:\n ``QWERTY``, ``Dvorak``, ``AZERTY``, ``QWERTZ``)\n\n Returns\n -------\n float\n Normalized typo similarity\n\n Examples\n --------\n >>> round(sim_typo('cat', 'hat'), 12)\n 0.472953716914\n >>> round(sim_typo('Niall', 'Neil'), 12)\n 0.434971857071\n >>> round(sim_typo('Colin', 'Cuilen'), 12)\n 0.430964390437\n >>> sim_typo('ATCG', 'TAGC')\n 0.375\n\n \"\"\"\n return Typo().sim(src, tar, metric, cost, layout)", "response": "Return the normalized typo similarity between two strings."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dist_abs(\n self,\n src,\n tar,\n metric='euclidean',\n cost=(1, 1, 0.5, 0.5),\n layout='QWERTY',\n ):\n \"\"\"Return the typo distance between two strings.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n metric : str\n Supported values include: ``euclidean``, ``manhattan``,\n ``log-euclidean``, and ``log-manhattan``\n cost : tuple\n A 4-tuple representing the cost of the four possible edits:\n inserts, deletes, substitutions, and shift, respectively (by\n default: (1, 1, 0.5, 0.5)) The substitution & shift costs should be\n significantly less than the cost of an insertion & deletion unless\n a log metric is used.\n layout : str\n Name of the keyboard layout to use (Currently supported:\n ``QWERTY``, ``Dvorak``, ``AZERTY``, ``QWERTZ``)\n\n Returns\n -------\n float\n Typo distance\n\n Raises\n ------\n ValueError\n char not found in any keyboard layouts\n\n Examples\n --------\n >>> cmp = Typo()\n >>> cmp.dist_abs('cat', 'hat')\n 1.5811388\n >>> cmp.dist_abs('Niall', 'Neil')\n 2.8251407\n >>> cmp.dist_abs('Colin', 'Cuilen')\n 3.4142137\n >>> cmp.dist_abs('ATCG', 'TAGC')\n 2.5\n\n >>> cmp.dist_abs('cat', 'hat', metric='manhattan')\n 2.0\n >>> cmp.dist_abs('Niall', 'Neil', metric='manhattan')\n 3.0\n >>> cmp.dist_abs('Colin', 'Cuilen', metric='manhattan')\n 3.5\n >>> cmp.dist_abs('ATCG', 'TAGC', metric='manhattan')\n 2.5\n\n >>> cmp.dist_abs('cat', 'hat', metric='log-manhattan')\n 0.804719\n >>> cmp.dist_abs('Niall', 'Neil', metric='log-manhattan')\n 2.2424533\n >>> cmp.dist_abs('Colin', 'Cuilen', metric='log-manhattan')\n 2.2424533\n >>> cmp.dist_abs('ATCG', 'TAGC', metric='log-manhattan')\n 2.3465736\n\n \"\"\"\n ins_cost, del_cost, sub_cost, shift_cost = cost\n\n if src == tar:\n return 0.0\n if not src:\n return len(tar) * ins_cost\n if not tar:\n return len(src) * del_cost\n\n keyboard = self._keyboard[layout]\n lowercase = {item for sublist in keyboard[0] for item in sublist}\n uppercase = {item for sublist in keyboard[1] for item in sublist}\n\n def _kb_array_for_char(char):\n \"\"\"Return the keyboard layout that contains ch.\n\n Parameters\n ----------\n char : str\n The character to lookup\n\n Returns\n -------\n tuple\n A keyboard\n\n Raises\n ------\n ValueError\n char not found in any keyboard layouts\n\n \"\"\"\n if char in lowercase:\n return keyboard[0]\n elif char in uppercase:\n return keyboard[1]\n raise ValueError(char + ' not found in any keyboard layouts')\n\n def _substitution_cost(char1, char2):\n cost = sub_cost\n cost *= metric_dict[metric](char1, char2) + shift_cost * (\n _kb_array_for_char(char1) != _kb_array_for_char(char2)\n )\n return cost\n\n def _get_char_coord(char, kb_array):\n \"\"\"Return the row & column of char in the keyboard.\n\n Parameters\n ----------\n char : str\n The character to search for\n kb_array : tuple of tuples\n The array of key positions\n\n Returns\n -------\n tuple\n The row & column of the key\n\n \"\"\"\n for row in kb_array: # pragma: no branch\n if char in row:\n return kb_array.index(row), row.index(char)\n\n def _euclidean_keyboard_distance(char1, char2):\n row1, col1 = _get_char_coord(char1, _kb_array_for_char(char1))\n row2, col2 = _get_char_coord(char2, _kb_array_for_char(char2))\n return ((row1 - row2) ** 2 + (col1 - col2) ** 2) ** 0.5\n\n def _manhattan_keyboard_distance(char1, char2):\n row1, col1 = _get_char_coord(char1, _kb_array_for_char(char1))\n row2, col2 = _get_char_coord(char2, _kb_array_for_char(char2))\n return abs(row1 - row2) + abs(col1 - col2)\n\n def _log_euclidean_keyboard_distance(char1, char2):\n return log(1 + _euclidean_keyboard_distance(char1, char2))\n\n def _log_manhattan_keyboard_distance(char1, char2):\n return log(1 + _manhattan_keyboard_distance(char1, char2))\n\n metric_dict = {\n 'euclidean': _euclidean_keyboard_distance,\n 'manhattan': _manhattan_keyboard_distance,\n 'log-euclidean': _log_euclidean_keyboard_distance,\n 'log-manhattan': _log_manhattan_keyboard_distance,\n }\n\n d_mat = np_zeros((len(src) + 1, len(tar) + 1), dtype=np_float32)\n for i in range(len(src) + 1):\n d_mat[i, 0] = i * del_cost\n for j in range(len(tar) + 1):\n d_mat[0, j] = j * ins_cost\n\n for i in range(len(src)):\n for j in range(len(tar)):\n d_mat[i + 1, j + 1] = min(\n d_mat[i + 1, j] + ins_cost, # ins\n d_mat[i, j + 1] + del_cost, # del\n d_mat[i, j]\n + (\n _substitution_cost(src[i], tar[j])\n if src[i] != tar[j]\n else 0\n ), # sub/==\n )\n\n return d_mat[len(src), len(tar)]", "response": "Return the typo distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dist(\n self,\n src,\n tar,\n metric='euclidean',\n cost=(1, 1, 0.5, 0.5),\n layout='QWERTY',\n ):\n \"\"\"Return the normalized typo distance between two strings.\n\n This is typo distance, normalized to [0, 1].\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n metric : str\n Supported values include: ``euclidean``, ``manhattan``,\n ``log-euclidean``, and ``log-manhattan``\n cost : tuple\n A 4-tuple representing the cost of the four possible edits:\n inserts, deletes, substitutions, and shift, respectively (by\n default: (1, 1, 0.5, 0.5)) The substitution & shift costs should be\n significantly less than the cost of an insertion & deletion unless\n a log metric is used.\n layout : str\n Name of the keyboard layout to use (Currently supported:\n ``QWERTY``, ``Dvorak``, ``AZERTY``, ``QWERTZ``)\n\n Returns\n -------\n float\n Normalized typo distance\n\n Examples\n --------\n >>> cmp = Typo()\n >>> round(cmp.dist('cat', 'hat'), 12)\n 0.527046283086\n >>> round(cmp.dist('Niall', 'Neil'), 12)\n 0.565028142929\n >>> round(cmp.dist('Colin', 'Cuilen'), 12)\n 0.569035609563\n >>> cmp.dist('ATCG', 'TAGC')\n 0.625\n\n \"\"\"\n if src == tar:\n return 0.0\n ins_cost, del_cost = cost[:2]\n return self.dist_abs(src, tar, metric, cost, layout) / (\n max(len(src) * del_cost, len(tar) * ins_cost)\n )", "response": "Return the normalized typo distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef manhattan(src, tar, qval=2, normalized=False, alphabet=None):\n return Manhattan().dist_abs(src, tar, qval, normalized, alphabet)", "response": "Return the Manhattan distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dist_manhattan(src, tar, qval=2, alphabet=None):\n return Manhattan().dist(src, tar, qval, alphabet)", "response": "Return the normalized Manhattan distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the normalized Manhattan similarity of two strings.", "response": "def sim_manhattan(src, tar, qval=2, alphabet=None):\n \"\"\"Return the normalized Manhattan similarity of two strings.\n\n This is a wrapper for :py:meth:`Manhattan.sim`.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n alphabet : collection or int\n The values or size of the alphabet\n\n Returns\n -------\n float\n The normalized Manhattan similarity\n\n Examples\n --------\n >>> sim_manhattan('cat', 'hat')\n 0.5\n >>> round(sim_manhattan('Niall', 'Neil'), 12)\n 0.363636363636\n >>> round(sim_manhattan('Colin', 'Cuilen'), 12)\n 0.307692307692\n >>> sim_manhattan('ATCG', 'TAGC')\n 0.0\n\n \"\"\"\n return Manhattan().sim(src, tar, qval, alphabet)"} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sim_jaro_winkler(\n src,\n tar,\n qval=1,\n mode='winkler',\n long_strings=False,\n boost_threshold=0.7,\n scaling_factor=0.1,\n):\n \"\"\"Return the Jaro or Jaro-Winkler similarity of two strings.\n\n This is a wrapper for :py:meth:`JaroWinkler.sim`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n qval : int\n The length of each q-gram (defaults to 1: character-wise matching)\n mode : str\n Indicates which variant of this distance metric to compute:\n\n - ``winkler`` -- computes the Jaro-Winkler distance (default) which\n increases the score for matches near the start of the word\n - ``jaro`` -- computes the Jaro distance\n\n long_strings : bool\n Set to True to \"Increase the probability of a match when the number of\n matched characters is large. This option allows for a little more\n tolerance when the strings are large. It is not an appropriate test\n when comparing fixedlength fields such as phone and social security\n numbers.\" (Used in 'winkler' mode only.)\n boost_threshold : float\n A value between 0 and 1, below which the Winkler boost is not applied\n (defaults to 0.7). (Used in 'winkler' mode only.)\n scaling_factor : float\n A value between 0 and 0.25, indicating by how much to boost scores for\n matching prefixes (defaults to 0.1). (Used in 'winkler' mode only.)\n\n Returns\n -------\n float\n Jaro or Jaro-Winkler similarity\n\n Examples\n --------\n >>> round(sim_jaro_winkler('cat', 'hat'), 12)\n 0.777777777778\n >>> round(sim_jaro_winkler('Niall', 'Neil'), 12)\n 0.805\n >>> round(sim_jaro_winkler('aluminum', 'Catalan'), 12)\n 0.60119047619\n >>> round(sim_jaro_winkler('ATCG', 'TAGC'), 12)\n 0.833333333333\n\n >>> round(sim_jaro_winkler('cat', 'hat', mode='jaro'), 12)\n 0.777777777778\n >>> round(sim_jaro_winkler('Niall', 'Neil', mode='jaro'), 12)\n 0.783333333333\n >>> round(sim_jaro_winkler('aluminum', 'Catalan', mode='jaro'), 12)\n 0.60119047619\n >>> round(sim_jaro_winkler('ATCG', 'TAGC', mode='jaro'), 12)\n 0.833333333333\n\n \"\"\"\n return JaroWinkler().sim(\n src, tar, qval, mode, long_strings, boost_threshold, scaling_factor\n )", "response": "Return the Jaro or Jaro - Winkler similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dist_jaro_winkler(\n src,\n tar,\n qval=1,\n mode='winkler',\n long_strings=False,\n boost_threshold=0.7,\n scaling_factor=0.1,\n):\n \"\"\"Return the Jaro or Jaro-Winkler distance between two strings.\n\n This is a wrapper for :py:meth:`JaroWinkler.dist`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n qval : int\n The length of each q-gram (defaults to 1: character-wise matching)\n mode : str\n Indicates which variant of this distance metric to compute:\n\n - ``winkler`` -- computes the Jaro-Winkler distance (default) which\n increases the score for matches near the start of the word\n - ``jaro`` -- computes the Jaro distance\n\n long_strings : bool\n Set to True to \"Increase the probability of a match when the number of\n matched characters is large. This option allows for a little more\n tolerance when the strings are large. It is not an appropriate test\n when comparing fixedlength fields such as phone and social security\n numbers.\" (Used in 'winkler' mode only.)\n boost_threshold : float\n A value between 0 and 1, below which the Winkler boost is not applied\n (defaults to 0.7). (Used in 'winkler' mode only.)\n scaling_factor : float\n A value between 0 and 0.25, indicating by how much to boost scores for\n matching prefixes (defaults to 0.1). (Used in 'winkler' mode only.)\n\n Returns\n -------\n float\n Jaro or Jaro-Winkler distance\n\n Examples\n --------\n >>> round(dist_jaro_winkler('cat', 'hat'), 12)\n 0.222222222222\n >>> round(dist_jaro_winkler('Niall', 'Neil'), 12)\n 0.195\n >>> round(dist_jaro_winkler('aluminum', 'Catalan'), 12)\n 0.39880952381\n >>> round(dist_jaro_winkler('ATCG', 'TAGC'), 12)\n 0.166666666667\n\n >>> round(dist_jaro_winkler('cat', 'hat', mode='jaro'), 12)\n 0.222222222222\n >>> round(dist_jaro_winkler('Niall', 'Neil', mode='jaro'), 12)\n 0.216666666667\n >>> round(dist_jaro_winkler('aluminum', 'Catalan', mode='jaro'), 12)\n 0.39880952381\n >>> round(dist_jaro_winkler('ATCG', 'TAGC', mode='jaro'), 12)\n 0.166666666667\n\n \"\"\"\n return JaroWinkler().dist(\n src, tar, qval, mode, long_strings, boost_threshold, scaling_factor\n )", "response": "Return the Jaro or Jaro - Winkler distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef sim(\n self,\n src,\n tar,\n qval=1,\n mode='winkler',\n long_strings=False,\n boost_threshold=0.7,\n scaling_factor=0.1,\n ):\n \"\"\"Return the Jaro or Jaro-Winkler similarity of two strings.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n qval : int\n The length of each q-gram (defaults to 1: character-wise matching)\n mode : str\n Indicates which variant of this distance metric to compute:\n\n - ``winkler`` -- computes the Jaro-Winkler distance (default)\n which increases the score for matches near the start of the\n word\n - ``jaro`` -- computes the Jaro distance\n\n long_strings : bool\n Set to True to \"Increase the probability of a match when the number\n of matched characters is large. This option allows for a little\n more tolerance when the strings are large. It is not an appropriate\n test when comparing fixed length fields such as phone and social\n security numbers.\" (Used in 'winkler' mode only.)\n boost_threshold : float\n A value between 0 and 1, below which the Winkler boost is not\n applied (defaults to 0.7). (Used in 'winkler' mode only.)\n scaling_factor : float\n A value between 0 and 0.25, indicating by how much to boost scores\n for matching prefixes (defaults to 0.1). (Used in 'winkler' mode\n only.)\n\n Returns\n -------\n float\n Jaro or Jaro-Winkler similarity\n\n Raises\n ------\n ValueError\n Unsupported boost_threshold assignment; boost_threshold must be\n between 0 and 1.\n ValueError\n Unsupported scaling_factor assignment; scaling_factor must be\n between 0 and 0.25.'\n\n Examples\n --------\n >>> round(sim_jaro_winkler('cat', 'hat'), 12)\n 0.777777777778\n >>> round(sim_jaro_winkler('Niall', 'Neil'), 12)\n 0.805\n >>> round(sim_jaro_winkler('aluminum', 'Catalan'), 12)\n 0.60119047619\n >>> round(sim_jaro_winkler('ATCG', 'TAGC'), 12)\n 0.833333333333\n\n >>> round(sim_jaro_winkler('cat', 'hat', mode='jaro'), 12)\n 0.777777777778\n >>> round(sim_jaro_winkler('Niall', 'Neil', mode='jaro'), 12)\n 0.783333333333\n >>> round(sim_jaro_winkler('aluminum', 'Catalan', mode='jaro'), 12)\n 0.60119047619\n >>> round(sim_jaro_winkler('ATCG', 'TAGC', mode='jaro'), 12)\n 0.833333333333\n\n \"\"\"\n if mode == 'winkler':\n if boost_threshold > 1 or boost_threshold < 0:\n raise ValueError(\n 'Unsupported boost_threshold assignment; '\n + 'boost_threshold must be between 0 and 1.'\n )\n if scaling_factor > 0.25 or scaling_factor < 0:\n raise ValueError(\n 'Unsupported scaling_factor assignment; '\n + 'scaling_factor must be between 0 and 0.25.'\n )\n\n if src == tar:\n return 1.0\n\n src = QGrams(src.strip(), qval)._ordered_list\n tar = QGrams(tar.strip(), qval)._ordered_list\n\n lens = len(src)\n lent = len(tar)\n\n # If either string is blank - return - added in Version 2\n if lens == 0 or lent == 0:\n return 0.0\n\n if lens > lent:\n search_range = lens\n minv = lent\n else:\n search_range = lent\n minv = lens\n\n # Zero out the flags\n src_flag = [0] * search_range\n tar_flag = [0] * search_range\n search_range = max(0, search_range // 2 - 1)\n\n # Looking only within the search range,\n # count and flag the matched pairs.\n num_com = 0\n yl1 = lent - 1\n for i in range(lens):\n low_lim = (i - search_range) if (i >= search_range) else 0\n hi_lim = (i + search_range) if ((i + search_range) <= yl1) else yl1\n for j in range(low_lim, hi_lim + 1):\n if (tar_flag[j] == 0) and (tar[j] == src[i]):\n tar_flag[j] = 1\n src_flag[i] = 1\n num_com += 1\n break\n\n # If no characters in common - return\n if num_com == 0:\n return 0.0\n\n # Count the number of transpositions\n k = n_trans = 0\n for i in range(lens):\n if src_flag[i] != 0:\n j = 0\n for j in range(k, lent): # pragma: no branch\n if tar_flag[j] != 0:\n k = j + 1\n break\n if src[i] != tar[j]:\n n_trans += 1\n n_trans //= 2\n\n # Main weight computation for Jaro distance\n weight = (\n num_com / lens + num_com / lent + (num_com - n_trans) / num_com\n )\n weight /= 3.0\n\n # Continue to boost the weight if the strings are similar\n # This is the Winkler portion of Jaro-Winkler distance\n if mode == 'winkler' and weight > boost_threshold:\n\n # Adjust for having up to the first 4 characters in common\n j = 4 if (minv >= 4) else minv\n i = 0\n while (i < j) and (src[i] == tar[i]):\n i += 1\n weight += i * scaling_factor * (1.0 - weight)\n\n # Optionally adjust for long strings.\n\n # After agreeing beginning chars, at least two more must agree and\n # the agreeing characters must be > .5 of remaining characters.\n if (\n long_strings\n and (minv > 4)\n and (num_com > i + 1)\n and (2 * num_com >= minv + i)\n ):\n weight += (1.0 - weight) * (\n (num_com - i - 1) / (lens + lent - i * 2 + 2)\n )\n\n return weight", "response": "Return the Jaro or Jaro - Winkler similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef hamming(src, tar, diff_lens=True):\n return Hamming().dist_abs(src, tar, diff_lens)", "response": "Returns the Hamming distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dist_hamming(src, tar, diff_lens=True):\n return Hamming().dist(src, tar, diff_lens)", "response": "Return the normalized Hamming distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sim_hamming(src, tar, diff_lens=True):\n return Hamming().sim(src, tar, diff_lens)", "response": "Return the normalized Hamming similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dist_abs(self, src, tar, diff_lens=True):\n if not diff_lens and len(src) != len(tar):\n raise ValueError(\n 'Undefined for sequences of unequal length; set diff_lens '\n + 'to True for Hamming distance between strings of unequal '\n + 'lengths.'\n )\n\n hdist = 0\n if diff_lens:\n hdist += abs(len(src) - len(tar))\n hdist += sum(c1 != c2 for c1, c2 in zip(src, tar))\n\n return hdist", "response": "Return the Hamming distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef dist(self, src, tar, diff_lens=True):\n if src == tar:\n return 0.0\n return self.dist_abs(src, tar, diff_lens) / max(len(src), len(tar))", "response": "Return the normalized Hamming distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the Metaphone code for a word.", "response": "def encode(self, word, max_length=-1):\n \"\"\"Return the Metaphone code for a word.\n\n Based on Lawrence Philips' Pick BASIC code from 1990\n :cite:`Philips:1990`, as described in :cite:`Philips:1990b`.\n This incorporates some corrections to the above code, particularly\n some of those suggested by Michael Kuhn in :cite:`Kuhn:1995`.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The maximum length of the returned Metaphone code (defaults to 64,\n but in Philips' original implementation this was 4)\n\n Returns\n -------\n str\n The Metaphone value\n\n Examples\n --------\n >>> pe = Metaphone()\n >>> pe.encode('Christopher')\n 'KRSTFR'\n >>> pe.encode('Niall')\n 'NL'\n >>> pe.encode('Smith')\n 'SM0'\n >>> pe.encode('Schmidt')\n 'SKMTT'\n\n \"\"\"\n # Require a max_length of at least 4\n if max_length != -1:\n max_length = max(4, max_length)\n else:\n max_length = 64\n\n # As in variable sound--those modified by adding an \"h\"\n ename = ''.join(c for c in word.upper() if c.isalnum())\n ename = ename.replace('\u00df', 'SS')\n\n # Delete non-alphanumeric characters and make all caps\n if not ename:\n return ''\n if ename[0:2] in {'PN', 'AE', 'KN', 'GN', 'WR'}:\n ename = ename[1:]\n elif ename[0] == 'X':\n ename = 'S' + ename[1:]\n elif ename[0:2] == 'WH':\n ename = 'W' + ename[2:]\n\n # Convert to metaphone\n elen = len(ename) - 1\n metaph = ''\n for i in range(len(ename)):\n if len(metaph) >= max_length:\n break\n if (\n ename[i] not in {'G', 'T'}\n and i > 0\n and ename[i - 1] == ename[i]\n ):\n continue\n\n if ename[i] in self._uc_v_set and i == 0:\n metaph = ename[i]\n\n elif ename[i] == 'B':\n if i != elen or ename[i - 1] != 'M':\n metaph += ename[i]\n\n elif ename[i] == 'C':\n if not (\n i > 0\n and ename[i - 1] == 'S'\n and ename[i + 1 : i + 2] in self._frontv\n ):\n if ename[i + 1 : i + 3] == 'IA':\n metaph += 'X'\n elif ename[i + 1 : i + 2] in self._frontv:\n metaph += 'S'\n elif i > 0 and ename[i - 1 : i + 2] == 'SCH':\n metaph += 'K'\n elif ename[i + 1 : i + 2] == 'H':\n if (\n i == 0\n and i + 1 < elen\n and ename[i + 2 : i + 3] not in self._uc_v_set\n ):\n metaph += 'K'\n else:\n metaph += 'X'\n else:\n metaph += 'K'\n\n elif ename[i] == 'D':\n if (\n ename[i + 1 : i + 2] == 'G'\n and ename[i + 2 : i + 3] in self._frontv\n ):\n metaph += 'J'\n else:\n metaph += 'T'\n\n elif ename[i] == 'G':\n if ename[i + 1 : i + 2] == 'H' and not (\n i + 1 == elen or ename[i + 2 : i + 3] not in self._uc_v_set\n ):\n continue\n elif i > 0 and (\n (i + 1 == elen and ename[i + 1] == 'N')\n or (i + 3 == elen and ename[i + 1 : i + 4] == 'NED')\n ):\n continue\n elif (\n i - 1 > 0\n and i + 1 <= elen\n and ename[i - 1] == 'D'\n and ename[i + 1] in self._frontv\n ):\n continue\n elif ename[i + 1 : i + 2] == 'G':\n continue\n elif ename[i + 1 : i + 2] in self._frontv:\n if i == 0 or ename[i - 1] != 'G':\n metaph += 'J'\n else:\n metaph += 'K'\n else:\n metaph += 'K'\n\n elif ename[i] == 'H':\n if (\n i > 0\n and ename[i - 1] in self._uc_v_set\n and ename[i + 1 : i + 2] not in self._uc_v_set\n ):\n continue\n elif i > 0 and ename[i - 1] in self._varson:\n continue\n else:\n metaph += 'H'\n\n elif ename[i] in {'F', 'J', 'L', 'M', 'N', 'R'}:\n metaph += ename[i]\n\n elif ename[i] == 'K':\n if i > 0 and ename[i - 1] == 'C':\n continue\n else:\n metaph += 'K'\n\n elif ename[i] == 'P':\n if ename[i + 1 : i + 2] == 'H':\n metaph += 'F'\n else:\n metaph += 'P'\n\n elif ename[i] == 'Q':\n metaph += 'K'\n\n elif ename[i] == 'S':\n if (\n i > 0\n and i + 2 <= elen\n and ename[i + 1] == 'I'\n and ename[i + 2] in 'OA'\n ):\n metaph += 'X'\n elif ename[i + 1 : i + 2] == 'H':\n metaph += 'X'\n else:\n metaph += 'S'\n\n elif ename[i] == 'T':\n if (\n i > 0\n and i + 2 <= elen\n and ename[i + 1] == 'I'\n and ename[i + 2] in {'A', 'O'}\n ):\n metaph += 'X'\n elif ename[i + 1 : i + 2] == 'H':\n metaph += '0'\n elif ename[i + 1 : i + 3] != 'CH':\n if ename[i - 1 : i] != 'T':\n metaph += 'T'\n\n elif ename[i] == 'V':\n metaph += 'F'\n\n elif ename[i] in 'WY':\n if ename[i + 1 : i + 2] in self._uc_v_set:\n metaph += ename[i]\n\n elif ename[i] == 'X':\n metaph += 'KS'\n\n elif ename[i] == 'Z':\n metaph += 'S'\n\n return metaph"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef pshp_soundex_last(lname, max_length=4, german=False):\n return PSHPSoundexLast().encode(lname, max_length, german)", "response": "This function returns the PSHP Soundex Last code for a given last name."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef encode(self, lname, max_length=4, german=False):\n lname = unicode_normalize('NFKD', text_type(lname.upper()))\n lname = lname.replace('\u00df', 'SS')\n lname = ''.join(c for c in lname if c in self._uc_set)\n\n # A. Prefix treatment\n if lname[:3] == 'VON' or lname[:3] == 'VAN':\n lname = lname[3:].strip()\n\n # The rule implemented below says \"MC, MAC become 1\". I believe it\n # meant to say they become M except in German data (where superscripted\n # 1 indicates \"except in German data\"). It doesn't make sense for them\n # to become 1 (BPFV -> 1) or to apply outside German. Unfortunately,\n # both articles have this error(?).\n if not german:\n if lname[:3] == 'MAC':\n lname = 'M' + lname[3:]\n elif lname[:2] == 'MC':\n lname = 'M' + lname[2:]\n\n # The non-German-only rule to strip ' is unnecessary due to filtering\n\n if lname[:1] in {'E', 'I', 'O', 'U'}:\n lname = 'A' + lname[1:]\n elif lname[:2] in {'GE', 'GI', 'GY'}:\n lname = 'J' + lname[1:]\n elif lname[:2] in {'CE', 'CI', 'CY'}:\n lname = 'S' + lname[1:]\n elif lname[:3] == 'CHR':\n lname = 'K' + lname[1:]\n elif lname[:1] == 'C' and lname[:2] != 'CH':\n lname = 'K' + lname[1:]\n\n if lname[:2] == 'KN':\n lname = 'N' + lname[1:]\n elif lname[:2] == 'PH':\n lname = 'F' + lname[1:]\n elif lname[:3] in {'WIE', 'WEI'}:\n lname = 'V' + lname[1:]\n\n if german and lname[:1] in {'W', 'M', 'Y', 'Z'}:\n lname = {'W': 'V', 'M': 'N', 'Y': 'J', 'Z': 'S'}[lname[0]] + lname[\n 1:\n ]\n\n code = lname[:1]\n\n # B. Postfix treatment\n if german: # moved from end of postfix treatment due to blocking\n if lname[-3:] == 'TES':\n lname = lname[:-3]\n elif lname[-2:] == 'TS':\n lname = lname[:-2]\n if lname[-3:] == 'TZE':\n lname = lname[:-3]\n elif lname[-2:] == 'ZE':\n lname = lname[:-2]\n if lname[-1:] == 'Z':\n lname = lname[:-1]\n elif lname[-2:] == 'TE':\n lname = lname[:-2]\n\n if lname[-1:] == 'R':\n lname = lname[:-1] + 'N'\n elif lname[-2:] in {'SE', 'CE'}:\n lname = lname[:-2]\n if lname[-2:] == 'SS':\n lname = lname[:-2]\n elif lname[-1:] == 'S':\n lname = lname[:-1]\n\n if not german:\n l5_repl = {'STOWN': 'SAWON', 'MPSON': 'MASON'}\n l4_repl = {\n 'NSEN': 'ASEN',\n 'MSON': 'ASON',\n 'STEN': 'SAEN',\n 'STON': 'SAON',\n }\n if lname[-5:] in l5_repl:\n lname = lname[:-5] + l5_repl[lname[-5:]]\n elif lname[-4:] in l4_repl:\n lname = lname[:-4] + l4_repl[lname[-4:]]\n\n if lname[-2:] in {'NG', 'ND'}:\n lname = lname[:-1]\n if not german and lname[-3:] in {'GAN', 'GEN'}:\n lname = lname[:-3] + 'A' + lname[-2:]\n\n # C. Infix Treatment\n lname = lname.replace('CK', 'C')\n lname = lname.replace('SCH', 'S')\n lname = lname.replace('DT', 'T')\n lname = lname.replace('ND', 'N')\n lname = lname.replace('NG', 'N')\n lname = lname.replace('LM', 'M')\n lname = lname.replace('MN', 'M')\n lname = lname.replace('WIE', 'VIE')\n lname = lname.replace('WEI', 'VEI')\n\n # D. Soundexing\n # code for X & Y are unspecified, but presumably are 2 & 0\n\n lname = lname.translate(self._trans)\n lname = self._delete_consecutive_repeats(lname)\n\n code += lname[1:]\n code = code.replace('0', '') # rule 1\n\n if max_length != -1:\n if len(code) < max_length:\n code += '0' * (max_length - len(code))\n else:\n code = code[:max_length]\n\n return code", "response": "This function encodes a last name into a set of PSHP soundex and viewex codes."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fingerprint(self, word):\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = ''.join(c for c in word if c in self._letters)\n start = word[0:1]\n consonant_part = ''\n vowel_part = ''\n\n # add consonants & vowels to to separate strings\n # (omitting the first char & duplicates)\n for char in word[1:]:\n if char != start:\n if char in self._vowels:\n if char not in vowel_part:\n vowel_part += char\n elif char not in consonant_part:\n consonant_part += char\n # return the first char followed by consonants followed by vowels\n return start + consonant_part + vowel_part", "response": "Return the skeleton key for the given word."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef nysiis(word, max_length=6, modified=False):\n return NYSIIS().encode(word, max_length, modified)", "response": "Return the NYSIIS code for a word."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nencoding a string into a NYSIIS code.", "response": "def encode(self, word, max_length=6, modified=False):\n \"\"\"Return the NYSIIS code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The maximum length (default 6) of the code to return\n modified : bool\n Indicates whether to use USDA modified NYSIIS\n\n Returns\n -------\n str\n The NYSIIS value\n\n Examples\n --------\n >>> pe = NYSIIS()\n >>> pe.encode('Christopher')\n 'CRASTA'\n >>> pe.encode('Niall')\n 'NAL'\n >>> pe.encode('Smith')\n 'SNAT'\n >>> pe.encode('Schmidt')\n 'SNAD'\n\n >>> pe.encode('Christopher', max_length=-1)\n 'CRASTAFAR'\n\n >>> pe.encode('Christopher', max_length=8, modified=True)\n 'CRASTAFA'\n >>> pe.encode('Niall', max_length=8, modified=True)\n 'NAL'\n >>> pe.encode('Smith', max_length=8, modified=True)\n 'SNAT'\n >>> pe.encode('Schmidt', max_length=8, modified=True)\n 'SNAD'\n\n \"\"\"\n # Require a max_length of at least 6\n if max_length > -1:\n max_length = max(6, max_length)\n\n word = ''.join(c for c in word.upper() if c.isalpha())\n word = word.replace('\u00df', 'SS')\n\n # exit early if there are no alphas\n if not word:\n return ''\n\n original_first_char = word[0]\n\n if word[:3] == 'MAC':\n word = 'MCC' + word[3:]\n elif word[:2] == 'KN':\n word = 'NN' + word[2:]\n elif word[:1] == 'K':\n word = 'C' + word[1:]\n elif word[:2] in {'PH', 'PF'}:\n word = 'FF' + word[2:]\n elif word[:3] == 'SCH':\n word = 'SSS' + word[3:]\n elif modified:\n if word[:2] == 'WR':\n word = 'RR' + word[2:]\n elif word[:2] == 'RH':\n word = 'RR' + word[2:]\n elif word[:2] == 'DG':\n word = 'GG' + word[2:]\n elif word[:1] in self._uc_v_set:\n word = 'A' + word[1:]\n\n if modified and word[-1:] in {'S', 'Z'}:\n word = word[:-1]\n\n if (\n word[-2:] == 'EE'\n or word[-2:] == 'IE'\n or (modified and word[-2:] == 'YE')\n ):\n word = word[:-2] + 'Y'\n elif word[-2:] in {'DT', 'RT', 'RD'}:\n word = word[:-2] + 'D'\n elif word[-2:] in {'NT', 'ND'}:\n word = word[:-2] + ('N' if modified else 'D')\n elif modified:\n if word[-2:] == 'IX':\n word = word[:-2] + 'ICK'\n elif word[-2:] == 'EX':\n word = word[:-2] + 'ECK'\n elif word[-2:] in {'JR', 'SR'}:\n return 'ERROR'\n\n key = word[:1]\n\n skip = 0\n for i in range(1, len(word)):\n if i >= len(word):\n continue\n elif skip:\n skip -= 1\n continue\n elif word[i : i + 2] == 'EV':\n word = word[:i] + 'AF' + word[i + 2 :]\n skip = 1\n elif word[i] in self._uc_v_set:\n word = word[:i] + 'A' + word[i + 1 :]\n elif modified and i != len(word) - 1 and word[i] == 'Y':\n word = word[:i] + 'A' + word[i + 1 :]\n elif word[i] == 'Q':\n word = word[:i] + 'G' + word[i + 1 :]\n elif word[i] == 'Z':\n word = word[:i] + 'S' + word[i + 1 :]\n elif word[i] == 'M':\n word = word[:i] + 'N' + word[i + 1 :]\n elif word[i : i + 2] == 'KN':\n word = word[:i] + 'N' + word[i + 2 :]\n elif word[i] == 'K':\n word = word[:i] + 'C' + word[i + 1 :]\n elif modified and i == len(word) - 3 and word[i : i + 3] == 'SCH':\n word = word[:i] + 'SSA'\n skip = 2\n elif word[i : i + 3] == 'SCH':\n word = word[:i] + 'SSS' + word[i + 3 :]\n skip = 2\n elif modified and i == len(word) - 2 and word[i : i + 2] == 'SH':\n word = word[:i] + 'SA'\n skip = 1\n elif word[i : i + 2] == 'SH':\n word = word[:i] + 'SS' + word[i + 2 :]\n skip = 1\n elif word[i : i + 2] == 'PH':\n word = word[:i] + 'FF' + word[i + 2 :]\n skip = 1\n elif modified and word[i : i + 3] == 'GHT':\n word = word[:i] + 'TTT' + word[i + 3 :]\n skip = 2\n elif modified and word[i : i + 2] == 'DG':\n word = word[:i] + 'GG' + word[i + 2 :]\n skip = 1\n elif modified and word[i : i + 2] == 'WR':\n word = word[:i] + 'RR' + word[i + 2 :]\n skip = 1\n elif word[i] == 'H' and (\n word[i - 1] not in self._uc_v_set\n or word[i + 1 : i + 2] not in self._uc_v_set\n ):\n word = word[:i] + word[i - 1] + word[i + 1 :]\n elif word[i] == 'W' and word[i - 1] in self._uc_v_set:\n word = word[:i] + word[i - 1] + word[i + 1 :]\n\n if word[i : i + skip + 1] != key[-1:]:\n key += word[i : i + skip + 1]\n\n key = self._delete_consecutive_repeats(key)\n\n if key[-1:] == 'S':\n key = key[:-1]\n if key[-2:] == 'AY':\n key = key[:-2] + 'Y'\n if key[-1:] == 'A':\n key = key[:-1]\n if modified and key[:1] == 'A':\n key = original_first_char + key[1:]\n\n if max_length > 0:\n key = key[:max_length]\n\n return key"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef chebyshev(src, tar, qval=2, alphabet=None):\n return Chebyshev().dist_abs(src, tar, qval, alphabet)", "response": "r Return the Chebyshev distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dist_abs(self, src, tar, qval=2, alphabet=None):\n return super(self.__class__, self).dist_abs(\n src, tar, qval, float('inf'), False, alphabet\n )", "response": "Return the Chebyshev distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef mean_pairwise_similarity(\n collection, metric=sim, mean_func=hmean, symmetric=False\n):\n \"\"\"Calculate the mean pairwise similarity of a collection of strings.\n\n Takes the mean of the pairwise similarity between each member of a\n collection, optionally in both directions (for asymmetric similarity\n metrics.\n\n Parameters\n ----------\n collection : list\n A collection of terms or a string that can be split\n metric : function\n A similarity metric function\n mean_func : function\n A mean function that takes a list of values and returns a float\n symmetric : bool\n Set to True if all pairwise similarities should be calculated in both\n directions\n\n Returns\n -------\n float\n The mean pairwise similarity of a collection of strings\n\n Raises\n ------\n ValueError\n mean_func must be a function\n ValueError\n metric must be a function\n ValueError\n collection is neither a string nor iterable type\n ValueError\n collection has fewer than two members\n\n Examples\n --------\n >>> round(mean_pairwise_similarity(['Christopher', 'Kristof',\n ... 'Christobal']), 12)\n 0.519801980198\n >>> round(mean_pairwise_similarity(['Niall', 'Neal', 'Neil']), 12)\n 0.545454545455\n\n \"\"\"\n if not callable(mean_func):\n raise ValueError('mean_func must be a function')\n if not callable(metric):\n raise ValueError('metric must be a function')\n\n if hasattr(collection, 'split'):\n collection = collection.split()\n if not hasattr(collection, '__iter__'):\n raise ValueError('collection is neither a string nor iterable type')\n elif len(collection) < 2:\n raise ValueError('collection has fewer than two members')\n\n collection = list(collection)\n\n pairwise_values = []\n\n for i in range(len(collection)):\n for j in range(i + 1, len(collection)):\n pairwise_values.append(metric(collection[i], collection[j]))\n if symmetric:\n pairwise_values.append(metric(collection[j], collection[i]))\n\n return mean_func(pairwise_values)", "response": "Calculates the mean pairwise similarity between each member of a list of strings."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\ncalculating the pairwise similarity statistics of a collection of strings.", "response": "def pairwise_similarity_statistics(\n src_collection,\n tar_collection,\n metric=sim,\n mean_func=amean,\n symmetric=False,\n):\n \"\"\"Calculate the pairwise similarity statistics a collection of strings.\n\n Calculate pairwise similarities among members of two collections,\n returning the maximum, minimum, mean (according to a supplied function,\n arithmetic mean, by default), and (population) standard deviation\n of those similarities.\n\n Parameters\n ----------\n src_collection : list\n A collection of terms or a string that can be split\n tar_collection : list\n A collection of terms or a string that can be split\n metric : function\n A similarity metric function\n mean_func : function\n A mean function that takes a list of values and returns a float\n symmetric : bool\n Set to True if all pairwise similarities should be calculated in both\n directions\n\n Returns\n -------\n tuple\n The max, min, mean, and standard deviation of similarities\n\n Raises\n ------\n ValueError\n mean_func must be a function\n ValueError\n metric must be a function\n ValueError\n src_collection is neither a string nor iterable\n ValueError\n tar_collection is neither a string nor iterable\n\n Example\n -------\n >>> tuple(round(_, 12) for _ in pairwise_similarity_statistics(\n ... ['Christopher', 'Kristof', 'Christobal'], ['Niall', 'Neal', 'Neil']))\n (0.2, 0.0, 0.118614718615, 0.075070477184)\n\n \"\"\"\n if not callable(mean_func):\n raise ValueError('mean_func must be a function')\n if not callable(metric):\n raise ValueError('metric must be a function')\n\n if hasattr(src_collection, 'split'):\n src_collection = src_collection.split()\n if not hasattr(src_collection, '__iter__'):\n raise ValueError('src_collection is neither a string nor iterable')\n\n if hasattr(tar_collection, 'split'):\n tar_collection = tar_collection.split()\n if not hasattr(tar_collection, '__iter__'):\n raise ValueError('tar_collection is neither a string nor iterable')\n\n src_collection = list(src_collection)\n tar_collection = list(tar_collection)\n\n pairwise_values = []\n\n for src in src_collection:\n for tar in tar_collection:\n pairwise_values.append(metric(src, tar))\n if symmetric:\n pairwise_values.append(metric(tar, src))\n\n return (\n max(pairwise_values),\n min(pairwise_values),\n mean_func(pairwise_values),\n std(pairwise_values, mean_func, 0),\n )"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef stem(self, word, early_english=False):\n # lowercase, normalize, and compose\n word = normalize('NFC', text_type(word.lower()))\n # replace apostrophe-like characters with U+0027, per\n # http://snowball.tartarus.org/texts/apostrophe.html\n word = word.replace('\u2019', '\\'')\n word = word.replace('\u2019', '\\'')\n\n # Exceptions 1\n if word in self._exception1dict:\n return self._exception1dict[word]\n elif word in self._exception1set:\n return word\n\n # Return word if stem is shorter than 3\n if len(word) < 3:\n return word\n\n # Remove initial ', if present.\n while word and word[0] == '\\'':\n word = word[1:]\n # Return word if stem is shorter than 2\n if len(word) < 2:\n return word\n\n # Re-map vocalic Y to y (Y will be C, y will be V)\n if word[0] == 'y':\n word = 'Y' + word[1:]\n for i in range(1, len(word)):\n if word[i] == 'y' and word[i - 1] in self._vowels:\n word = word[:i] + 'Y' + word[i + 1 :]\n\n r1_start = self._sb_r1(word, self._r1_prefixes)\n r2_start = self._sb_r2(word, self._r1_prefixes)\n\n # Step 0\n if word[-3:] == '\\'s\\'':\n word = word[:-3]\n elif word[-2:] == '\\'s':\n word = word[:-2]\n elif word[-1:] == '\\'':\n word = word[:-1]\n # Return word if stem is shorter than 2\n if len(word) < 3:\n return word\n\n # Step 1a\n if word[-4:] == 'sses':\n word = word[:-2]\n elif word[-3:] in {'ied', 'ies'}:\n if len(word) > 4:\n word = word[:-2]\n else:\n word = word[:-1]\n elif word[-2:] in {'us', 'ss'}:\n pass\n elif word[-1] == 's':\n if self._sb_has_vowel(word[:-2]):\n word = word[:-1]\n\n # Exceptions 2\n if word in self._exception2set:\n return word\n\n # Step 1b\n step1b_flag = False\n if word[-5:] == 'eedly':\n if len(word[r1_start:]) >= 5:\n word = word[:-3]\n elif word[-5:] == 'ingly':\n if self._sb_has_vowel(word[:-5]):\n word = word[:-5]\n step1b_flag = True\n elif word[-4:] == 'edly':\n if self._sb_has_vowel(word[:-4]):\n word = word[:-4]\n step1b_flag = True\n elif word[-3:] == 'eed':\n if len(word[r1_start:]) >= 3:\n word = word[:-1]\n elif word[-3:] == 'ing':\n if self._sb_has_vowel(word[:-3]):\n word = word[:-3]\n step1b_flag = True\n elif word[-2:] == 'ed':\n if self._sb_has_vowel(word[:-2]):\n word = word[:-2]\n step1b_flag = True\n elif early_english:\n if word[-3:] == 'est':\n if self._sb_has_vowel(word[:-3]):\n word = word[:-3]\n step1b_flag = True\n elif word[-3:] == 'eth':\n if self._sb_has_vowel(word[:-3]):\n word = word[:-3]\n step1b_flag = True\n\n if step1b_flag:\n if word[-2:] in {'at', 'bl', 'iz'}:\n word += 'e'\n elif word[-2:] in self._doubles:\n word = word[:-1]\n elif self._sb_short_word(word, self._r1_prefixes):\n word += 'e'\n\n # Step 1c\n if (\n len(word) > 2\n and word[-1] in {'Y', 'y'}\n and word[-2] not in self._vowels\n ):\n word = word[:-1] + 'i'\n\n # Step 2\n if word[-2] == 'a':\n if word[-7:] == 'ational':\n if len(word[r1_start:]) >= 7:\n word = word[:-5] + 'e'\n elif word[-6:] == 'tional':\n if len(word[r1_start:]) >= 6:\n word = word[:-2]\n elif word[-2] == 'c':\n if word[-4:] in {'enci', 'anci'}:\n if len(word[r1_start:]) >= 4:\n word = word[:-1] + 'e'\n elif word[-2] == 'e':\n if word[-4:] == 'izer':\n if len(word[r1_start:]) >= 4:\n word = word[:-1]\n elif word[-2] == 'g':\n if word[-3:] == 'ogi':\n if (\n r1_start >= 1\n and len(word[r1_start:]) >= 3\n and word[-4] == 'l'\n ):\n word = word[:-1]\n elif word[-2] == 'l':\n if word[-6:] == 'lessli':\n if len(word[r1_start:]) >= 6:\n word = word[:-2]\n elif word[-5:] in {'entli', 'fulli', 'ousli'}:\n if len(word[r1_start:]) >= 5:\n word = word[:-2]\n elif word[-4:] == 'abli':\n if len(word[r1_start:]) >= 4:\n word = word[:-1] + 'e'\n elif word[-4:] == 'alli':\n if len(word[r1_start:]) >= 4:\n word = word[:-2]\n elif word[-3:] == 'bli':\n if len(word[r1_start:]) >= 3:\n word = word[:-1] + 'e'\n elif word[-2:] == 'li':\n if (\n r1_start >= 1\n and len(word[r1_start:]) >= 2\n and word[-3] in self._li\n ):\n word = word[:-2]\n elif word[-2] == 'o':\n if word[-7:] == 'ization':\n if len(word[r1_start:]) >= 7:\n word = word[:-5] + 'e'\n elif word[-5:] == 'ation':\n if len(word[r1_start:]) >= 5:\n word = word[:-3] + 'e'\n elif word[-4:] == 'ator':\n if len(word[r1_start:]) >= 4:\n word = word[:-2] + 'e'\n elif word[-2] == 's':\n if word[-7:] in {'fulness', 'ousness', 'iveness'}:\n if len(word[r1_start:]) >= 7:\n word = word[:-4]\n elif word[-5:] == 'alism':\n if len(word[r1_start:]) >= 5:\n word = word[:-3]\n elif word[-2] == 't':\n if word[-6:] == 'biliti':\n if len(word[r1_start:]) >= 6:\n word = word[:-5] + 'le'\n elif word[-5:] == 'aliti':\n if len(word[r1_start:]) >= 5:\n word = word[:-3]\n elif word[-5:] == 'iviti':\n if len(word[r1_start:]) >= 5:\n word = word[:-3] + 'e'\n\n # Step 3\n if word[-7:] == 'ational':\n if len(word[r1_start:]) >= 7:\n word = word[:-5] + 'e'\n elif word[-6:] == 'tional':\n if len(word[r1_start:]) >= 6:\n word = word[:-2]\n elif word[-5:] in {'alize', 'icate', 'iciti'}:\n if len(word[r1_start:]) >= 5:\n word = word[:-3]\n elif word[-5:] == 'ative':\n if len(word[r2_start:]) >= 5:\n word = word[:-5]\n elif word[-4:] == 'ical':\n if len(word[r1_start:]) >= 4:\n word = word[:-2]\n elif word[-4:] == 'ness':\n if len(word[r1_start:]) >= 4:\n word = word[:-4]\n elif word[-3:] == 'ful':\n if len(word[r1_start:]) >= 3:\n word = word[:-3]\n\n # Step 4\n for suffix in (\n 'ement',\n 'ance',\n 'ence',\n 'able',\n 'ible',\n 'ment',\n 'ant',\n 'ent',\n 'ism',\n 'ate',\n 'iti',\n 'ous',\n 'ive',\n 'ize',\n 'al',\n 'er',\n 'ic',\n ):\n if word[-len(suffix) :] == suffix:\n if len(word[r2_start:]) >= len(suffix):\n word = word[: -len(suffix)]\n break\n else:\n if word[-3:] == 'ion':\n if (\n len(word[r2_start:]) >= 3\n and len(word) >= 4\n and word[-4] in tuple('st')\n ):\n word = word[:-3]\n\n # Step 5\n if word[-1] == 'e':\n if len(word[r2_start:]) >= 1 or (\n len(word[r1_start:]) >= 1\n and not self._sb_ends_in_short_syllable(word[:-1])\n ):\n word = word[:-1]\n elif word[-1] == 'l':\n if len(word[r2_start:]) >= 1 and word[-2] == 'l':\n word = word[:-1]\n\n # Change 'Y' back to 'y' if it survived stemming\n for i in range(0, len(word)):\n if word[i] == 'Y':\n word = word[:i] + 'y' + word[i + 1 :]\n\n return word", "response": "Returns the Porter2 stem."} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn the Synoname toolcode for the given name.", "response": "def synoname_toolcode(lname, fname='', qual='', normalize=0):\n \"\"\"Build the Synoname toolcode.\n\n This is a wrapper for :py:meth:`SynonameToolcode.fingerprint`.\n\n Parameters\n ----------\n lname : str\n Last name\n fname : str\n First name (can be blank)\n qual : str\n Qualifier\n normalize : int\n Normalization mode (0, 1, or 2)\n\n Returns\n -------\n tuple\n The transformed names and the synoname toolcode\n\n Examples\n --------\n >>> synoname_toolcode('hat')\n ('hat', '', '0000000003$$h')\n >>> synoname_toolcode('niall')\n ('niall', '', '0000000005$$n')\n >>> synoname_toolcode('colin')\n ('colin', '', '0000000005$$c')\n >>> synoname_toolcode('atcg')\n ('atcg', '', '0000000004$$a')\n >>> synoname_toolcode('entreatment')\n ('entreatment', '', '0000000011$$e')\n\n >>> synoname_toolcode('Ste.-Marie', 'Count John II', normalize=2)\n ('ste.-marie ii', 'count john', '0200491310$015b049a127c$smcji')\n >>> synoname_toolcode('Michelangelo IV', '', 'Workshop of')\n ('michelangelo iv', '', '3000550015$055b$mi')\n\n \"\"\"\n return SynonameToolcode().fingerprint(lname, fname, qual, normalize)"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn the synoname toolcode for a given set of names.", "response": "def fingerprint(self, lname, fname='', qual='', normalize=0):\n \"\"\"Build the Synoname toolcode.\n\n Parameters\n ----------\n lname : str\n Last name\n fname : str\n First name (can be blank)\n qual : str\n Qualifier\n normalize : int\n Normalization mode (0, 1, or 2)\n\n Returns\n -------\n tuple\n The transformed names and the synoname toolcode\n\n Examples\n --------\n >>> st = SynonameToolcode()\n >>> st.fingerprint('hat')\n ('hat', '', '0000000003$$h')\n >>> st.fingerprint('niall')\n ('niall', '', '0000000005$$n')\n >>> st.fingerprint('colin')\n ('colin', '', '0000000005$$c')\n >>> st.fingerprint('atcg')\n ('atcg', '', '0000000004$$a')\n >>> st.fingerprint('entreatment')\n ('entreatment', '', '0000000011$$e')\n\n >>> st.fingerprint('Ste.-Marie', 'Count John II', normalize=2)\n ('ste.-marie ii', 'count john', '0200491310$015b049a127c$smcji')\n >>> st.fingerprint('Michelangelo IV', '', 'Workshop of')\n ('michelangelo iv', '', '3000550015$055b$mi')\n\n \"\"\"\n lname = lname.lower()\n fname = fname.lower()\n qual = qual.lower()\n\n # Start with the basic code\n toolcode = ['0', '0', '0', '000', '00', '00', '$', '', '$', '']\n\n full_name = ' '.join((lname, fname))\n\n if qual in self._qual_3:\n toolcode[0] = '3'\n elif qual in self._qual_2:\n toolcode[0] = '2'\n elif qual in self._qual_1:\n toolcode[0] = '1'\n\n # Fill field 1 (punctuation)\n if '.' in full_name:\n toolcode[1] = '2'\n else:\n for punct in ',-/:;\"&\\'()!{|}?$%*+<=>[\\\\]^_`~':\n if punct in full_name:\n toolcode[1] = '1'\n break\n\n elderyounger = '' # save elder/younger for possible movement later\n for gen in self._gen_1:\n if gen in full_name:\n toolcode[2] = '1'\n elderyounger = gen\n break\n else:\n for gen in self._gen_2:\n if gen in full_name:\n toolcode[2] = '2'\n elderyounger = gen\n break\n\n # do comma flip\n if normalize:\n comma = lname.find(',')\n if comma != -1:\n lname_end = lname[comma + 1 :]\n while lname_end[0] in {' ', ','}:\n lname_end = lname_end[1:]\n fname = lname_end + ' ' + fname\n lname = lname[:comma].strip()\n\n # do elder/younger move\n if normalize == 2 and elderyounger:\n elderyounger_loc = fname.find(elderyounger)\n if elderyounger_loc != -1:\n lname = ' '.join((lname, elderyounger.strip()))\n fname = ' '.join(\n (\n fname[:elderyounger_loc].strip(),\n fname[elderyounger_loc + len(elderyounger) :],\n )\n ).strip()\n\n toolcode[4] = '{:02d}'.format(len(fname))\n toolcode[5] = '{:02d}'.format(len(lname))\n\n # strip punctuation\n for char in ',/:;\"&()!{|}?$%*+<=>[\\\\]^_`~':\n full_name = full_name.replace(char, '')\n for pos, char in enumerate(full_name):\n if char == '-' and full_name[pos - 1 : pos + 2] != 'b-g':\n full_name = full_name[:pos] + ' ' + full_name[pos + 1 :]\n\n # Fill field 9 (search range)\n for letter in [_[0] for _ in full_name.split()]:\n if letter not in toolcode[9]:\n toolcode[9] += letter\n if len(toolcode[9]) == 15:\n break\n\n def roman_check(numeral, fname, lname):\n \"\"\"Move Roman numerals from first name to last.\n\n Parameters\n ----------\n numeral : str\n Roman numeral\n fname : str\n First name\n lname : str\n Last name\n\n Returns\n -------\n tuple\n First and last names with Roman numeral moved\n\n \"\"\"\n loc = fname.find(numeral)\n if fname and (\n loc != -1\n and (len(fname[loc:]) == len(numeral))\n or fname[loc + len(numeral)] in {' ', ','}\n ):\n lname = ' '.join((lname, numeral))\n fname = ' '.join(\n (\n fname[:loc].strip(),\n fname[loc + len(numeral) :].lstrip(' ,'),\n )\n )\n return fname.strip(), lname.strip()\n\n # Fill fields 7 (specials) and 3 (roman numerals)\n for num, special in enumerate(self._synoname_special_table):\n roman, match, extra, method = special\n if method & self._method_dict['end']:\n match_context = ' ' + match\n loc = full_name.find(match_context)\n if (len(full_name) > len(match_context)) and (\n loc == len(full_name) - len(match_context)\n ):\n if roman:\n if not any(\n abbr in fname for abbr in ('i.', 'v.', 'x.')\n ):\n full_name = full_name[:loc]\n toolcode[7] += '{:03d}'.format(num) + 'a'\n if toolcode[3] == '000':\n toolcode[3] = '{:03d}'.format(num)\n if normalize == 2:\n fname, lname = roman_check(match, fname, lname)\n else:\n full_name = full_name[:loc]\n toolcode[7] += '{:03d}'.format(num) + 'a'\n if method & self._method_dict['middle']:\n match_context = ' ' + match + ' '\n loc = 0\n while loc != -1:\n loc = full_name.find(match_context, loc + 1)\n if loc > 0:\n if roman:\n if not any(\n abbr in fname for abbr in ('i.', 'v.', 'x.')\n ):\n full_name = (\n full_name[:loc]\n + full_name[loc + len(match) + 1 :]\n )\n toolcode[7] += '{:03d}'.format(num) + 'b'\n if toolcode[3] == '000':\n toolcode[3] = '{:03d}'.format(num)\n if normalize == 2:\n fname, lname = roman_check(\n match, fname, lname\n )\n else:\n full_name = (\n full_name[:loc]\n + full_name[loc + len(match) + 1 :]\n )\n toolcode[7] += '{:03d}'.format(num) + 'b'\n if method & self._method_dict['beginning']:\n match_context = match + ' '\n loc = full_name.find(match_context)\n if loc == 0:\n full_name = full_name[len(match) + 1 :]\n toolcode[7] += '{:03d}'.format(num) + 'c'\n if method & self._method_dict['beginning_no_space']:\n loc = full_name.find(match)\n if loc == 0:\n toolcode[7] += '{:03d}'.format(num) + 'd'\n if full_name[: len(match)] not in toolcode[9]:\n toolcode[9] += full_name[: len(match)]\n\n if extra:\n loc = full_name.find(extra)\n if loc != -1:\n toolcode[7] += '{:03d}'.format(num) + 'X'\n # Since extras are unique, we only look for each of them\n # once, and they include otherwise impossible characters\n # for this field, it's not possible for the following line\n # to have ever been false.\n # if full_name[loc:loc+len(extra)] not in toolcode[9]:\n toolcode[9] += full_name[loc : loc + len(match)]\n\n return lname, fname, ''.join(toolcode)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef uealite(\n word,\n max_word_length=20,\n max_acro_length=8,\n return_rule_no=False,\n var='standard',\n):\n \"\"\"Return UEA-Lite stem.\n\n This is a wrapper for :py:meth:`UEALite.stem`.\n\n Parameters\n ----------\n word : str\n The word to stem\n max_word_length : int\n The maximum word length allowed\n max_acro_length : int\n The maximum acronym length allowed\n return_rule_no : bool\n If True, returns the stem along with rule number\n var : str\n Variant rules to use:\n\n - ``Adams`` to use Jason Adams' rules\n - ``Perl`` to use the original Perl rules\n\n Returns\n -------\n str or (str, int)\n Word stem\n\n Examples\n --------\n >>> uealite('readings')\n 'read'\n >>> uealite('insulted')\n 'insult'\n >>> uealite('cussed')\n 'cuss'\n >>> uealite('fancies')\n 'fancy'\n >>> uealite('eroded')\n 'erode'\n\n \"\"\"\n return UEALite().stem(\n word, max_word_length, max_acro_length, return_rule_no, var\n )", "response": "Return UEA - Lite stem."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef stem(\n self,\n word,\n max_word_length=20,\n max_acro_length=8,\n return_rule_no=False,\n var='standard',\n ):\n \"\"\"Return UEA-Lite stem.\n\n Parameters\n ----------\n word : str\n The word to stem\n max_word_length : int\n The maximum word length allowed\n max_acro_length : int\n The maximum acronym length allowed\n return_rule_no : bool\n If True, returns the stem along with rule number\n var : str\n Variant rules to use:\n\n - ``Adams`` to use Jason Adams' rules\n - ``Perl`` to use the original Perl rules\n\n Returns\n -------\n str or (str, int)\n Word stem\n\n Examples\n --------\n >>> uealite('readings')\n 'read'\n >>> uealite('insulted')\n 'insult'\n >>> uealite('cussed')\n 'cuss'\n >>> uealite('fancies')\n 'fancy'\n >>> uealite('eroded')\n 'erode'\n\n \"\"\"\n\n def _stem_with_duplicate_character_check(word, del_len):\n if word[-1] == 's':\n del_len += 1\n stemmed_word = word[:-del_len]\n if re_match(r'.*(\\w)\\1$', stemmed_word):\n stemmed_word = stemmed_word[:-1]\n return stemmed_word\n\n def _stem(word):\n stemmed_word = word\n rule_no = 0\n\n if not word:\n return word, 0\n if word in self._problem_words or (\n word == 'menses' and var == 'Adams'\n ):\n return word, 90\n if max_word_length and len(word) > max_word_length:\n return word, 95\n\n if \"'\" in word:\n if word[-2:] in {\"'s\", \"'S\"}:\n stemmed_word = word[:-2]\n if word[-1:] == \"'\":\n stemmed_word = word[:-1]\n stemmed_word = stemmed_word.replace(\"n't\", 'not')\n stemmed_word = stemmed_word.replace(\"'ve\", 'have')\n stemmed_word = stemmed_word.replace(\"'re\", 'are')\n stemmed_word = stemmed_word.replace(\"'m\", 'am')\n return stemmed_word, 94\n\n if word.isdigit():\n return word, 90.3\n else:\n hyphen = word.find('-')\n if len(word) > hyphen > 0:\n if (\n word[:hyphen].isalpha()\n and word[hyphen + 1 :].isalpha()\n ):\n return word, 90.2\n else:\n return word, 90.1\n elif '_' in word:\n return word, 90\n elif word[-1] == 's' and word[:-1].isupper():\n if var == 'Adams' and len(word) - 1 > max_acro_length:\n return word, 96\n return word[:-1], 91.1\n elif word.isupper():\n if var == 'Adams' and len(word) > max_acro_length:\n return word, 96\n return word, 91\n elif re_match(r'^.*[A-Z].*[A-Z].*$', word):\n return word, 92\n elif word[0].isupper():\n return word, 93\n elif var == 'Adams' and re_match(\n r'^[a-z](|[rl])(ing|ed)$', word\n ):\n return word, 97\n\n for n in range(7, 1, -1):\n if word[-n:] in self._rules[var][n]:\n rule_no, del_len, add_str = self._rules[var][n][word[-n:]]\n if del_len:\n stemmed_word = word[:-del_len]\n else:\n stemmed_word = word\n if add_str:\n stemmed_word += add_str\n break\n\n if not rule_no:\n if re_match(r'.*\\w\\wings?$', word): # rule 58\n stemmed_word = _stem_with_duplicate_character_check(\n word, 3\n )\n rule_no = 58\n elif re_match(r'.*\\w\\weds?$', word): # rule 62\n stemmed_word = _stem_with_duplicate_character_check(\n word, 2\n )\n rule_no = 62\n elif word[-1] == 's': # rule 68\n stemmed_word = word[:-1]\n rule_no = 68\n\n return stemmed_word, rule_no\n\n stem, rule_no = _stem(word)\n if return_rule_no:\n return stem, rule_no\n return stem", "response": "Return UEA - Lite stem."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the R1 region as defined in the Porter2 specification.", "response": "def _sb_r1(self, term, r1_prefixes=None):\n \"\"\"Return the R1 region, as defined in the Porter2 specification.\n\n Parameters\n ----------\n term : str\n The term to examine\n r1_prefixes : set\n Prefixes to consider\n\n Returns\n -------\n int\n Length of the R1 region\n\n \"\"\"\n vowel_found = False\n if hasattr(r1_prefixes, '__iter__'):\n for prefix in r1_prefixes:\n if term[: len(prefix)] == prefix:\n return len(prefix)\n\n for i in range(len(term)):\n if not vowel_found and term[i] in self._vowels:\n vowel_found = True\n elif vowel_found and term[i] not in self._vowels:\n return i + 1\n return len(term)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef _sb_r2(self, term, r1_prefixes=None):\n r1_start = self._sb_r1(term, r1_prefixes)\n return r1_start + self._sb_r1(term[r1_start:])", "response": "Return the R2 region as defined in the Porter2 specification."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns True iff term ends in a short syllable.", "response": "def _sb_ends_in_short_syllable(self, term):\n \"\"\"Return True iff term ends in a short syllable.\n\n (...according to the Porter2 specification.)\n\n NB: This is akin to the CVC test from the Porter stemmer. The\n description is unfortunately poor/ambiguous.\n\n Parameters\n ----------\n term : str\n The term to examine\n\n Returns\n -------\n bool\n True iff term ends in a short syllable\n\n \"\"\"\n if not term:\n return False\n if len(term) == 2:\n if term[-2] in self._vowels and term[-1] not in self._vowels:\n return True\n elif len(term) >= 3:\n if (\n term[-3] not in self._vowels\n and term[-2] in self._vowels\n and term[-1] in self._codanonvowels\n ):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn True iff term is a short word.", "response": "def _sb_short_word(self, term, r1_prefixes=None):\n \"\"\"Return True iff term is a short word.\n\n (...according to the Porter2 specification.)\n\n Parameters\n ----------\n term : str\n The term to examine\n r1_prefixes : set\n Prefixes to consider\n\n Returns\n -------\n bool\n True iff term is a short word\n\n \"\"\"\n if self._sb_r1(term, r1_prefixes) == len(\n term\n ) and self._sb_ends_in_short_syllable(term):\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Implement a function in Python 3 to\nreturn True iff a vowel exists in the term.", "response": "def _sb_has_vowel(self, term):\n \"\"\"Return Porter helper function _sb_has_vowel value.\n\n Parameters\n ----------\n term : str\n The term to examine\n\n Returns\n -------\n bool\n True iff a vowel exists in the term (as defined in the Porter\n stemmer definition)\n\n \"\"\"\n for letter in term:\n if letter in self._vowels:\n return True\n return False"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef encode(self, word, max_length=8):\n # Lowercase input & filter unknown characters\n word = ''.join(\n char for char in word.lower() if char in self._initial_phones\n )\n\n if not word:\n word = '\u00f7'\n\n # Perform initial eudex coding of each character\n values = [self._initial_phones[word[0]]]\n values += [self._trailing_phones[char] for char in word[1:]]\n\n # Right-shift by one to determine if second instance should be skipped\n shifted_values = [_ >> 1 for _ in values]\n condensed_values = [values[0]]\n for n in range(1, len(shifted_values)):\n if shifted_values[n] != shifted_values[n - 1]:\n condensed_values.append(values[n])\n\n # Add padding after first character & trim beyond max_length\n values = (\n [condensed_values[0]]\n + [0] * max(0, max_length - len(condensed_values))\n + condensed_values[1:max_length]\n )\n\n # Combine individual character values into eudex hash\n hash_value = 0\n for val in values:\n hash_value = (hash_value << 8) | val\n\n return hash_value", "response": "Encodes a string into a set of eudex phonetic hash values."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef roger_root(word, max_length=5, zero_pad=True):\n return RogerRoot().encode(word, max_length, zero_pad)", "response": "Return the Roger Root code for a word."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the Roger Root code for a word.", "response": "def encode(self, word, max_length=5, zero_pad=True):\n \"\"\"Return the Roger Root code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The maximum length (default 5) of the code to return\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length\n string\n\n Returns\n -------\n str\n The Roger Root code\n\n Examples\n --------\n >>> roger_root('Christopher')\n '06401'\n >>> roger_root('Niall')\n '02500'\n >>> roger_root('Smith')\n '00310'\n >>> roger_root('Schmidt')\n '06310'\n\n \"\"\"\n # uppercase, normalize, decompose, and filter non-A-Z out\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n\n code = ''\n pos = 0\n\n # Do first digit(s) first\n for num in range(4, 0, -1):\n if word[:num] in self._init_patterns[num]:\n code = self._init_patterns[num][word[:num]]\n pos += num\n break\n\n # Then code subsequent digits\n while pos < len(word):\n for num in range(4, 0, -1): # pragma: no branch\n if word[pos : pos + num] in self._med_patterns[num]:\n code += self._med_patterns[num][word[pos : pos + num]]\n pos += num\n break\n\n code = self._delete_consecutive_repeats(code)\n code = code.replace('*', '')\n\n if zero_pad:\n code += '0' * max_length\n\n return code[:max_length]"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef encode(self, word):\n\n def _after(word, pos, letters):\n \"\"\"Return True if word[pos] follows one of the supplied letters.\n\n Parameters\n ----------\n word : str\n The word to check\n pos : int\n Position within word to check\n letters : str\n Letters to confirm precede word[pos]\n\n Returns\n -------\n bool\n True if word[pos] follows a value in letters\n\n \"\"\"\n return pos > 0 and word[pos - 1] in letters\n\n def _before(word, pos, letters):\n \"\"\"Return True if word[pos] precedes one of the supplied letters.\n\n Parameters\n ----------\n word : str\n The word to check\n pos : int\n Position within word to check\n letters : str\n Letters to confirm follow word[pos]\n\n Returns\n -------\n bool\n True if word[pos] precedes a value in letters\n\n \"\"\"\n return pos + 1 < len(word) and word[pos + 1] in letters\n\n sdx = ''\n\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n\n word = word.replace('\u00c4', 'AE')\n word = word.replace('\u00d6', 'OE')\n word = word.replace('\u00dc', 'UE')\n word = ''.join(c for c in word if c in self._uc_set)\n\n # Nothing to convert, return base case\n if not word:\n return sdx\n\n for i in range(len(word)):\n if word[i] in self._uc_v_set:\n sdx += '0'\n elif word[i] == 'B':\n sdx += '1'\n elif word[i] == 'P':\n if _before(word, i, {'H'}):\n sdx += '3'\n else:\n sdx += '1'\n elif word[i] in {'D', 'T'}:\n if _before(word, i, {'C', 'S', 'Z'}):\n sdx += '8'\n else:\n sdx += '2'\n elif word[i] in {'F', 'V', 'W'}:\n sdx += '3'\n elif word[i] in {'G', 'K', 'Q'}:\n sdx += '4'\n elif word[i] == 'C':\n if _after(word, i, {'S', 'Z'}):\n sdx += '8'\n elif i == 0:\n if _before(\n word, i, {'A', 'H', 'K', 'L', 'O', 'Q', 'R', 'U', 'X'}\n ):\n sdx += '4'\n else:\n sdx += '8'\n elif _before(word, i, {'A', 'H', 'K', 'O', 'Q', 'U', 'X'}):\n sdx += '4'\n else:\n sdx += '8'\n elif word[i] == 'X':\n if _after(word, i, {'C', 'K', 'Q'}):\n sdx += '8'\n else:\n sdx += '48'\n elif word[i] == 'L':\n sdx += '5'\n elif word[i] in {'M', 'N'}:\n sdx += '6'\n elif word[i] == 'R':\n sdx += '7'\n elif word[i] in {'S', 'Z'}:\n sdx += '8'\n\n sdx = self._delete_consecutive_repeats(sdx)\n\n if sdx:\n sdx = sdx[:1] + sdx[1:].replace('0', '')\n\n return sdx", "response": "Encodes a word into a K\u00f6lner Phonetik value."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nconvert a K\u00f6lner Phonetik code from numeric to alphabetic.", "response": "def _to_alpha(self, num):\n \"\"\"Convert a K\u00f6lner Phonetik code from numeric to alphabetic.\n\n Parameters\n ----------\n num : str or int\n A numeric K\u00f6lner Phonetik representation\n\n Returns\n -------\n str\n An alphabetic representation of the same word\n\n Examples\n --------\n >>> pe = Koelner()\n >>> pe._to_alpha('862')\n 'SNT'\n >>> pe._to_alpha('657')\n 'NLR'\n >>> pe._to_alpha('86766')\n 'SNRNN'\n\n \"\"\"\n num = ''.join(c for c in text_type(num) if c in self._num_set)\n return num.translate(self._num_trans)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef main(argv):\n first_col = 3\n last_col = -1\n\n def print_usage():\n \"\"\"Print usage statement.\"\"\"\n sys.stdout.write(\n 'features_csv_to_dict.py -i ' + '[-o ]\\n'\n )\n sys.exit(2)\n\n def binarize(num):\n \"\"\"Replace 0, -1, 1, 2 with 00, 10, 01, 11.\n\n Parameters\n ----------\n num : str\n The number to binarize\n\n Returns\n -------\n str\n A binarized number\n\n \"\"\"\n if num == '0': # 0\n return '00'\n elif num == '-1': # -\n return '10'\n elif num == '1': # +\n return '01'\n elif num == '2': # \u00b1 (segmental) or copy from base (non-segmental)\n return '11'\n\n def init_termdicts():\n \"\"\"Initialize the terms dict.\n\n Returns\n -------\n (dict, dict)\n Term & feature mask dictionaries\n\n \"\"\"\n ifile = codecs.open('features_terms.csv', 'r', 'utf-8')\n\n feature_mask = {}\n keyline = ifile.readline().strip().split(',')[first_col:last_col]\n mag = len(keyline)\n for i in range(len(keyline)):\n features = '0b' + ('00' * i) + '11' + ('00' * (mag - i - 1))\n feature_mask[keyline[i]] = int(features, 2)\n\n termdict = {}\n for line in ifile:\n line = line.strip().rstrip(',')\n if '#' in line:\n line = line[: line.find('#')].strip()\n if line:\n line = line.split(',')\n term = line[last_col]\n features = '0b' + ''.join(\n [binarize(val) for val in line[first_col:last_col]]\n )\n termdict[term] = int(features, 2)\n\n return termdict, feature_mask\n\n def check_terms(sym, features, name, termdict):\n \"\"\"Check terms.\n\n Check each term of the phone name to confirm that it matches\n the expected features implied by that feature.\n\n Parameters\n ----------\n sym : str\n Symbol to check\n features : int\n Phone features\n name : str\n Phone name\n termdict : dict\n Dictionary of terms\n\n \"\"\"\n if '#' in name:\n name = name[: name.find('#')].strip()\n for term in name.split():\n if term in termdict:\n if termdict[term] & features != termdict[term]:\n sys.stdout.write(\n 'Feature mismatch for term \"'\n + term\n + '\" in '\n + sym\n + '\\n'\n )\n else:\n sys.stdout.write(\n 'Unknown term \"'\n + term\n + '\" in '\n + name\n + ' : '\n + sym\n + '\\n'\n )\n\n def check_entailments(sym, features, feature_mask):\n \"\"\"Check entailments.\n\n Check for necessary feature assignments (entailments)\n For example, [+round] necessitates [+labial].\n\n Parameters\n ----------\n sym : str\n Symbol to check\n features : int\n Phone features\n feature_mask : dict\n The feature mask\n\n \"\"\"\n entailments = {\n '+labial': ('\u00b1round', '\u00b1protruded', '\u00b1compressed', '\u00b1labiodental'),\n '-labial': ('0round', '0protruded', '0compressed', '0labiodental'),\n '+coronal': ('\u00b1anterior', '\u00b1distributed'),\n '-coronal': ('0anterior', '0distributed'),\n '+dorsal': ('\u00b1high', '\u00b1low', '\u00b1front', '\u00b1back', '\u00b1tense'),\n '-dorsal': ('0high', '0low', '0front', '0back', '0tense'),\n '+pharyngeal': ('\u00b1atr', '\u00b1rtr'),\n '-pharyngeal': ('0atr', '0rtr'),\n '+protruded': ('+labial', '+round', '-compressed'),\n '+compressed': ('+labial', '+round', '-protruded'),\n '+glottalic_suction': ('-velaric_suction',),\n '+velaric_suction': ('-glottalic_suction',),\n }\n\n for feature in entailments:\n fname = feature[1:]\n if feature[0] == '+':\n fm = (feature_mask[fname] >> 1) & feature_mask[fname]\n else:\n fm = (feature_mask[fname] << 1) & feature_mask[fname]\n if (features & fm) == fm:\n for ent in entailments[feature]:\n ename = ent[1:]\n if ent[0] == '+':\n efm = (feature_mask[ename] >> 1) & feature_mask[ename]\n elif ent[0] == '-':\n efm = (feature_mask[ename] << 1) & feature_mask[ename]\n elif ent[0] == '0':\n efm = 0\n elif ent[0] == '\u00b1':\n efm = feature_mask[ename]\n\n if ent[0] == '\u00b1':\n if (features & efm) == 0:\n sys.stdout.write(\n 'Incorrect entailment for '\n + sym\n + ' for feature '\n + fname\n + ' and entailment '\n + ename\n )\n else:\n if (features & efm) != efm:\n sys.stdout.write(\n 'Incorrect entailment for '\n + sym\n + ' for feature '\n + fname\n + ' and entailment '\n + ename\n )\n\n checkdict = {} # a mapping of symbol to feature\n checkset_s = set() # a set of the symbols seen\n checkset_f = set() # a set of the feature values seen\n\n termdict, feature_mask = init_termdicts()\n\n ifile = ''\n ofile = ''\n try:\n opts = getopt.getopt(argv, 'hi:o:', ['ifile=', 'ofile='])[0]\n except getopt.GetoptError:\n print_usage()\n for opt, arg in opts:\n if opt == '-h':\n print_usage()\n elif opt in ('-i', '--ifile'):\n ifile = codecs.open(arg, 'r', 'utf-8')\n elif opt in ('-o', '--ofile'):\n ofile = codecs.open(arg, 'w', 'utf-8')\n if not ifile:\n print_usage()\n\n oline = 'PHONETIC_FEATURES = {'\n if not ofile:\n ofile = sys.stdout\n\n ofile.write(oline + '\\n')\n\n keyline = ifile.readline().strip().split(',')[first_col:last_col]\n for line in ifile:\n line = line.strip().rstrip(',')\n\n if line.startswith('####'):\n break\n\n line = unicodedata.normalize('NFC', line)\n\n if not line or line.startswith('#'):\n oline = ' ' + line\n\n else:\n line = line.strip().split(',')\n if '#' in line:\n line = line[: line.find('#')]\n symbol = line[0]\n variant = int(line[1])\n segmental = bool(line[2])\n features = '0b' + ''.join(\n [binarize(val) for val in line[first_col:last_col]]\n )\n name = line[-1].strip()\n if not segmental:\n features = '-' + features\n\n featint = int(features, 2)\n check_terms(symbol, featint, name, termdict)\n check_entailments(symbol, featint, feature_mask)\n if symbol in checkset_s:\n sys.stdout.write(\n 'Symbol ' + symbol + ' appears twice in CSV.\\n'\n )\n else:\n checkset_s.add(symbol)\n\n if variant < 2:\n if featint in checkset_f:\n sys.stdout.write(\n 'Feature set '\n + str(featint)\n + ' appears in CSV for two primary IPA '\n + 'symbols: '\n + symbol\n + ' and '\n + checkdict[featint]\n )\n else:\n checkdict[featint] = symbol\n checkset_f.add(featint)\n\n if variant < 5:\n oline = ' \\'{}\\': {},'.format(\n symbol, featint\n )\n else:\n oline = ''\n\n if oline:\n ofile.write(oline + '\\n')\n\n ofile.write(' }\\n\\nFEATURE_MASK = {')\n\n mag = len(keyline)\n for i in range(len(keyline)):\n features = int('0b' + ('00' * i) + '11' + ('00' * (mag - i - 1)), 2)\n oline = ' \\'{}\\': {},'.format(keyline[i], features)\n ofile.write(oline + '\\n')\n\n ofile.write(' }\\n')", "response": "This function is the main function of the features_terms. csv script."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef lein(word, max_length=4, zero_pad=True):\n return Lein().encode(word, max_length, zero_pad)", "response": "Return the Lein code for a word."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the Lein code for a word.", "response": "def encode(self, word, max_length=4, zero_pad=True):\n \"\"\"Return the Lein code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to 4)\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length\n string\n\n Returns\n -------\n str\n The Lein code\n\n Examples\n --------\n >>> pe = Lein()\n >>> pe.encode('Christopher')\n 'C351'\n >>> pe.encode('Niall')\n 'N300'\n >>> pe.encode('Smith')\n 'S210'\n >>> pe.encode('Schmidt')\n 'S521'\n\n \"\"\"\n # uppercase, normalize, decompose, and filter non-A-Z out\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n\n code = word[:1] # Rule 1\n word = word[1:].translate(self._del_trans) # Rule 2\n word = self._delete_consecutive_repeats(word) # Rule 3\n code += word.translate(self._trans) # Rule 4\n\n if zero_pad:\n code += '0' * max_length # Rule 4\n\n return code[:max_length]"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the Q - Grams in src & tar.", "response": "def _get_qgrams(self, src, tar, qval=0, skip=0):\n \"\"\"Return the Q-Grams in src & tar.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n skip : int\n The number of characters to skip (only works when src and tar are\n strings)\n\n Returns\n -------\n tuple of Counters\n Q-Grams\n\n Examples\n --------\n >>> pe = _TokenDistance()\n >>> pe._get_qgrams('AT', 'TT', qval=2)\n (QGrams({'$A': 1, 'AT': 1, 'T#': 1}),\n QGrams({'$T': 1, 'TT': 1, 'T#': 1}))\n\n \"\"\"\n if isinstance(src, Counter) and isinstance(tar, Counter):\n return src, tar\n if qval > 0:\n return QGrams(src, qval, '$#', skip), QGrams(tar, qval, '$#', skip)\n return Counter(src.strip().split()), Counter(tar.strip().split())"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef rle_decode(text, use_bwt=True):\n text = RLE().decode(text)\n if use_bwt:\n text = BWT().decode(text)\n return text", "response": "r A wrapper for RLE. decode that does the actual decoding."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nperform encoding of run - length - encoding of a string.", "response": "def encode(self, text):\n r\"\"\"Perform encoding of run-length-encoding (RLE).\n\n Parameters\n ----------\n text : str\n A text string to encode\n\n Returns\n -------\n str\n Word decoded by RLE\n\n Examples\n --------\n >>> rle = RLE()\n >>> bwt = BWT()\n >>> rle.encode(bwt.encode('align'))\n 'n\\x00ilag'\n >>> rle.encode('align')\n 'align'\n\n >>> rle.encode(bwt.encode('banana'))\n 'annb\\x00aa'\n >>> rle.encode('banana')\n 'banana'\n\n >>> rle.encode(bwt.encode('aaabaabababa'))\n 'ab\\x00abbab5a'\n >>> rle.encode('aaabaabababa')\n '3abaabababa'\n\n \"\"\"\n if text:\n text = ((len(list(g)), k) for k, g in groupby(text))\n text = (\n (str(n) + k if n > 2 else (k if n == 1 else 2 * k))\n for n, k in text\n )\n return ''.join(text)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef decode(self, text):\n mult = ''\n decoded = []\n for letter in list(text):\n if not letter.isdigit():\n if mult:\n decoded.append(int(mult) * letter)\n mult = ''\n else:\n decoded.append(letter)\n else:\n mult += letter\n\n text = ''.join(decoded)\n return text", "response": "Perform decoding of run - length - encoding of a string."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the SoundD code for the given word.", "response": "def encode(self, word, max_length=4):\n \"\"\"Return the SoundD code.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to 4)\n\n Returns\n -------\n str\n The SoundD code\n\n Examples\n --------\n >>> sound_d('Gough')\n '2000'\n >>> sound_d('pneuma')\n '5500'\n >>> sound_d('knight')\n '5300'\n >>> sound_d('trice')\n '3620'\n >>> sound_d('judge')\n '2200'\n\n \"\"\"\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n\n if word[:2] in {'KN', 'GN', 'PN', 'AC', 'WR'}:\n word = word[1:]\n elif word[:1] == 'X':\n word = 'S' + word[1:]\n elif word[:2] == 'WH':\n word = 'W' + word[2:]\n\n word = (\n word.replace('DGE', '20').replace('DGI', '20').replace('GH', '0')\n )\n\n word = word.translate(self._trans)\n word = self._delete_consecutive_repeats(word)\n word = word.replace('0', '')\n\n if max_length != -1:\n if len(word) < max_length:\n word += '0' * (max_length - len(word))\n else:\n word = word[:max_length]\n\n return word"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef dist(self, src, tar):\n if src == tar:\n return 0.0\n\n src = src.encode('utf-8')\n tar = tar.encode('utf-8')\n\n if lzma is not None:\n src_comp = lzma.compress(src)[14:]\n tar_comp = lzma.compress(tar)[14:]\n concat_comp = lzma.compress(src + tar)[14:]\n concat_comp2 = lzma.compress(tar + src)[14:]\n else: # pragma: no cover\n raise ValueError(\n 'Install the PylibLZMA module in order to use LZMA'\n )\n\n return (\n min(len(concat_comp), len(concat_comp2))\n - min(len(src_comp), len(tar_comp))\n ) / max(len(src_comp), len(tar_comp))", "response": "Return the NCD between two strings using LZMA compression."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the Smith - Waterman score of two strings.", "response": "def smith_waterman(src, tar, gap_cost=1, sim_func=sim_ident):\n \"\"\"Return the Smith-Waterman score of two strings.\n\n This is a wrapper for :py:meth:`SmithWaterman.dist_abs`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n gap_cost : float\n The cost of an alignment gap (1 by default)\n sim_func : function\n A function that returns the similarity of two characters (identity\n similarity by default)\n\n Returns\n -------\n float\n Smith-Waterman score\n\n Examples\n --------\n >>> smith_waterman('cat', 'hat')\n 2.0\n >>> smith_waterman('Niall', 'Neil')\n 1.0\n >>> smith_waterman('aluminum', 'Catalan')\n 0.0\n >>> smith_waterman('ATCG', 'TAGC')\n 1.0\n\n \"\"\"\n return SmithWaterman().dist_abs(src, tar, gap_cost, sim_func)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the Smith - Waterman score of two strings.", "response": "def dist_abs(self, src, tar, gap_cost=1, sim_func=sim_ident):\n \"\"\"Return the Smith-Waterman score of two strings.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n gap_cost : float\n The cost of an alignment gap (1 by default)\n sim_func : function\n A function that returns the similarity of two characters (identity\n similarity by default)\n\n Returns\n -------\n float\n Smith-Waterman score\n\n Examples\n --------\n >>> cmp = SmithWaterman()\n >>> cmp.dist_abs('cat', 'hat')\n 2.0\n >>> cmp.dist_abs('Niall', 'Neil')\n 1.0\n >>> cmp.dist_abs('aluminum', 'Catalan')\n 0.0\n >>> cmp.dist_abs('ATCG', 'TAGC')\n 1.0\n\n \"\"\"\n d_mat = np_zeros((len(src) + 1, len(tar) + 1), dtype=np_float32)\n\n for i in range(len(src) + 1):\n d_mat[i, 0] = 0\n for j in range(len(tar) + 1):\n d_mat[0, j] = 0\n for i in range(1, len(src) + 1):\n for j in range(1, len(tar) + 1):\n match = d_mat[i - 1, j - 1] + sim_func(src[i - 1], tar[j - 1])\n delete = d_mat[i - 1, j] - gap_cost\n insert = d_mat[i, j - 1] - gap_cost\n d_mat[i, j] = max(0, match, delete, insert)\n return d_mat[d_mat.shape[0] - 1, d_mat.shape[1] - 1]"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the Levenshtein distance between two strings.", "response": "def levenshtein(src, tar, mode='lev', cost=(1, 1, 1, 1)):\n \"\"\"Return the Levenshtein distance between two strings.\n\n This is a wrapper of :py:meth:`Levenshtein.dist_abs`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n mode : str\n Specifies a mode for computing the Levenshtein distance:\n\n - ``lev`` (default) computes the ordinary Levenshtein distance, in\n which edits may include inserts, deletes, and substitutions\n - ``osa`` computes the Optimal String Alignment distance, in which\n edits may include inserts, deletes, substitutions, and\n transpositions but substrings may only be edited once\n\n cost : tuple\n A 4-tuple representing the cost of the four possible edits: inserts,\n deletes, substitutions, and transpositions, respectively (by default:\n (1, 1, 1, 1))\n\n Returns\n -------\n int (may return a float if cost has float values)\n The Levenshtein distance between src & tar\n\n Examples\n --------\n >>> levenshtein('cat', 'hat')\n 1\n >>> levenshtein('Niall', 'Neil')\n 3\n >>> levenshtein('aluminum', 'Catalan')\n 7\n >>> levenshtein('ATCG', 'TAGC')\n 3\n\n >>> levenshtein('ATCG', 'TAGC', mode='osa')\n 2\n >>> levenshtein('ACTG', 'TAGC', mode='osa')\n 4\n\n \"\"\"\n return Levenshtein().dist_abs(src, tar, mode, cost)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dist_levenshtein(src, tar, mode='lev', cost=(1, 1, 1, 1)):\n return Levenshtein().dist(src, tar, mode, cost)", "response": "Return the normalized Levenshtein distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef sim_levenshtein(src, tar, mode='lev', cost=(1, 1, 1, 1)):\n return Levenshtein().sim(src, tar, mode, cost)", "response": "Return the Levenshtein similarity between two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dist_abs(self, src, tar, mode='lev', cost=(1, 1, 1, 1)):\n ins_cost, del_cost, sub_cost, trans_cost = cost\n\n if src == tar:\n return 0\n if not src:\n return len(tar) * ins_cost\n if not tar:\n return len(src) * del_cost\n\n d_mat = np_zeros((len(src) + 1, len(tar) + 1), dtype=np_int)\n for i in range(len(src) + 1):\n d_mat[i, 0] = i * del_cost\n for j in range(len(tar) + 1):\n d_mat[0, j] = j * ins_cost\n\n for i in range(len(src)):\n for j in range(len(tar)):\n d_mat[i + 1, j + 1] = min(\n d_mat[i + 1, j] + ins_cost, # ins\n d_mat[i, j + 1] + del_cost, # del\n d_mat[i, j]\n + (sub_cost if src[i] != tar[j] else 0), # sub/==\n )\n\n if mode == 'osa':\n if (\n i + 1 > 1\n and j + 1 > 1\n and src[i] == tar[j - 1]\n and src[i - 1] == tar[j]\n ):\n # transposition\n d_mat[i + 1, j + 1] = min(\n d_mat[i + 1, j + 1],\n d_mat[i - 1, j - 1] + trans_cost,\n )\n\n return d_mat[len(src), len(tar)]", "response": "Return the Levenshtein distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef dist(self, src, tar, mode='lev', cost=(1, 1, 1, 1)):\n if src == tar:\n return 0\n ins_cost, del_cost = cost[:2]\n return levenshtein(src, tar, mode, cost) / (\n max(len(src) * del_cost, len(tar) * ins_cost)\n )", "response": "Return the normalized Levenshtein distance between two strings src and tar."} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns the omission key for the given word.", "response": "def fingerprint(self, word):\n \"\"\"Return the omission key.\n\n Parameters\n ----------\n word : str\n The word to transform into its omission key\n\n Returns\n -------\n str\n The omission key\n\n Examples\n --------\n >>> ok = OmissionKey()\n >>> ok.fingerprint('The quick brown fox jumped over the lazy dog.')\n 'JKQXZVWYBFMGPDHCLNTREUIOA'\n >>> ok.fingerprint('Christopher')\n 'PHCTSRIOE'\n >>> ok.fingerprint('Niall')\n 'LNIA'\n\n \"\"\"\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = ''.join(c for c in word if c in self._letters)\n\n key = ''\n\n # add consonants in order supplied by _consonants (no duplicates)\n for char in self._consonants:\n if char in word:\n key += char\n\n # add vowels in order they appeared in the word (no duplicates)\n for char in word:\n if char not in self._consonants and char not in key:\n key += char\n\n return key"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the Minkowski distance of two strings.", "response": "def minkowski(src, tar, qval=2, pval=1, normalized=False, alphabet=None):\n \"\"\"Return the Minkowski distance (:math:`L^p`-norm) of two strings.\n\n This is a wrapper for :py:meth:`Minkowski.dist_abs`.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n pval : int or float\n The :math:`p`-value of the :math:`L^p`-space\n normalized : bool\n Normalizes to [0, 1] if True\n alphabet : collection or int\n The values or size of the alphabet\n\n Returns\n -------\n float\n The Minkowski distance\n\n Examples\n --------\n >>> minkowski('cat', 'hat')\n 4.0\n >>> minkowski('Niall', 'Neil')\n 7.0\n >>> minkowski('Colin', 'Cuilen')\n 9.0\n >>> minkowski('ATCG', 'TAGC')\n 10.0\n\n \"\"\"\n return Minkowski().dist_abs(src, tar, qval, pval, normalized, alphabet)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning normalized Minkowski distance of two strings.", "response": "def dist_minkowski(src, tar, qval=2, pval=1, alphabet=None):\n \"\"\"Return normalized Minkowski distance of two strings.\n\n This is a wrapper for :py:meth:`Minkowski.dist`.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n pval : int or float\n The :math:`p`-value of the :math:`L^p`-space\n alphabet : collection or int\n The values or size of the alphabet\n\n Returns\n -------\n float\n The normalized Minkowski distance\n\n Examples\n --------\n >>> dist_minkowski('cat', 'hat')\n 0.5\n >>> round(dist_minkowski('Niall', 'Neil'), 12)\n 0.636363636364\n >>> round(dist_minkowski('Colin', 'Cuilen'), 12)\n 0.692307692308\n >>> dist_minkowski('ATCG', 'TAGC')\n 1.0\n\n \"\"\"\n return Minkowski().dist(src, tar, qval, pval, alphabet)"} {"SOURCE": "codesearchnet", "instruction": "Can you write a function in Python 3 where it\nreturns normalized Minkowski similarity of two strings.", "response": "def sim_minkowski(src, tar, qval=2, pval=1, alphabet=None):\n \"\"\"Return normalized Minkowski similarity of two strings.\n\n This is a wrapper for :py:meth:`Minkowski.sim`.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n pval : int or float\n The :math:`p`-value of the :math:`L^p`-space\n alphabet : collection or int\n The values or size of the alphabet\n\n Returns\n -------\n float\n The normalized Minkowski similarity\n\n Examples\n --------\n >>> sim_minkowski('cat', 'hat')\n 0.5\n >>> round(sim_minkowski('Niall', 'Neil'), 12)\n 0.363636363636\n >>> round(sim_minkowski('Colin', 'Cuilen'), 12)\n 0.307692307692\n >>> sim_minkowski('ATCG', 'TAGC')\n 0.0\n\n \"\"\"\n return Minkowski().sim(src, tar, qval, pval, alphabet)"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef dist_abs(\n self, src, tar, qval=2, pval=1, normalized=False, alphabet=None\n ):\n \"\"\"Return the Minkowski distance (:math:`L^p`-norm) of two strings.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n pval : int or float\n The :math:`p`-value of the :math:`L^p`-space\n normalized : bool\n Normalizes to [0, 1] if True\n alphabet : collection or int\n The values or size of the alphabet\n\n Returns\n -------\n float\n The Minkowski distance\n\n Examples\n --------\n >>> cmp = Minkowski()\n >>> cmp.dist_abs('cat', 'hat')\n 4.0\n >>> cmp.dist_abs('Niall', 'Neil')\n 7.0\n >>> cmp.dist_abs('Colin', 'Cuilen')\n 9.0\n >>> cmp.dist_abs('ATCG', 'TAGC')\n 10.0\n\n \"\"\"\n q_src, q_tar = self._get_qgrams(src, tar, qval)\n diffs = ((q_src - q_tar) + (q_tar - q_src)).values()\n\n normalizer = 1\n if normalized:\n totals = (q_src + q_tar).values()\n if alphabet is not None:\n # noinspection PyTypeChecker\n normalizer = (\n alphabet if isinstance(alphabet, Number) else len(alphabet)\n )\n elif pval == 0:\n normalizer = len(totals)\n else:\n normalizer = sum(_ ** pval for _ in totals) ** (1 / pval)\n\n if len(diffs) == 0:\n return 0.0\n if pval == float('inf'):\n # Chebyshev distance\n return max(diffs) / normalizer\n if pval == 0:\n # This is the l_0 \"norm\" as developed by David Donoho\n return len(diffs) / normalizer\n return sum(_ ** pval for _ in diffs) ** (1 / pval) / normalizer", "response": "Return the Minkowski distance of two strings."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning normalized Minkowski distance of two strings.", "response": "def dist(self, src, tar, qval=2, pval=1, alphabet=None):\n \"\"\"Return normalized Minkowski distance of two strings.\n\n The normalized Minkowski distance :cite:`Minkowski:1910` is a distance\n metric in :math:`L^p`-space, normalized to [0, 1].\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n pval : int or float\n The :math:`p`-value of the :math:`L^p`-space\n alphabet : collection or int\n The values or size of the alphabet\n\n Returns\n -------\n float\n The normalized Minkowski distance\n\n Examples\n --------\n >>> cmp = Minkowski()\n >>> cmp.dist('cat', 'hat')\n 0.5\n >>> round(cmp.dist('Niall', 'Neil'), 12)\n 0.636363636364\n >>> round(cmp.dist('Colin', 'Cuilen'), 12)\n 0.692307692308\n >>> cmp.dist('ATCG', 'TAGC')\n 1.0\n\n \"\"\"\n return self.dist_abs(src, tar, qval, pval, True, alphabet)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef occurrence_halved_fingerprint(\n word, n_bits=16, most_common=MOST_COMMON_LETTERS_CG\n):\n \"\"\"Return the occurrence halved fingerprint.\n\n This is a wrapper for :py:meth:`OccurrenceHalved.fingerprint`.\n\n Parameters\n ----------\n word : str\n The word to fingerprint\n n_bits : int\n Number of bits in the fingerprint returned\n most_common : list\n The most common tokens in the target language, ordered by frequency\n\n Returns\n -------\n int\n The occurrence halved fingerprint\n\n Examples\n --------\n >>> bin(occurrence_halved_fingerprint('hat'))\n '0b1010000000010'\n >>> bin(occurrence_halved_fingerprint('niall'))\n '0b10010100000'\n >>> bin(occurrence_halved_fingerprint('colin'))\n '0b1001010000'\n >>> bin(occurrence_halved_fingerprint('atcg'))\n '0b10100000000000'\n >>> bin(occurrence_halved_fingerprint('entreatment'))\n '0b1111010000110000'\n\n \"\"\"\n return OccurrenceHalved().fingerprint(word, n_bits, most_common)", "response": "Return the occurrence halved fingerprint."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef fingerprint(self, word, n_bits=16, most_common=MOST_COMMON_LETTERS_CG):\n if n_bits % 2:\n n_bits += 1\n\n w_len = len(word) // 2\n w_1 = set(word[:w_len])\n w_2 = set(word[w_len:])\n fingerprint = 0\n\n for letter in most_common:\n if n_bits:\n fingerprint <<= 1\n if letter in w_1:\n fingerprint += 1\n fingerprint <<= 1\n if letter in w_2:\n fingerprint += 1\n n_bits -= 2\n else:\n break\n\n if n_bits > 0:\n fingerprint <<= n_bits\n\n return fingerprint", "response": "Return the occurrence halved fingerprint based on the occurrence halved fingerprint from Cislak 2017."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the early version of the Henry code for a word. Parameters ---------- word : str The word to transform max_length : int The length of the code returned (defaults to 3) Returns ------- str The early Henry code Examples -------- >>> henry_early('Marchand') 'MRC' >>> henry_early('Beaulieu') 'BL' >>> henry_early('Beaumont') 'BM' >>> henry_early('Legrand') 'LGR' >>> henry_early('Pelletier') 'PLT'", "response": "def encode(self, word, max_length=3):\n \"\"\"Calculate the early version of the Henry code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to 3)\n\n Returns\n -------\n str\n The early Henry code\n\n Examples\n --------\n >>> henry_early('Marchand')\n 'MRC'\n >>> henry_early('Beaulieu')\n 'BL'\n >>> henry_early('Beaumont')\n 'BM'\n >>> henry_early('Legrand')\n 'LGR'\n >>> henry_early('Pelletier')\n 'PLT'\n\n \"\"\"\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = ''.join(c for c in word if c in self._uc_set)\n\n if not word:\n return ''\n\n # Rule Ia seems to be covered entirely in II\n\n # Rule Ib\n if word[0] in self._uc_vy_set:\n # Ib1\n if (\n word[1:2] in self._uc_c_set - {'M', 'N'}\n and word[2:3] in self._uc_c_set\n ) or (\n word[1:2] in self._uc_c_set and word[2:3] not in self._uc_c_set\n ):\n if word[0] == 'Y':\n word = 'I' + word[1:]\n # Ib2\n elif word[1:2] in {'M', 'N'} and word[2:3] in self._uc_c_set:\n if word[0] == 'E':\n word = 'A' + word[1:]\n elif word[0] in {'I', 'U', 'Y'}:\n word = 'E' + word[1:]\n # Ib3\n elif word[:2] in self._diph:\n word = self._diph[word[:2]] + word[2:]\n # Ib4\n elif word[1:2] in self._uc_vy_set and word[0] == 'Y':\n word = 'I' + word[1:]\n\n code = ''\n skip = 0\n\n # Rule II\n for pos, char in enumerate(word):\n nxch = word[pos + 1 : pos + 2]\n prev = word[pos - 1 : pos]\n\n if skip:\n skip -= 1\n elif char in self._uc_vy_set:\n code += char\n # IIc\n elif char == nxch:\n skip = 1\n code += char\n elif word[pos : pos + 2] in {'CQ', 'DT', 'SC'}:\n continue\n # IIb\n elif char in self._simple:\n code += self._simple[char]\n elif char in {'C', 'G', 'P', 'Q', 'S'}:\n if char == 'C':\n if nxch in {'A', 'O', 'U', 'L', 'R'}:\n code += 'K'\n elif nxch in {'E', 'I', 'Y'}:\n code += 'S'\n elif nxch == 'H':\n if word[pos + 2 : pos + 3] in self._uc_vy_set:\n code += 'C'\n else: # CHR, CHL, etc.\n code += 'K'\n else:\n code += 'C'\n elif char == 'G':\n if nxch in {'A', 'O', 'U', 'L', 'R'}:\n code += 'G'\n elif nxch in {'E', 'I', 'Y'}:\n code += 'J'\n elif nxch == 'N':\n code += 'N'\n elif char == 'P':\n if nxch != 'H':\n code += 'P'\n else:\n code += 'F'\n elif char == 'Q':\n if word[pos + 1 : pos + 3] in {'UE', 'UI', 'UY'}:\n code += 'G'\n else: # QUA, QUO, etc.\n code += 'K'\n else: # S...\n if word[pos : pos + 6] == 'SAINTE':\n code += 'X'\n skip = 5\n elif word[pos : pos + 5] == 'SAINT':\n code += 'X'\n skip = 4\n elif word[pos : pos + 3] == 'STE':\n code += 'X'\n skip = 2\n elif word[pos : pos + 2] == 'ST':\n code += 'X'\n skip = 1\n elif nxch in self._uc_c_set:\n continue\n else:\n code += 'S'\n # IId\n elif char == 'H' and prev in self._uc_c_set:\n continue\n elif char in self._uc_c_set - {\n 'L',\n 'R',\n } and nxch in self._uc_c_set - {'L', 'R'}:\n continue\n elif char == 'L' and nxch in {'M', 'N'}:\n continue\n elif (\n char in {'M', 'N'}\n and prev in self._uc_vy_set\n and nxch in self._uc_c_set\n ):\n continue\n # IIa\n else:\n code += char\n\n # IIe1\n if code[-4:] in {'AULT', 'EULT', 'OULT'}:\n code = code[:-2]\n # The following are blocked by rules above\n # elif code[-4:-3] in _vows and code[-3:] == 'MPS':\n # code = code[:-3]\n # elif code[-3:-2] in _vows and code[-2:] in {'MB', 'MP', 'ND',\n # 'NS', 'NT'}:\n # code = code[:-2]\n elif code[-2:-1] == 'R' and code[-1:] in self._uc_c_set:\n code = code[:-1]\n # IIe2\n elif code[-2:-1] in self._uc_vy_set and code[-1:] in {\n 'D',\n 'M',\n 'N',\n 'S',\n 'T',\n }:\n code = code[:-1]\n elif code[-2:] == 'ER':\n code = code[:-1]\n\n # Drop non-initial vowels\n code = code[:1] + code[1:].translate(\n {65: '', 69: '', 73: '', 79: '', 85: '', 89: ''}\n )\n\n if max_length != -1:\n code = code[:max_length]\n\n return code"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef pshp_soundex_first(fname, max_length=4, german=False):\n return PSHPSoundexFirst().encode(fname, max_length, german)", "response": "This function returns the PSHP Soundex First code for a given first name."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef encode(self, fname, max_length=4, german=False):\n fname = unicode_normalize('NFKD', text_type(fname.upper()))\n fname = fname.replace('\u00df', 'SS')\n fname = ''.join(c for c in fname if c in self._uc_set)\n\n # special rules\n if fname == 'JAMES':\n code = 'J7'\n elif fname == 'PAT':\n code = 'P7'\n\n else:\n # A. Prefix treatment\n if fname[:2] in {'GE', 'GI', 'GY'}:\n fname = 'J' + fname[1:]\n elif fname[:2] in {'CE', 'CI', 'CY'}:\n fname = 'S' + fname[1:]\n elif fname[:3] == 'CHR':\n fname = 'K' + fname[1:]\n elif fname[:1] == 'C' and fname[:2] != 'CH':\n fname = 'K' + fname[1:]\n\n if fname[:2] == 'KN':\n fname = 'N' + fname[1:]\n elif fname[:2] == 'PH':\n fname = 'F' + fname[1:]\n elif fname[:3] in {'WIE', 'WEI'}:\n fname = 'V' + fname[1:]\n\n if german and fname[:1] in {'W', 'M', 'Y', 'Z'}:\n fname = {'W': 'V', 'M': 'N', 'Y': 'J', 'Z': 'S'}[\n fname[0]\n ] + fname[1:]\n\n code = fname[:1]\n\n # B. Soundex coding\n # code for Y unspecified, but presumably is 0\n fname = fname.translate(self._trans)\n fname = self._delete_consecutive_repeats(fname)\n\n code += fname[1:]\n syl_ptr = code.find('0')\n syl2_ptr = code[syl_ptr + 1 :].find('0')\n if syl_ptr != -1 and syl2_ptr != -1 and syl2_ptr - syl_ptr > -1:\n code = code[: syl_ptr + 2]\n\n code = code.replace('0', '') # rule 1\n\n if max_length != -1:\n if len(code) < max_length:\n code += '0' * (max_length - len(code))\n else:\n code = code[:max_length]\n\n return code", "response": "This function encodes a first name and returns the code of the first name."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nconverting camelCase to snake_case", "response": "def c2u(name):\n \"\"\"Convert camelCase (used in PHP) to Python-standard snake_case.\n\n Src:\n https://stackoverflow.com/questions/1175208/elegant-python-function-to-convert-camelcase-to-snake-case\n\n Parameters\n ----------\n name: A function or variable name in camelCase\n\n Returns\n -------\n str: The name in snake_case\n\n \"\"\"\n s1 = re.sub('(.)([A-Z][a-z]+)', r'\\1_\\2', name)\n s1 = re.sub('([a-z0-9])([A-Z])', r'\\1_\\2', s1).lower()\n return s1"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nconverting a line of BMPM code from PHP to Python.", "response": "def pythonize(line, fn='', subdir='gen'):\n \"\"\"Convert a line of BMPM code from PHP to Python.\n\n Parameters\n ----------\n line : str\n A line of code\n fn : str\n A filename\n subdir : str\n The file's subdirectory\n\n Returns\n -------\n The code in Python\n\n \"\"\"\n global array_seen, nl, sd\n\n if '$all' in line:\n return ''\n if 'make the sum of all languages be visible in the function' in line:\n return ''\n\n line = line.strip()\n\n if 'array' in line and not line.startswith('//'):\n array_seen = True\n\n line = re.sub('//+', '#', line)\n # line = re.sub('\"\\.\\((\\$.+?)\\)\\.\"', r'\\1', line)\n if line and re.search(r'array\\(\"[^\"]+?\"\\)', line):\n # print(\"### \" + line)\n line = ''\n line = line.replace('array', '')\n line = re.sub(r'^\\s*', '', line)\n line = re.sub(';$', '', line)\n line = re.sub('^include_.+', '', line)\n\n line = re.sub(\n r'\\$(approx|rules|exact)\\[LanguageIndex\\(\"([^\"]+)\", '\n + r'\\$languages\\)\\] = \\$([a-zA-Z]+)',\n lambda m: (\n \"BMDATA['\"\n + subdir\n + \"']['\"\n + m.group(1)\n + \"'][L_\"\n + m.group(2).upper()\n + '] = _'\n + subdir.upper()\n + '_'\n + c2u(m.group(3)).upper()\n ),\n line,\n )\n\n line = re.sub(\n r'\\$(approx|rules|exact|hebrew)([A-Za-z]+) = _merge'\n + r'\\(\\$([a-zA-Z]+), \\$([a-zA-Z]+)\\)',\n lambda m: (\n \"BMDATA['\"\n + subdir\n + \"']['\"\n + m.group(1)\n + \"'][L_\"\n + c2u(m.group(2)).upper()\n + '] = _'\n + subdir.upper()\n + '_'\n + c2u(m.group(3)).upper()\n + ' + _'\n + subdir.upper()\n + '_'\n + c2u(m.group(4)).upper()\n ),\n line,\n )\n\n line = re.sub(\n r'\\$(approx|rules|exact)\\[LanguageIndex\\(\"([^\"]+)\", '\n + r'\\$languages\\)\\] = _merge\\(\\$([a-zA-Z]+), \\$([a-zA-Z]+)\\)',\n lambda m: (\n \"BMDATA['\"\n + subdir\n + \"']['\"\n + m.group(1)\n + \"'][L_\"\n + c2u(m.group(2)).upper()\n + '] = _'\n + subdir.upper()\n + '_'\n + c2u(m.group(3)).upper()\n + ' + _'\n + subdir.upper()\n + '_'\n + c2u(m.group(4)).upper()\n ),\n line,\n )\n\n line = re.sub(\n r'^\\$([a-zA-Z]+)',\n lambda m: '_' + sd.upper() + '_' + c2u(m.group(1)).upper(),\n line,\n )\n\n for _ in range(len(lang_tuple)):\n line = re.sub(r'($[a-zA-Z]+) *\\+ *($[a-zA-Z]+)', r'\\1\\+\\2', line)\n\n line = re.sub(\n r'\\$([a-zA-Z]+)',\n lambda m: (\n 'L_' + m.group(1).upper()\n if m.group(1) in lang_dict\n else '$' + m.group(1)\n ),\n line,\n )\n line = re.sub(r'\\[\\\"\\.\\((L_[A-Z_+]+)\\)\\.\\\"\\]', r'[\\1]', line)\n\n line = re.sub(\n 'L_([A-Z]+)', lambda m: str(lang_dict[m.group(1).lower()]), line\n )\n for _ in range(4):\n line = re.sub(\n r'([0-9]+) *\\+ *([0-9]+)',\n lambda m: str(int(m.group(1)) + int(m.group(2))),\n line,\n )\n\n if fn == 'lang':\n if len(line.split(',')) >= 3:\n parts = line.split(',')\n parts[0] = re.sub('/(.+?)/', r'\\1', parts[0])\n # parts[1] = re.sub('\\$', 'L_', parts[1])\n # parts[1] = re.sub(' *\\+ *', '|', parts[1])\n parts[2] = parts[2].title()\n line = ','.join(parts)\n\n if 'languagenames' in fn:\n line = line.replace('\"', \"'\")\n line = line.replace(\"','\", \"', '\")\n if line and line[0] == \"'\":\n line = ' ' * 14 + line\n\n # fix upstream\n # line = line.replace('\u00eb', '\u00fc')\n\n comment = ''\n if '#' in line:\n hashsign = line.find('#')\n comment = line[hashsign:]\n code = line[:hashsign]\n else:\n code = line\n\n code = code.rstrip()\n comment = comment.strip()\n if not re.match(r'^\\s*$', code):\n comment = ' ' + comment\n\n if '(' in code and ')' in code:\n prefix = code[: code.find('(') + 1]\n suffix = code[code.rfind(')') :]\n tuplecontent = code[len(prefix) : len(code) - len(suffix)]\n\n elts = tuplecontent.split(',')\n for i in range(len(elts)):\n elts[i] = elts[i].strip()\n if elts[i][0] == '\"' and elts[i][-1] == '\"':\n elts[i] = \"'\" + elts[i][1:-1].replace(\"'\", \"\\\\'\") + \"'\"\n tuplecontent = ', '.join(elts)\n\n code = prefix + tuplecontent + suffix\n\n line = code + comment\n line = re.sub('# *', '# ', line)\n\n if line:\n nl = False\n if array_seen and not (line[0] == '_' or line.startswith('BMDATA')):\n line = ' ' * 4 + line\n return line + '\\n'\n elif not nl:\n nl = True\n return '\\n'\n else:\n return ''"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the Monge - Elkan similarity of two strings.", "response": "def sim_monge_elkan(src, tar, sim_func=sim_levenshtein, symmetric=False):\n \"\"\"Return the Monge-Elkan similarity of two strings.\n\n This is a wrapper for :py:meth:`MongeElkan.sim`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n sim_func : function\n Rhe internal similarity metric to employ\n symmetric : bool\n Return a symmetric similarity measure\n\n Returns\n -------\n float\n Monge-Elkan similarity\n\n Examples\n --------\n >>> sim_monge_elkan('cat', 'hat')\n 0.75\n >>> round(sim_monge_elkan('Niall', 'Neil'), 12)\n 0.666666666667\n >>> round(sim_monge_elkan('aluminum', 'Catalan'), 12)\n 0.388888888889\n >>> sim_monge_elkan('ATCG', 'TAGC')\n 0.5\n\n \"\"\"\n return MongeElkan().sim(src, tar, sim_func, symmetric)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the Monge - Elkan distance between two strings.", "response": "def dist_monge_elkan(src, tar, sim_func=sim_levenshtein, symmetric=False):\n \"\"\"Return the Monge-Elkan distance between two strings.\n\n This is a wrapper for :py:meth:`MongeElkan.dist`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n sim_func : function\n The internal similarity metric to employ\n symmetric : bool\n Return a symmetric similarity measure\n\n Returns\n -------\n float\n Monge-Elkan distance\n\n Examples\n --------\n >>> dist_monge_elkan('cat', 'hat')\n 0.25\n >>> round(dist_monge_elkan('Niall', 'Neil'), 12)\n 0.333333333333\n >>> round(dist_monge_elkan('aluminum', 'Catalan'), 12)\n 0.611111111111\n >>> dist_monge_elkan('ATCG', 'TAGC')\n 0.5\n\n \"\"\"\n return MongeElkan().dist(src, tar, sim_func, symmetric)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef sim(self, src, tar, sim_func=sim_levenshtein, symmetric=False):\n if src == tar:\n return 1.0\n\n q_src = sorted(QGrams(src).elements())\n q_tar = sorted(QGrams(tar).elements())\n\n if not q_src or not q_tar:\n return 0.0\n\n sum_of_maxes = 0\n for q_s in q_src:\n max_sim = float('-inf')\n for q_t in q_tar:\n max_sim = max(max_sim, sim_func(q_s, q_t))\n sum_of_maxes += max_sim\n sim_em = sum_of_maxes / len(q_src)\n\n if symmetric:\n sim_em = (sim_em + self.sim(tar, src, sim_func, False)) / 2\n\n return sim_em", "response": "Return the Monge - Elkan similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Here you have a function in Python 3, explain what it does\ndef encode(self, word):\n word = unicode_normalize('NFC', text_type(word.upper()))\n for i, j in self._substitutions:\n word = word.replace(i, j)\n word = word.translate(self._trans)\n\n return ''.join(\n c\n for c in self._delete_consecutive_repeats(word)\n if c in self._uc_set\n )", "response": "Return the Phonem code for a word."} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef stem(self, word):\n wlen = len(word) - 2\n\n if wlen > 2 and word[-1] == 's':\n word = word[:-1]\n wlen -= 1\n\n _endings = {\n 5: {'elser', 'heten'},\n 4: {'arne', 'erna', 'ande', 'else', 'aste', 'orna', 'aren'},\n 3: {'are', 'ast', 'het'},\n 2: {'ar', 'er', 'or', 'en', 'at', 'te', 'et'},\n 1: {'a', 'e', 'n', 't'},\n }\n\n for end_len in range(5, 0, -1):\n if wlen > end_len and word[-end_len:] in _endings[end_len]:\n return word[:-end_len]\n return word", "response": "Return the stem of the word."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef _undouble(self, word):\n if (\n len(word) > 1\n and word[-1] == word[-2]\n and word[-1] in {'d', 'k', 't'}\n ):\n return word[:-1]\n return word", "response": "Undouble endings - kk dd and - tt."} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn Snowball Dutch stem.", "response": "def stem(self, word):\n \"\"\"Return Snowball Dutch stem.\n\n Parameters\n ----------\n word : str\n The word to stem\n\n Returns\n -------\n str\n Word stem\n\n Examples\n --------\n >>> stmr = SnowballDutch()\n >>> stmr.stem('lezen')\n 'lez'\n >>> stmr.stem('opschorting')\n 'opschort'\n >>> stmr.stem('ongrijpbaarheid')\n 'ongrijp'\n\n \"\"\"\n # lowercase, normalize, decompose, filter umlauts & acutes out, and\n # compose\n word = normalize('NFC', text_type(word.lower()))\n word = word.translate(self._accented)\n\n for i in range(len(word)):\n if i == 0 and word[0] == 'y':\n word = 'Y' + word[1:]\n elif word[i] == 'y' and word[i - 1] in self._vowels:\n word = word[:i] + 'Y' + word[i + 1 :]\n elif (\n word[i] == 'i'\n and word[i - 1] in self._vowels\n and i + 1 < len(word)\n and word[i + 1] in self._vowels\n ):\n word = word[:i] + 'I' + word[i + 1 :]\n\n r1_start = max(3, self._sb_r1(word))\n r2_start = self._sb_r2(word)\n\n # Step 1\n if word[-5:] == 'heden':\n if len(word[r1_start:]) >= 5:\n word = word[:-3] + 'id'\n elif word[-3:] == 'ene':\n if len(word[r1_start:]) >= 3 and (\n word[-4] not in self._vowels and word[-6:-3] != 'gem'\n ):\n word = self._undouble(word[:-3])\n elif word[-2:] == 'en':\n if len(word[r1_start:]) >= 2 and (\n word[-3] not in self._vowels and word[-5:-2] != 'gem'\n ):\n word = self._undouble(word[:-2])\n elif word[-2:] == 'se':\n if (\n len(word[r1_start:]) >= 2\n and word[-3] not in self._not_s_endings\n ):\n word = word[:-2]\n elif word[-1:] == 's':\n if (\n len(word[r1_start:]) >= 1\n and word[-2] not in self._not_s_endings\n ):\n word = word[:-1]\n\n # Step 2\n e_removed = False\n if word[-1:] == 'e':\n if len(word[r1_start:]) >= 1 and word[-2] not in self._vowels:\n word = self._undouble(word[:-1])\n e_removed = True\n\n # Step 3a\n if word[-4:] == 'heid':\n if len(word[r2_start:]) >= 4 and word[-5] != 'c':\n word = word[:-4]\n if word[-2:] == 'en':\n if len(word[r1_start:]) >= 2 and (\n word[-3] not in self._vowels and word[-5:-2] != 'gem'\n ):\n word = self._undouble(word[:-2])\n\n # Step 3b\n if word[-4:] == 'lijk':\n if len(word[r2_start:]) >= 4:\n word = word[:-4]\n # Repeat step 2\n if word[-1:] == 'e':\n if (\n len(word[r1_start:]) >= 1\n and word[-2] not in self._vowels\n ):\n word = self._undouble(word[:-1])\n elif word[-4:] == 'baar':\n if len(word[r2_start:]) >= 4:\n word = word[:-4]\n elif word[-3:] in ('end', 'ing'):\n if len(word[r2_start:]) >= 3:\n word = word[:-3]\n if (\n word[-2:] == 'ig'\n and len(word[r2_start:]) >= 2\n and word[-3] != 'e'\n ):\n word = word[:-2]\n else:\n word = self._undouble(word)\n elif word[-3:] == 'bar':\n if len(word[r2_start:]) >= 3 and e_removed:\n word = word[:-3]\n elif word[-2:] == 'ig':\n if len(word[r2_start:]) >= 2 and word[-3] != 'e':\n word = word[:-2]\n\n # Step 4\n if (\n len(word) >= 4\n and word[-3] == word[-2]\n and word[-2] in {'a', 'e', 'o', 'u'}\n and word[-4] not in self._vowels\n and word[-1] not in self._vowels\n and word[-1] != 'I'\n ):\n word = word[:-2] + word[-1]\n\n # Change 'Y' and 'U' back to lowercase if survived stemming\n for i in range(0, len(word)):\n if word[i] == 'Y':\n word = word[:i] + 'y' + word[i + 1 :]\n elif word[i] == 'I':\n word = word[:i] + 'i' + word[i + 1 :]\n\n return word"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef soundex(word, max_length=4, var='American', reverse=False, zero_pad=True):\n return Soundex().encode(word, max_length, var, reverse, zero_pad)", "response": "Return the Soundex code for a word."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef encode(\n self, word, max_length=4, var='American', reverse=False, zero_pad=True\n ):\n \"\"\"Return the Soundex code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to 4)\n var : str\n The variant of the algorithm to employ (defaults to ``American``):\n\n - ``American`` follows the American Soundex algorithm, as\n described at :cite:`US:2007` and in :cite:`Knuth:1998`; this\n is also called Miracode\n - ``special`` follows the rules from the 1880-1910 US Census\n retrospective re-analysis, in which h & w are not treated as\n blocking consonants but as vowels. Cf. :cite:`Repici:2013`.\n - ``Census`` follows the rules laid out in GIL 55\n :cite:`US:1997` by the US Census, including coding prefixed\n and unprefixed versions of some names\n\n reverse : bool\n Reverse the word before computing the selected Soundex (defaults to\n False); This results in \"Reverse Soundex\", which is useful for\n blocking in cases where the initial elements may be in error.\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length\n string\n\n Returns\n -------\n str\n The Soundex value\n\n Examples\n --------\n >>> pe = Soundex()\n >>> pe.encode(\"Christopher\")\n 'C623'\n >>> pe.encode(\"Niall\")\n 'N400'\n >>> pe.encode('Smith')\n 'S530'\n >>> pe.encode('Schmidt')\n 'S530'\n\n >>> pe.encode('Christopher', max_length=-1)\n 'C623160000000000000000000000000000000000000000000000000000000000'\n >>> pe.encode('Christopher', max_length=-1, zero_pad=False)\n 'C62316'\n\n >>> pe.encode('Christopher', reverse=True)\n 'R132'\n\n >>> pe.encode('Ashcroft')\n 'A261'\n >>> pe.encode('Asicroft')\n 'A226'\n >>> pe.encode('Ashcroft', var='special')\n 'A226'\n >>> pe.encode('Asicroft', var='special')\n 'A226'\n\n \"\"\"\n # Require a max_length of at least 4 and not more than 64\n if max_length != -1:\n max_length = min(max(4, max_length), 64)\n else:\n max_length = 64\n\n # uppercase, normalize, decompose, and filter non-A-Z out\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n\n if var == 'Census':\n if word[:3] in {'VAN', 'CON'} and len(word) > 4:\n return (\n soundex(word, max_length, 'American', reverse, zero_pad),\n soundex(\n word[3:], max_length, 'American', reverse, zero_pad\n ),\n )\n if word[:2] in {'DE', 'DI', 'LA', 'LE'} and len(word) > 3:\n return (\n soundex(word, max_length, 'American', reverse, zero_pad),\n soundex(\n word[2:], max_length, 'American', reverse, zero_pad\n ),\n )\n # Otherwise, proceed as usual (var='American' mode, ostensibly)\n\n word = ''.join(c for c in word if c in self._uc_set)\n\n # Nothing to convert, return base case\n if not word:\n if zero_pad:\n return '0' * max_length\n return '0'\n\n # Reverse word if computing Reverse Soundex\n if reverse:\n word = word[::-1]\n\n # apply the Soundex algorithm\n sdx = word.translate(self._trans)\n\n if var == 'special':\n sdx = sdx.replace('9', '0') # special rule for 1880-1910 census\n else:\n sdx = sdx.replace('9', '') # rule 1\n sdx = self._delete_consecutive_repeats(sdx) # rule 3\n\n if word[0] in 'HW':\n sdx = word[0] + sdx\n else:\n sdx = word[0] + sdx[1:]\n sdx = sdx.replace('0', '') # rule 1\n\n if zero_pad:\n sdx += '0' * max_length # rule 4\n\n return sdx[:max_length]", "response": "Encodes a string into a Soundex code."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef fingerprint(self, phrase, joiner=' '):\n phrase = unicode_normalize('NFKD', text_type(phrase.strip().lower()))\n phrase = ''.join([c for c in phrase if c.isalnum() or c.isspace()])\n phrase = joiner.join(sorted(list(set(phrase.split()))))\n return phrase", "response": "Return string fingerprint.\n\n Parameters\n ----------\n phrase : str\n The string from which to calculate the fingerprint\n joiner : str\n The string that will be placed between each word\n\n Returns\n -------\n str\n The fingerprint of the phrase\n\n Example\n -------\n >>> sf = String()\n >>> sf.fingerprint('The quick brown fox jumped over the lazy dog.')\n 'brown dog fox jumped lazy over quick the'"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the Russell Index output of a word.", "response": "def encode(self, word):\n \"\"\"Return the Russell Index (integer output) of a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n\n Returns\n -------\n int\n The Russell Index value\n\n Examples\n --------\n >>> pe = RussellIndex()\n >>> pe.encode('Christopher')\n 3813428\n >>> pe.encode('Niall')\n 715\n >>> pe.encode('Smith')\n 3614\n >>> pe.encode('Schmidt')\n 3614\n\n \"\"\"\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = word.replace('GH', '') # discard gh (rule 3)\n word = word.rstrip('SZ') # discard /[sz]$/ (rule 3)\n\n # translate according to Russell's mapping\n word = ''.join(c for c in word if c in self._uc_set)\n sdx = word.translate(self._trans)\n\n # remove any 1s after the first occurrence\n one = sdx.find('1') + 1\n if one:\n sdx = sdx[:one] + ''.join(c for c in sdx[one:] if c != '1')\n\n # remove repeating characters\n sdx = self._delete_consecutive_repeats(sdx)\n\n # return as an int\n return int(sdx) if sdx else float('NaN')"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef ipa_to_features(ipa):\n features = []\n pos = 0\n ipa = normalize('NFD', text_type(ipa.lower()))\n\n maxsymlen = max(len(_) for _ in _PHONETIC_FEATURES)\n\n while pos < len(ipa):\n found_match = False\n for i in range(maxsymlen, 0, -1):\n if (\n pos + i - 1 <= len(ipa)\n and ipa[pos : pos + i] in _PHONETIC_FEATURES\n ):\n features.append(_PHONETIC_FEATURES[ipa[pos : pos + i]])\n pos += i\n found_match = True\n\n if not found_match:\n features.append(-1)\n pos += 1\n\n return features", "response": "Convert an IPA string of one or more phones to a list of ints."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nget a list of ints representing presence and absence and neutral features of a particular phonetic feature.", "response": "def get_feature(vector, feature):\n \"\"\"Get a feature vector.\n\n This returns a list of ints, equal in length to the vector input,\n representing presence/absence/neutrality with respect to a particular\n phonetic feature.\n\n Parameters\n ----------\n vector : list\n A tuple or list of ints representing the phonetic features of a phone\n or series of phones (such as is returned by the ipa_to_features\n function)\n feature : str\n A feature name from the set:\n\n - ``consonantal``\n - ``sonorant``\n - ``syllabic``\n - ``labial``\n - ``round``\n - ``coronal``\n - ``anterior``\n - ``distributed``\n - ``dorsal``\n - ``high``\n - ``low``\n - ``back``\n - ``tense``\n - ``pharyngeal``\n - ``ATR``\n - ``voice``\n - ``spread_glottis``\n - ``constricted_glottis``\n - ``continuant``\n - ``strident``\n - ``lateral``\n - ``delayed_release``\n - ``nasal``\n\n Returns\n -------\n list of ints\n A list indicating presence/absence/neutrality with respect to the\n feature\n\n Raises\n ------\n AttributeError\n feature must be one of ...\n\n Examples\n --------\n >>> tails = ipa_to_features('telz')\n >>> get_feature(tails, 'consonantal')\n [1, -1, 1, 1]\n >>> get_feature(tails, 'sonorant')\n [-1, 1, 1, -1]\n >>> get_feature(tails, 'nasal')\n [-1, -1, -1, -1]\n >>> get_feature(tails, 'coronal')\n [1, -1, 1, 1]\n\n \"\"\"\n # :param bool binary: if False, -1, 0, & 1 represent -, 0, & +\n # if True, only binary oppositions are allowed:\n # 0 & 1 represent - & + and 0s are mapped to -\n\n if feature not in _FEATURE_MASK:\n raise AttributeError(\n \"feature must be one of: '\"\n + \"', '\".join(\n (\n 'consonantal',\n 'sonorant',\n 'syllabic',\n 'labial',\n 'round',\n 'coronal',\n 'anterior',\n 'distributed',\n 'dorsal',\n 'high',\n 'low',\n 'back',\n 'tense',\n 'pharyngeal',\n 'ATR',\n 'voice',\n 'spread_glottis',\n 'constricted_glottis',\n 'continuant',\n 'strident',\n 'lateral',\n 'delayed_release',\n 'nasal',\n )\n )\n + \"'\"\n )\n\n # each feature mask contains two bits, one each for - and +\n mask = _FEATURE_MASK[feature]\n # the lower bit represents +\n pos_mask = mask >> 1\n retvec = []\n for char in vector:\n if char < 0:\n retvec.append(float('NaN'))\n else:\n masked = char & mask\n if masked == 0:\n retvec.append(0) # 0\n elif masked == mask:\n retvec.append(2) # +/-\n elif masked & pos_mask:\n retvec.append(1) # +\n else:\n retvec.append(-1) # -\n\n return retvec"} {"SOURCE": "codesearchnet", "instruction": "How would you explain what the following Python 3 function does\ndef cmp_features(feat1, feat2):\n if feat1 < 0 or feat2 < 0:\n return -1.0\n if feat1 == feat2:\n return 1.0\n\n magnitude = len(_FEATURE_MASK)\n featxor = feat1 ^ feat2\n diffbits = 0\n # print(featxor)\n while featxor:\n if featxor & 0b1:\n diffbits += 1\n featxor >>= 1\n # print(diffbits)\n return 1 - (diffbits / (2 * magnitude))", "response": "Compares two feature bundles and returns a number representing a comparison of the two features."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sim(self, src, tar):\n if src == tar:\n return 1.0\n if not src or not tar:\n return 0.0\n return (\n len(src) / len(tar) if len(src) < len(tar) else len(tar) / len(src)\n )", "response": "Return the length similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nencoding a string into a Caverphone code.", "response": "def encode(self, word, version=2):\n \"\"\"Return the Caverphone code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n version : int\n The version of Caverphone to employ for encoding (defaults to 2)\n\n Returns\n -------\n str\n The Caverphone value\n\n Examples\n --------\n >>> pe = Caverphone()\n >>> pe.encode('Christopher')\n 'KRSTFA1111'\n >>> pe.encode('Niall')\n 'NA11111111'\n >>> pe.encode('Smith')\n 'SMT1111111'\n >>> pe.encode('Schmidt')\n 'SKMT111111'\n\n >>> pe.encode('Christopher', 1)\n 'KRSTF1'\n >>> pe.encode('Niall', 1)\n 'N11111'\n >>> pe.encode('Smith', 1)\n 'SMT111'\n >>> pe.encode('Schmidt', 1)\n 'SKMT11'\n\n \"\"\"\n word = word.lower()\n word = ''.join(c for c in word if c in self._lc_set)\n\n def _squeeze_replace(word, char):\n \"\"\"Convert strings of char in word to one instance.\n\n Parameters\n ----------\n word : str\n The partially converted word\n char : str\n A character to 'squeeze'\n\n Returns\n -------\n str\n The word with instances of char squeezed down to one\n\n \"\"\"\n while char * 2 in word:\n word = word.replace(char * 2, char)\n return word.replace(char, char.upper())\n\n # the main replacement algorithm\n if version != 1 and word[-1:] == 'e':\n word = word[:-1]\n if word:\n if word[:5] == 'cough':\n word = 'cou2f' + word[5:]\n if word[:5] == 'rough':\n word = 'rou2f' + word[5:]\n if word[:5] == 'tough':\n word = 'tou2f' + word[5:]\n if word[:6] == 'enough':\n word = 'enou2f' + word[6:]\n if version != 1 and word[:6] == 'trough':\n word = 'trou2f' + word[6:]\n if word[:2] == 'gn':\n word = '2n' + word[2:]\n if word[-2:] == 'mb':\n word = word[:-1] + '2'\n for src, tar in (\n ('cq', '2q'),\n ('ci', 'si'),\n ('ce', 'se'),\n ('cy', 'sy'),\n ('tch', '2ch'),\n ('c', 'k'),\n ('q', 'k'),\n ('x', 'k'),\n ('v', 'f'),\n ('dg', '2g'),\n ('tio', 'sio'),\n ('tia', 'sia'),\n ('d', 't'),\n ('ph', 'fh'),\n ('b', 'p'),\n ('sh', 's2'),\n ('z', 's'),\n ):\n word = word.replace(src, tar)\n if word[0] in self._lc_v_set:\n word = 'A' + word[1:]\n for vowel in 'aeiou':\n word = word.replace(vowel, '3')\n if version != 1:\n word = word.replace('j', 'y')\n if word[:2] == 'y3':\n word = 'Y3' + word[2:]\n if word[:1] == 'y':\n word = 'A' + word[1:]\n word = word.replace('y', '3')\n for src, tar in (('3gh3', '3kh3'), ('gh', '22'), ('g', 'k')):\n word = word.replace(src, tar)\n\n for char in 'stpkfmn':\n word = _squeeze_replace(word, char)\n\n word = word.replace('w3', 'W3')\n if version == 1:\n word = word.replace('wy', 'Wy')\n word = word.replace('wh3', 'Wh3')\n if version == 1:\n word = word.replace('why', 'Why')\n if version != 1 and word[-1:] == 'w':\n word = word[:-1] + '3'\n word = word.replace('w', '2')\n if word[:1] == 'h':\n word = 'A' + word[1:]\n word = word.replace('h', '2')\n word = word.replace('r3', 'R3')\n if version == 1:\n word = word.replace('ry', 'Ry')\n if version != 1 and word[-1:] == 'r':\n word = word[:-1] + '3'\n word = word.replace('r', '2')\n word = word.replace('l3', 'L3')\n if version == 1:\n word = word.replace('ly', 'Ly')\n if version != 1 and word[-1:] == 'l':\n word = word[:-1] + '3'\n word = word.replace('l', '2')\n if version == 1:\n word = word.replace('j', 'y')\n word = word.replace('y3', 'Y3')\n word = word.replace('y', '2')\n word = word.replace('2', '')\n if version != 1 and word[-1:] == '3':\n word = word[:-1] + 'A'\n word = word.replace('3', '')\n\n # pad with 1s, then extract the necessary length of code\n word += '1' * 10\n if version != 1:\n word = word[:10]\n else:\n word = word[:6]\n\n return word"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef imean(nums):\n if len(nums) == 1:\n return nums[0]\n if len(nums) > 2:\n raise AttributeError('imean supports no more than two values')\n if nums[0] <= 0 or nums[1] <= 0:\n return float('NaN')\n elif nums[0] == nums[1]:\n return nums[0]\n return (1 / math.e) * (nums[0] ** nums[0] / nums[1] ** nums[1]) ** (\n 1 / (nums[0] - nums[1])\n )", "response": "r Return the identric mean of two numbers x and y."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef seiffert_mean(nums):\n if len(nums) == 1:\n return nums[0]\n if len(nums) > 2:\n raise AttributeError('seiffert_mean supports no more than two values')\n if nums[0] + nums[1] == 0 or nums[0] - nums[1] == 0:\n return float('NaN')\n return (nums[0] - nums[1]) / (\n 2 * math.asin((nums[0] - nums[1]) / (nums[0] + nums[1]))\n )", "response": "r Returns Seiffert s mean of two numbers x and y."} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the Lehmer mean of a series of numbers and exponent.", "response": "def lehmer_mean(nums, exp=2):\n r\"\"\"Return Lehmer mean.\n\n The Lehmer mean is:\n :math:`\\frac{\\sum\\limits_i{x_i^p}}{\\sum\\limits_i{x_i^(p-1)}}`\n\n Cf. https://en.wikipedia.org/wiki/Lehmer_mean\n\n Parameters\n ----------\n nums : list\n A series of numbers\n exp : numeric\n The exponent of the Lehmer mean\n\n Returns\n -------\n float\n The Lehmer mean of nums for the given exponent\n\n Examples\n --------\n >>> lehmer_mean([1, 2, 3, 4])\n 3.0\n >>> lehmer_mean([1, 2])\n 1.6666666666666667\n >>> lehmer_mean([0, 5, 1000])\n 995.0497512437811\n\n \"\"\"\n return sum(x ** exp for x in nums) / sum(x ** (exp - 1) for x in nums)"} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef heronian_mean(nums):\n mag = len(nums)\n rolling_sum = 0\n for i in range(mag):\n for j in range(i, mag):\n if nums[i] == nums[j]:\n rolling_sum += nums[i]\n else:\n rolling_sum += (nums[i] * nums[j]) ** 0.5\n return rolling_sum * 2 / (mag * (mag + 1))", "response": "Return the Heronian mean of a series of numbers."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the H\u00f6lder mean of a series of numbers and exponent.", "response": "def hoelder_mean(nums, exp=2):\n r\"\"\"Return H\u00f6lder (power/generalized) mean.\n\n The H\u00f6lder mean is defined as:\n :math:`\\sqrt[p]{\\frac{1}{|nums|} \\cdot \\sum\\limits_i{x_i^p}}`\n for :math:`p \\ne 0`, and the geometric mean for :math:`p = 0`\n\n Cf. https://en.wikipedia.org/wiki/Generalized_mean\n\n Parameters\n ----------\n nums : list\n A series of numbers\n exp : numeric\n The exponent of the H\u00f6lder mean\n\n Returns\n -------\n float\n The H\u00f6lder mean of nums for the given exponent\n\n Examples\n --------\n >>> hoelder_mean([1, 2, 3, 4])\n 2.7386127875258306\n >>> hoelder_mean([1, 2])\n 1.5811388300841898\n >>> hoelder_mean([0, 5, 1000])\n 577.3574860228857\n\n \"\"\"\n if exp == 0:\n return gmean(nums)\n return ((1 / len(nums)) * sum(i ** exp for i in nums)) ** (1 / exp)"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the arithmetic - geometric mean of the given numbers.", "response": "def agmean(nums):\n \"\"\"Return arithmetic-geometric mean.\n\n Iterates between arithmetic & geometric means until they converge to\n a single value (rounded to 12 digits).\n\n Cf. https://en.wikipedia.org/wiki/Arithmetic-geometric_mean\n\n Parameters\n ----------\n nums : list\n A series of numbers\n\n Returns\n -------\n float\n The arithmetic-geometric mean of nums\n\n Examples\n --------\n >>> agmean([1, 2, 3, 4])\n 2.3545004777751077\n >>> agmean([1, 2])\n 1.4567910310469068\n >>> agmean([0, 5, 1000])\n 2.9753977059954195e-13\n\n \"\"\"\n m_a = amean(nums)\n m_g = gmean(nums)\n if math.isnan(m_a) or math.isnan(m_g):\n return float('nan')\n while round(m_a, 12) != round(m_g, 12):\n m_a, m_g = (m_a + m_g) / 2, (m_a * m_g) ** (1 / 2)\n return m_a"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef ghmean(nums):\n m_g = gmean(nums)\n m_h = hmean(nums)\n if math.isnan(m_g) or math.isnan(m_h):\n return float('nan')\n while round(m_h, 12) != round(m_g, 12):\n m_g, m_h = (m_g * m_h) ** (1 / 2), (2 * m_g * m_h) / (m_g + m_h)\n return m_g", "response": "Returns the geometric - harmonic mean of the given numbers."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef aghmean(nums):\n m_a = amean(nums)\n m_g = gmean(nums)\n m_h = hmean(nums)\n if math.isnan(m_a) or math.isnan(m_g) or math.isnan(m_h):\n return float('nan')\n while round(m_a, 12) != round(m_g, 12) and round(m_g, 12) != round(\n m_h, 12\n ):\n m_a, m_g, m_h = (\n (m_a + m_g + m_h) / 3,\n (m_a * m_g * m_h) ** (1 / 3),\n 3 / (1 / m_a + 1 / m_g + 1 / m_h),\n )\n return m_a", "response": "Return an arithmetic - geometric - harmonic mean."} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef median(nums):\n nums = sorted(nums)\n mag = len(nums)\n if mag % 2:\n mag = int((mag - 1) / 2)\n return nums[mag]\n mag = int(mag / 2)\n med = (nums[mag - 1] + nums[mag]) / 2\n return med if not med.is_integer() else int(med)", "response": "Return the median of a list of numbers sorted by value."} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the stem of a word according to the Schinke stemmer.", "response": "def stem(self, word):\n \"\"\"Return the stem of a word according to the Schinke stemmer.\n\n Parameters\n ----------\n word : str\n The word to stem\n\n Returns\n -------\n str\n Word stem\n\n Examples\n --------\n >>> stmr = Schinke()\n >>> stmr.stem('atque')\n {'n': 'atque', 'v': 'atque'}\n >>> stmr.stem('census')\n {'n': 'cens', 'v': 'censu'}\n >>> stmr.stem('virum')\n {'n': 'uir', 'v': 'uiru'}\n >>> stmr.stem('populusque')\n {'n': 'popul', 'v': 'populu'}\n >>> stmr.stem('senatus')\n {'n': 'senat', 'v': 'senatu'}\n\n \"\"\"\n word = normalize('NFKD', text_type(word.lower()))\n word = ''.join(\n c\n for c in word\n if c\n in {\n 'a',\n 'b',\n 'c',\n 'd',\n 'e',\n 'f',\n 'g',\n 'h',\n 'i',\n 'j',\n 'k',\n 'l',\n 'm',\n 'n',\n 'o',\n 'p',\n 'q',\n 'r',\n 's',\n 't',\n 'u',\n 'v',\n 'w',\n 'x',\n 'y',\n 'z',\n }\n )\n\n # Rule 2\n word = word.replace('j', 'i').replace('v', 'u')\n\n # Rule 3\n if word[-3:] == 'que':\n # This diverges from the paper by also returning 'que' itself\n # unstemmed\n if word[:-3] in self._keep_que or word == 'que':\n return {'n': word, 'v': word}\n else:\n word = word[:-3]\n\n # Base case will mean returning the words as is\n noun = word\n verb = word\n\n # Rule 4\n for endlen in range(4, 0, -1):\n if word[-endlen:] in self._n_endings[endlen]:\n if len(word) - 2 >= endlen:\n noun = word[:-endlen]\n else:\n noun = word\n break\n\n for endlen in range(6, 0, -1):\n if word[-endlen:] in self._v_endings_strip[endlen]:\n if len(word) - 2 >= endlen:\n verb = word[:-endlen]\n else:\n verb = word\n break\n if word[-endlen:] in self._v_endings_alter[endlen]:\n if word[-endlen:] in {\n 'iuntur',\n 'erunt',\n 'untur',\n 'iunt',\n 'unt',\n }:\n new_word = word[:-endlen] + 'i'\n addlen = 1\n elif word[-endlen:] in {'beris', 'bor', 'bo'}:\n new_word = word[:-endlen] + 'bi'\n addlen = 2\n else:\n new_word = word[:-endlen] + 'eri'\n addlen = 3\n\n # Technically this diverges from the paper by considering the\n # length of the stem without the new suffix\n if len(new_word) >= 2 + addlen:\n verb = new_word\n else:\n verb = word\n break\n\n return {'n': noun, 'v': verb}"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the Oxford Name Compression Algorithm ( ONCA ) code for a word.", "response": "def onca(word, max_length=4, zero_pad=True):\n \"\"\"Return the Oxford Name Compression Algorithm (ONCA) code for a word.\n\n This is a wrapper for :py:meth:`ONCA.encode`.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The maximum length (default 5) of the code to return\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length string\n\n Returns\n -------\n str\n The ONCA code\n\n Examples\n --------\n >>> onca('Christopher')\n 'C623'\n >>> onca('Niall')\n 'N400'\n >>> onca('Smith')\n 'S530'\n >>> onca('Schmidt')\n 'S530'\n\n \"\"\"\n return ONCA().encode(word, max_length, zero_pad)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the Oxford Name Compression Algorithm for a word.", "response": "def encode(self, word, max_length=4, zero_pad=True):\n \"\"\"Return the Oxford Name Compression Algorithm (ONCA) code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The maximum length (default 5) of the code to return\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length\n string\n\n Returns\n -------\n str\n The ONCA code\n\n Examples\n --------\n >>> pe = ONCA()\n >>> pe.encode('Christopher')\n 'C623'\n >>> pe.encode('Niall')\n 'N400'\n >>> pe.encode('Smith')\n 'S530'\n >>> pe.encode('Schmidt')\n 'S530'\n\n \"\"\"\n # In the most extreme case, 3 characters of NYSIIS input can be\n # compressed to one character of output, so give it triple the\n # max_length.\n return self._soundex.encode(\n self._nysiis.encode(word, max_length=max_length * 3),\n max_length,\n zero_pad=zero_pad,\n )"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef phonix(word, max_length=4, zero_pad=True):\n return Phonix().encode(word, max_length, zero_pad)", "response": "Returns the Phonix code for a word."} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef encode(self, word, max_length=4, zero_pad=True):\n\n def _start_repl(word, src, tar, post=None):\n \"\"\"Replace src with tar at the start of word.\n\n Parameters\n ----------\n word : str\n The word to modify\n src : str\n Substring to match\n tar : str\n Substring to substitute\n post : set\n Following characters\n\n Returns\n -------\n str\n Modified string\n\n \"\"\"\n if post:\n for i in post:\n if word.startswith(src + i):\n return tar + word[len(src) :]\n elif word.startswith(src):\n return tar + word[len(src) :]\n return word\n\n def _end_repl(word, src, tar, pre=None):\n \"\"\"Replace src with tar at the end of word.\n\n Parameters\n ----------\n word : str\n The word to modify\n src : str\n Substring to match\n tar : str\n Substring to substitute\n pre : set\n Preceding characters\n\n Returns\n -------\n str\n Modified string\n\n \"\"\"\n if pre:\n for i in pre:\n if word.endswith(i + src):\n return word[: -len(src)] + tar\n elif word.endswith(src):\n return word[: -len(src)] + tar\n return word\n\n def _mid_repl(word, src, tar, pre=None, post=None):\n \"\"\"Replace src with tar in the middle of word.\n\n Parameters\n ----------\n word : str\n The word to modify\n src : str\n Substring to match\n tar : str\n Substring to substitute\n pre : set\n Preceding characters\n post : set\n Following characters\n\n Returns\n -------\n str\n Modified string\n\n \"\"\"\n if pre or post:\n if not pre:\n return word[0] + _all_repl(word[1:], src, tar, pre, post)\n elif not post:\n return _all_repl(word[:-1], src, tar, pre, post) + word[-1]\n return _all_repl(word, src, tar, pre, post)\n return (\n word[0] + _all_repl(word[1:-1], src, tar, pre, post) + word[-1]\n )\n\n def _all_repl(word, src, tar, pre=None, post=None):\n \"\"\"Replace src with tar anywhere in word.\n\n Parameters\n ----------\n word : str\n The word to modify\n src : str\n Substring to match\n tar : str\n Substring to substitute\n pre : set\n Preceding characters\n post : set\n Following characters\n\n Returns\n -------\n str\n Modified string\n\n \"\"\"\n if pre or post:\n if post:\n post = post\n else:\n post = frozenset(('',))\n if pre:\n pre = pre\n else:\n pre = frozenset(('',))\n\n for i, j in ((i, j) for i in pre for j in post):\n word = word.replace(i + src + j, i + tar + j)\n return word\n else:\n return word.replace(src, tar)\n\n repl_at = (_start_repl, _end_repl, _mid_repl, _all_repl)\n\n sdx = ''\n\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n if word:\n for trans in self._substitutions:\n word = repl_at[trans[0]](word, *trans[1:])\n if word[0] in self._uc_vy_set:\n sdx = 'v' + word[1:].translate(self._trans)\n else:\n sdx = word[0] + word[1:].translate(self._trans)\n sdx = self._delete_consecutive_repeats(sdx)\n sdx = sdx.replace('0', '')\n\n # Clamp max_length to [4, 64]\n if max_length != -1:\n max_length = min(max(4, max_length), 64)\n else:\n max_length = 64\n\n if zero_pad:\n sdx += '0' * max_length\n if not sdx:\n sdx = '0'\n return sdx[:max_length]", "response": "Encodes a word into a Phonix code."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef editex(src, tar, cost=(0, 1, 2), local=False):\n return Editex().dist_abs(src, tar, cost, local)", "response": "Returns the distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the normalized distance between two strings.", "response": "def dist_editex(src, tar, cost=(0, 1, 2), local=False):\n \"\"\"Return the normalized Editex distance between two strings.\n\n This is a wrapper for :py:meth:`Editex.dist`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n cost : tuple\n A 3-tuple representing the cost of the four possible edits: match,\n same-group, and mismatch respectively (by default: (0, 1, 2))\n local : bool\n If True, the local variant of Editex is used\n\n Returns\n -------\n int\n Normalized Editex distance\n\n Examples\n --------\n >>> round(dist_editex('cat', 'hat'), 12)\n 0.333333333333\n >>> round(dist_editex('Niall', 'Neil'), 12)\n 0.2\n >>> dist_editex('aluminum', 'Catalan')\n 0.75\n >>> dist_editex('ATCG', 'TAGC')\n 0.75\n\n \"\"\"\n return Editex().dist(src, tar, cost, local)"} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef sim_editex(src, tar, cost=(0, 1, 2), local=False):\n return Editex().sim(src, tar, cost, local)", "response": "Return the normalized Editex similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you tell what is the following Python 3 function doing\ndef dist_abs(self, src, tar, cost=(0, 1, 2), local=False):\n match_cost, group_cost, mismatch_cost = cost\n\n def r_cost(ch1, ch2):\n \"\"\"Return r(a,b) according to Zobel & Dart's definition.\n\n Parameters\n ----------\n ch1 : str\n The first character to compare\n ch2 : str\n The second character to compare\n\n Returns\n -------\n int\n r(a,b) according to Zobel & Dart's definition\n\n \"\"\"\n if ch1 == ch2:\n return match_cost\n if ch1 in self._all_letters and ch2 in self._all_letters:\n for group in self._letter_groups:\n if ch1 in group and ch2 in group:\n return group_cost\n return mismatch_cost\n\n def d_cost(ch1, ch2):\n \"\"\"Return d(a,b) according to Zobel & Dart's definition.\n\n Parameters\n ----------\n ch1 : str\n The first character to compare\n ch2 : str\n The second character to compare\n\n Returns\n -------\n int\n d(a,b) according to Zobel & Dart's definition\n\n \"\"\"\n if ch1 != ch2 and (ch1 == 'H' or ch1 == 'W'):\n return group_cost\n return r_cost(ch1, ch2)\n\n # convert both src & tar to NFKD normalized unicode\n src = unicode_normalize('NFKD', text_type(src.upper()))\n tar = unicode_normalize('NFKD', text_type(tar.upper()))\n # convert \u00df to SS (for Python2)\n src = src.replace('\u00df', 'SS')\n tar = tar.replace('\u00df', 'SS')\n\n if src == tar:\n return 0.0\n if not src:\n return len(tar) * mismatch_cost\n if not tar:\n return len(src) * mismatch_cost\n\n d_mat = np_zeros((len(src) + 1, len(tar) + 1), dtype=np_int)\n lens = len(src)\n lent = len(tar)\n src = ' ' + src\n tar = ' ' + tar\n\n if not local:\n for i in range(1, lens + 1):\n d_mat[i, 0] = d_mat[i - 1, 0] + d_cost(src[i - 1], src[i])\n for j in range(1, lent + 1):\n d_mat[0, j] = d_mat[0, j - 1] + d_cost(tar[j - 1], tar[j])\n\n for i in range(1, lens + 1):\n for j in range(1, lent + 1):\n d_mat[i, j] = min(\n d_mat[i - 1, j] + d_cost(src[i - 1], src[i]),\n d_mat[i, j - 1] + d_cost(tar[j - 1], tar[j]),\n d_mat[i - 1, j - 1] + r_cost(src[i], tar[j]),\n )\n\n return d_mat[lens, lent]", "response": "Return the distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dist(self, src, tar, cost=(0, 1, 2), local=False):\n if src == tar:\n return 0.0\n mismatch_cost = cost[2]\n return self.dist_abs(src, tar, cost, local) / (\n max(len(src) * mismatch_cost, len(tar) * mismatch_cost)\n )", "response": "Return the normalized distance between two strings src and tar."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef position_fingerprint(\n word, n_bits=16, most_common=MOST_COMMON_LETTERS_CG, bits_per_letter=3\n):\n \"\"\"Return the position fingerprint.\n\n This is a wrapper for :py:meth:`Position.fingerprint`.\n\n Parameters\n ----------\n word : str\n The word to fingerprint\n n_bits : int\n Number of bits in the fingerprint returned\n most_common : list\n The most common tokens in the target language, ordered by frequency\n bits_per_letter : int\n The bits to assign for letter position\n\n Returns\n -------\n int\n The position fingerprint\n\n Examples\n --------\n >>> bin(position_fingerprint('hat'))\n '0b1110100011111111'\n >>> bin(position_fingerprint('niall'))\n '0b1111110101110010'\n >>> bin(position_fingerprint('colin'))\n '0b1111111110010111'\n >>> bin(position_fingerprint('atcg'))\n '0b1110010001111111'\n >>> bin(position_fingerprint('entreatment'))\n '0b101011111111'\n\n \"\"\"\n return Position().fingerprint(word, n_bits, most_common, bits_per_letter)", "response": "Return the position fingerprint."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the position fingerprint of the current language.", "response": "def fingerprint(\n self,\n word,\n n_bits=16,\n most_common=MOST_COMMON_LETTERS_CG,\n bits_per_letter=3,\n ):\n \"\"\"Return the position fingerprint.\n\n Parameters\n ----------\n word : str\n The word to fingerprint\n n_bits : int\n Number of bits in the fingerprint returned\n most_common : list\n The most common tokens in the target language, ordered by frequency\n bits_per_letter : int\n The bits to assign for letter position\n\n Returns\n -------\n int\n The position fingerprint\n\n Examples\n --------\n >>> bin(position_fingerprint('hat'))\n '0b1110100011111111'\n >>> bin(position_fingerprint('niall'))\n '0b1111110101110010'\n >>> bin(position_fingerprint('colin'))\n '0b1111111110010111'\n >>> bin(position_fingerprint('atcg'))\n '0b1110010001111111'\n >>> bin(position_fingerprint('entreatment'))\n '0b101011111111'\n\n \"\"\"\n position = {}\n for pos, letter in enumerate(word):\n if letter not in position and letter in most_common:\n position[letter] = min(pos, 2 ** bits_per_letter - 1)\n\n fingerprint = 0\n\n for letter in most_common:\n if n_bits:\n fingerprint <<= min(bits_per_letter, n_bits)\n if letter in position:\n fingerprint += min(position[letter], 2 ** n_bits - 1)\n else:\n fingerprint += min(\n 2 ** bits_per_letter - 1, 2 ** n_bits - 1\n )\n n_bits -= min(bits_per_letter, n_bits)\n else:\n break\n\n for _ in range(n_bits):\n fingerprint <<= 1\n fingerprint += 1\n\n return fingerprint"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function to\nreturn the stemmed version of the given word.", "response": "def stem(self, word):\n \"\"\"Return Caumanns German stem.\n\n Parameters\n ----------\n word : str\n The word to stem\n\n Returns\n -------\n str\n Word stem\n\n Examples\n --------\n >>> stmr = Caumanns()\n >>> stmr.stem('lesen')\n 'les'\n >>> stmr.stem('graues')\n 'grau'\n >>> stmr.stem('buchstabieren')\n 'buchstabier'\n\n \"\"\"\n if not word:\n return ''\n\n upper_initial = word[0].isupper()\n word = normalize('NFC', text_type(word.lower()))\n\n # # Part 2: Substitution\n # 1. Change umlauts to corresponding vowels & \u00df to ss\n word = word.translate(self._umlauts)\n word = word.replace('\u00df', 'ss')\n\n # 2. Change second of doubled characters to *\n new_word = word[0]\n for i in range(1, len(word)):\n if new_word[i - 1] == word[i]:\n new_word += '*'\n else:\n new_word += word[i]\n word = new_word\n\n # 3. Replace sch, ch, ei, ie with $, \u00a7, %, &\n word = word.replace('sch', '$')\n word = word.replace('ch', '\u00a7')\n word = word.replace('ei', '%')\n word = word.replace('ie', '&')\n word = word.replace('ig', '#')\n word = word.replace('st', '!')\n\n # # Part 1: Recursive Context-Free Stripping\n # 1. Remove the following 7 suffixes recursively\n while len(word) > 3:\n if (len(word) > 4 and word[-2:] in {'em', 'er'}) or (\n len(word) > 5 and word[-2:] == 'nd'\n ):\n word = word[:-2]\n elif (word[-1] in {'e', 's', 'n'}) or (\n not upper_initial and word[-1] in {'t', '!'}\n ):\n word = word[:-1]\n else:\n break\n\n # Additional optimizations:\n if len(word) > 5 and word[-5:] == 'erin*':\n word = word[:-1]\n if word[-1] == 'z':\n word = word[:-1] + 'x'\n\n # Reverse substitutions:\n word = word.replace('$', 'sch')\n word = word.replace('\u00a7', 'ch')\n word = word.replace('%', 'ei')\n word = word.replace('&', 'ie')\n word = word.replace('#', 'ig')\n word = word.replace('!', 'st')\n\n # Expand doubled\n word = ''.join(\n [word[0]]\n + [\n word[i - 1] if word[i] == '*' else word[i]\n for i in range(1, len(word))\n ]\n )\n\n # Finally, convert gege to ge\n if len(word) > 4:\n word = word.replace('gege', 'ge', 1)\n\n return word"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the NCD between two strings using arithmetic coding.", "response": "def dist(self, src, tar, probs=None):\n \"\"\"Return the NCD between two strings using arithmetic coding.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n probs : dict\n A dictionary trained with :py:meth:`Arithmetic.train`\n\n Returns\n -------\n float\n Compression distance\n\n Examples\n --------\n >>> cmp = NCDarith()\n >>> cmp.dist('cat', 'hat')\n 0.5454545454545454\n >>> cmp.dist('Niall', 'Neil')\n 0.6875\n >>> cmp.dist('aluminum', 'Catalan')\n 0.8275862068965517\n >>> cmp.dist('ATCG', 'TAGC')\n 0.6923076923076923\n\n \"\"\"\n if src == tar:\n return 0.0\n\n if probs is None:\n # lacking a reasonable dictionary, train on the strings themselves\n self._coder.train(src + tar)\n else:\n self._coder.set_probs(probs)\n\n src_comp = self._coder.encode(src)[1]\n tar_comp = self._coder.encode(tar)[1]\n concat_comp = self._coder.encode(src + tar)[1]\n concat_comp2 = self._coder.encode(tar + src)[1]\n\n return (\n min(concat_comp, concat_comp2) - min(src_comp, tar_comp)\n ) / max(src_comp, tar_comp)"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreads the contents of a file and return the contents.", "response": "def readfile(fn):\n \"\"\"Read fn and return the contents.\n\n Parameters\n ----------\n fn : str\n A filename\n\n Returns\n -------\n str\n The content of the file\n\n \"\"\"\n with open(path.join(HERE, fn), 'r', encoding='utf-8') as f:\n return f.read()"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef spanish_metaphone(word, max_length=6, modified=False):\n return SpanishMetaphone().encode(word, max_length, modified)", "response": "Returns the Spanish Metaphone code of a word."} {"SOURCE": "codesearchnet", "instruction": "Can you generate the documentation for the following Python 3 function\ndef encode(self, word, max_length=6, modified=False):\n\n def _is_vowel(pos):\n \"\"\"Return True if the character at word[pos] is a vowel.\n\n Parameters\n ----------\n pos : int\n Position to check for a vowel\n\n Returns\n -------\n bool\n True if word[pos] is a vowel\n\n \"\"\"\n return pos < len(word) and word[pos] in {'A', 'E', 'I', 'O', 'U'}\n\n word = unicode_normalize('NFC', text_type(word.upper()))\n\n meta_key = ''\n pos = 0\n\n # do some replacements for the modified version\n if modified:\n word = word.replace('MB', 'NB')\n word = word.replace('MP', 'NP')\n word = word.replace('BS', 'S')\n if word[:2] == 'PS':\n word = word[1:]\n\n # simple replacements\n word = word.replace('\u00c1', 'A')\n word = word.replace('CH', 'X')\n word = word.replace('\u00c7', 'S')\n word = word.replace('\u00c9', 'E')\n word = word.replace('\u00cd', 'I')\n word = word.replace('\u00d3', 'O')\n word = word.replace('\u00da', 'U')\n word = word.replace('\u00d1', 'NY')\n word = word.replace('G\u00dc', 'W')\n word = word.replace('\u00dc', 'U')\n word = word.replace('B', 'V')\n word = word.replace('LL', 'Y')\n\n while len(meta_key) < max_length:\n if pos >= len(word):\n break\n\n # get the next character\n current_char = word[pos]\n\n # if a vowel in pos 0, add to key\n if _is_vowel(pos) and pos == 0:\n meta_key += current_char\n pos += 1\n # otherwise, do consonant rules\n else:\n # simple consonants (unmutated)\n if current_char in {\n 'D',\n 'F',\n 'J',\n 'K',\n 'M',\n 'N',\n 'P',\n 'T',\n 'V',\n 'L',\n 'Y',\n }:\n meta_key += current_char\n # skip doubled consonants\n if word[pos + 1 : pos + 2] == current_char:\n pos += 2\n else:\n pos += 1\n else:\n if current_char == 'C':\n # special case 'acci\u00f3n', 'reacci\u00f3n',etc.\n if word[pos + 1 : pos + 2] == 'C':\n meta_key += 'X'\n pos += 2\n # special case 'cesar', 'cien', 'cid', 'conciencia'\n elif word[pos + 1 : pos + 2] in {'E', 'I'}:\n meta_key += 'Z'\n pos += 2\n # base case\n else:\n meta_key += 'K'\n pos += 1\n elif current_char == 'G':\n # special case 'gente', 'ecologia',etc\n if word[pos + 1 : pos + 2] in {'E', 'I'}:\n meta_key += 'J'\n pos += 2\n # base case\n else:\n meta_key += 'G'\n pos += 1\n elif current_char == 'H':\n # since the letter 'H' is silent in Spanish,\n # set the meta key to the vowel after the letter 'H'\n if _is_vowel(pos + 1):\n meta_key += word[pos + 1]\n pos += 2\n else:\n meta_key += 'H'\n pos += 1\n elif current_char == 'Q':\n if word[pos + 1 : pos + 2] == 'U':\n pos += 2\n else:\n pos += 1\n meta_key += 'K'\n elif current_char == 'W':\n meta_key += 'U'\n pos += 1\n elif current_char == 'R':\n meta_key += 'R'\n pos += 1\n elif current_char == 'S':\n if not _is_vowel(pos + 1) and pos == 0:\n meta_key += 'ES'\n pos += 1\n else:\n meta_key += 'S'\n pos += 1\n elif current_char == 'Z':\n meta_key += 'Z'\n pos += 1\n elif current_char == 'X':\n if (\n len(word) > 1\n and pos == 0\n and not _is_vowel(pos + 1)\n ):\n meta_key += 'EX'\n pos += 1\n else:\n meta_key += 'X'\n pos += 1\n else:\n pos += 1\n\n # Final change from S to Z in modified version\n if modified:\n meta_key = meta_key.replace('S', 'Z')\n\n return meta_key", "response": "Encodes a single word into a Spanish Metaphone code."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nreturning the FONEM code of a word.", "response": "def encode(self, word):\n \"\"\"Return the FONEM code of a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n\n Returns\n -------\n str\n The FONEM code\n\n Examples\n --------\n >>> pe = FONEM()\n >>> pe.encode('Marchand')\n 'MARCHEN'\n >>> pe.encode('Beaulieu')\n 'BOLIEU'\n >>> pe.encode('Beaumont')\n 'BOMON'\n >>> pe.encode('Legrand')\n 'LEGREN'\n >>> pe.encode('Pelletier')\n 'PELETIER'\n\n \"\"\"\n # normalize, upper-case, and filter non-French letters\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.translate({198: 'AE', 338: 'OE'})\n word = ''.join(c for c in word if c in self._uc_set)\n\n for rule in self._rule_order:\n regex, repl = self._rule_table[rule]\n if isinstance(regex, text_type):\n word = word.replace(regex, repl)\n else:\n word = regex.sub(repl, word)\n\n return word"} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef encode(self, word, max_length=4):\n # uppercase, normalize, decompose, and filter non-A-Z out\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n if not word:\n return ''\n\n code = word[1:]\n for vowel in self._uc_vy_set:\n code = code.replace(vowel, '')\n code = word[0] + code\n code = self._delete_consecutive_repeats(code)\n code = code.replace(' ', '')\n\n return code[:max_length]", "response": "Return the Statistics Canada code for a word."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the Synoname similarity type of two words.", "response": "def synoname(\n src,\n tar,\n word_approx_min=0.3,\n char_approx_min=0.73,\n tests=2 ** 12 - 1,\n ret_name=False,\n):\n \"\"\"Return the Synoname similarity type of two words.\n\n This is a wrapper for :py:meth:`Synoname.dist_abs`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n word_approx_min : float\n The minimum word approximation value to signal a 'word_approx' match\n char_approx_min : float\n The minimum character approximation value to signal a 'char_approx'\n match\n tests : int or Iterable\n Either an integer indicating tests to perform or a list of test names\n to perform (defaults to performing all tests)\n ret_name : bool\n If True, returns the match name rather than its integer equivalent\n\n Returns\n -------\n int (or str if ret_name is True)\n Synoname value\n\n Examples\n --------\n >>> synoname(('Breghel', 'Pieter', ''), ('Brueghel', 'Pieter', ''))\n 2\n >>> synoname(('Breghel', 'Pieter', ''), ('Brueghel', 'Pieter', ''),\n ... ret_name=True)\n 'omission'\n >>> synoname(('Dore', 'Gustave', ''),\n ... ('Dore', 'Paul Gustave Louis Christophe', ''), ret_name=True)\n 'inclusion'\n >>> synoname(('Pereira', 'I. R.', ''), ('Pereira', 'I. Smith', ''),\n ... ret_name=True)\n 'word_approx'\n\n \"\"\"\n return Synoname().dist_abs(\n src, tar, word_approx_min, char_approx_min, tests, ret_name\n )"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning a word with punctuation stripped out.", "response": "def _synoname_strip_punct(self, word):\n \"\"\"Return a word with punctuation stripped out.\n\n Parameters\n ----------\n word : str\n A word to strip punctuation from\n\n Returns\n -------\n str\n The word stripped of punctuation\n\n Examples\n --------\n >>> pe = Synoname()\n >>> pe._synoname_strip_punct('AB;CD EF-GH$IJ')\n 'ABCD EFGHIJ'\n\n \"\"\"\n stripped = ''\n for char in word:\n if char not in set(',-./:;\"&\\'()!{|}?$%*+<=>[\\\\]^_`~'):\n stripped += char\n return stripped.strip()"} {"SOURCE": "codesearchnet", "instruction": "Can you implement a function in Python 3 that\nreturns the Synoname word approximation score for two names.", "response": "def _synoname_word_approximation(\n self, src_ln, tar_ln, src_fn='', tar_fn='', features=None\n ):\n \"\"\"Return the Synoname word approximation score for two names.\n\n Parameters\n ----------\n src_ln : str\n Last name of the source\n tar_ln : str\n Last name of the target\n src_fn : str\n First name of the source (optional)\n tar_fn : str\n First name of the target (optional)\n features : dict\n A dict containing special features calculated using\n :py:class:`fingerprint.SynonameToolcode` (optional)\n\n Returns\n -------\n float\n The word approximation score\n\n Examples\n --------\n >>> pe = Synoname()\n >>> pe._synoname_word_approximation('Smith Waterman', 'Waterman',\n ... 'Tom Joe Bob', 'Tom Joe')\n 0.6\n\n \"\"\"\n if features is None:\n features = {}\n if 'src_specials' not in features:\n features['src_specials'] = []\n if 'tar_specials' not in features:\n features['tar_specials'] = []\n\n src_len_specials = len(features['src_specials'])\n tar_len_specials = len(features['tar_specials'])\n\n # 1\n if ('gen_conflict' in features and features['gen_conflict']) or (\n 'roman_conflict' in features and features['roman_conflict']\n ):\n return 0\n\n # 3 & 7\n full_tar1 = ' '.join((tar_ln, tar_fn)).replace('-', ' ').strip()\n for s_pos, s_type in features['tar_specials']:\n if s_type == 'a':\n full_tar1 = full_tar1[\n : -(\n 1\n + len(\n self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n )\n )\n ]\n elif s_type == 'b':\n loc = (\n full_tar1.find(\n ' '\n + self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n + ' '\n )\n + 1\n )\n full_tar1 = (\n full_tar1[:loc]\n + full_tar1[\n loc\n + len(\n self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n ) :\n ]\n )\n elif s_type == 'c':\n full_tar1 = full_tar1[\n 1\n + len(\n self._stc._synoname_special_table[s_pos][ # noqa: SF01\n 1\n ]\n ) :\n ]\n\n full_src1 = ' '.join((src_ln, src_fn)).replace('-', ' ').strip()\n for s_pos, s_type in features['src_specials']:\n if s_type == 'a':\n full_src1 = full_src1[\n : -(\n 1\n + len(\n self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n )\n )\n ]\n elif s_type == 'b':\n loc = (\n full_src1.find(\n ' '\n + self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n + ' '\n )\n + 1\n )\n full_src1 = (\n full_src1[:loc]\n + full_src1[\n loc\n + len(\n self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n ) :\n ]\n )\n elif s_type == 'c':\n full_src1 = full_src1[\n 1\n + len(\n self._stc._synoname_special_table[s_pos][ # noqa: SF01\n 1\n ]\n ) :\n ]\n\n full_tar2 = full_tar1\n for s_pos, s_type in features['tar_specials']:\n if s_type == 'd':\n full_tar2 = full_tar2[\n len(\n self._stc._synoname_special_table[s_pos][ # noqa: SF01\n 1\n ]\n ) :\n ]\n elif (\n s_type == 'X'\n and self._stc._synoname_special_table[s_pos][1] # noqa: SF01\n in full_tar2\n ):\n loc = full_tar2.find(\n ' '\n + self._stc._synoname_special_table[s_pos][1] # noqa: SF01\n )\n full_tar2 = (\n full_tar2[:loc]\n + full_tar2[\n loc\n + len(\n self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n ) :\n ]\n )\n\n full_src2 = full_src1\n for s_pos, s_type in features['src_specials']:\n if s_type == 'd':\n full_src2 = full_src2[\n len(\n self._stc._synoname_special_table[s_pos][ # noqa: SF01\n 1\n ]\n ) :\n ]\n elif (\n s_type == 'X'\n and self._stc._synoname_special_table[s_pos][1] # noqa: SF01\n in full_src2\n ):\n loc = full_src2.find(\n ' '\n + self._stc._synoname_special_table[s_pos][1] # noqa: SF01\n )\n full_src2 = (\n full_src2[:loc]\n + full_src2[\n loc\n + len(\n self._stc._synoname_special_table[ # noqa: SF01\n s_pos\n ][1]\n ) :\n ]\n )\n\n full_tar1 = self._synoname_strip_punct(full_tar1)\n tar1_words = full_tar1.split()\n tar1_num_words = len(tar1_words)\n\n full_src1 = self._synoname_strip_punct(full_src1)\n src1_words = full_src1.split()\n src1_num_words = len(src1_words)\n\n full_tar2 = self._synoname_strip_punct(full_tar2)\n tar2_words = full_tar2.split()\n tar2_num_words = len(tar2_words)\n\n full_src2 = self._synoname_strip_punct(full_src2)\n src2_words = full_src2.split()\n src2_num_words = len(src2_words)\n\n # 2\n if (\n src1_num_words < 2\n and src_len_specials == 0\n and src2_num_words < 2\n and tar_len_specials == 0\n ):\n return 0\n\n # 4\n if (\n tar1_num_words == 1\n and src1_num_words == 1\n and tar1_words[0] == src1_words[0]\n ):\n return 1\n if tar1_num_words < 2 and tar_len_specials == 0:\n return 0\n\n # 5\n last_found = False\n for word in tar1_words:\n if src_ln.endswith(word) or word + ' ' in src_ln:\n last_found = True\n\n if not last_found:\n for word in src1_words:\n if tar_ln.endswith(word) or word + ' ' in tar_ln:\n last_found = True\n\n # 6\n matches = 0\n if last_found:\n for i, s_word in enumerate(src1_words):\n for j, t_word in enumerate(tar1_words):\n if s_word == t_word:\n src1_words[i] = '@'\n tar1_words[j] = '@'\n matches += 1\n w_ratio = matches / max(tar1_num_words, src1_num_words)\n if matches > 1 or (\n matches == 1\n and src1_num_words == 1\n and tar1_num_words == 1\n and (tar_len_specials > 0 or src_len_specials > 0)\n ):\n return w_ratio\n\n # 8\n if (\n tar2_num_words == 1\n and src2_num_words == 1\n and tar2_words[0] == src2_words[0]\n ):\n return 1\n # I see no way that the following can be True if the equivalent in\n # #4 was False.\n if tar2_num_words < 2 and tar_len_specials == 0: # pragma: no cover\n return 0\n\n # 9\n last_found = False\n for word in tar2_words:\n if src_ln.endswith(word) or word + ' ' in src_ln:\n last_found = True\n\n if not last_found:\n for word in src2_words:\n if tar_ln.endswith(word) or word + ' ' in tar_ln:\n last_found = True\n\n if not last_found:\n return 0\n\n # 10\n matches = 0\n if last_found:\n for i, s_word in enumerate(src2_words):\n for j, t_word in enumerate(tar2_words):\n if s_word == t_word:\n src2_words[i] = '@'\n tar2_words[j] = '@'\n matches += 1\n w_ratio = matches / max(tar2_num_words, src2_num_words)\n if matches > 1 or (\n matches == 1\n and src2_num_words == 1\n and tar2_num_words == 1\n and (tar_len_specials > 0 or src_len_specials > 0)\n ):\n return w_ratio\n\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the Synoname similarity type of two words.", "response": "def dist_abs(\n self,\n src,\n tar,\n word_approx_min=0.3,\n char_approx_min=0.73,\n tests=2 ** 12 - 1,\n ret_name=False,\n ):\n \"\"\"Return the Synoname similarity type of two words.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n word_approx_min : float\n The minimum word approximation value to signal a 'word_approx'\n match\n char_approx_min : float\n The minimum character approximation value to signal a 'char_approx'\n match\n tests : int or Iterable\n Either an integer indicating tests to perform or a list of test\n names to perform (defaults to performing all tests)\n ret_name : bool\n If True, returns the match name rather than its integer equivalent\n\n Returns\n -------\n int (or str if ret_name is True)\n Synoname value\n\n Examples\n --------\n >>> cmp = Synoname()\n >>> cmp.dist_abs(('Breghel', 'Pieter', ''), ('Brueghel', 'Pieter', ''))\n 2\n >>> cmp.dist_abs(('Breghel', 'Pieter', ''), ('Brueghel', 'Pieter', ''),\n ... ret_name=True)\n 'omission'\n >>> cmp.dist_abs(('Dore', 'Gustave', ''),\n ... ('Dore', 'Paul Gustave Louis Christophe', ''), ret_name=True)\n 'inclusion'\n >>> cmp.dist_abs(('Pereira', 'I. R.', ''), ('Pereira', 'I. Smith', ''),\n ... ret_name=True)\n 'word_approx'\n\n \"\"\"\n if isinstance(tests, Iterable):\n new_tests = 0\n for term in tests:\n if term in self._test_dict:\n new_tests += self._test_dict[term]\n tests = new_tests\n\n if isinstance(src, tuple):\n src_ln, src_fn, src_qual = src\n elif '#' in src:\n src_ln, src_fn, src_qual = src.split('#')[-3:]\n else:\n src_ln, src_fn, src_qual = src, '', ''\n\n if isinstance(tar, tuple):\n tar_ln, tar_fn, tar_qual = tar\n elif '#' in tar:\n tar_ln, tar_fn, tar_qual = tar.split('#')[-3:]\n else:\n tar_ln, tar_fn, tar_qual = tar, '', ''\n\n def _split_special(spec):\n spec_list = []\n while spec:\n spec_list.append((int(spec[:3]), spec[3:4]))\n spec = spec[4:]\n return spec_list\n\n def _fmt_retval(val):\n if ret_name:\n return self._match_name[val]\n return val\n\n # 1. Preprocessing\n\n # Lowercasing\n src_fn = src_fn.strip().lower()\n src_ln = src_ln.strip().lower()\n src_qual = src_qual.strip().lower()\n\n tar_fn = tar_fn.strip().lower()\n tar_ln = tar_ln.strip().lower()\n tar_qual = tar_qual.strip().lower()\n\n # Create toolcodes\n src_ln, src_fn, src_tc = self._stc.fingerprint(\n src_ln, src_fn, src_qual\n )\n tar_ln, tar_fn, tar_tc = self._stc.fingerprint(\n tar_ln, tar_fn, tar_qual\n )\n\n src_generation = int(src_tc[2])\n src_romancode = int(src_tc[3:6])\n src_len_fn = int(src_tc[6:8])\n src_tc = src_tc.split('$')\n src_specials = _split_special(src_tc[1])\n\n tar_generation = int(tar_tc[2])\n tar_romancode = int(tar_tc[3:6])\n tar_len_fn = int(tar_tc[6:8])\n tar_tc = tar_tc.split('$')\n tar_specials = _split_special(tar_tc[1])\n\n gen_conflict = (src_generation != tar_generation) and bool(\n src_generation or tar_generation\n )\n roman_conflict = (src_romancode != tar_romancode) and bool(\n src_romancode or tar_romancode\n )\n\n ln_equal = src_ln == tar_ln\n fn_equal = src_fn == tar_fn\n\n # approx_c\n def _approx_c():\n if gen_conflict or roman_conflict:\n return False, 0\n\n full_src = ' '.join((src_ln, src_fn))\n if full_src.startswith('master '):\n full_src = full_src[len('master ') :]\n for intro in [\n 'of the ',\n 'of ',\n 'known as the ',\n 'with the ',\n 'with ',\n ]:\n if full_src.startswith(intro):\n full_src = full_src[len(intro) :]\n\n full_tar = ' '.join((tar_ln, tar_fn))\n if full_tar.startswith('master '):\n full_tar = full_tar[len('master ') :]\n for intro in [\n 'of the ',\n 'of ',\n 'known as the ',\n 'with the ',\n 'with ',\n ]:\n if full_tar.startswith(intro):\n full_tar = full_tar[len(intro) :]\n\n loc_ratio = sim_ratcliff_obershelp(full_src, full_tar)\n return loc_ratio >= char_approx_min, loc_ratio\n\n approx_c_result, ca_ratio = _approx_c()\n\n if tests & self._test_dict['exact'] and fn_equal and ln_equal:\n return _fmt_retval(self._match_type_dict['exact'])\n if tests & self._test_dict['omission']:\n if (\n fn_equal\n and levenshtein(src_ln, tar_ln, cost=(1, 1, 99, 99)) == 1\n ):\n if not roman_conflict:\n return _fmt_retval(self._match_type_dict['omission'])\n elif (\n ln_equal\n and levenshtein(src_fn, tar_fn, cost=(1, 1, 99, 99)) == 1\n ):\n return _fmt_retval(self._match_type_dict['omission'])\n if tests & self._test_dict['substitution']:\n if (\n fn_equal\n and levenshtein(src_ln, tar_ln, cost=(99, 99, 1, 99)) == 1\n ):\n return _fmt_retval(self._match_type_dict['substitution'])\n elif (\n ln_equal\n and levenshtein(src_fn, tar_fn, cost=(99, 99, 1, 99)) == 1\n ):\n return _fmt_retval(self._match_type_dict['substitution'])\n if tests & self._test_dict['transposition']:\n if fn_equal and (\n levenshtein(src_ln, tar_ln, mode='osa', cost=(99, 99, 99, 1))\n == 1\n ):\n return _fmt_retval(self._match_type_dict['transposition'])\n elif ln_equal and (\n levenshtein(src_fn, tar_fn, mode='osa', cost=(99, 99, 99, 1))\n == 1\n ):\n return _fmt_retval(self._match_type_dict['transposition'])\n if tests & self._test_dict['punctuation']:\n np_src_fn = self._synoname_strip_punct(src_fn)\n np_tar_fn = self._synoname_strip_punct(tar_fn)\n np_src_ln = self._synoname_strip_punct(src_ln)\n np_tar_ln = self._synoname_strip_punct(tar_ln)\n\n if (np_src_fn == np_tar_fn) and (np_src_ln == np_tar_ln):\n return _fmt_retval(self._match_type_dict['punctuation'])\n\n np_src_fn = self._synoname_strip_punct(src_fn.replace('-', ' '))\n np_tar_fn = self._synoname_strip_punct(tar_fn.replace('-', ' '))\n np_src_ln = self._synoname_strip_punct(src_ln.replace('-', ' '))\n np_tar_ln = self._synoname_strip_punct(tar_ln.replace('-', ' '))\n\n if (np_src_fn == np_tar_fn) and (np_src_ln == np_tar_ln):\n return _fmt_retval(self._match_type_dict['punctuation'])\n\n if tests & self._test_dict['initials'] and ln_equal:\n if src_fn and tar_fn:\n src_initials = self._synoname_strip_punct(src_fn).split()\n tar_initials = self._synoname_strip_punct(tar_fn).split()\n initials = bool(\n (len(src_initials) == len(''.join(src_initials)))\n or (len(tar_initials) == len(''.join(tar_initials)))\n )\n if initials:\n src_initials = ''.join(_[0] for _ in src_initials)\n tar_initials = ''.join(_[0] for _ in tar_initials)\n if src_initials == tar_initials:\n return _fmt_retval(self._match_type_dict['initials'])\n initial_diff = abs(len(src_initials) - len(tar_initials))\n if initial_diff and (\n (\n initial_diff\n == levenshtein(\n src_initials,\n tar_initials,\n cost=(1, 99, 99, 99),\n )\n )\n or (\n initial_diff\n == levenshtein(\n tar_initials,\n src_initials,\n cost=(1, 99, 99, 99),\n )\n )\n ):\n return _fmt_retval(self._match_type_dict['initials'])\n if tests & self._test_dict['extension']:\n if src_ln[1] == tar_ln[1] and (\n src_ln.startswith(tar_ln) or tar_ln.startswith(src_ln)\n ):\n if (\n (not src_len_fn and not tar_len_fn)\n or (tar_fn and src_fn.startswith(tar_fn))\n or (src_fn and tar_fn.startswith(src_fn))\n ) and not roman_conflict:\n return _fmt_retval(self._match_type_dict['extension'])\n if tests & self._test_dict['inclusion'] and ln_equal:\n if (src_fn and src_fn in tar_fn) or (tar_fn and tar_fn in src_ln):\n return _fmt_retval(self._match_type_dict['inclusion'])\n if tests & self._test_dict['no_first'] and ln_equal:\n if src_fn == '' or tar_fn == '':\n return _fmt_retval(self._match_type_dict['no_first'])\n if tests & self._test_dict['word_approx']:\n ratio = self._synoname_word_approximation(\n src_ln,\n tar_ln,\n src_fn,\n tar_fn,\n {\n 'gen_conflict': gen_conflict,\n 'roman_conflict': roman_conflict,\n 'src_specials': src_specials,\n 'tar_specials': tar_specials,\n },\n )\n if ratio == 1 and tests & self._test_dict['confusions']:\n if (\n ' '.join((src_fn, src_ln)).strip()\n == ' '.join((tar_fn, tar_ln)).strip()\n ):\n return _fmt_retval(self._match_type_dict['confusions'])\n if ratio >= word_approx_min:\n return _fmt_retval(self._match_type_dict['word_approx'])\n if tests & self._test_dict['char_approx']:\n if ca_ratio >= char_approx_min:\n return _fmt_retval(self._match_type_dict['char_approx'])\n return _fmt_retval(self._match_type_dict['no_match'])"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the normalized Synoname distance between two words.", "response": "def dist(\n self,\n src,\n tar,\n word_approx_min=0.3,\n char_approx_min=0.73,\n tests=2 ** 12 - 1,\n ):\n \"\"\"Return the normalized Synoname distance between two words.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n word_approx_min : float\n The minimum word approximation value to signal a 'word_approx'\n match\n char_approx_min : float\n The minimum character approximation value to signal a 'char_approx'\n match\n tests : int or Iterable\n Either an integer indicating tests to perform or a list of test\n names to perform (defaults to performing all tests)\n\n Returns\n -------\n float\n Normalized Synoname distance\n\n \"\"\"\n return (\n synoname(src, tar, word_approx_min, char_approx_min, tests, False)\n / 14\n )"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sim(self, src, tar, qval=2):\n return super(self.__class__, self).sim(src, tar, qval, 1, 1)", "response": "Return the Jaccard similarity of two strings."} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the Tanimoto distance between two strings.", "response": "def tanimoto_coeff(self, src, tar, qval=2):\n \"\"\"Return the Tanimoto distance between two strings.\n\n Tanimoto distance :cite:`Tanimoto:1958` is\n :math:`-log_{2} sim_{Tanimoto}(X, Y)`.\n\n Parameters\n ----------\n src : str\n Source string (or QGrams/Counter objects) for comparison\n tar : str\n Target string (or QGrams/Counter objects) for comparison\n qval : int\n The length of each q-gram; 0 for non-q-gram version\n\n Returns\n -------\n float\n Tanimoto distance\n\n Examples\n --------\n >>> cmp = Jaccard()\n >>> cmp.tanimoto_coeff('cat', 'hat')\n -1.5849625007211563\n >>> cmp.tanimoto_coeff('Niall', 'Neil')\n -2.1699250014423126\n >>> cmp.tanimoto_coeff('aluminum', 'Catalan')\n -4.0\n >>> cmp.tanimoto_coeff('ATCG', 'TAGC')\n -inf\n\n \"\"\"\n coeff = self.sim(src, tar, qval)\n if coeff != 0:\n return log(coeff, 2)\n\n return float('-inf')"} {"SOURCE": "codesearchnet", "instruction": "Implement a Python 3 function for\nreturning the common Sift4 distance between two terms.", "response": "def sift4_common(src, tar, max_offset=5, max_distance=0):\n \"\"\"Return the \"common\" Sift4 distance between two terms.\n\n This is a wrapper for :py:meth:`Sift4.dist_abs`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n max_offset : int\n The number of characters to search for matching letters\n max_distance : int\n The distance at which to stop and exit\n\n Returns\n -------\n int\n The Sift4 distance according to the common formula\n\n Examples\n --------\n >>> sift4_common('cat', 'hat')\n 1\n >>> sift4_common('Niall', 'Neil')\n 2\n >>> sift4_common('Colin', 'Cuilen')\n 3\n >>> sift4_common('ATCG', 'TAGC')\n 2\n\n \"\"\"\n return Sift4().dist_abs(src, tar, max_offset, max_distance)"} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef dist_sift4(src, tar, max_offset=5, max_distance=0):\n return Sift4().dist(src, tar, max_offset, max_distance)", "response": "Return the normalized common Sift4 distance between two terms."} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the normalized common Sift4 similarity of two terms.", "response": "def sim_sift4(src, tar, max_offset=5, max_distance=0):\n \"\"\"Return the normalized \"common\" Sift4 similarity of two terms.\n\n This is a wrapper for :py:meth:`Sift4.sim`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n max_offset : int\n The number of characters to search for matching letters\n max_distance : int\n The distance at which to stop and exit\n\n Returns\n -------\n float\n The normalized Sift4 similarity\n\n Examples\n --------\n >>> round(sim_sift4('cat', 'hat'), 12)\n 0.666666666667\n >>> sim_sift4('Niall', 'Neil')\n 0.6\n >>> sim_sift4('Colin', 'Cuilen')\n 0.5\n >>> sim_sift4('ATCG', 'TAGC')\n 0.5\n\n \"\"\"\n return Sift4().sim(src, tar, max_offset, max_distance)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the common Sift4 distance between two strings.", "response": "def dist_abs(self, src, tar, max_offset=5, max_distance=0):\n \"\"\"Return the \"common\" Sift4 distance between two terms.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n max_offset : int\n The number of characters to search for matching letters\n max_distance : int\n The distance at which to stop and exit\n\n Returns\n -------\n int\n The Sift4 distance according to the common formula\n\n Examples\n --------\n >>> cmp = Sift4()\n >>> cmp.dist_abs('cat', 'hat')\n 1\n >>> cmp.dist_abs('Niall', 'Neil')\n 2\n >>> cmp.dist_abs('Colin', 'Cuilen')\n 3\n >>> cmp.dist_abs('ATCG', 'TAGC')\n 2\n\n \"\"\"\n if not src:\n return len(tar)\n\n if not tar:\n return len(src)\n\n src_len = len(src)\n tar_len = len(tar)\n\n src_cur = 0\n tar_cur = 0\n lcss = 0\n local_cs = 0\n trans = 0\n offset_arr = []\n\n while (src_cur < src_len) and (tar_cur < tar_len):\n if src[src_cur] == tar[tar_cur]:\n local_cs += 1\n is_trans = False\n i = 0\n while i < len(offset_arr):\n ofs = offset_arr[i]\n if src_cur <= ofs['src_cur'] or tar_cur <= ofs['tar_cur']:\n is_trans = abs(tar_cur - src_cur) >= abs(\n ofs['tar_cur'] - ofs['src_cur']\n )\n if is_trans:\n trans += 1\n elif not ofs['trans']:\n ofs['trans'] = True\n trans += 1\n break\n elif src_cur > ofs['tar_cur'] and tar_cur > ofs['src_cur']:\n del offset_arr[i]\n else:\n i += 1\n\n offset_arr.append(\n {'src_cur': src_cur, 'tar_cur': tar_cur, 'trans': is_trans}\n )\n else:\n lcss += local_cs\n local_cs = 0\n if src_cur != tar_cur:\n src_cur = tar_cur = min(src_cur, tar_cur)\n for i in range(max_offset):\n if not (\n (src_cur + i < src_len) or (tar_cur + i < tar_len)\n ):\n break\n if (src_cur + i < src_len) and (\n src[src_cur + i] == tar[tar_cur]\n ):\n src_cur += i - 1\n tar_cur -= 1\n break\n if (tar_cur + i < tar_len) and (\n src[src_cur] == tar[tar_cur + i]\n ):\n src_cur -= 1\n tar_cur += i - 1\n break\n\n src_cur += 1\n tar_cur += 1\n\n if max_distance:\n temporary_distance = max(src_cur, tar_cur) - lcss + trans\n if temporary_distance >= max_distance:\n return round(temporary_distance)\n\n if (src_cur >= src_len) or (tar_cur >= tar_len):\n lcss += local_cs\n local_cs = 0\n src_cur = tar_cur = min(src_cur, tar_cur)\n\n lcss += local_cs\n return round(max(src_len, tar_len) - lcss + trans)"} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the normalized common Sift4 distance between two terms.", "response": "def dist(self, src, tar, max_offset=5, max_distance=0):\n \"\"\"Return the normalized \"common\" Sift4 distance between two terms.\n\n This is Sift4 distance, normalized to [0, 1].\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n max_offset : int\n The number of characters to search for matching letters\n max_distance : int\n The distance at which to stop and exit\n\n Returns\n -------\n float\n The normalized Sift4 distance\n\n Examples\n --------\n >>> cmp = Sift4()\n >>> round(cmp.dist('cat', 'hat'), 12)\n 0.333333333333\n >>> cmp.dist('Niall', 'Neil')\n 0.4\n >>> cmp.dist('Colin', 'Cuilen')\n 0.5\n >>> cmp.dist('ATCG', 'TAGC')\n 0.5\n\n \"\"\"\n return self.dist_abs(src, tar, max_offset, max_distance) / (\n max(len(src), len(tar), 1)\n )"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function for\nreturning the NCD between two strings using bzip2 compression.", "response": "def dist(self, src, tar):\n \"\"\"Return the NCD between two strings using bzip2 compression.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n float\n Compression distance\n\n Examples\n --------\n >>> cmp = NCDbz2()\n >>> cmp.dist('cat', 'hat')\n 0.06666666666666667\n >>> cmp.dist('Niall', 'Neil')\n 0.03125\n >>> cmp.dist('aluminum', 'Catalan')\n 0.17647058823529413\n >>> cmp.dist('ATCG', 'TAGC')\n 0.03125\n\n \"\"\"\n if src == tar:\n return 0.0\n\n src = src.encode('utf-8')\n tar = tar.encode('utf-8')\n\n src_comp = bz2.compress(src, self._level)[10:]\n tar_comp = bz2.compress(tar, self._level)[10:]\n concat_comp = bz2.compress(src + tar, self._level)[10:]\n concat_comp2 = bz2.compress(tar + src, self._level)[10:]\n\n return (\n min(len(concat_comp), len(concat_comp2))\n - min(len(src_comp), len(tar_comp))\n ) / max(len(src_comp), len(tar_comp))"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\nreturn the MetaSoundex code for a word.", "response": "def encode(self, word, lang='en'):\n \"\"\"Return the MetaSoundex code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n lang : str\n Either ``en`` for English or ``es`` for Spanish\n\n Returns\n -------\n str\n The MetaSoundex code\n\n Examples\n --------\n >>> pe = MetaSoundex()\n >>> pe.encode('Smith')\n '4500'\n >>> pe.encode('Waters')\n '7362'\n >>> pe.encode('James')\n '1520'\n >>> pe.encode('Schmidt')\n '4530'\n >>> pe.encode('Ashcroft')\n '0261'\n >>> pe.encode('Perez', lang='es')\n '094'\n >>> pe.encode('Martinez', lang='es')\n '69364'\n >>> pe.encode('Gutierrez', lang='es')\n '83994'\n >>> pe.encode('Santiago', lang='es')\n '4638'\n >>> pe.encode('Nicol\u00e1s', lang='es')\n '6754'\n\n \"\"\"\n if lang == 'es':\n return self._phonetic_spanish.encode(\n self._spanish_metaphone.encode(word)\n )\n\n word = self._soundex.encode(self._metaphone.encode(word))\n word = word[0].translate(self._trans) + word[1:]\n return word"} {"SOURCE": "codesearchnet", "instruction": "Explain what the following Python 3 code does\ndef stem(self, word):\n lowered = word.lower()\n if lowered[-3:] == 'ies' and lowered[-4:-3] not in {'e', 'a'}:\n return word[:-3] + ('Y' if word[-1:].isupper() else 'y')\n if lowered[-2:] == 'es' and lowered[-3:-2] not in {'a', 'e', 'o'}:\n return word[:-1]\n if lowered[-1:] == 's' and lowered[-2:-1] not in {'u', 's'}:\n return word[:-1]\n return word", "response": "Returns the S - stemmed form of a word."} {"SOURCE": "codesearchnet", "instruction": "Create a Python 3 function for\nreturning the similarity of two strings src and tar.", "response": "def sim(self, src, tar):\n \"\"\"Return the Ratcliff-Obershelp similarity of two strings.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n float\n Ratcliff-Obershelp similarity\n\n Examples\n --------\n >>> cmp = RatcliffObershelp()\n >>> round(cmp.sim('cat', 'hat'), 12)\n 0.666666666667\n >>> round(cmp.sim('Niall', 'Neil'), 12)\n 0.666666666667\n >>> round(cmp.sim('aluminum', 'Catalan'), 12)\n 0.4\n >>> cmp.sim('ATCG', 'TAGC')\n 0.5\n\n \"\"\"\n\n def _lcsstr_stl(src, tar):\n \"\"\"Return start positions & length for Ratcliff-Obershelp.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n tuple\n The start position in the source string, start position in the\n target string, and length of the longest common substring of\n strings src and tar.\n\n \"\"\"\n lengths = np_zeros((len(src) + 1, len(tar) + 1), dtype=np_int)\n longest, src_longest, tar_longest = 0, 0, 0\n for i in range(1, len(src) + 1):\n for j in range(1, len(tar) + 1):\n if src[i - 1] == tar[j - 1]:\n lengths[i, j] = lengths[i - 1, j - 1] + 1\n if lengths[i, j] > longest:\n longest = lengths[i, j]\n src_longest = i\n tar_longest = j\n else:\n lengths[i, j] = 0\n return src_longest - longest, tar_longest - longest, longest\n\n def _sstr_matches(src, tar):\n \"\"\"Return the sum of substring match lengths.\n\n This follows the Ratcliff-Obershelp algorithm\n :cite:`Ratcliff:1988`:\n 1. Find the length of the longest common substring in src &\n tar.\n 2. Recurse on the strings to the left & right of each this\n substring in src & tar.\n 3. Base case is a 0 length common substring, in which case,\n return 0.\n 4. Return the sum.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n int\n Sum of substring match lengths\n\n \"\"\"\n src_start, tar_start, length = _lcsstr_stl(src, tar)\n if length == 0:\n return 0\n return (\n _sstr_matches(src[:src_start], tar[:tar_start])\n + length\n + _sstr_matches(\n src[src_start + length :], tar[tar_start + length :]\n )\n )\n\n if src == tar:\n return 1.0\n elif not src or not tar:\n return 0.0\n return 2 * _sstr_matches(src, tar) / (len(src) + len(tar))"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the Refined Soundex code for a word.", "response": "def refined_soundex(word, max_length=-1, zero_pad=False, retain_vowels=False):\n \"\"\"Return the Refined Soundex code for a word.\n\n This is a wrapper for :py:meth:`RefinedSoundex.encode`.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to unlimited)\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length string\n retain_vowels : bool\n Retain vowels (as 0) in the resulting code\n\n Returns\n -------\n str\n The Refined Soundex value\n\n Examples\n --------\n >>> refined_soundex('Christopher')\n 'C393619'\n >>> refined_soundex('Niall')\n 'N87'\n >>> refined_soundex('Smith')\n 'S386'\n >>> refined_soundex('Schmidt')\n 'S386'\n\n \"\"\"\n return RefinedSoundex().encode(word, max_length, zero_pad, retain_vowels)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns the Refined Soundex code for a word.", "response": "def encode(self, word, max_length=-1, zero_pad=False, retain_vowels=False):\n \"\"\"Return the Refined Soundex code for a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n max_length : int\n The length of the code returned (defaults to unlimited)\n zero_pad : bool\n Pad the end of the return value with 0s to achieve a max_length\n string\n retain_vowels : bool\n Retain vowels (as 0) in the resulting code\n\n Returns\n -------\n str\n The Refined Soundex value\n\n Examples\n --------\n >>> pe = RefinedSoundex()\n >>> pe.encode('Christopher')\n 'C393619'\n >>> pe.encode('Niall')\n 'N87'\n >>> pe.encode('Smith')\n 'S386'\n >>> pe.encode('Schmidt')\n 'S386'\n\n \"\"\"\n # uppercase, normalize, decompose, and filter non-A-Z out\n word = unicode_normalize('NFKD', text_type(word.upper()))\n word = word.replace('\u00df', 'SS')\n word = ''.join(c for c in word if c in self._uc_set)\n\n # apply the Soundex algorithm\n sdx = word[:1] + word.translate(self._trans)\n sdx = self._delete_consecutive_repeats(sdx)\n if not retain_vowels:\n sdx = sdx.replace('0', '') # Delete vowels, H, W, Y\n\n if max_length > 0:\n if zero_pad:\n sdx += '0' * max_length\n sdx = sdx[:max_length]\n\n return sdx"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\nreturns the MRA comparison rating of two strings.", "response": "def dist_abs(self, src, tar):\n \"\"\"Return the MRA comparison rating of two strings.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n\n Returns\n -------\n int\n MRA comparison rating\n\n Examples\n --------\n >>> cmp = MRA()\n >>> cmp.dist_abs('cat', 'hat')\n 5\n >>> cmp.dist_abs('Niall', 'Neil')\n 6\n >>> cmp.dist_abs('aluminum', 'Catalan')\n 0\n >>> cmp.dist_abs('ATCG', 'TAGC')\n 5\n\n \"\"\"\n if src == tar:\n return 6\n if src == '' or tar == '':\n return 0\n src = list(mra(src))\n tar = list(mra(tar))\n\n if abs(len(src) - len(tar)) > 2:\n return 0\n\n length_sum = len(src) + len(tar)\n if length_sum < 5:\n min_rating = 5\n elif length_sum < 8:\n min_rating = 4\n elif length_sum < 12:\n min_rating = 3\n else:\n min_rating = 2\n\n for _ in range(2):\n new_src = []\n new_tar = []\n minlen = min(len(src), len(tar))\n for i in range(minlen):\n if src[i] != tar[i]:\n new_src.append(src[i])\n new_tar.append(tar[i])\n src = new_src + src[minlen:]\n tar = new_tar + tar[minlen:]\n src.reverse()\n tar.reverse()\n\n similarity = 6 - max(len(src), len(tar))\n\n if similarity >= min_rating:\n return similarity\n return 0"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 function that can\nreturn the Parmar - Kumbharana encoding of a word.", "response": "def encode(self, word):\n \"\"\"Return the Parmar-Kumbharana encoding of a word.\n\n Parameters\n ----------\n word : str\n The word to transform\n\n Returns\n -------\n str\n The Parmar-Kumbharana encoding\n\n Examples\n --------\n >>> pe = ParmarKumbharana()\n >>> pe.encode('Gough')\n 'GF'\n >>> pe.encode('pneuma')\n 'NM'\n >>> pe.encode('knight')\n 'NT'\n >>> pe.encode('trice')\n 'TRS'\n >>> pe.encode('judge')\n 'JJ'\n\n \"\"\"\n word = word.upper() # Rule 3\n word = self._delete_consecutive_repeats(word) # Rule 4\n\n # Rule 5\n i = 0\n while i < len(word):\n for match_len in range(4, 1, -1):\n if word[i : i + match_len] in self._rules[match_len]:\n repl = self._rules[match_len][word[i : i + match_len]]\n word = word[:i] + repl + word[i + match_len :]\n i += len(repl)\n break\n else:\n i += 1\n\n word = word[:1] + word[1:].translate(self._del_trans) # Rule 6\n return word"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script to\ncalculate the Hamming distance between the Eudex hashes of two terms. This is a wrapper for :py:meth:`Eudex.eudex_hamming`. Parameters ---------- src : str Source string for comparison tar : str Target string for comparison weights : str, iterable, or generator function The weights or weights generator function max_length : int The number of characters to encode as a eudex hash normalized : bool Normalizes to [0, 1] if True Returns ------- int The Eudex Hamming distance Examples -------- >>> eudex_hamming('cat', 'hat') 128 >>> eudex_hamming('Niall', 'Neil') 2 >>> eudex_hamming('Colin', 'Cuilen') 10 >>> eudex_hamming('ATCG', 'TAGC') 403 >>> eudex_hamming('cat', 'hat', weights='fibonacci') 34 >>> eudex_hamming('Niall', 'Neil', weights='fibonacci') 2 >>> eudex_hamming('Colin', 'Cuilen', weights='fibonacci') 7 >>> eudex_hamming('ATCG', 'TAGC', weights='fibonacci') 117 >>> eudex_hamming('cat', 'hat', weights=None) 1 >>> eudex_hamming('Niall', 'Neil', weights=None) 1 >>> eudex_hamming('Colin', 'Cuilen', weights=None) 2 >>> eudex_hamming('ATCG', 'TAGC', weights=None) 9 >>> # Using the OEIS A000142: >>> eudex_hamming('cat', 'hat', [1, 1, 2, 6, 24, 120, 720, 5040]) 1 >>> eudex_hamming('Niall', 'Neil', [1, 1, 2, 6, 24, 120, 720, 5040]) 720 >>> eudex_hamming('Colin', 'Cuilen', [1, 1, 2, 6, 24, 120, 720, 5040]) 744 >>> eudex_hamming('ATCG', 'TAGC', [1, 1, 2, 6, 24, 120, 720, 5040]) 6243", "response": "def eudex_hamming(\n src, tar, weights='exponential', max_length=8, normalized=False\n):\n \"\"\"Calculate the Hamming distance between the Eudex hashes of two terms.\n\n This is a wrapper for :py:meth:`Eudex.eudex_hamming`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n weights : str, iterable, or generator function\n The weights or weights generator function\n max_length : int\n The number of characters to encode as a eudex hash\n normalized : bool\n Normalizes to [0, 1] if True\n\n Returns\n -------\n int\n The Eudex Hamming distance\n\n Examples\n --------\n >>> eudex_hamming('cat', 'hat')\n 128\n >>> eudex_hamming('Niall', 'Neil')\n 2\n >>> eudex_hamming('Colin', 'Cuilen')\n 10\n >>> eudex_hamming('ATCG', 'TAGC')\n 403\n\n >>> eudex_hamming('cat', 'hat', weights='fibonacci')\n 34\n >>> eudex_hamming('Niall', 'Neil', weights='fibonacci')\n 2\n >>> eudex_hamming('Colin', 'Cuilen', weights='fibonacci')\n 7\n >>> eudex_hamming('ATCG', 'TAGC', weights='fibonacci')\n 117\n\n >>> eudex_hamming('cat', 'hat', weights=None)\n 1\n >>> eudex_hamming('Niall', 'Neil', weights=None)\n 1\n >>> eudex_hamming('Colin', 'Cuilen', weights=None)\n 2\n >>> eudex_hamming('ATCG', 'TAGC', weights=None)\n 9\n\n >>> # Using the OEIS A000142:\n >>> eudex_hamming('cat', 'hat', [1, 1, 2, 6, 24, 120, 720, 5040])\n 1\n >>> eudex_hamming('Niall', 'Neil', [1, 1, 2, 6, 24, 120, 720, 5040])\n 720\n >>> eudex_hamming('Colin', 'Cuilen', [1, 1, 2, 6, 24, 120, 720, 5040])\n 744\n >>> eudex_hamming('ATCG', 'TAGC', [1, 1, 2, 6, 24, 120, 720, 5040])\n 6243\n\n \"\"\"\n return Eudex().dist_abs(src, tar, weights, max_length, normalized)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns normalized Hamming distance between two Eudex hashes of two terms.", "response": "def dist_eudex(src, tar, weights='exponential', max_length=8):\n \"\"\"Return normalized Hamming distance between Eudex hashes of two terms.\n\n This is a wrapper for :py:meth:`Eudex.dist`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n weights : str, iterable, or generator function\n The weights or weights generator function\n max_length : int\n The number of characters to encode as a eudex hash\n\n Returns\n -------\n int\n The normalized Eudex Hamming distance\n\n Examples\n --------\n >>> round(dist_eudex('cat', 'hat'), 12)\n 0.062745098039\n >>> round(dist_eudex('Niall', 'Neil'), 12)\n 0.000980392157\n >>> round(dist_eudex('Colin', 'Cuilen'), 12)\n 0.004901960784\n >>> round(dist_eudex('ATCG', 'TAGC'), 12)\n 0.197549019608\n\n \"\"\"\n return Eudex().dist(src, tar, weights, max_length)"} {"SOURCE": "codesearchnet", "instruction": "How would you implement a function in Python 3 that\nreturns normalized Eudex hamming similarity between two terms.", "response": "def sim_eudex(src, tar, weights='exponential', max_length=8):\n \"\"\"Return normalized Hamming similarity between Eudex hashes of two terms.\n\n This is a wrapper for :py:meth:`Eudex.sim`.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n weights : str, iterable, or generator function\n The weights or weights generator function\n max_length : int\n The number of characters to encode as a eudex hash\n\n Returns\n -------\n int\n The normalized Eudex Hamming similarity\n\n Examples\n --------\n >>> round(sim_eudex('cat', 'hat'), 12)\n 0.937254901961\n >>> round(sim_eudex('Niall', 'Neil'), 12)\n 0.999019607843\n >>> round(sim_eudex('Colin', 'Cuilen'), 12)\n 0.995098039216\n >>> round(sim_eudex('ATCG', 'TAGC'), 12)\n 0.802450980392\n\n \"\"\"\n return Eudex().sim(src, tar, weights, max_length)"} {"SOURCE": "codesearchnet", "instruction": "Write a Python 3 script for\nyielding the next Fibonacci number.", "response": "def gen_fibonacci():\n \"\"\"Yield the next Fibonacci number.\n\n Based on https://www.python-course.eu/generators.php\n Starts at Fibonacci number 3 (the second 1)\n\n Yields\n ------\n int\n The next Fibonacci number\n\n \"\"\"\n num_a, num_b = 1, 2\n while True:\n yield num_a\n num_a, num_b = num_b, num_a + num_b"} {"SOURCE": "codesearchnet", "instruction": "Can you create a Python 3 function that\ncalculates the distance between two strings in an absolute form.", "response": "def dist_abs(\n self, src, tar, weights='exponential', max_length=8, normalized=False\n ):\n \"\"\"Calculate the distance between the Eudex hashes of two terms.\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n weights : str, iterable, or generator function\n The weights or weights generator function\n\n - If set to ``None``, a simple Hamming distance is calculated.\n - If set to ``exponential``, weight decays by powers of 2, as\n proposed in the eudex specification:\n https://github.com/ticki/eudex.\n - If set to ``fibonacci``, weight decays through the Fibonacci\n series, as in the eudex reference implementation.\n - If set to a callable function, this assumes it creates a\n generator and the generator is used to populate a series of\n weights.\n - If set to an iterable, the iterable's values should be\n integers and will be used as the weights.\n\n max_length : int\n The number of characters to encode as a eudex hash\n normalized : bool\n Normalizes to [0, 1] if True\n\n Returns\n -------\n int\n The Eudex Hamming distance\n\n Examples\n --------\n >>> cmp = Eudex()\n >>> cmp.dist_abs('cat', 'hat')\n 128\n >>> cmp.dist_abs('Niall', 'Neil')\n 2\n >>> cmp.dist_abs('Colin', 'Cuilen')\n 10\n >>> cmp.dist_abs('ATCG', 'TAGC')\n 403\n\n >>> cmp.dist_abs('cat', 'hat', weights='fibonacci')\n 34\n >>> cmp.dist_abs('Niall', 'Neil', weights='fibonacci')\n 2\n >>> cmp.dist_abs('Colin', 'Cuilen', weights='fibonacci')\n 7\n >>> cmp.dist_abs('ATCG', 'TAGC', weights='fibonacci')\n 117\n\n >>> cmp.dist_abs('cat', 'hat', weights=None)\n 1\n >>> cmp.dist_abs('Niall', 'Neil', weights=None)\n 1\n >>> cmp.dist_abs('Colin', 'Cuilen', weights=None)\n 2\n >>> cmp.dist_abs('ATCG', 'TAGC', weights=None)\n 9\n\n >>> # Using the OEIS A000142:\n >>> cmp.dist_abs('cat', 'hat', [1, 1, 2, 6, 24, 120, 720, 5040])\n 1\n >>> cmp.dist_abs('Niall', 'Neil', [1, 1, 2, 6, 24, 120, 720, 5040])\n 720\n >>> cmp.dist_abs('Colin', 'Cuilen',\n ... [1, 1, 2, 6, 24, 120, 720, 5040])\n 744\n >>> cmp.dist_abs('ATCG', 'TAGC', [1, 1, 2, 6, 24, 120, 720, 5040])\n 6243\n\n \"\"\"\n # Calculate the eudex hashes and XOR them\n xored = eudex(src, max_length=max_length) ^ eudex(\n tar, max_length=max_length\n )\n\n # Simple hamming distance (all bits are equal)\n if not weights:\n binary = bin(xored)\n distance = binary.count('1')\n if normalized:\n return distance / (len(binary) - 2)\n return distance\n\n # If weights is a function, it should create a generator,\n # which we now use to populate a list\n if callable(weights):\n weights = weights()\n elif weights == 'exponential':\n weights = Eudex.gen_exponential()\n elif weights == 'fibonacci':\n weights = Eudex.gen_fibonacci()\n if isinstance(weights, GeneratorType):\n weights = [next(weights) for _ in range(max_length)][::-1]\n\n # Sum the weighted hamming distance\n distance = 0\n max_distance = 0\n while (xored or normalized) and weights:\n max_distance += 8 * weights[-1]\n distance += bin(xored & 0xFF).count('1') * weights.pop()\n xored >>= 8\n\n if normalized:\n distance /= max_distance\n\n return distance"} {"SOURCE": "codesearchnet", "instruction": "How would you code a function in Python 3 to\nreturn normalized distance between the Eudex hashes of two terms.", "response": "def dist(self, src, tar, weights='exponential', max_length=8):\n \"\"\"Return normalized distance between the Eudex hashes of two terms.\n\n This is Eudex distance normalized to [0, 1].\n\n Parameters\n ----------\n src : str\n Source string for comparison\n tar : str\n Target string for comparison\n weights : str, iterable, or generator function\n The weights or weights generator function\n max_length : int\n The number of characters to encode as a eudex hash\n\n Returns\n -------\n int\n The normalized Eudex Hamming distance\n\n Examples\n --------\n >>> cmp = Eudex()\n >>> round(cmp.dist('cat', 'hat'), 12)\n 0.062745098039\n >>> round(cmp.dist('Niall', 'Neil'), 12)\n 0.000980392157\n >>> round(cmp.dist('Colin', 'Cuilen'), 12)\n 0.004901960784\n >>> round(cmp.dist('ATCG', 'TAGC'), 12)\n 0.197549019608\n\n \"\"\"\n return self.dist_abs(src, tar, weights, max_length, True)"} {"SOURCE": "codesearchnet", "instruction": "Make a summary of the following Python 3 code\ndef euclidean(src, tar, qval=2, normalized=False, alphabet=None):\n return Euclidean().dist_abs(src, tar, qval, normalized, alphabet)", "response": "Return the Euclidean distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Can you generate a brief explanation for the following Python 3 code\ndef dist_euclidean(src, tar, qval=2, alphabet=None):\n return Euclidean().dist(src, tar, qval, alphabet)", "response": "Return the normalized Euclidean distance between two strings."} {"SOURCE": "codesearchnet", "instruction": "Given the following Python 3 function, write the documentation\ndef sim_euclidean(src, tar, qval=2, alphabet=None):\n return Euclidean().sim(src, tar, qval, alphabet)", "response": "Return the normalized Euclidean similarity of two strings."}